At the April 27 Data Architecture call, I volunteered to write a use case
for replication of subsets of data. Below is my attempt to capture the
basic idea in a few examples..
Allen
Data Subset Replication
Replication of entire objects (e.g., a file, an entire database) is a
natural, and obvious, place to start considering replication. However
there is a real need to replicate subsets of data. Here are a number of
motivating examples (use cases):
Company A has an employee database containing all information about its
employees. The database is multiple terabytes in size and updates are
frequent. Suppose that the payroll information is contained in a single
table in that database. The payroll department needs to have fast access
to the payroll table. This table is only a few tens of gigabytes in size
and updates are infrequent. Replicating just this subset reduces storage
consumption on the payroll system, reduces the bandwidth used to maintain
the replica and reduces the processing power used to create and handle
updates to the payroll system. This is an example of replicating a single
table instead of an entire database.
A Life Sciences example. Suppose that there is a large file that
describes the entire human genome held on some server at UCLA. This file
is multiple terabytes in size. A researcher in Paris desires to perform
some computation on genes 14 through 19. For efficiency the data being
processed must be at a server in Paris. Instead of expending considerable
resource to move the entire file, relatively small amounts of resource are
used to move just that portion of the file containing genes 14 through 19.
Now suppose that a second researcher in Paris desires to perform a
computation on genes 18 and 19. Instead of moving the entire file, or
even moving the subset of the file containing genes 18 and 19, that
researcher can reuse the partial replica already held in Parsis since it
contains genes 14 through 19 and the genes of interest are a subset of
those.
A final example from the database world. Suppose that a multi-site
hospital keeps a database of its patients and that the database contains
all patient information including voluminous information such as x-ray
images and MRI scans. Thus it is very large. One of the hospitals in the
system, located in Boston, is very specialized - it only sees local
elderly cancer patients. This hospital needs to have a replica of the
database that contains only those patients - replicating the entire
database is, as above, a waste of precious resources. So the hospital
needs a replica that contains only those patient records that match a
query that might look something like: "Patient.age > 65 AND
DISTANCE(Patient.address, "Boston") < '50 miles' AND (Patient.illnesses
INCLUDES "cancer" OR Patient.past Illnesses INCLUDES "cancer")"
