Lustre at LS-CAT
LS-CAT uses the Lustre
file system to manage data storage. This file system was
developed by Cluster File Systems, Inc (now owned
by Sun) for use at supercomputer facilities and, happily, is
perfect for our situation. On the server side we have one computer,
"piano", that acts as a "Meta Data Server" (or an "MDS" as the lustre
geeks like to say). This handles all the directory information but
does not store any of the actual data. The meta data is continually
backed up on a second server "forte" using the DRBD block device. The plan is to make
piano and forte into a fail-over pair we'll call pianoforte to provide
a highly available system, but we are not there yet.
The data are normally striped across the data servers (aka Object
Storage Targets [OST]) and hence even large files can be read and
written very quickly as the load is distributed across many fast
servers.
This file system model scales very well: as we add more OST's the
access speed increases and is limited by the ethernet connections to
the clients and these connection can be bonded so that multiple ports
work together to form a single fast connection.
The illustration below shows the Lustre implementation at LS-CAT as
of early 2008. Each of our stations has a "V" computer that writes
the data directly from the detector to the Lustre file system. There
are also two processing computers per experimental station (the "G"
computers). As of this writing there are only 3 OST's (the "P"
computers) and these provide a total of 11 TB of storage.
Our Lustre system is several orders of magnitude smaller
than other Lustre systems in production use: we have plenty of room to
grow before we'll run into any limits.
| 
