WARNING: This is the _old_ Lustre wiki, and it is in the process of being retired. The information found here is all likely to be out of date. Please search the new wiki for more up to date information.
Lustre FAQ
Glossary
- ACL: Access Control List
- DLM: Distributed Lock Manager
- EA: Extended Attribute
- FC: Fibrechannel
- HPC: High-Performance Computing
- IB: InfiniBand
- MDS: Metadata Server
- NAL: Network Abstraction Layer; a software module which provides support for a particular interconnect
- OSS: Object Storage Server
- OST: Object Storage Target (what's the difference? )
Fundamentals
Can you describe the data caching and cache coherency method?
There is complete cache coherence for both metadata (names, readdir lists, inode attributes) and file data. Clients and servers both take locks with a distributed lock management service; caches are flushed before locks are released.
Does Lustre separate metadata and file data?
Yes. The entire namespace is stored on Lustre Metadata Servers (MDSs); file data is stored on Lustre Object Storage Servers (OSSs).
The data for each file may reside in multiple objects on separate servers. Lustre 1.x manages these objects in a RAID-0 (striping) configuration, so each object in a multi-object file contains only a part of the file's data. Future versions of Lustre will allow the user or administrator to choose other striping =methods, such as RAID-1 or RAID-5 redundancy.
What is the difference between an OST and an OSS?
There is a lot of confusion, and it's mostly our fault; as the architecture evolved, we refined these terms, and it has been difficult to enforce.
An Object Storage Server (OSS) is a server node, running the Lustre software stack. It has one or more network interfaces, and usually one or more disks.
An Object Storage Target (OST) is an interface to a single exported backend volume. It is conceptually similar to an NFS export, except that an OST does not contain a whole namespace, but rather file system objects.
It is common for a single OSS to export more than one OST? Yes, for example to get around the Linux 2.6 maximum 8 TB partition size. Although Lustre will aggregate multiple OSTs into a single large file system, the individual OST partitions are 8 TB.
Does Lustre perform high-level I/O load balancing?
Yes. Because a single file can reside in pieces on many servers, the I/O load for even a single file can be distributed.
By default, objects are randomly distributed amongst OSTs.
Is there a common synchronized namespace for files and directories?
Yes. All clients which mount the file system will see a single, coherent, synchronized namespace at all times.
Can Lustre be used as part of a "single system image" installation?
Yes. Lustre as the root file system is being used by some installations on both clients and servers, although it will not be officially supported until Lustre 1.6.x (see the roadmap ).
Do Lustre clients use NFS to reach the servers?
No. Client nodes run a native Lustre client file system driver, which uses the Lustre metadata and object protocols to communicate with the servers. The NFS protocol is not suitable to meet Lustre's metadata, I/O, locking, recovery, or performance requirements.
Does Lustre use/provide a single security domain?
Current versions of Lustre expect the clients and servers to have an identical understanding of UIDs and GIDs, but security is enforced on the Metadata Server by a server-local PAM-managed group database. Strong security and POSIX Access Control Lists are coming in future versions (see the roadmap ).
Does Lustre support the standard POSIX file system APIs?
Yes. Applications which use standard POSIX file system APIs can run on Lustre without modifications.
Is Lustre "POSIX compliant"? Are there any exceptions?
POSIX does not, strictly speaking, say anything about how a file system will operate on multiple clients. However, Lustre conforms to the most reasonable interpretation of what the single-node POSIX requirements would mean in a clustered environment.
For example, the atomicity of read and write operations are enforced through the Lustre distributed lock manager; if application threads running on multiple nodes were to try to read and write the same part of a file at the same time, they would both see consistent results. This is true of all I/O and metadata operations, with two exceptions:
- 1. atime updates
It is not practical to maintain fully coherent atime updates in a high-performance cluster file system. Lustre will update the atime of files lazily -- if an inode needs to be changed on disk anyways, we will piggy-back an atime update if needed -- and when files are closed.
- 2. flock/lockf
POSIX and BSD flock/lockf system calls will be completely coherent across the cluster, using the Lustre lock manager, but are not yet supported today. They will be soon (see the roadmap).
Prior to Lustre 1.4.1, there was one additional deviation from POSIX, in the area of mmap I/O. In 1.4.1, changes were made to support cache-coherent mmap I/O and robust execution of binaries and shared libraries residing in Lustre. mmap() I/O is now coherent and synchronized via the Lustre lock manager, although there may be pathological cases that remain hazardous for some time.
Can you grow/shrink file systems online?
Lustre 1.6 contains support for online addition of OST targets either on a new or on an existing OSS. In an upcoming version of Lustre the recently added support for online resizing of ext3 volumes will provide an additional way of growing file systems. Shrinking is not supported.
Which disk file systems are supported as Lustre backend file systems?
Lustre includes a patched version of the ext3 file system, with additional features such as extents, an efficient multi-block allocator, htree directories, large inodes, extended attributes, transaction optimizations, fine-grained locking, and CPU affinity for critical operations. This is the only supported backend file system today.
Why did Lustre choose ext3? Do you ever plan to support others?
There are many reasons to choose ext3. One is size; at under 15,000 lines of code, it is extremely understandable, maintainable, and modifiable. Another is reliability; ext3 is proven stable by millions of users, with an excellent file system repair tool.
When we began, there was a big difference between the various Linux file systems, particularly with respect to performance. In the last three years, however, Lustre has carried ext3 very substantially forward, and it is now extremely competitive with other Linux file systems.
Nevertheless, we had originally planned to support multiple file systems, so Lustre does contain a file system abstraction. The Solaris Port of the OSS will support ZFS.
Why didn't you use IBM's distributed lock manager?
The design of the Lustre DLM borrows heavily from the VAX Clusters DLM, plus extensions that are not found in others. Although we have received some reasonable criticism for not using an existing package (such as IBM's DLM), experience thus far has seemed to indicate that we've made the correct choice: it's smaller, simpler and, at least for our needs, more extensible.
The Lustre DLM, at around 6,000 lines of code, has proven to be an overseeable maintenance task, despite its somewhat daunting complexity. The IBM DLM, by comparison, was nearly the size of all of Lustre combined. This is not necessarily a criticism of the IBM DLM, however; to its credit, it is a complete DLM which implements many features which we do not require in Lustre.
In particular, Lustre's DLM is not really distributed, at least not when compared to other such systems. Locks in the Lustre DLM are always managed by the service node, and do not change masters as other systems allow. Omitting features of this type has allowed us to rapidly develop and stabilize the core DLM functionality required by the file system, plus add several extensions of our own (extent locking, intent locking, policy functions, glimpse ASTs, and a different take on lock value blocks).
Are services at user or kernel level? How do they communicate?
All daemons on a single node run in the kernel, and therefore share a single address space. Daemons on different nodes communicate through RPC messages; large messages are sent using remote DMA if the fabric supports it.
Sizing
What is the maximum file system size? What is the largest file system you've tested?
Each backend OST file system is restricted to a maximum of 2 TB on Linux 2.4 (imposed by the kernel block device), or 8 TB on Linux 2.6 (imposed by ext3). Of course, it is possible to have multiple OST file systems on a single OSS, and to aggregate multiple OSSs. Running with almost 4000 thousand OST's has been tried - hence 32PB file systems can be achieved today.
Lustre users already run single production filesystems of 1.4PB.
What is the maximum file system block size?
The basic ext3 block size is 4096 bytes, although this could in principle be easily changed to a large PAGE_SIZE (on IA64, for example) with a few modifications to ext3. It is not clear, however, that this is necessary.
Some people confuse block size with extent size or I/O request size -- they are not the same thing. The block size is the basic unit of disk allocation, and for our purposes it seems that 4kB is as good as any. The size of a single file extent, by definition, is almost always larger than 4kB, and ext3 improvements in Lustre 1.4.x do a good job of using contiguous disk blocks whenever possible. The I/O request size (the amount of data that we try to read or write in a single request) is much much larger.
What is the maximum single-file size?
On 32-bit clients, the page cache makes it quite difficult to read or write a single file larger than 8 TB. On 64 bit clusters, the maximum file size is 2^64. A current Lustre limit for allocated file space arises from a maximum of 160 stripes and about 8TB per stripe, leading to about 1.28PB per file.
What is the maximum number of files in a single file system? In a single directory?
We use the ext3 hashed directory code, which has a theoretical limit of ~134 million files per directory, at which point the directory grows to more than 2 GB. The maximum number of subdirectories is 32,000 in versions prior to Lustre 1.2.6 and is unlimited in later versions (small ext3 format change).
More realistically, we regularly run tests with ten million files in a single directory. On a properly-configured dual-CPU MDS with 4 GB of ram, it is possible to do random lookups in this directory at a rate of 5,000/second.
A single MDS imposes an upper limit of 4 billion inodes, but the default limit is slightly less than the device size / 4kB, so about 512 million inodes for a 2 TB MDS file system. This can be increased at initial file system creation time by specifying mkfs options. We regularly test with file systems containing approximately 100 million files.
With the introduction of clustered metadata servers (Lustre 2.0), these limits will disappear.
How many OSSs do I need?
The short answer is: as many as you need to achieve the required aggregate I/O throughput.
The long answer is: Each OSS contributes to the total capacity and the aggregate throughput. For example, a 100 TB file system may use 100 single-gige-connected OSS nodes with 1 TB of 100 MB/sec storage each, providing 10 GB/sec of aggregate bandwidth. The same bandwidth and capacity could be provided with four heavy-duty 25 TB OSS servers with three Elan 4 interfaces and 16 FC2 channels, each providing ~2.5 GB/s in aggregate bandwidth. The 25 TB of storage must be capable of 2.5 GB/s.
Each OSS can support a very large number of clients, so we do not advise our customers to use any particular client-to-OSS ratio. Nevertheless, it is common to deploy 1 GB/s of OSS throughput per 1 TFLOP/s of compute power.
What is the largest possible I/O request?
When most people ask this question, they are asking what is the maximum buffer size that can be safely passed to a read() or write() system call. In principle this is limited only by the address space on the client, although in practice it is possible to request so much I/O in a single atomic unit that the cluster infrastructure will see this as non-responsiveness and generate timeouts. Depending on your I/O infrastructure this can be 100 MB or more, however, so it has not been an issue in reality. In any case, we are aware of this limitation, and will work to remove it in a future release.
The Lustre I/O subsystem is designed with the understanding that an I/O request travels through many pipelines, and that it's important to keep all pipelines full for maximum performance. So it is not necessary to teach your application to do I/O in very large chunks; Lustre and the page cache will aggregate I/O for you.
Typically, Lustre client nodes will do their best to aggregate I/O into 1 MB chunks on the wire, and to keep between 5 and 10 I/O requests "in flight" at a time, per server.
On the OSS, we have gone to significant effort to ensure that these large 1 MB buffers do not get unnecessarily broken up by lower kernel layers. In Linux 2.4, modifications were required to the SCSI layer, block devices, and the QLogic fibrechannel driver. Because nearly 100% of our customers use QLogic fibrechannel cards, we have not yet optimized other drivers in this way.
How many nodes can connect to a single Lustre file system?
The largest single production Lustre installation is approximately 25,000 nodes today (2006). A Cray XT3 cluster of almost 12,000 nodes is currently being deployed, although it will likely run in production as two separate 6,000-node clusters.
Although these are the largest clusters available to us today, we believe that the architecture is fundamentally capable of supporting many tens of thousands of clients.
Installation
Which operating systems are supported as clients and servers?
Please see OS Support .
Can you use NFS or CIFS to reach a Lustre volume?
Yes. Any native Lustre client (running Linux today, by definition) can export a volume using Samba (or NFS, but keep reading). Some people have even built small clusters of these export nodes, to improve overall performance to their non-native clients.
Although NFS export works today, we don't support it because it is extremely slow. Lustre will require some specific optimizations to work around the behaviour of the Linux kernel NFS server. We're working on these, but in the meantime, we suggest Samba.
CIFS export with Samba, even a cluster of such nodes, is possible with one caveat: oplocks and Windows share modes. If you connect to these Samba shares with Windows clients, they will probably make heavy use of share modes and oplocks for locking and synchronization. Samba implements them internally, and does not yet have a clustered mode to coordinate them between multiple servers running on separate nodes. So if you rely on the consistency of these share modes and oplocks, you should use a single node to export CIFS.
What is the typical MDS node configuration?
1,000-node clusters with moderate metadata loads are commonly supported with a dual-Xeon node with 2GB of RAM, providing sustained throughput of over 1,000 ops/second. It is common for these systems to have roughly 20 million files. Even in 1,000-client clusters, the single MDS has been shown not to be a bottleneck under typical loads.
High throughput with very large directories is possible with 64-bit architectures and extra RAM. Typically, write I/O is low, but seek latency is very important, hence RAID-1 mirrored storage can be recommended.
Storage requirements should be sized at approximately 4kB per file, except in unusual circumstances.
What is the typical OSS node configuration?
IA32 systems support up to 150 MB/s over dual-GigE, and DMA-capable networks are typically limited only by the bus. 64-bit servers with good buses are capable of saturating multiple interconnects of any type. These nodes are often dual-CPU and support up to 4 fibrechannel channels. RAM is used for locks and read caching, but large amounts of RAM are usually not necessary, even for 1,000-node clusters.
Which architectures are interoperable?
Lustre 1.4 requires the page size on server nodes (MDS and OSS) to be smaller or the same size as client nodes. Except for this, there are no known obstacles to interoperability, even among heterogeneous client groups and systems with different endian.
Which storage devices are supported, on MDS and OSS nodes?
Servers support all block storage: fibrechannel, SCSI, SATA, ATA and exotic storage (NVRAM) are supported.
Which storage interconnects are supported?
Just to be clear: Lustre does not require a SAN, nor does it require a fabric like iSCSI. It will work just fine over simple IDE block devices. But because many people already have SANs, or want some amount of shared storage for failover, this is a common question.
For storage behind server nodes, FibreChannel, InfiniBand, iSCSI, or any other block storage protocol can be used. Failover functionality requires shared storage (each partition used active/passive) between a pair of nodes on a fabric like SCSI, FC or SATA.
Typically, fibrechannel switches are not necessary. Multi-port shared storage for failover is normally configured to be shared between two server nodes on a FC-AL. Shared SCSI and future shared SATA devices will also work.
Backend storage is expected to be cache-coherent between multiple channels reaching the devices. Servers in an OSS failover pair are normally both active in the file system, and will take over partitions for each other in the case of a failure. MDS failover pairs can also both be active, but only if they serve multiple separate file systems.
Can you put the file system journal on a separate device?
Yes. This can be configured when the backend ext3 file systems are created.
Can you run Lustre on LVM volumes, software RAID, etc?
Yes. You can use any Linux block device as storage for a backend Lustre server file system, including LVM or software RAID devices.
Can you describe the installation process?
The current installation process is straightforward, but manual:
1. Install the provided kernel and Lustre RPMs. 2. A configuration tool assistant can generate a configuration file for simple configurations, or you can build more complex configurations with relatively simple shell scripts. 3. Format and start the object servers and metadata servers. The command is usually identical on all nodes, so it's easy to use a utility like pdsh/prun to execute it. 4. Start the clients with "mount", similar to how NFS is mounted.
We are in the process of building new tools for our enterprise customers. New un-configured servers will announce themselves to the management node for template/profile-based addition to the cluster. The first such tools will appear in 2005.
What is the estimated installation time per compute node?
Assuming that node doesn't require special drivers or kernel configuration, 5 minutes. Compute nodes can be installed and started in parallel.
What is the estimated installation time per I/O node?
5 minutes, plus formatting time, which can also be done in parallel.
Networking (5)
Which interconnects and protocols are currently supported?
Today, Lustre supports TCP/IP (commonly over gigabit or 10-gigabit ethernet), Quadrics Elan 3 and 4, OpenIB generation 1, Voltaire IB (3.4.5+), OFED(1.1), Topspin(3.2.0),GM (Myrinet GM) (2.1.22+), CISCO and Cray's Rapid Array and Seastar networks.
Can I use more than one interface of the same type on the same node?
Yes, with Lustre 1.4.6 and later.
Can I use two or more different interconnects on the same node?
Yes, with Lustre 1.4.x, subject to the particular limitations of the interconnect. For example, we are told that it is not possible to use both Elan 3 and Elan 4 in the same node at the same time.
Can I use TCP offload cards?
Probably -- but we've tried many of these cards, and for various reasons we didn't see much improvement, if any. First, because Lustre runs entirely in the kernel, it uses kernel networking APIs which are often not supported (or at least not optimized) by the offload drivers.
Second, the problem isn't the overhead of checksum calculation or the need for interrupt coalescing; lots of commodity ethernet cards already support these features. The big overhead is memory copying and buffering, which these cards rarely do anything to address.
Does Lustre support crazy heterogeneous network topologies?
Yes, although the craziest of them are not yet fully supported.
Because Lustre supports native protocols on top of high speed cluster interconnects (in addition to TCP/IP), some special infrastructure is necessary.
Lustre uses its own implementation of the Portals message passing API, upon which we have implemented Gateway nodes, to route between two native protocols. These are commodity nodes with, for example, both gigabit ethernet and Quadrics Elan interfaces. The gateway software translates the Portals packets between the interfaces to bridge the two networks.
These routers are in use today, and may become more popular as more enterprises connect multiple clusters with special interconnects to a single global Lustre file system. On the other hand, TCP/IP on gig-e is the interconnect of choice for most organizations, which requires no additional Portals routing.
Metadata Servers (10)
How many metadata servers does Lustre support?
Lustre 1.x supports up to two metadata servers (MDSs) per file system in an active/passive failover configuration, meaning that only one server is actually servicing requests as a time.
You can configure multiple active metadata servers today, but they must each serve separate file systems.
Lustre 2.x will introduced the clustered metadata feature, which will permit dozens or hundreds of metadata servers working in parallel for a single file system.
How will clustered metadata work?
At a high level, it is reasonably simple: each directory can be striped over multiple metadata servers, each of which contains a disjoint portion of the namespace. When a client wants to lookup or create a name in that namespace, it uses a hashing algorithm to determine which metadata server holds the information for that name.
When you consider the details of doing this efficiently, coherently, and completely recoverably in the face of any number of different failures, it becomes more complicated.
We have already demonstrated very substantial portions of this functionality, including recovery, as a part of our DoE/NNSA PathForward effort. It will not be production-quality, however, for some time.
Isn't the single metadata server a bottleneck?
Not so far. We regularly perform tests with single directories containing millions of files, and we have many customers with 1,000-node clusters and a single metadata server.
Lustre is carefully designed to place the entire burden of file I/O on the Object Storage Servers (OSSs): locking, disk allocation, storage and retrieval, everything. Once the file is opened and the striping information obtained by the client, the metadata server is no longer involved in the business of file I/O.
The Lustre metadata server software is extremely multithreaded, and we have made substantial modifications to ext3 and the Linux VFS (2.6) to enable fine-grained locking of a single directory. These are the kinds of optimizations which make it possible to do random creations or lookups in a single 10-million-file directory at a rate of more than 5,000 per second.
If there is a customer need for massive metadata servers prior to the release of clustered metadata, it should be possible to scale quite far using large SMP systems (such as the SGI Altix or Bull NovaScale) with large memory spaces which can utilize large caches to speed MDS operations.
What is the typical MDS node configuration?
Please see Installation.
How do I automate failover of my MDSs?
Please see Recovery.
No. The extreme complexity that this would introduce does not seem to be warranted by the relatively modest cost of a small amount of shared storage for the metadata servers.
How is metadata allocated on disk?
The standard way that Lustre formats the MDS file system is with 256-byte ext3 inodes, which contain extended attributes (EAs) embedded in the inodes. One use of such an EA is for the file striping data, which tells the clients on which object servers to find the file data. For very widely striped files, this EA may be too large to store in the inode and will be stored in separate blocks. By storing the EA in the inode when possible, we avoid an extra very expensive disk seek.
What is intent locking?
Most file systems operate in one of two modes: a mode in which the server does the metadata modifications, or a mode in which the client can cache metadata updates itself. Both ways have their advantages and disadvantages in certain situations.
Consider the case of 1,000 clients all chdir'ing to /tmp and creating their own output files. If each client locks the directory, adds their file, uploads the modification, and releases the lock, this simple operation will take forever. If the metadata server is able to execute the operations locally and return the results, it should all happen in less than a second.
This is not a contrived example -- Lustre users run applications which do this very thing every hour of every day, for example, to write checkpoint data of a long-running scientific computation.
Consider another very common case, like a user's home directory being used only by one node. In this case, it would be extremely advantageous to allow that node to cache metadata updates in ram, then lazily propagate them back to the MDS. This allows the user to make updates as fast as they can be recorded in ram (until ram is full).
The moral of the store is: in cases of high concurrency, do the updates on the metadata server. In single-user cases, cache updates on the client.
What does this have to do with intent locking? Our protocol bundles up the information for the entire operation with the initial lock request, in the form of a metadata intent, and gives the metadata server the option of whether to execute the operation immediately (and return only a result code), or to return a lock to allow the client to perform writeback caching.
Lustre 1.x does not include a metadata writeback cache on the client, so today's metadata server always executes the operation on the client's behalf. Even without a writeback cache, however, the intent locking infrastructure still provides value. By having all of the information available during the initial lock request, we are able to perform all metadata operations in a single RPC.
How does the metadata locking protocol work?
Prior to Lustre 1.4.x, each metadata server inode was locked as a single unit. When the client wished to cache the existence of a name, or the attributes or directory pages of an inode, it would take and hold such a read lock. When the metadata server modified an inode, it would take a write lock on the same.
There are common cases when even one lock per inode is not enough, however. For example, consider the case of creating the file "dir/file", as would happen during the unpacking of a tar archive. The client would first lookup "dir", and in doing so take a lock to cache the result. It would then ask the metadata server to create "file" inside of it, which would lock "dir" to modify it, thus yanking the lock back from the client. This "ping-pong" back and forth is unnecessary and very inefficient, so a Lustre 1.4.x release will introduce the separate locking of different parts of the inode (simple existence, directory pages, and attributes).
Does the MDS do any pre-allocation?
Yes. To enable very fast file creation, the metadata server asks the individual OSTs to pre-create some number of objects, which the MDS can then allocate as file stripes without additional RPCs. These preallocations are replenished asynchronously.
Object Servers and I/O Throughput (13)
How do I configure failover services?
Typical failover configurations couple two Lustre MDS or OSS nodes in pairs directly to a multi-port disk array. Object servers are typically active/active, with each serving half of the array, while metadata servers must be active/passive. These array devices typically have redundancy internally, to eliminate them as single points of failure.
This does not typically require a fibrechannel switch.
How do I automate failover of my MDSs/OSSs?
The actual business of automating the decisions about whether a server has failed, and which server should take over the load, is managed by a separate package (our customers have used Red Hat's Cluster Manager and SuSE's Heartbeat).
Completely automated failover also requires some kind of programmatically controllable power switch, because the new "active" MDS must be able to completely power off the failed node. Otherwise, there is a chance that the "dead" node could wake up, start using the disk at the same time, and cause massive corruption.
How necessary is failover, really?
The answer depends on how close to 100% uptime you need to achieve. Failover doesn't protect against the failure of individual disks -- that is handled by software or hardware RAID. Failover is to handle the failure of an MDS or OSS node as a whole which, in our experience, is not very common.
We would suggest that simple RAID-5 is sufficient for most users, but that the most important production systems should consider failover.
If Lustre is configured without shared storage for failover, and a server node fails, then a client that tries to use that node will pause until the failed server is returned to operation. After a short delay (a configurable timeout value), applications waiting for those nodes can be aborted with a signal (kill or Ctrl-C), similar to the NFS soft-mount mode.
When the node is returned to service, applications which have not been aborted will continue to run without errors or data loss.
If a node suffers a connection failure, will the node select an alternate route for recovery?
Yes. If a node has multiple network paths, and one fails, it can continue to use the others.
What are the supported hardware methods for HBA, switch, and controller failover?
These are supported to the extent supported by the HBA drivers. If arrays with multiple ports are shared by multiple I/O nodes, Lustre offers 100% transparent failover for I/O and metadata nodes. (Applications see a delay, but system calls complete without errors.)
Can you describe an example failure scenario, and its resolution?
Although failures are becoming more rare, it is more likely that a node will hang or timeout rather than crash. If a client node hangs or crashes, usually all other client and server nodes are not affected. Normally such a client is rebooted and rejoins the file system. When server nodes hang, they are commonly restarted, merely causing a short delay to applications which try to use that node. Other server nodes or clients are not usually affected.
How are power failures, disk or RAID controller failures, etc. addressed?
If I/O to the storage is interrupted AND the storage device guarantees strict ordering of transactions, then the ext3 journal recovery will restore the file system in a few seconds.
If the file system is damaged through device failures, unordered transactions, or a power loss affecting a storage device's caches, Lustre requires a file system repair. Lustre's tools will reliably repair any damage it can. It will run in parallel on all nodes, but can still be very time consuming for large file systems.
Recovery (8)
How do I configure failover services?
Typical failover configurations couple two Lustre MDS or OSS nodes in pairs directly to a multi-port disk array. Object servers are typically active/active, with each serving half of the array, while metadata servers must be active/passive. These array devices typically have redundancy internally, to eliminate them as single points of failure.
This does not typically require a fibrechannel switch.
How do I automate failover of my MDSs/OSSs?
The actual business of automating the decisions about whether a server has failed, and which server should take over the load, is managed by a separate package (our customers have used Red Hat's Cluster Manager and SuSE's Heartbeat).
Completely automated failover also requires some kind of programmatically controllable power switch, because the new "active" MDS must be able to completely power off the failed node. Otherwise, there is a chance that the "dead" node could wake up, start using the disk at the same time, and cause massive corruption.
How necessary is failover, really?
The answer depends on how close to 100% uptime you need to achieve. Failover doesn't protect against the failure of individual disks -- that is handled by software or hardware RAID. Failover is to handle the failure of an MDS or OSS node as a whole which, in our experience, is not very common.
We would suggest that simple RAID-5 is sufficient for most users, but that the most important production systems should consider failover.
If Lustre is configured without shared storage for failover, and a server node fails, then a client that tries to use that node will pause until the failed server is returned to operation. After a short delay (a configurable timeout value), applications waiting for those nodes can be aborted with a signal (kill or Ctrl-C), similar to the NFS soft-mount mode.
When the node is returned to service, applications which have not been aborted will continue to run without errors or data loss.
If a node suffers a connection failure, will the node select an alternate route for recovery?
Yes. If a node has multiple network paths, and one fails, it can continue to use the others.
What are the supported hardware methods for HBA, switch, and controller failover?
These are supported to the extent supported by the HBA drivers. If arrays with multiple ports are shared by multiple I/O nodes, Lustre offers 100% transparent failover for I/O and metadata nodes. (Applications see a delay, but system calls complete without errors.
Can you describe an example failure scenario, and its resolution?
Although failures are becoming more rare, it is more likely that a node will hang or timeout rather than crash. If a client node hangs or crashes, usually all other client and server nodes are not affected. Normally such a client is rebooted and rejoins the file system. When server nodes hang, they are commonly restarted, merely causing a short delay to applications which try to use that node. Other server nodes or clients are not usually affected.
How are power failures, disk or RAID controller failures, etc. addressed?
If I/O to the storage is interrupted AND the storage device guarantees strict ordering of transactions, then the ext3 journal recovery will restore the file system in a few seconds.
If the file system is damaged through device failures, unordered transactions, or a power loss affecting a storage device's caches, Lustre requires a file system repair. Lustre's tools will reliably repair any damage it can. It will run in parallel on all nodes, but can still be very time consuming for large file systems.
OS Support (9)
Which operating systems are/will be supported?
There are three ways for a file system client to access a Lustre volume: directly, with a native kernel driver; directly, via a userspace library (liblustre); or indirectly, via an NFS or CIFS export.
Today, native kernel drivers exist only for Linux (2.4 and 2.6). Other native ports, such as to Windows, AIX, or Solaris, are being considered. Your feedback about the desirability of such ports would be appreciated.
liblustre is not yet as robust or well-tested as Lustre in the Linux kernel, but it is fundamentally almost entirely the same code. The Cray Red Storm supercomputer (2005) will run liblustre on the Catamount operating system to provide all client and server storage, including root FS and swap space -- so very serious use of liblustre is right around the corner. It could in principle be used on any Unix-like operating system, or even Windows.
CIFS or NFS export will be the solution of choice for most of our customers wishing to integration non-Linux platforms in the short term.
We currently support only Linux as Lustre server nodes. Support for other operating systems such as Solaris will be available in late 2008.
Beginning with Lustre 1.6.0, Linux 2.4 servers will no longer be supported, and with 1.8 clients with 2.4 kernels will not be supported.
Why has you decided to patch the Linux kernel?
Lustre's goals are extremely ambitious; there are few, if any, other systems which attempt such scalability, performance, and consistency guarantees in a single package. Because this hadn't been done before, the infrastructure was not present in the kernel for such a file system.
The majority of the changes were in the VFS layer, implementing an API extension to make intent locking possible. Another very substantial set of changes were made to ext3, to make it more scalable and performant. Some extra symbols need to be exported.
Are there plans to get these patches into the kernel.org/OSDL kernel?
Yes. The Lustre patches have been extensively reviewed by the Linux kernel community and vendors. Many of the changes are already present in Linux 2.6, and our contributions to ext3 have been delivered upstream (although many are not yet incorporated).
We have made several changes based on the feedback from these reviews, and Lustre 1.8 will be based on this new set of patches. It is not possible to switch to these patches until we no longer support Linux 2.4, because many of these changes are not practically possible in Linux 2.4.
Based on the reaction from the kernel community principals, and the customers putting pressure on vendors to ship Lustre, we expect that these patches will be incorporated into the kernel.org tree before the release of Lustre 1.8. We will continue to make reasonable changes which may be necessary to conform to the Linux kernel.
Can I run Lustre without patching my kernel?
Certainly -- you can run SuSE Linux Enterprise Server.
SLES 9 includes a kernel that supports Lustre out of the box, and SLES 9 service pack 2 supports the Lustre 1.4.x series. Our close partnership with Novell ensures that there is always a vendor-supported kernel for the latest version of Lustre.
Which Linux kernels are supported?
We currently support three kernels: for Linux 2.4, the Red Hat Enterprise Linux 3 kernel (based on 2.4.21); for Linux 2.6, the SuSE Linux Enterprise Server 9 kernel (based on 2.6.5),the SuSE Linux Enterprise Server 10 kernel (based on 2.6.16), and the Red Hat Enterprise Linux 4 kernel (based on 2.6.16). Enterprise support customers can download complete binary packages for many architectures.
Which Linux distributions are supported?
Because Lustre runs almost entirely in the kernel, there are practically no distribution-specific issues.
Lustre enterprise support customers can download packages which have been tested on RHEL3, RHEL4, SLES9 and SLES10 systems, but are likely to work elsewhere. You are welcome to use whichever distribution you wish, although we do ask that you use one of the three Lustre-supported kernels.
What if I don't run one of those kernels?
If you can't or won't run one of the supported kernels, then there is not much that we can do for you. It's not a matter of "just building from source", because you need patches for your particular kernel, and given the differences between most kernels, these are fairly non-trivial to port. Until there is a customer demand for a given kernel series, Lustre group simply does not have the resources to maintain those patches.
We expect that this will no longer be an issue in the timeframe of Lustre 1.8, and in the meantime we will support the kernels that our customers use.
Do you support Lustre on an SGI Altix?
We expect that, if you use the SLES9 kernel, it should pretty much work out of the box.
When was Lustre for Linux first used in production?
A pre-1.0 version of Lustre for Linux was first used in a production cluster environment in March 2003.
Release Testing and Upgrading (4)
How does the Lustre group fix issues?
The Lustre group approaches bug tracking and fixing seriously and methodically:
- Regression testing: A test is written to reproduce the problem, which is added to the ongoing test suite.
- Architecture and Design: Depending on the severity and invasiveness, an update to the architecture description may be written and reviewed by senior management. A detailed design description for the patch is written and reviewed by principal engineers.
- Implementation: Fixes are implemented according to the design description, and added to a bug for review and inspection.
- Review and Inspection: A developer or development team will review the code first, and then submit it for a methodical inspection by senior and principal engineers.
- Testing: The developer runs a small suite of tests before the code leaves his or her desk, then it's added to a branch for regression testing. Ongoing test results are available at https://buffalo.lustre.org/
What testing does each version undergo prior to release?
Sun and its vendor and customer partners run a large suite of tests on a number of systems, architectures, kernels, and interconnects, including clusters as large as 400 nodes. Major updates receive testing on the largest clusters available to us, around 1,000 nodes.
Are Lustre releases backwards and forward compatible on the disk? On the wire?
Special care is taken to ensure that any disk format changes -- which are rare to begin with -- are handled transparently in previous and subsequent releases. Before the disk format changes, we release versions which support the new format, so you can safely roll back in case of problems. After the format change, new versions continue to support the old formats for some time, and transparently update disk structures when old versions are encountered.
So far, the same care has not been taken for wire structures, because we've been adding features so quickly that it hasn't been practical.
Some people ask "why don't you just have separate handlers for different versions of the protocol?" but it's unfortunately not that simple. The protocol rarely changes for aesthetic reasons, but rather because some part of the underlying infrastructure has been changed or extended in a substantive way. It is often not possible to trivially map the old behaviour to the new behaviour, or to do so in a way that preserves proper semantics.
There will come a time when we make that effort, but there is a finite pool of resources, so we will rely on our customers to tell us when that work is more important than adding features or fixing bugs. Beginning with Lustre 1.4.0, we will make clear in our release notes when the wire protocol changes, and with which versions it is backwards-compatible, if any.
Do you have to reboot to upgrade?
Not unless you upgrade your kernel. It's usually a simple matter of unmounting the file system or stopping the server, as the case may be, installing the new RPMs, and restarting it.
Some of our customers upgrade servers between wire-compatible releases using failover; a service is failed over, the software is updated on the stopped node, the service is failed back, and the failover partner is upgraded in the same way.
Licensing and Support
What is the licensing model for the Lustre file system for Linux?
The Lustre file system for Linux is an Open Source product.
New releases are made available to the general public at the same time as to our paying customers and partners, under the terms and conditions of the GNU GPL.
As we develop ports for operating systems other than Linux, it is highly likely that these will be proprietary. We may also decide to develop some future features as proprietary add-ons (outlined in red on the roadmap. These are easily separable, non-core features primarily of interest to enterprise customers.
Virtually every Open Source company has gone out of business or had to change to a proprietary model; this is our way of balancing the realities of business with the desire to provide an excellent Open Source cluster file system for Linux.
How can you add proprietary features? If you release part of Lustre under the GPL, you have to distribute the code for the entire thing.
Sun owns the copyright to virtually 100% of the relevant Lustre source code, and has a very liberal license to the remainder. This means that we can distribute Lustre under whichever license (or licenses) we choose.
If public money helped fund the development of Lustre, don't the taxpayers own that code?
Some portions of Lustre were developed under the sponsorship of the US Government under Subcontract nos. B514193, B525177, B523817, B536384, 2204-10713, and others. Under the terms of those subcontracts, Sun retained the copyright to all software developed. We released all of this software to the US Government under the GNU GPL.
Many of these efforts were to produce prototypes or beta implementations, not production-quality software. Sun has invested considerable resources in productizing these features, and developing new features entirely on our own.
If Sun goes out of business, the taxpayers' money will have been wasted. To that end, our government sponsors strongly encourage us to build a sustainable business model.
How does the commercial distribution of Lustre work?
Our Lustre enterprise support customers receive unlimited access to the Lustre technical support team, and through them, the Lustre developers. Support contracts are priced according to the number of Lustre clients and servers, with discounts that grow as the cluster size increases.
For more information about Lustre support contracts, please contact sales@clusterfs.com. This e-mail address is being protected from spam bots, you need JavaScript enabled to view it.
Will Sun develop custom features for a proprietary product?
You can ask, but generally speaking, no. There are several potential reasons for this:
Ideological: We want to provide a fully-featured cluster file system for Linux which is available to everyone. If some features were developed for your proprietary product, it undermines that effort.
Pragmatic: We don't have the resources to test and support very many different versions. It's best for us if everyone runs software with the same features, because it very substantially reduces the release engineering burden.
Selfish: It would be unfortunate for us if the code that we developed for you, under a proprietary license, was code that we later wanted to use in a different way. We'd rather not paint ourselves into that corner.
Which Lustre support services are available?
Sun provides worldwide, 24/7 enterprise support services for the Lustre file system, with guaranteed response times as low as one hour. You can contact us via telephone, email, or our web site, and speak directly to the experts who designed and implemented the file system.
Sun also supports many partners and their products with licensing and support services. For partners with demonstrated expertise and Lustre support experience, substantial discounts are available.
Contract development of new features, or acceleration of roadmap features to a guaranteed delivery date, are possible.
Sun also provides public and private on-site training services for your system administrators or support staff.
