Clustered Metadata

Clustered Metadata Design (in progress)

This document describes the design of the clustered metadata handling for Lustre. This material depends on other Lustre design, such as:

• General recovery
• Orphan Recovery
• Metadata Write Back caching

The draft Clustered Metadata Design is available as a PDF:

• Clustered Metadata Design

Introduction

Overall the clustered metadata handling is structured as follows.

• A cluster of metadata servers manage a collection of inode groups. Each inode group is a Lustre device exporting the usual metadata api, augmented with a few operations specifically crafted for metadata clustering. We call these collections of inodes inode groups.
• Directory formats for file systems used on the MDS devices are changed to introduce a allow directory entries to contain an inode group and identifier of the inode.
• A logical clustered metadata driver is introduced below the client Lustre file system write back cache driver that maintains connections with the MDS servers.
• There is a single metadata protocol that is used by the client file system to make updates on the MDS's and by the MDS's to make updates involving other MDS's.
• There is a single recovery protocol that is used by the clients - MDS and MDS-MDS service.
• Directories can be split across multiple MDS nodes. In that case a primary MDS directory inode contains an extended attribute that points at other MDS inodes which we call directory objects.

Configuration management and Startup

The configuration will name an MDS server, and optionally a failover node, which hold the root inode for a fileset. Clients will contact that MDS for the root inode during mount, as they do already.

They will also fetch from it a clustering descriptor. The clustering descriptor contains a header and an array lists what inode groups are served by what server.

Through normal mechanisms clients will wait and probe for available metadata servers, during startup and cluster transitions. When new servers are found or configurations have changed they can update their clustering descriptor as they update the LOV striping descriptor for OST's.

Data Structures

The fid will contain a new 32 bit integer to name the inode group.

Directory entries will contain a new 32 bit integer to name the inode group.

Directory inodes on the MDS, when large, contain a new EA which is a descriptor of how the directory is split over directory objects, residing on other MDS's. This EA is subject to ordinary concurrency control by the MDS holding the inode. The EA is virtually identical to the LOV EA.

The clustered metadata client (CMC)

Client systems will have the write back client (WBD) or client file system directly communicate with the CMC driver: it offers the metadata api to the file system and uses the metadata api offered by a collection of MDC drivers. Each MDC driver managed the metadata traffic to one.

The function of the CMC is very simple: it figures out from the command issued what MDC to use. This is based on:

• the inode groups in the request
• a hash value of names used in the request, combined with the EA of a primary inode involved in the request.
• for readdir the directory offset combined with the EA of the primary inode
• the clustering descriptor

In any case every command is dispatched to a single metadata server, the clients will not engage more than one metadata server for a single request.

The api changes here are minimal and the client part of the implementation is very trivial.

MDS implementation

For the most part, operations are extremely similar or identical to what they were before. In some cases multiple mds servers are involved in updates.

Getattr, open, readdir, setattr and lookup methods are unaffected.

Methods adding entries to directories are modified in some cases:

• mkdir always create the new directory on another MDS
• unlink, rmdir, rename: may involve more than one MDS
• large directories all operations making updates to directories can cause a directory split. The directory split is discussed below.
• other operations If no splits large directories are encountered all other operations proceed as they are executed on one MDS.

Directory Split

A directory that is growing larger will be split. There is a fairly heavy penalty associated with splitting the directory and also with renames in within split directories. Moreover, at the point of splitting, inodes become remote and will incur a penalty upon unlink.

Probably it is best to delay the split until the directory is fairly large, and then to split over several nodes, to avoid further splits being necessary soon afterwards.

Recovery

Transaction Replay

The MDS - MDS interaction is managed as follows. The node approached with a request change is made the coordinator of the transaction.

Mechanisms

The coordinator will first establish that the transaction can commit on all nodes, by acquiring locks on directories and checking for available space existing entries with the same name etc. It may also first perform a directory split if the size is becoming too large, and more MDS nodes are still available.

All nodes involved in the transaction need to have a transaction sequence number to place the transaction into their sequence and allow correctly replay.

At this point the coordinator will:

• start a transaction locally.
• It will then report the transaction sequence number to all other nodes involved in the transaction.
• These nodes will commit (in memory as usual), write a journal record for replay and reply to the coordinator.
• The coordinator will then commit its own transaction.
• The replay log records are subject to normal log commit cancelation messages, but on the coordinator commit messages must be received from all other nodes before the record will be canceled.

In this way if the results of the transaction survive on any of the nodes, they can be replayed on all.

Cluster crashes and the transaction sequence

If the cluster crashes abruptly, there is the opportunity for transactions to be in progress affecting multiple nodes. Dependencies between the transactions must be managed to ensure serializibility of the protocol.

Example: In transaction one, a node X creates directories a. Then in transaction 2 a cross MDS node rename moves a file with a directory entry on node Y into this directory. It is now possible for this file to lose its directory entry on Y and for the transaction on X not to commit. More complex examples exist.

We see 4 solutions:

• Disk commit delay locks: dependent transactions many not commit before the parent transaction commits.
• Commit acks: transactions may not proceed until previous pre-requisite transactions have committed.
• Synchronous NVRAM journal on all MDS nodes
• Shared journal among all MDS nodes

The first and last method offer the most opportunity to proceed without synchronous disk writes. The last method involves contention on a shared resource.

Although exhaustive analysis remains, it is clear that rename and splitting the directories are the primary culprits. Hence, we wonder about the following policy related issues:

• Only split really large directories, say after 1M entries.
• Do not needlessly create subdirectories on other nodes. A much better policy is likely to keep directories with one owner, or possibly one client system generating them together.

Replay

To order transaction sequences Lustre uses reply ack's: the acks server only one purpose to release a lock that enforces ordering of the transaction sequence. In the case where MDS operations involve more than server, the reply "ack" from the primary to secondary servers should only be sent after the client has sent the ack to the first server. This MDS-MDS reply ack is now not really an ack anymore but a simple lock cancelation review.

Clients will replay lost transactions to the mds which they originally engaged for the request.

Orphaned children will be cleaned up only after replay completes to allow orphaned objects to be re-used during replay.

Failover

The configuration data can designate a standby MDS that will take over from a failed MDS. By organizing the servers in one or more rings, the nearest working left neighbor MDS can be the failover node. This leads to a simple scheme with multiple failover nodes, avoiding quorum and other complications beyond what is needed for two node clusters.

Locking

We believe locking can be done in fid order as it is currently done on the MDS.