Obsolete Lustre Wiki - User contributions [en]

Clustered Metadata

2007-06-21T07:36:02Z

Yep:

= Clustered Metadata Design (in progress) =

This document describes the design of the clustered metadata handling
for Lustre. This material depends on other Lustre design, such as:
:
* General recovery
* Orphan Recovery
* Metadata Write Back caching

= Introduction =

Overall the clustered metadata handling is structured as follows.

* A cluster of metadata servers manage a collection of inode groups. Each inode group is a Lustre device exporting the usual metadata api, augmented with a few operations specifically crafted for metadata clustering. We call these collections of inodes inode groups.
* Directory formats for file systems used on the MDS devices are changed to introduce a allow directory entries to contain an inode group and identifier of the inode.
* A logical clustered metadata driver is introduced below the client Lustre file system write back cache driver that maintains connections with the MDS servers.
* There is a single metadata protocol that is used by the client file system to make updates on the MDS's and by the MDS's to make updates involving other MDS's.
* There is a single recovery protocol that is used by the clients - MDS and MDS-MDS service.
* Directories can be split across multiple MDS nodes. In that case a primary MDS directory inode contains an extended attribute that points at other MDS inodes which we call directory objects.

= Configuration management and Startup =

The configuration will name an MDS server, and optionally a failover
node, which hold the root inode for a fileset. Clients will contact
that MDS for the root inode during mount, as they do already.

They will also fetch from it a clustering descriptor. The clustering
descriptor contains a header and an array lists what inode groups are
served by what server.

Through normal mechanisms clients will wait and probe for available
metadata servers, during startup and cluster transitions. When new
servers are found or configurations have changed they can update their
clustering descriptor as they update the LOV striping descriptor for
OST's.

= Data Structures =

The fid will contain a new 32 bit integer to name the inode group.

Directory entries will contain a new 32 bit integer to name the inode
group.

Directory inodes on the MDS, when large, contain a new EA which is a
descriptor of how the directory is split over directory objects,
residing on other MDS's. This EA is subject to ordinary concurrency
control by the MDS holding the inode. The EA is virtually identical
to the LOV EA.

= The clustered metadata client (CMC) =

Client systems will have the write back client (WBD) or client file
system directly communicate with the CMC driver: it offers the
metadata api to the file system and uses the metadata api offered by a
collection of MDC drivers. Each MDC driver managed the metadata
traffic to one.

The function of the CMC is very simple: it figures out from the
command issued what MDC to use. This is based on:
* the inode groups in the request
* a hash value of names used in the request, combined with the EA of a primary inode involved in the request.
* for readdir the directory offset combined with the EA of the primary inode
* the clustering descriptor

In any case every command is dispatched to a single metadata server,
the clients will not engage more than one metadata server for a single
request.

The api changes here are minimal and the client part of the
implementation is very trivial.

= MDS implementation =

For the most part, operations are extremely similar or identical to
what they were before. In some cases multiple mds servers are
involved in updates.

Getattr, open, readdir, setattr and lookup methods are unaffected.

Methods adding entries to directories are modified in some cases:

* '''mkdir''' always create the new directory on another MDS
* '''unlink, rmdir, rename''': may involve more than one MDS
* '''large directories''' all operations making updates to directories can cause a directory split. The directory split is discussed below.
* '''other operations''' If no splits large directories are encountered all other operations proceed as they are executed on one MDS.

== Directory Split ==

A directory that is growing larger will be split. There is a fairly heavy penalty associated with splitting the directory and also with renames in within split directories. Moreover, at the point of splitting, inodes become remote and will incur a penalty upon unlink.

Probably it is best to delay the split until the directory is fairly large, and then to split over several nodes, to avoid further splits being necessary soon afterwards.

= Recovery =

== Transaction Replay ==

The MDS - MDS interaction is managed as follows. The node approached
with a request change is made the coordinator of the transaction.

=== Mechanisms ===

The coordinator will first establish that the transaction can commit on all nodes, by acquiring locks on directories and checking for available space existing entries with the same name etc. It may also first perform a directory split if the size is becoming too large, and more MDS nodes are still available.

All nodes involved in the transaction need to have a transaction sequence number to place the transaction into their sequence and allow correctly replay.

At this point the coordinator will:
* start a transaction locally.
* It will then report the transaction sequence number to all other nodes involved in the transaction.
* These nodes will commit (in memory as usual), write a journal record for replay and reply to the coordinator.
* The coordinator will then commit its own transaction.
* The replay log records are subject to normal log commit cancelation messages, but on the coordinator commit messages must be received from all other nodes before the record will be canceled.

In this way if the results of the transaction survive on any of the nodes, they can be replayed on all.

=== Cluster crashes and the transaction sequence ===

If the cluster crashes abruptly, there is the opportunity for transactions to be in progress affecting multiple nodes. Dependencies between the transactions must be managed to ensure serializibility of the protocol.

Example: In transaction one, a node X creates directories a. Then in transaction 2 a cross MDS node rename moves a file with a directory entry on node Y into this directory. It is now possible for this file to lose its directory entry on Y and for the transaction on X not to commit. More complex examples exist.

We see 4 solutions:

* Disk commit delay locks: dependent transactions many not commit before the parent transaction commits.
* Commit acks: transactions may not proceed until previous pre-requisite transactions have committed.
* Synchronous NVRAM journal on all MDS nodes
* Shared journal among all MDS nodes

The first and last method offer the most opportunity to proceed without synchronous disk writes. The last method involves contention on a shared resource.

Although exhaustive analysis remains, it is clear that rename and splitting the directories are the primary culprits. Hence, we wonder about the following policy related issues:
* Only split really large directories, say after 1M entries.
* Do not needlessly create subdirectories on other nodes. A much better policy is likely to keep directories with one owner, or possibly one client system generating them together.

=== Replay ===

To order transaction sequences Lustre uses reply ack's: the acks
server only one purpose to release a lock that enforces ordering of
the transaction sequence. In the case where MDS operations involve
more than server, the reply "ack" from the primary to secondary
servers should only be sent after the client has sent the ack to the
first server. This MDS-MDS reply ack is now not really an ack anymore
but a simple lock cancelation review.

Clients will replay lost transactions to the mds which they originally
engaged for the request.

Orphaned children will be cleaned up only
after replay completes to allow orphaned objects to be re-used during
replay.

== Failover ==

The configuration data can designate a standby MDS that will take over
from a failed MDS. By organizing the servers in one or more rings,
the nearest working left neighbor MDS can be the failover node. This
leads to a simple scheme with multiple failover nodes, avoiding quorum
and other complications beyond what is needed for two node clusters.

= Locking =

We believe locking can be done in fid order as it is currently done on the MDS.

Logging API

2007-06-21T06:19:32Z

Yep:

= Lustre Logging API =

In numerous places Lustre needs a logging API. Generally log records are written transactionally and canceled when a commit on another system completes. There are many uses for this, all associated with updates of persistent information on multiple systems.

Log records are stored in log objects. Log objects are currently implemented as files, and possibly, in some minor ways, the api below reflects this. We speak of log objects and sometimes of llog's (lustre-logs).

= API Requirements =

* The api should be usable through methods.
* The methods should mask wether the api is used remotely or locally.
* Logs only grow
* Logs can be removed, and remote callers may not assume open logs remain available

* Access to logs should be mostly through a stateless api that can be called remotely
* Access to logs should go through some kind of authentication/authorization system
:

= Fundamental Data Structures =

# Log objects can be identified in two ways:
* Through a name:
* the interpretation of the name is up to the driver servicing the call.
* typical examples of named logs are files identified by a pathname
* text versions of UUID's
* profile names
* Through an object id, the llog_logid.
: A directory of llogs which can look up a name to get an id can provide translation from a naming system to an id based access system. In our implementation we use a file system directory to provide this catalog function.
# Logs only contain records
# Records in logs have the following structure:
* llog_rec_hdr: a header, indicating the index, length and type; the header is 16 bytes in length
* body which is opaque 32 bit aligned blob ; the body is 32 byte aligned
* llog_rec_tail: length and index of the record again, for walking backward ; the tail is 16 byes length.
# The first record in every log is a 4K llog_log_rec
* The body of this record contains:
* a bitmap of records that are allocated; bit 0 is set immediately because the header itself occupies it
* A collection of log records behind the header
# Records can be accessed:
* by iterating through a specific log
* by providing a llog_cookie
: 4. Some logs are potentially very large, for example, replication logs, and require a hierarchical structure, where a catalog of logs is held at the top level. In some cases the catalog structure is two deep.
* A catalog api is provided which exploits the lower lustre log api
* Catalog entries

= Cancellation API =

This describes typical use of the logging api to manage distributed commit of related distributed persistent updates.

We consider systems that make related updates

: originator:: the first system performing a transaction
: replicators:: one or more other systems performing a related update

The key requirement is that the replicators must complete their updates. This is accomplished by transactionally recording the originators action in a log. When the replicators related action completes, it cancels the log entry on the originator. Here we describe the handshakes and protocols involved.

== Normal Operation ==

* Originator performs a transaction and as part of the transaction writes a log record for each replicator.
* The log record creation produces a ''log_cookie''.
* The log_cookie is sent to the replicator.
* The replicator performs the related transaction and executes a commit callback for that.
* The call back to put the log_cookie up for cancellation
* When the replicator has a page full of cancellation cookies it sends the cookies to the originator
* The originator cancels the log records associated with the cookies and cleans up empty log files

'''Notes'''

# These replication scenarios are closely related to commit callbacks and rpc's Key differences are
* The commit callbacks with transaction numbers involve a volatile client and persistent server
* The transaction sequence is determined by the server in the volatile-persistent case by the originator in the replicating case

: === Examples ===
:
: ==== deletion of files ====

* change needs to be replicated from mds to ost's
* osc's on mds act as originator for the change log, using storage from the mds
* osc's write log records
* osc's generate cancel cookies
* osc's send cookies to ost's
* ost's act as replicators
* remove objects
* pass osc generated cookies as parameters to obd_destroy transactions
* collect cookies in pages for bulk cancellation rpcs to the osc on mds
* cancel records on the osc's on mds

: ==== file size changes ====

* changes orginate on OST's need to be implemented on the MDS
* Upon the first file size change in an io epoch on OST:
* writes a new size change record for new epoch
* records the size of the previous epoch in the record
* records the object id of the previous epoch in the record
* it generates a cancellation cookie
* when MDS knows epoch has ended
* it obtains the size at completion of epoch from client (or exceptionally from the OST)
* it obtains cancellation cookies for each OST from the client or from the OST's
* it postpones starting a new epoch until the size is known
* it starts a setattr transaction to store the size
* when it commits, it cancels the records on the OST's

* RAID1 OST
* The primary is the originator, the secondary the replica
* Writes on the primary are accompanied by a change record for an extent
*

== Connections ==

In order to process cancellation and recovery actions the originators and replicators use a ptlrpc connection to execute remote procedure calls. The connection can be set up on the originator or the replicator and we call the system setting up the connection the initiator.

* The originator and the replicator establish a connection.
* The logging subsystem on each end of the connection receives a log connection active completion event
* if connections are active log_cookies can be accepted for cancellation by the cancellation daemons
* if connections are not active the log_cookies are not collected for cancellation
* A generation number is associated with the connections
* it increases for each connection complete event
:
== Cancellation timeout ==

* If the replicator times out during cancellation
* It will continue to process the transactions with cookies
* The cancellation daemon will drop the cookies

== Initiator Connect Event ==

When the replicator receives a connect complete event it
:
* Fetches the catalogs log_id for the catalog it needs to process through a get_cat_info call
* The replicator needs to know what the catalog logid is for processing
* When it has received the catalog it calls process catalog:
* The replicator calls sync
* It only processes records in logs from previous log connection generations
* The catalog processing repeats operations that should have been performed by the initiator earlier
* The replicator must be able to distinguish:
* the operation already took place
* the operation was not performed
* If the operations are performed:
* During this processing the replicator generates cookies as it normally does
* Upon commit cookies are queued on the replicator for bulk cancellation of log records
* If the update was already performed:
* the replicator can queue a cancellation cookie back immediately

== Log Removal Failure ==
:
* If an initiator crashes during log removal, the log entries may re-appear after recovery
* It is important that the removal of a log from a catalog and the removal of the log file are atomic
* Upon reconnection the replicator will again process the log

= Log API's =

== Low Level Log Operations ==

Logs can be opened and/or created. This fills in a log handle. The log handle can be used through the log handle api.

: <pre>
: int llog_create(struct obd_device *obd, struct llog_handle **, struct llog_logid *, char *name);
: </pre>

: If llog_id is not null open an existing log with this ID. If name is not null, open or create a log with that name. Otherwise open a nameless log. The object id of the log is stored in the handle upon success of opening or creation.

: <pre>
: int llog_close(struct llog_handle *loghandle);
: </pre>

: Close the log and free the handle. Remove the handle from the catalogs list of open handles. If the log has a flag set of destroy if empty the log may be zapped.

: <pre>
: int llog_destroy(struct llog_handle *);
: </pre>

: Destroy the log object and close the handle.

: <pre>
: int llog_write_rec(struct llog_handle *handle, struct llog_rec_hdr *rec, struct llog_cookie *cookie, int cookie_count, void *buf);
: </pre>

: Write a record in the log. If buf is null the record is complete, if buf is not null it inserted in the middle. Records are multiples of 128 bits in size and have a hdr and tail. Write the cookie for the entry into the cookie pointer. (cookie_count is probably a mistake).

: <pre>
: int llog_next_block(struct llog_handle *h, int curr_idx, int next_idx, __u64 *offset, void *buf, int len);
: </pre>

: Index curr_idx is in the block at offset *offset. Set *offset to the block offset of record next_idx. Copy len bytes from the start of that block into the buffer buf.

: <pre>
: int (*lop_read_header)(struct llog_handle *handle);
: </pre>

: Read the header of the log into the handle and also read the last rec_tail in the log to find the last index that was used in the log.

== Higher Level Log Operations ==

: <pre>
: int llog_init_handle(struct llog_handle *handle, int flags, struct obd_uuid *uuid);
: </pre>

: Initialize the handle. Try to read it from the log file, but if the log has no header yet, build it from the arguments. If the header is read, verify the flags and UUID in the log equal those of the arguments.

== OBD Level Log Operations ==

<pre>
: int llog_add_record(struct llog_handle *cathandle, struct llog_trans_hdr *rec,
: struct llog_cookie *logcookies, void *buf)
</pre>

<pre>
: int llog_delete_log(struct llog_handle *cathandle,struct llog_handle *loghandle)
</pre>

<pre>
: int llog_cancel_records(struct llog_handle *cathandle, int count, struct llog_cookie *cookies)
</pre>
* For each cookie in the cookie array, we clear the log in-use bit and either:
* the log is empty, so mark it free in the catalog header and delete it
* the log is not empty, just write out the log header
* The cookies may be in different log files, so we need to get new logs each time.

<pre>
: int llog_next_block(struct llog_handle *loghandle, int cur_idx, int next_idx,
: __u64 *cur_offset, void *buf, int len)
</pre>
* Return the block in the log that contains record with index next_idx. The current idx at offset cur_offset is used to optimize the search.

<pre>
: typedef int (*llog_cb_t)(struct llog_handle *, struct llog_trans_hdr *rec, void *data);
: int llog_process_log(struct llog_handle *loghandle, llog_cb_t cb, void *data)
</pre>
* Call the callback function cb

Fsck Support

2007-06-19T07:34:11Z

Yep:

== Where to get e2fsck support for Lustre ==

In order to support some of the more advanced features of Lustre (e.g. extents and large inode/EA support) you need a patched e2fsprogs. This can be downloaded (RPM, SRPM) from the lfsck directory of the customer download site. See http://clusterfs.com/download.html

=== Code for e2fsprogs-lustre ===
We maintain a quilt patchset of all the CFS changes to the upstream e2fsprogs. The patches are available in the .src.rpm in the ''e2fsprogs/patches'' directory.

== Using e2fsck and lfsck ==
In a node crash case it is not neccesary to run e2fsck on the filesystem - the ext3 journaling will ensure that the filesystem remains coherent. The only time we really required that e2fsck is run on a device is when there is some sort of event that may cause problems outside of what ext3 journaling can handle, such as hardware device failure, IO errors, etc. If the ext3 kernel code detects corruption on the disk it will mount the filesystem read-only to prevent further corruption but still allow read access to the device.

In such a situation, it is normally only ''required'' that e2fsck be run on the bad device before placing it back into service. In the vast majority of cases Lustre will be able to cope with inconsistencies it finds on the disk and between other devices in the filesystem. For good problem analysis it is always strongly recommended that e2fsck is run under a logger like '''script''' to record all of the output and changes that are made to the filesystem in case this information is needed later. If time permits, it is also a good idea to first run e2fsck in non-fixing mode (-n option) to first assess the type/extent of damage to the filesystem. The drawback is that in this mode e2fsck doesn't recover the filesystem journal, so there may appear to be filesystem corruption when none really exists. Briefly mounting and unmounting the ext3 filesystem (directly on the node like "mount -t ldiskfs /dev/{ostdev} /mnt/ost; umount /mnt/ost", with Lustre stopped, '''NOT''' via Lustre) will cause the journal to be recovered if there is concern/confusion about whether corruption is real or only due to the journal not being replayed.

While e2fsck is very good at fixing filesystem corruption (better than any other similar filesystem recovery tool, and a primary reason why ext3 was chosen over other filesystems for Lustre), it is often useful to know what type of damage there is, and an ext3 expert may be able to make more intelligent decisions about what needs fixing compared to e2fsck. CFS support is available for such situations.

<pre>
root# {stop lustre services for this device, if running}
root# script /tmp/e2fsck.sda
Script started, file is /tmp/e2fsck.sda
root# mount -t ldiskfs /dev/sda /mnt/ost
root# umount /mnt/ost
root# e2fsck -fn /dev/sda # don't fix filesystem, just check for corruption
root# e2fsck -fp /dev/sda # fix filesystem using "prudent" answers (usually 'y')
</pre>

In addition, the e2fsprogs package contains an '''lfsck''' tool which does distributed coherency checking for the Lustre filesystem, after e2fsck has been run. This is not required in a large majority of cases, at the small chance of having some leaked space in the filesystem. It can also be run once Lustre is already started (with care) to avoid a lengthy downtime.

=== How to run e2fsck+lfsck on a corrupted Lustre filesystem ===

In cases where the MDS or an OST become corrupted for some reason, it is possible to run a distributed check on the filesystem to determine what sort of problems exist.

The first step is to run 'e2fsck -f' on the individual MDS/OST '''with Lustre stopped''' that had problems in order to fix any local filesystem damage. It is a very good idea to run this e2fsck under "script" as shown above so that you have a log of whatever changes it made to the filesystem in case this is needed later. After this is complete it is then possible to bring the filesystem up if necessary to reduce the outage window.

Next, a full e2fsck of the MDS is necessary in order to create a databse for lfsck. The use of the '-n' option is critical for a mounted filesystem (i.e. if Lustre is running), otherwise you will corrupt your filesystem. The mdsdb file can grow fairly large, depending on the number of files in the filesystem (10GB or more for millions of files, though the actual file size is larger because the file is sparse). It is fastest if this is written to a local filesystem because of the seeking and small writes. Depending on the number of files, this step can take several hours.

<pre>
e2fsck -n -v --mdsdb /tmp/mdsdb /dev/{mdsdev}
</pre>

Next this file must be made accessible on all of the OSTs (either via a shared filesystem, or by copying it to the OSTs - pdcp is very useful here), and the OSTs need to run a similar e2fsck step. There is a stub mdsdb file generated called {mdsdb}.mdshdr that can be used instead of the full mdsdb file, if the OSTs do not have shared filesystem access to the MDS filesystem. The mdsdb is only used for reading so it does not need to be in shared storage for all of the OSTs. The OST e2fsck --ostdb step can be run in parallel on all of the OSTs.

<pre>
e2fsck -n -v --mdsdb /tmp/mdsdb --ostdb /tmp/{ostNdb} /dev/{ostNdev}
</pre>

Finally, the mdsdb and all of the ostdb files need to be made available on a mounted client so that lfsck can be run to examine the filesystem and optionally correct defects it finds.

<pre>
lfsck -n -v --mdsdb /tmp/mdsdb --ostdb /tmp/{ost1db} /tmp/{ost2db} ... /lustre/mount/point
</pre>

By default lfsck does not repair any inconsistencies it finds, only reporting the errors. It checks for 3 kinds of inconsistencies:

# Inode exists but has missing objects = dangling inode. This normally happens if there was a problem with an OST.
# Inode is missing but OST has unreferenced objects = orphan object. This normally happens if there was a problem with the MDS.
# Multiple inodes reference the same objects. This can happen if there was corruption on the MDS, or if the MDS storage is cached and loses some but not all writes.

If the filesystem is in use and being modified while the --mdsdb and --ostdb steps are running, lfsck may report inconsistencies where none exist because of files and objects being created/removed after the database files were collected, so the results should be examined closely and you may want to re-run the test and/or contact CFS support for guidance.

The easiest problem to resolve is that of orphaned objects. Using the '-l' option to lfsck it can link these objects to new files and put them into lost+found in the Lustre filesystem, where they can be examined and saved or deleted as necessary. If you are certain the objects are not useful, lfsck can run with the '-d' option to delete orphaned objects and free up any space they are using.

To fix dangling inodes, lfsck will create new zero-length objects are created on the OSTs if the '-c' option is given. These files will read back with binary zeros for the stripes that had objects recreated. Such files can also be read even without lfsck repair by using '''dd if=/lustre/bad/file of=/new/file bs=4k conv=sync,noerror'''. Because this is rarely useful to have files with large holes in them, most users delete these files after reading them (if useful) and/or restoring them from backup. Note that it is not possible to write to the holes of such a file without having lfsck recreate the objects, so it is generally easier to delete these files and restore them from backup.

To fix inodes with duplicate objects, lfsck will copy the duplicate object to a new object and assign that to one of the files if the '-c' option is given. One of the files will be OK, and one will likely contain garbage, but lfsck cannot tell by itself which one is correct.

File:Software breakdown.jpg

2007-06-11T08:36:34Z

Yep:

Coding Guidelines

2007-06-11T07:46:02Z

Yep:

All Lustre developers should follow the guidelines in this page very strictly to avoid problems during code merges later on. Please make the required changes to the default formatting rules in the editor you use to comply to the guidelines below.

== Guidelines ==

1. There should be no tabs in any files.

2. Blocks should be indented by 8 spaces.

3. New files should contain the following along with the license boilerplate. This will cause vim and emacs to use spaces instead of tabs for indenting. If you use a different editor, it also needs to be set to use spaces for indening Lustre code.
<pre style="background:lightgrey;">
/* -*- mode: c; c-basic-offset: 8; indent-tabs-mode: nil; -*-
* vim:expandtab:shiftwidth=8:tabstop=8:
*/
</pre>
4. All lines should wrap at 80 characters. If it's getting too hard to wrap there, you probably need to break it up into more functions. In some cases, it is acceptable to remove a few spaces between function arguments to avoid overflowing onto the next line.

5. Don't have spaces or tabs on blank lines or at the end of lines.

6. Don't use "inline" unless you're doing something so performance critical that the function call overhead will make a difference -- in other words: never. It makes debugging harder.

All of our wrapping, parenthesis, brace placement, etc. rules are basically Linux kernel rules, which are basically K&R. For those of you in need of a refresher, great detail is provided below.

== Great detail ==

a. When you wrap, the next line should start after the parenthesis:
<pre style="background:lightgrey;">
right:

variable = do_something_complicated(long_argument, longer_argument,
longest_argument(sub_argument,
foo_argument),
last_argument);

if (some_long_condition(arg1, arg2, arg3) < some_long_value &&
another_long_condition(very_long_argument_name,
another_long_argument_name) >
second_long_value) {
}

wrong:

variable = do_something_complicated(long_argument, longer_argument,
longest_argument(sub_argument, foo_argument),
last_argument);

if (some_long_condition(arg1, arg2, arg3) < some_long_value &&
another_long_condition(very_long_argument_name,
another_long_argument_name) >
second_long_value) {
}
</pre>
b. If you're wrapping put the operators at the end of the line, and if there are no parentheses indent 8 more:
<pre style="background:lightgrey;">
off = le32_to_cpu(fsd->fsd_client_start) +
cl_idx * le16_to_cpu(fsd->fsd_client_size);
</pre>
c. Binary and ternary (but not unary) operators should be separated from their arguments by one space.
<pre style="background:lightgrey;">
a++;
b |= c;
d = f > g ? 0 : 1;
</pre>
d. Function calls should be nestled against the parentheses, the parentheses should crowd the arguments, and one space after commas:
<pre style="background:lightgrey;">
right: do_foo(bar, baz);
wrong: do_foo ( bar,baz );
</pre>

e. All ''if'', ''for'', ''while'', etc. expressions should be separated by a space from the parenthesis, one space after the semicolons:
<pre style="background:lightgrey;">
for (a = 0; a < b; a++)
if (a < b || a == c)
while (1)
</pre>
f. Opening braces should be on the same line as the line that introduces the block, except for function calls. Closing braces get their own line, except for "else".
<pre style="background:lightgrey;">
int foo(void)
{
if (bar) {
this();
that();
} else if (baz) {
;
} else {
;
}
do {
cow();
} while (0);
}
</pre>
g. If one part of a compound ''if'' block has braces, all should.
<pre style="background:lightgrey;">
right:

if (foo) {
bar();
baz();
} else {
salmon();
}

wrong:

if (foo) {
bar();
baz();
} else
moose();
</pre>
h. When you make a macro, protect those who might call it by using do/while and parentheses; line up your backslashes:
<pre style="background:lightgrey;">
right:

#define DO_STUFF(a) \
do { \
int b = (a) + MAGIC; \
do_other_stuff(b); \
} while (0)

wrong:

#define DO_STUFF(a) \
{ \
int b = a + MAGIC; \
do_other_stuff(b); \
}
</pre>
i. If you nest preprocessor commands, use spaces to visually delineate:
<pre style="background:lightgrey;">
#ifdef __KERNEL__
# include <goose>
# define MOOSE steak
#else
# include <mutton>
# define MOOSE prancing
#endif
</pre>
j. For very long #ifdefs include the conditional with each #endif to make it readable:
<pre style="background:lightgrey;">
#ifdef __KERNEL__
# if LINUX_VERSION_CODE >= KERNEL_VERSION(2,5,0)
/* lots
of
stuff */
# endif /* KERNEL_VERSION(2,5,0) */
#else /* !__KERNEL__ */
# if HAVE_FEATURE
/* more
* stuff */
# endif
#endif /* __KERNEL__ */
</pre>
k. Comments should have the leading '''/*''' on the same line as the comment, and the trailing '''*/''' at the end of the last comment line. Intermediate lines should start with a '''*''' aligned with the first line's '''*''':
<pre style="background:lightgrey;">
/* This is a short comment */

/* This is a multi-line comment. I wish the line would wrap already,
* as I don't have much to write about. */
</pre>
l. Function declarations absolutely should NOT go into .c files, unless they are forward declarations for static functions that can't otherwise be moved before the caller. Instead, the declaration should go into the most "local" header available (preferrably *_internal.h for a given piece of code).

m. Structure and constant declarations should not be declared in multiple places. Put the struct into the most "local" header possible. If it is something that is passed over the wire it needs to go into lustre_idl.h, and needs to be correctly swabbed when the RPC message is unpacked.

n. The types and printf/printk formats used by Lustre code are:
<pre style="background:lightgrey;">
__u64 LPU64/LPX64/LPD64 (unsigned, hex, signed)
size_t LPSZ (or cast to int and use %u / %d)
__u32/int %u/%x/%d (unsigned, hex, signed)
(unsigned) long long %llu/%llx/%lld
loff_t %lld after a cast to long long (unfortunately)
</pre>
----
*'''[http://wiki.lustre.org/index.php?title=Front_Page FrontPage]'''

Coding Guidelines

2007-06-11T07:45:00Z

Yep:

All Lustre developers should follow the guidelines in this page very strictly to avoid problems during code merges later on. Please make the required changes to the default formatting rules in the editor you use to comply to the guidelines below.

== Guidelines ==

1. There should be no tabs in any files.

2. Blocks should be indented by 8 spaces.

3. New files should contain the following along with the license boilerplate. This will cause vim and emacs to use spaces instead of tabs for indenting. If you use a different editor, it also needs to be set to use spaces for indening Lustre code.
<pre style="background:lightgrey;">
/* -*- mode: c; c-basic-offset: 8; indent-tabs-mode: nil; -*-
* vim:expandtab:shiftwidth=8:tabstop=8:
*/
</pre>
4. All lines should wrap at 80 characters. If it's getting too hard to wrap there, you probably need to break it up into more functions. In some cases, it is acceptable to remove a few spaces between function arguments to avoid overflowing onto the next line.

5. Don't have spaces or tabs on blank lines or at the end of lines.

6. Don't use "inline" unless you're doing something so performance critical that the function call overhead will make a difference -- in other words: never. It makes debugging harder.

All of our wrapping, parenthesis, brace placement, etc. rules are basically Linux kernel rules, which are basically K&R. For those of you in need of a refresher, great detail is provided below.

== Great detail ==

a. When you wrap, the next line should start after the parenthesis:
<pre style="background:lightgrey;">
right:

variable = do_something_complicated(long_argument, longer_argument,
longest_argument(sub_argument,
foo_argument),
last_argument);

if (some_long_condition(arg1, arg2, arg3) < some_long_value &&
another_long_condition(very_long_argument_name,
another_long_argument_name) >
second_long_value) {
}

wrong:

variable = do_something_complicated(long_argument, longer_argument,
longest_argument(sub_argument, foo_argument),
last_argument);

if (some_long_condition(arg1, arg2, arg3) < some_long_value &&
another_long_condition(very_long_argument_name,
another_long_argument_name) >
second_long_value) {
}
</pre>
b. If you're wrapping put the operators at the end of the line, and if there are no parentheses indent 8 more:
<pre style="background:lightgrey;">
off = le32_to_cpu(fsd->fsd_client_start) +
cl_idx * le16_to_cpu(fsd->fsd_client_size);
</pre>
c. Binary and ternary (but not unary) operators should be separated from their arguments by one space.
<pre style="background:lightgrey;">
a++;
b |= c;
d = f > g ? 0 : 1;
</pre>
d. Function calls should be nestled against the parentheses, the parentheses should crowd the arguments, and one space after commas:
<pre style="background:lightgrey;">
right: do_foo(bar, baz);
wrong: do_foo ( bar,baz );
</pre>

e. All ''if'', ''for'', ''while'', etc. expressions should be separated by a space from the parenthesis, one space after the semicolons:
<pre style="background:lightgrey;">
for (a = 0; a < b; a++)
if (a < b || a == c)
while (1)
</pre>
f. Opening braces should be on the same line as the line that introduces the block, except for function calls. Closing braces get their own line, except for "else".
<pre style="background:lightgrey;">
int foo(void)
{
if (bar) {
this();
that();
} else if (baz) {
;
} else {
;
}
do {
cow();
} while (0);
}
</pre>
g. If one part of a compound ''if'' block has braces, all should.
<pre style="background:lightgrey;">
right:

if (foo) {
bar();
baz();
} else {
salmon();
}

wrong:

if (foo) {
bar();
baz();
} else
moose();
</pre>
h. When you make a macro, protect those who might call it by using do/while and parentheses; line up your backslashes:
<pre style="background:lightgrey;">
right:

#define DO_STUFF(a) \
do { \
int b = (a) + MAGIC; \
do_other_stuff(b); \
} while (0)

wrong:

#define DO_STUFF(a) \
{ \
int b = a + MAGIC; \
do_other_stuff(b); \
}
</pre>
i. If you nest preprocessor commands, use spaces to visually delineate:
<pre style="background:lightgrey;">
#ifdef __KERNEL__
# include <goose>
# define MOOSE steak
#else
# include <mutton>
# define MOOSE prancing
#endif
</pre>
j. For very long #ifdefs include the conditional with each #endif to make it readable:
<pre style="background:lightgrey;">
#ifdef __KERNEL__
# if LINUX_VERSION_CODE >= KERNEL_VERSION(2,5,0)
/* lots
of
stuff */
# endif /* KERNEL_VERSION(2,5,0) */
#else /* !__KERNEL__ */
# if HAVE_FEATURE
/* more
* stuff */
# endif
#endif /* __KERNEL__ */
</pre>
k. Comments should have the leading '''/*''' on the same line as the comment, and the trailing '''*/''' at the end of the last comment line. Intermediate lines should start with a '''*''' aligned with the first line's '''*''':
/* This is a short comment */

/* This is a multi-line comment. I wish the line would wrap already,
* as I don't have much to write about. */

l. Function declarations absolutely should NOT go into .c files, unless they are forward declarations for static functions that can't otherwise be moved before the caller. Instead, the declaration should go into the most "local" header available (preferrably *_internal.h for a given piece of code).

m. Structure and constant declarations should not be declared in multiple places. Put the struct into the most "local" header possible. If it is something that is passed over the wire it needs to go into lustre_idl.h, and needs to be correctly swabbed when the RPC message is unpacked.

n. The types and printf/printk formats used by Lustre code are:
<pre style="background:lightgrey;">
__u64 LPU64/LPX64/LPD64 (unsigned, hex, signed)
size_t LPSZ (or cast to int and use %u / %d)
__u32/int %u/%x/%d (unsigned, hex, signed)
(unsigned) long long %llu/%llx/%lld
loff_t %lld after a cast to long long (unfortunately)
</pre>
----
*'''[http://wiki.lustre.org/index.php?title=Front_Page FrontPage]'''

Coding Guidelines

2007-06-11T07:43:22Z

Yep:

All Lustre developers should follow the guidelines in this page very strictly to avoid problems during code merges later on. Please make the required changes to the default formatting rules in the editor you use to comply to the guidelines below.

== Guidelines ==

1. There should be no tabs in any files.

2. Blocks should be indented by 8 spaces.

3. New files should contain the following along with the license boilerplate. This will cause vim and emacs to use spaces instead of tabs for indenting. If you use a different editor, it also needs to be set to use spaces for indening Lustre code.
<pre style="background:lightgrey;">
/* -*- mode: c; c-basic-offset: 8; indent-tabs-mode: nil; -*-
* vim:expandtab:shiftwidth=8:tabstop=8:
*/
</pre>
4. All lines should wrap at 80 characters. If it's getting too hard to wrap there, you probably need to break it up into more functions. In some cases, it is acceptable to remove a few spaces between function arguments to avoid overflowing onto the next line.

5. Don't have spaces or tabs on blank lines or at the end of lines.

6. Don't use "inline" unless you're doing something so performance critical that the function call overhead will make a difference -- in other words: never. It makes debugging harder.

All of our wrapping, parenthesis, brace placement, etc. rules are basically Linux kernel rules, which are basically K&R. For those of you in need of a refresher, great detail is provided below.

== Great detail ==

a. When you wrap, the next line should start after the parenthesis:
<pre style="background:lightgrey;">
right:

variable = do_something_complicated(long_argument, longer_argument,
longest_argument(sub_argument,
foo_argument),
last_argument);

if (some_long_condition(arg1, arg2, arg3) < some_long_value &&
another_long_condition(very_long_argument_name,
another_long_argument_name) >
second_long_value) {
}

wrong:

variable = do_something_complicated(long_argument, longer_argument,
longest_argument(sub_argument, foo_argument),
last_argument);

if (some_long_condition(arg1, arg2, arg3) < some_long_value &&
another_long_condition(very_long_argument_name,
another_long_argument_name) >
second_long_value) {
}
</pre>
b. If you're wrapping put the operators at the end of the line, and if there are no parentheses indent 8 more:
<pre style="background:lightgrey;">
off = le32_to_cpu(fsd->fsd_client_start) +
cl_idx * le16_to_cpu(fsd->fsd_client_size);
</pre>
c. Binary and ternary (but not unary) operators should be separated from their arguments by one space.
<pre style="background:lightgrey;">
a++;
b |= c;
d = f > g ? 0 : 1;
</pre>
d. Function calls should be nestled against the parentheses, the parentheses should crowd the arguments, and one space after commas:
<pre style="background:lightgrey;">
right: do_foo(bar, baz);
wrong: do_foo ( bar,baz );
</pre>

e. All ''if'', ''for'', ''while'', etc. expressions should be separated by a space from the parenthesis, one space after the semicolons:
<pre style="background:lightgrey;">
for (a = 0; a < b; a++)
if (a < b || a == c)
while (1)
</pre>
f. Opening braces should be on the same line as the line that introduces the block, except for function calls. Closing braces get their own line, except for "else".
<pre style="background:lightgrey;">
int foo(void)
{
if (bar) {
this();
that();
} else if (baz) {
;
} else {
;
}
do {
cow();
} while (0);
}
</pre>
g. If one part of a compound ''if'' block has braces, all should.
<pre style="background:lightgrey;">
right:

if (foo) {
bar();
baz();
} else {
salmon();
}

wrong:

if (foo) {
bar();
baz();
} else
moose();
</pre>
h. When you make a macro, protect those who might call it by using do/while and parentheses; line up your backslashes:
<pre style="background:lightgrey;">
right:

#define DO_STUFF(a) \
do { \
int b = (a) + MAGIC; \
do_other_stuff(b); \
} while (0)

wrong:

#define DO_STUFF(a) \
{ \
int b = a + MAGIC; \
do_other_stuff(b); \
}
</pre>
i. If you nest preprocessor commands, use spaces to visually delineate:

#ifdef __KERNEL__
# include <goose>
# define MOOSE steak
#else
# include <mutton>
# define MOOSE prancing
#endif

j. For very long #ifdefs include the conditional with each #endif to make it readable:

#ifdef __KERNEL__
# if LINUX_VERSION_CODE >= KERNEL_VERSION(2,5,0)
/* lots
of
stuff */
# endif /* KERNEL_VERSION(2,5,0) */
#else /* !__KERNEL__ */
# if HAVE_FEATURE
/* more
* stuff */
# endif
#endif /* __KERNEL__ */

k. Comments should have the leading '''/*''' on the same line as the comment, and the trailing '''*/''' at the end of the last comment line. Intermediate lines should start with a '''*''' aligned with the first line's '''*''':
/* This is a short comment */

/* This is a multi-line comment. I wish the line would wrap already,
* as I don't have much to write about. */

l. Function declarations absolutely should NOT go into .c files, unless they are forward declarations for static functions that can't otherwise be moved before the caller. Instead, the declaration should go into the most "local" header available (preferrably *_internal.h for a given piece of code).

m. Structure and constant declarations should not be declared in multiple places. Put the struct into the most "local" header possible. If it is something that is passed over the wire it needs to go into lustre_idl.h, and needs to be correctly swabbed when the RPC message is unpacked.

n. The types and printf/printk formats used by Lustre code are:

__u64 LPU64/LPX64/LPD64 (unsigned, hex, signed)
size_t LPSZ (or cast to int and use %u / %d)
__u32/int %u/%x/%d (unsigned, hex, signed)
(unsigned) long long %llu/%llx/%lld
loff_t %lld after a cast to long long (unfortunately)

----
*'''[http://wiki.lustre.org/index.php?title=Front_Page FrontPage]'''

Coding Guidelines

2007-06-11T07:39:27Z

Yep: /* Guidelines */

All Lustre developers should follow the guidelines in this page very strictly to avoid problems during code merges later on. Please make the required changes to the default formatting rules in the editor you use to comply to the guidelines below.

== Guidelines ==

1. There should be no tabs in any files.

2. Blocks should be indented by 8 spaces.

3. New files should contain the following along with the license boilerplate. This will cause vim and emacs to use spaces instead of tabs for indenting. If you use a different editor, it also needs to be set to use spaces for indening Lustre code.
<pre style="background:lightgrey;"
/* -*- mode: c; c-basic-offset: 8; indent-tabs-mode: nil; -*-
* vim:expandtab:shiftwidth=8:tabstop=8:
*/
</pre>
4. All lines should wrap at 80 characters. If it's getting too hard to wrap there, you probably need to break it up into more functions. In some cases, it is acceptable to remove a few spaces between function arguments to avoid overflowing onto the next line.

5. Don't have spaces or tabs on blank lines or at the end of lines.

6. Don't use "inline" unless you're doing something so performance critical that the function call overhead will make a difference -- in other words: never. It makes debugging harder.

All of our wrapping, parenthesis, brace placement, etc. rules are basically Linux kernel rules, which are basically K&R. For those of you in need of a refresher, great detail is provided below.

== Great detail ==

a. When you wrap, the next line should start after the parenthesis:

right:

variable = do_something_complicated(long_argument, longer_argument,
longest_argument(sub_argument,
foo_argument),
last_argument);

if (some_long_condition(arg1, arg2, arg3) < some_long_value &&
another_long_condition(very_long_argument_name,
another_long_argument_name) >
second_long_value) {
}

wrong:

variable = do_something_complicated(long_argument, longer_argument,
longest_argument(sub_argument, foo_argument),
last_argument);

if (some_long_condition(arg1, arg2, arg3) < some_long_value &&
another_long_condition(very_long_argument_name,
another_long_argument_name) >
second_long_value) {
}

b. If you're wrapping put the operators at the end of the line, and if there are no parentheses indent 8 more:

off = le32_to_cpu(fsd->fsd_client_start) +
cl_idx * le16_to_cpu(fsd->fsd_client_size);

c. Binary and ternary (but not unary) operators should be separated from their arguments by one space.

a++;
b |= c;
d = f > g ? 0 : 1;

d. Function calls should be nestled against the parentheses, the parentheses should crowd the arguments, and one space after commas:

right: do_foo(bar, baz);
wrong: do_foo ( bar,baz );

e. All ''if'', ''for'', ''while'', etc. expressions should be separated by a space from the parenthesis, one space after the semicolons:

for (a = 0; a < b; a++)
if (a < b || a == c)
while (1)

f. Opening braces should be on the same line as the line that introduces the block, except for function calls. Closing braces get their own line, except for "else".

int foo(void)
{
if (bar) {
this();
that();
} else if (baz) {
;
} else {
;
}
do {
cow();
} while (0);
}

g. If one part of a compound ''if'' block has braces, all should.

right:

if (foo) {
bar();
baz();
} else {
salmon();
}

wrong:

if (foo) {
bar();
baz();
} else
moose();

h. When you make a macro, protect those who might call it by using do/while and parentheses; line up your backslashes:

right:

#define DO_STUFF(a) \
do { \
int b = (a) + MAGIC; \
do_other_stuff(b); \
} while (0)

wrong:

#define DO_STUFF(a) \
{ \
int b = a + MAGIC; \
do_other_stuff(b); \
}

i. If you nest preprocessor commands, use spaces to visually delineate:

#ifdef __KERNEL__
# include <goose>
# define MOOSE steak
#else
# include <mutton>
# define MOOSE prancing
#endif

j. For very long #ifdefs include the conditional with each #endif to make it readable:

#ifdef __KERNEL__
# if LINUX_VERSION_CODE >= KERNEL_VERSION(2,5,0)
/* lots
of
stuff */
# endif /* KERNEL_VERSION(2,5,0) */
#else /* !__KERNEL__ */
# if HAVE_FEATURE
/* more
* stuff */
# endif
#endif /* __KERNEL__ */

k. Comments should have the leading '''/*''' on the same line as the comment, and the trailing '''*/''' at the end of the last comment line. Intermediate lines should start with a '''*''' aligned with the first line's '''*''':
/* This is a short comment */

/* This is a multi-line comment. I wish the line would wrap already,
* as I don't have much to write about. */

l. Function declarations absolutely should NOT go into .c files, unless they are forward declarations for static functions that can't otherwise be moved before the caller. Instead, the declaration should go into the most "local" header available (preferrably *_internal.h for a given piece of code).

m. Structure and constant declarations should not be declared in multiple places. Put the struct into the most "local" header possible. If it is something that is passed over the wire it needs to go into lustre_idl.h, and needs to be correctly swabbed when the RPC message is unpacked.

n. The types and printf/printk formats used by Lustre code are:

__u64 LPU64/LPX64/LPD64 (unsigned, hex, signed)
size_t LPSZ (or cast to int and use %u / %d)
__u32/int %u/%x/%d (unsigned, hex, signed)
(unsigned) long long %llu/%llx/%lld
loff_t %lld after a cast to long long (unfortunately)

----
*'''[http://wiki.lustre.org/index.php?title=Front_Page FrontPage]'''

Main Page

2007-05-22T07:51:09Z

Yep:

== Links to Lustre resources and knowledge ==

* Lustre Architecture Wiki - http://arch.lustre.org
* Lustre Manual and Documentation - http://manual.lustre.org
* Lustre Centre of Excellence Wiki Sites
** ORNL LCE Wiki - http://ornl-lce.clusterfs.com
* Lustre IO Performance Project Page - http://wiki.lustre.org/index.php?title=IOPerformanceProject

== Documentation ==

*[http://wiki.lustre.org/index.php?title=Lustre_Documentation Lustre Documentation] - Lustre Manual and also links to older documents

== Lustre User Group ==

* LUG Requirements Forum - [http://wiki.lustre.org/images/7/78/LUG-Requirements-060420-final.pdf LUG-Requirements-060420-final.pdf] | [http://wiki.lustre.org/images/7/78/LUG-Requirements-060420-final.xls LUG-Requirements-060420-final.xls]
* Lustre Users Group (http://lustreusers.org)

== Community Development Activities ==

* [http://wiki.lustre.org/index.php?title=Networking_Development NetworkingDevelopment]
* [http://wiki.lustre.org/index.php?title=Diskless_Booting DisklessBooting]
* [http://wiki.lustre.org/index.php?title=Parallel_Io_Enhancements ParallelIoEnhancements]
* [http://wiki.lustre.org/index.php?title=Drbd_And_Lustre DrbdAndLustre]
* Bull- Open Source tools for Lustre http://www.bullopensource.org/lustre
* LLNL- Lustre Monitoring Tool http://www.sourceforge.net/projects/lmt/
* [http://wiki.lustre.org/index.php?title=Contributed_Patches_And_Code ContributedPatchesAndCode]

== Lustre Testing & QA ==

* [http://wiki.lustre.org/images/7/78/vision-scope.html Vision & Scope of New Testing System] and [http://wiki.lustre.org/images/7/78/use-cases.html Use Cases]
* Buffalo Test Reports (http://buffalo.lustre.org)
* '''[http://wiki.lustre.org/index.php?title=Lustre_Testing LustreTesting]''' - Our QA practices, procedures and support infrastructure.
* '''[http://wiki.lustre.org/index.php?title=Testing_Tools TestingTools]''' - Help on some of available Lustre testing tools.(l2dbench, echo client/server etc..)

== Development Practices ==

* '''Bugs'''
** '''[http://wiki.lustre.org/index.php?title=Bug_Filing BugFiling]''' - A guide for testers.
** '''[http://wiki.lustre.org/index.php?title=Bug_Management BugManagement]''' - How to accept and manage bug fixings with Bugzilla.
** Lustre Bugzilla (http://bugzilla.lustre.org)
* '''Development'''
** '''[http://wiki.lustre.org/index.php?title=Coding_Guide_lines CodingGuidelines]''' - Formatting and indentation that should strictly be followed for any code checked into any branch.
** '''[http://wiki.lustre.org/index.php?title=Contribution_Policy ContributionPolicy]''' - What to do to get your changes in.
** '''[http://wiki.lustre.org/index.php?title=Cvs_Branches CvsBranches]''' - How to manage branches with CVS.
** '''[http://wiki.lustre.org/index.php?title=Cvs_Tips CvsTips]''' - Helpful things to know while using Lustre CVS.
** '''[http://wiki.lustre.org/index.php?title=Kernel_Patch_Management KernelPatchManagement]''' - How to handle the kernel patches.
** '''[http://wiki.lustre.org/index.php?title=Lustre_Debugging LustreDebugging]''' - A guide to Lustre debugging.
** '''[http://wiki.lustre.org/index.php?title=Lustre_Uml LustreUml]''' - Setting up and running Lustre on UMLs.
** '''[http://wiki.lustre.org/index.php?title=Development_Cluster DevelopmentCluster]''' - Properties of a cluster that offers a good debugging environment.
** '''[http://wiki.lustre.org/index.php?title=Linux_Debugging LinuxDebugging]''' - Debugging tools available for Linux kernel debugging(Kgdb, vmware, netdump, netconsole).
* '''Release Engineering'''
** '''[http://wiki.lustre.org/index.php?title=Release_Planning ReleasePlanning]''' - What QA goals accompany merges and releases.
** '''[http://wiki.lustre.org/index.php?title=Kernel_Patch_Management KernelPatchManagement]''' - How to handle the kernel patches.
** '''[http://wiki.lustre.org/index.php?title=Lustre_Statuson_Linux26 LustreStatusonLinux26]''' - Contributed patches and debugging hints

== Some Design Information ==

* '''[http://wiki.lustre.org/index.php?title=Ext3_Development Ext3Development]''' - community ext3 development work
* '''[http://wiki.lustre.org/index.php?title=File_Creation FileCreation]''' - Alternative file create path
* '''[http://wiki.lustre.org/index.php?title=Design_Changes DesignChanges]''' - Design work summary of last 6 months
** '''[http://wiki.lustre.org/index.php?title=Lock_Refinements LockRefinements]''' - Lustre Lite Performance lock refinements
** '''[http://wiki.lustre.org/index.php?title=Import_Statest ImportStates]''' - Refine client import states.
* '''[http://wiki.lustre.org/index.php?title=Recovery_Overview RecoveryOverview]''' - How Lustre recovers from various failure cases.
* '''[http://wiki.lustre.org/index.php?title=Service_Monitors ServiceMonitors]''' - Design for permanent Lustre service monitoring.
* '''[http://wiki.lustre.org/index.php?title=Raid5_Patches Raid5Patches]''' - patches to Linux RAID5 code to improve Lustre performance

----

Main Page

2007-05-22T07:48:05Z

Yep:

== Main Pages ==
This is the new Lustre Wiki. Here you find links to Lustre resources and knowledge.
* Lustre Architecture Wiki - http://arch.lustre.org
* Lustre Manual and Documentation - http://manual.lustre.org
* Lustre Centre of Excellence Wiki Sites
** ORNL LCE Wiki - http://ornl-lce.clusterfs.com
* Lustre IO Performance Project Page - http://wiki.lustre.org/index.php?title=IOPerformanceProject

== Documentation ==

*[http://wiki.lustre.org/index.php?title=Lustre_Documentation Lustre Documentation] - Lustre Manual and also links to older documents

== Lustre User Group ==

* LUG Requirements Forum - [http://wiki.lustre.org/images/7/78/LUG-Requirements-060420-final.pdf LUG-Requirements-060420-final.pdf] | [http://wiki.lustre.org/images/7/78/LUG-Requirements-060420-final.xls LUG-Requirements-060420-final.xls]
* Lustre Users Group (http://lustreusers.org)

== Community Development Activities ==

* [http://wiki.lustre.org/index.php?title=Networking_Development NetworkingDevelopment]
* [http://wiki.lustre.org/index.php?title=Diskless_Booting DisklessBooting]
* [http://wiki.lustre.org/index.php?title=Parallel_Io_Enhancements ParallelIoEnhancements]
* [http://wiki.lustre.org/index.php?title=Drbd_And_Lustre DrbdAndLustre]
* Bull- Open Source tools for Lustre http://www.bullopensource.org/lustre
* LLNL- Lustre Monitoring Tool http://www.sourceforge.net/projects/lmt/
* [http://wiki.lustre.org/index.php?title=Contributed_Patches_And_Code ContributedPatchesAndCode]

== Lustre Testing & QA ==

* [http://wiki.lustre.org/images/7/78/vision-scope.html Vision & Scope of New Testing System] and [http://wiki.lustre.org/images/7/78/use-cases.html Use Cases]
* Buffalo Test Reports (http://buffalo.lustre.org)
* '''[http://wiki.lustre.org/index.php?title=Lustre_Testing LustreTesting]''' - Our QA practices, procedures and support infrastructure.
* '''[http://wiki.lustre.org/index.php?title=Testing_Tools TestingTools]''' - Help on some of available Lustre testing tools.(l2dbench, echo client/server etc..)

== Development Practices ==

* '''Bugs'''
** '''[http://wiki.lustre.org/index.php?title=Bug_Filing BugFiling]''' - A guide for testers.
** '''[http://wiki.lustre.org/index.php?title=Bug_Management BugManagement]''' - How to accept and manage bug fixings with Bugzilla.
** Lustre Bugzilla (http://bugzilla.lustre.org)
* '''Development'''
** '''[http://wiki.lustre.org/index.php?title=Coding_Guide_lines CodingGuidelines]''' - Formatting and indentation that should strictly be followed for any code checked into any branch.
** '''[http://wiki.lustre.org/index.php?title=Contribution_Policy ContributionPolicy]''' - What to do to get your changes in.
** '''[http://wiki.lustre.org/index.php?title=Cvs_Branches CvsBranches]''' - How to manage branches with CVS.
** '''[http://wiki.lustre.org/index.php?title=Cvs_Tips CvsTips]''' - Helpful things to know while using Lustre CVS.
** '''[http://wiki.lustre.org/index.php?title=Kernel_Patch_Management KernelPatchManagement]''' - How to handle the kernel patches.
** '''[http://wiki.lustre.org/index.php?title=Lustre_Debugging LustreDebugging]''' - A guide to Lustre debugging.
** '''[http://wiki.lustre.org/index.php?title=Lustre_Uml LustreUml]''' - Setting up and running Lustre on UMLs.
** '''[http://wiki.lustre.org/index.php?title=Development_Cluster DevelopmentCluster]''' - Properties of a cluster that offers a good debugging environment.
** '''[http://wiki.lustre.org/index.php?title=Linux_Debugging LinuxDebugging]''' - Debugging tools available for Linux kernel debugging(Kgdb, vmware, netdump, netconsole).
* '''Release Engineering'''
** '''[http://wiki.lustre.org/index.php?title=Release_Planning ReleasePlanning]''' - What QA goals accompany merges and releases.
** '''[http://wiki.lustre.org/index.php?title=Kernel_Patch_Management KernelPatchManagement]''' - How to handle the kernel patches.
** '''[http://wiki.lustre.org/index.php?title=Lustre_Statuson_Linux26 LustreStatusonLinux26]''' - Contributed patches and debugging hints

== Some Design Information ==

* '''[http://wiki.lustre.org/index.php?title=Ext3_Development Ext3Development]''' - community ext3 development work
* '''[http://wiki.lustre.org/index.php?title=File_Creation FileCreation]''' - Alternative file create path
* '''[http://wiki.lustre.org/index.php?title=Design_Changes DesignChanges]''' - Design work summary of last 6 months
** '''[http://wiki.lustre.org/index.php?title=Lock_Refinements LockRefinements]''' - Lustre Lite Performance lock refinements
** '''[http://wiki.lustre.org/index.php?title=Import_Statest ImportStates]''' - Refine client import states.
* '''[http://wiki.lustre.org/index.php?title=Recovery_Overview RecoveryOverview]''' - How Lustre recovers from various failure cases.
* '''[http://wiki.lustre.org/index.php?title=Service_Monitors ServiceMonitors]''' - Design for permanent Lustre service monitoring.
* '''[http://wiki.lustre.org/index.php?title=Raid5_Patches Raid5Patches]''' - patches to Linux RAID5 code to improve Lustre performance

----

Lustre Documentation

2007-05-15T02:40:28Z

Yep: /* Older Documentation */

= Lustre Documentation =
== Lustre Manual ==

=== Lustre 1.6 ===
* '''Lustre Manual - HTML '''[http://wiki.lustre.org/images/7/78/LustreManual1_6.html Lustre 1.6 manual V1.1]
* '''Lustre Manual - PDF'''
** ''''''[http://wiki.lustre.org/images/7/78/LustreManual1_6.pdf Lustre 1.6 manual V1.1(A4)]
**''''''[http://wiki.lustre.org/images/7/78/LustreManual-letter1_6.pdf Lustre 1.6 manual V1.1(Letter)]
* '''Lustre Manual Changelog '''[http://wiki.lustre.org/images/7/78/manual-changelog1_6.html Lustre Manual-Change log]
* '''[http://wiki.lustre.org/index.php?title=Mount_Conf MountConf wiki]''' - A quick reference for those familiar with older versions of Lustre

=== Lustre 1.4.8 ===
* '''Lustre Manual - HTML '''[http://wiki.lustre.org/images/7/78/LustreManual.html Lustre 1.4.8 manual V1.37]
* '''Lustre Manual - PDF'''
** ''''''[http://wiki.lustre.org/images/7/78/LustreManual37.pdf Lustre 1.4.8 manual V1.37(A4)]
** '''Lustre Manual Changelog '''[http://wiki.lustre.org/images/7/78/manual-changelog.html Lustre Manual-Change log]

== Interim Lustre 1.8 Documentation ==
* '''[http://wiki.lustre.org/index.php?title=Kerb_Lustre KerbLustre]''' - The 1.8 interim Lustre documentation (Kerberos....)

== Older Documentation ==
This documentation will be incorporated into the manual. Until that is done you may find useful bits and pieces here.

* '''Frequently Asked Questions'''
** [http://www.clusterfs.com/faq.html Lustre Faq]
* '''Knowledge Base'''
** [http://bugzilla.lustre.org/showdependencytree.cgi?id=2374 Lustre Knowledge Base] - questions and answers
* '''Configuration'''
** [http://wiki.lustre.org/index.php?title=Lustre_Howto LustreHowto]''' - A guide to getting Lustre cluster started.
** [http://wiki.lustre.org/index.php?title=Lustre_LDAP LustreLDAP]''' - A guide for using LDAP with Lustre.
** [http://wiki.lustre.org/index.php?title=Lustre_Wizard LustreWizard]''' - The ''Lustre wizard'' or ''lwizard'' is a utility that helps with creation of configuration file for a cluster through asking some simple questions.
** [http://wiki.lustre.org/index.php?title=Lustre_Failover LustreFailover]''' - Managing failover
** [http://wiki.lustre.org/index.php?title=Filesystem_Backup FilesystemBackup]''' - How to back up a Lustre filesystem
** [http://wiki.lustre.org/index.php?title=Fsck_Suppor FsckSupport]''' - The lustre fsck tool and Lustre-patched e2fsck (extents, large inode/EA support)
** '''Filesystem Tuning'''
*[http://wiki.lustre.org/images/7/78/LustreManual.html#Chapter_III-2._LustreProc LustreProc] - A guide on the '''proc''' tunables parameters for Lustre and their usage. It describes several of the ''proc'' tunables including those that effect the client's RPC behaviour and prepare for a substantial reorganization of proc entries.
*'''[http://wiki.lustre.org/images/7/78/LustreManual.html#3.2_DDN_Tuning LustreDdnTuning]''' - A brief guide on tuning DDN S2A 8500 (and maybe 9500) storage optimally for Lustre

File:Mrtg-2.13.2.tar.gz

2007-05-07T10:13:20Z

Yep:

Lustre Documentation

2007-05-05T22:41:48Z

Yep: /* Lustre 1.4.8 */

= Lustre Documentation =
== Lustre Manual ==

=== Lustre 1.6 ===
* '''Lustre Manual - HTML '''[http://wiki.lustre.org/images/7/78/LustreManual1_6.html Lustre 1.6 manual V1.1]
* '''Lustre Manual - PDF'''
. ''''''[http://wiki.lustre.org/images/7/78/LustreManual1_6.pdf Lustre 1.6 manual V1.1(A4)]
. ''''''[http://wiki.lustre.org/images/7/78/LustreManual-letter1_6.pdf Lustre 1.6 manual V1.1(Letter)]
* '''Lustre Manual Changelog '''[http://wiki.lustre.org/images/7/78/manual-changelog1_6.html Lustre Manual-Change log]
* '''[http://wiki.lustre.org/index.php?title=Mount_Conf MountConf wiki]''' - A quick reference for those familiar with older versions of Lustre

=== Lustre 1.4.8 ===
* '''Lustre Manual - HTML '''[http://wiki.lustre.org/images/7/78/LustreManual.html Lustre 1.4.8 manual V1.37]
* '''Lustre Manual - PDF'''
. ''''''[http://wiki.lustre.org/images/7/78/LustreManual37.pdf Lustre 1.4.8 manual V1.37(A4)]
* '''Lustre Manual Changelog '''[http://wiki.lustre.org/images/7/78/manual-changelog.html Lustre Manual-Change log]

== Interim Lustre 1.8 Documentation ==
* '''[http://wiki.lustre.org/index.php?title=Kerb_Lustre KerbLustre]''' - The 1.8 interim Lustre documentation (Kerberos....)

== Older Documentation ==
This documentation will be incorporated into the manual. Until that is done you may find useful bits and pieces here.

* '''Frequently Asked Questions'''
. [http://www.clusterfs.com/faq.html Lustre Faq]
* '''Knowledge Base'''
. [http://bugzilla.lustre.org/showdependencytree.cgi?id=2374 Lustre Knowledge Base] - questions and answers
* '''Configuration'''
* '''[http://wiki.lustre.org/index.php?title=Lustre_Howto LustreHowto]''' - A guide to getting Lustre cluster started.
* '''[http://wiki.lustre.org/index.php?title=Lustre_LDAP LustreLDAP]''' - A guide for using LDAP with Lustre.
* '''[http://wiki.lustre.org/index.php?title=Lustre_Wizard LustreWizard]''' - The ''Lustre wizard'' or ''lwizard'' is a utility that helps with creation of configuration file for a cluster through asking some simple questions.
* '''[http://wiki.lustre.org/index.php?title=Lustre_Failover LustreFailover]''' - Managing failover
* '''[http://wiki.lustre.org/index.php?title=Filesystem_Backup FilesystemBackup]''' - How to back up a Lustre filesystem
* '''[http://wiki.lustre.org/index.php?title=Fsck_Suppor FsckSupport]''' - The lustre fsck tool and Lustre-patched e2fsck (extents, large inode/EA support)
* '''Filesystem Tuning'''
* [https://mail.clusterfs.com/wikis/attachments/LustreManual.html#Chapter_III-2._LustreProc LustreProc] - A guide on the '''proc''' tunables parameters for Lustre and their usage. It describes several of the ''proc'' tunables including those that effect the client's RPC behaviour and prepare for a substantial reorganization of proc entries.
* '''[https://mail.clusterfs.com/wikis/attachments/LustreManual.html#3.2_DDN_Tuning LustreDdnTuning]''' - A brief guide on tuning DDN S2A 8500 (and maybe 9500) storage optimally for Lustre

Lustre Documentation

2007-05-05T22:38:36Z

Yep: /* Lustre 1.6 */

= Lustre Documentation =
== Lustre Manual ==

=== Lustre 1.6 ===
* '''Lustre Manual - HTML '''[http://wiki.lustre.org/images/7/78/LustreManual1_6.html Lustre 1.6 manual V1.1]
* '''Lustre Manual - PDF'''
. ''''''[http://wiki.lustre.org/images/7/78/LustreManual1_6.pdf Lustre 1.6 manual V1.1(A4)]
. ''''''[http://wiki.lustre.org/images/7/78/LustreManual-letter1_6.pdf Lustre 1.6 manual V1.1(Letter)]
* '''Lustre Manual Changelog '''[http://wiki.lustre.org/images/7/78/manual-changelog1_6.html Lustre Manual-Change log]
* '''[http://wiki.lustre.org/index.php?title=Mount_Conf MountConf wiki]''' - A quick reference for those familiar with older versions of Lustre

=== Lustre 1.4.8 ===
* '''Lustre Manual - HTML '''[https://mail.clusterfs.com/wikis/attachments/LustreManual.html Lustre 1.4.8 manual V1.37]
* '''Lustre Manual - PDF'''
. ''''''[https://mail.clusterfs.com/wikis/attachments/LustreManual37.pdf Lustre 1.4.8 manual V1.37(A4)]
* '''Lustre Manual Changelog '''[https://mail.clusterfs.com/wikis/attachments/manual-changelog.html Lustre Manual-Change log]

== Interim Lustre 1.8 Documentation ==
* '''[http://wiki.lustre.org/index.php?title=Kerb_Lustre KerbLustre]''' - The 1.8 interim Lustre documentation (Kerberos....)

== Older Documentation ==
This documentation will be incorporated into the manual. Until that is done you may find useful bits and pieces here.

* '''Frequently Asked Questions'''
. [http://www.clusterfs.com/faq.html Lustre Faq]
* '''Knowledge Base'''
. [http://bugzilla.lustre.org/showdependencytree.cgi?id=2374 Lustre Knowledge Base] - questions and answers
* '''Configuration'''
* '''[http://wiki.lustre.org/index.php?title=Lustre_Howto LustreHowto]''' - A guide to getting Lustre cluster started.
* '''[http://wiki.lustre.org/index.php?title=Lustre_LDAP LustreLDAP]''' - A guide for using LDAP with Lustre.
* '''[http://wiki.lustre.org/index.php?title=Lustre_Wizard LustreWizard]''' - The ''Lustre wizard'' or ''lwizard'' is a utility that helps with creation of configuration file for a cluster through asking some simple questions.
* '''[http://wiki.lustre.org/index.php?title=Lustre_Failover LustreFailover]''' - Managing failover
* '''[http://wiki.lustre.org/index.php?title=Filesystem_Backup FilesystemBackup]''' - How to back up a Lustre filesystem
* '''[http://wiki.lustre.org/index.php?title=Fsck_Suppor FsckSupport]''' - The lustre fsck tool and Lustre-patched e2fsck (extents, large inode/EA support)
* '''Filesystem Tuning'''
* [https://mail.clusterfs.com/wikis/attachments/LustreManual.html#Chapter_III-2._LustreProc LustreProc] - A guide on the '''proc''' tunables parameters for Lustre and their usage. It describes several of the ''proc'' tunables including those that effect the client's RPC behaviour and prepare for a substantial reorganization of proc entries.
* '''[https://mail.clusterfs.com/wikis/attachments/LustreManual.html#3.2_DDN_Tuning LustreDdnTuning]''' - A brief guide on tuning DDN S2A 8500 (and maybe 9500) storage optimally for Lustre

File:LUG-Requirements-060420-final.pdf

2007-04-30T07:25:00Z

Yep:

RAID5 Patches

2007-04-30T07:11:12Z

Yep: /* Structures */

= Notes about RAID5 internals =

== Structures ==

In Linux RAID5 handles all incoming requests by small units called '''stripes'''.
Stripe is a set of '''blocks''' taken from all disks at the same position.
Block is defined as unit of PAGE_SIZE bytes.

For example, you have 3 disks and specified 8K chunksize. Then RAID5 will be looking internally as the following:
{|border=1 cellspacing=1
|-
| || S0 || S8 || S32 || S40 ||
|-
| Disk1 || #0 || #8 || #32 || #40 ||
|-
| Disk2 || #16 || #24 || #48 || #56 ||
|-
| Disk3 || P0 || P8 || P32 || P40 ||
|}
where:
* Sn -- number of internal stripe
* #n -- is an offset in sectors (512bytes)
* Pn -- parity for other blocks in the stripe (actually it floats among disks)

As you can see, 8K chunksize means 2 contig. blocks.

== Logic ==

''make_request()'' goes through incoming request, breaking it into '''blocks'''
(PAGE_SIZE) and handling them separately. given bio with bi_sector = 0
bi_size = 24K and array described above, ''make_request()'' would handle #0,
#8 and #16.

For every block, ''add_stripe_bio()'' and ''handle_stripe()'' are called.

''add_stripe_bio()'' intention is to add bio to given stripe, later in
''handle_stripe()'' we'll be able to use bio and its data for serving requests.

''handle_stripe()'' is a core of raid5, we'll discsuss it in the next part.

== handle_stripe() ==

the routine works with a stripe. it checks what should be done, learns current
state of a stripe in the internal cache, makes decision what I/O is needed to
satisfy users requests and does recovery.

say, user wants to write block #0 (8 sectors starting from sector 0). raid5's
responsibility is to store new data and update parity P0. there are few
possibilities here:
1. delay serving till data for block #16 is ready -- probably user will want to write #16 very soon?
2. read #16, make a new parity P0; write #0 and P0
3. read P0, rollback old #0 from P0 (so, it will look like we did parity with #0) and re-compute parity with new #0

1st way looks the better because it doesn't require very expensive read, but
the problem is that user may need to write only #0 and not #16 in near future.
also, the queue can get unplugged meaning that user wants all requests to
complete (unfortunately, in current block layer there is no way to specify
which exact request user is interested in, so any completion interest means
immediate serving of the whole queue).

== Problems ==

Short list of the problem in raid5 we met in Thumper project:

* order of handling isn't good for large requests
As ''handle_stripe()'' goes in logical block order, it
handles S0, then S8, then again S0 and S8. After the first touch
S0 is left with block #0 uptodate, while #16 and P0 are not. Thus
if the stripe is forced for completion, we'd need to read block
#16 or P0 to get full uptodate stripe. Such reads hurt throughput
almost to death. If just a single process writes, then things are
OK, because nobody unplugs the queue and there is no requests to
force completion of pending request. But the more writers, the
often queue unplug happens and the often pending requests are forced
for completion. Take into account that in reallity we use large
chuck size (128K, 256K and even larger), hence tons of non-uptodate
stripes in the cache and tons of reads in the end.

* memcpy() is top consumer
all requests go via internal cache. on dual-core 2way opteron
it takes up to 30-33% of CPU doing 1GB/s write

* small requests
to fill I/O pipes and reach good throughput we need quite large
I/O requests. Lustre does this using bio subsystem on 2.6. but
as it was mentioned, raid5 handles all blocks separately and
issues for every block separate I/O (bio). this is solved partial
by I/O scheduler that merges small requests into bigger ones, but
due to nature of block subsystem, any process that wants I/O to
get completed, ''unplug'' queue and we can get many small requests
in the pipe.

We developed patches that address described problems. You can find
them in ftp://ftp.clusterfs.com/pub/people/alex/raid5