Configuring Lustre File Striping

(Updated: Oct 2009)

One of the main factors leading to the high performance of Lustre™ file systems is the ability to stripe data over multiple OSTs. The stripe count can be set on a file system, directory, or file level. An example showing the use of striping is provided below.

For additional information, see Chapter 18: Managing File Striping and Free Space in the Lustre Operations Manual

Setting Up Striping
To see the current stripe size, use the command lfs getstripe [file, dir, fs]. This command will produce output similar to the following:

root@LustreClient01 lustre]# lfs getstripe /mnt/lustre OBDS: 0: lustre-OST0000_UUID ACTIVE 1: lustre-OST0001_UUID ACTIVE 2: lustre-OST0002_UUID ACTIVE 3: lustre-OST0003_UUID ACTIVE 4: lustre-OST0004_UUID ACTIVE 5: lustre-OST0005_UUID ACTIVE /mnt/lustre (Default) stripe_count: 2 stripe_size: 4M stripe_offset: 0

In this example, the default stripe count is 2 (that is, data blocks are striped over two OSTs), the default stripe size is 4 MB (the stripe size can be set in K, M or G), and all writes start from the first OST.

Note: When setting the stripe, the offset is set before the stripe count.

The command to set a new stripe pattern on the file system may look like this:

[root@LustreClient01 lustre]# lfs setstripe -s 4M -c 0 -i 1 /mnt/lustre

This example command sets the stripe of /mnt/lustre to 4 MB blocks starting at OST0 and spanning over one OST. If a new file is created with these settings, the following results are seen:

[root@LustreClient01 lustre]# dd if=/dev/zero of=/mnt/lustre/test1 bs=10M count=100

root@LustreClient01 lustre]# lfs df -h UUID                 bytes     Used  Available   Use%   Mounted on lustre-MDT0000_UUID    4.4G   214.5M       3.9G     4%   /mnt/lustre[MDT:0] lustre-OST0000_UUID   2.0G     1.1G     830.1M    53%   /mnt/lustre[OST:0] lustre-OST0001_UUID   2.0G    83.3M       1.8G     4%   /mnt/lustre[OST:1] lustre-OST0002_UUID   2.0G    83.3M       1.8G     4%   /mnt/lustre[OST:2] lustre-OST0003_UUID   2.0G    83.3M       1.8G     4%   /mnt/lustre[OST:3] lustre-OST0004_UUID   2.0G    83.3M       1.8G     4%   /mnt/lustre[OST:4] lustre-OST0005_UUID   2.0G    83.3M       1.8G     4%   /mnt/lustre[OST:5]

filesystem summary:  11.8G     1.5G       9.7G    12%   /mnt/lustre

In this example, the entire file was written to the first OST with a very uneven distribution of data blocks.

Continuing with this example, the file is removed and the stripe count is changed to a value of -1 to specify striping over all available OSTs:

[root@LustreClient01 lustre]# lfs setstripe -s 4M -c 0 -i -1 /mnt/lustre

Now, when a file is created, the new stripe setting evenly distributes the data over all the available OSTs:

[root@LustreClient01 lustre]# dd if=/dev/zero of=/mnt/lustre/test1 bs=10M count=100 100+0 records in 100+0 records out 1048576000 bytes (1.0 GB) copied, 20.2589 seconds, 51.8 MB/s

[root@LustreClient01 lustre]# lfs df -h UUID                 bytes     Used  Available   Use%   Mounted on lustre-MDT0000_UUID    4.4G   214.5M       3.9G     4%  /mnt/lustre[MDT:0] lustre-OST0000_UUID   2.0G   251.3M       1.6G    12%  /mnt/lustre[OST:0] lustre-OST0001_UUID   2.0G   251.3M       1.6G    12%  /mnt/lustre[OST:1] lustre-OST0002_UUID   2.0G   251.3M       1.6G    12%  /mnt/lustre[OST:2] lustre-OST0003_UUID   2.0G   251.3M       1.6G    12%  /mnt/lustre[OST:3] lustre-OST0004_UUID   2.0G   247.3M       1.6G    12%  /mnt/lustre[OST:4] lustre-OST0005_UUID   2.0G   247.3M       1.6G    12%  /mnt/lustre[OST:5]

filesystem summary:  11.8G     1.5G       9.7G    12%  /mnt/lustre

Displaying Stripe Information for a File
The lfs getstripe command can be used to display information that shows over which OSTs a file is distributed. For example, the output from the following command (showing multiple obdidx entries) indicates that the file test1 is striped over all six active OSTs in the configuration:

[root@LustreClient01 ~]# lfs getstripe /mnt/lustre/test1 OBDS: 0: lustre-OST0000_UUID ACTIVE 1: lustre-OST0001_UUID ACTIVE 2: lustre-OST0002_UUID ACTIVE 3: lustre-OST0003_UUID ACTIVE 4: lustre-OST0004_UUID ACTIVE 5: lustre-OST0005_UUID ACTIVE /mnt/lustre/test1 obdidx     objid     objid      group 0         8       0x8          0 1         4       0x4          0 2         5       0x5          0 3         5       0x5          0 4         4       0x4          0 5         2       0x2          0

In contrast, the output from the following command, which lists just a single obdidx entry, indicates that the file test2 is contained on a single OST:

[root@LustreClient01 ~]# lfs getstripe /mnt/lustre/test_2 OBDS: 0: lustre-OST0000_UUID ACTIVE 1: lustre-OST0001_UUID ACTIVE 2: lustre-OST0002_UUID ACTIVE 3: lustre-OST0003_UUID ACTIVE 4: lustre-OST0004_UUID ACTIVE 5: lustre-OST0005_UUID ACTIVE /mnt/lustre/test_2 obdidx     objid     objid      group 2         8       0x8          0