Configuring Lustre File Striping

(Updated: Oct 2009)

One of the main factors leading to the high performance of Lustre™ file systems is the ability to stripe data over multiple OSTs. The stripe count can be set on a file system, directory, or file level. An example showing the use of striping is provided below.

For additional information, see Chapter 18: Managing File Striping and Free Space in the Lustre Operations Manual

Setting Up Striping

To see the current stripe size, use the command lfs getstripe [file, dir, fs]. This command will produce output similar to the following:

root@LustreClient01 lustre]# lfs getstripe /mnt/lustre
OBDS:
0: lustre-OST0000_UUID ACTIVE
1: lustre-OST0001_UUID ACTIVE
2: lustre-OST0002_UUID ACTIVE
3: lustre-OST0003_UUID ACTIVE
4: lustre-OST0004_UUID ACTIVE
5: lustre-OST0005_UUID ACTIVE
/mnt/lustre
(Default) stripe_count: 2 stripe_size: 4M stripe_offset: 0

In this example, the default stripe count is 2 (that is, data blocks are striped over two OSTs), the default stripe size is 4 MB (the stripe size can be set in K, M or G), and all writes start from the first OST.

Note: When setting the stripe, the offset is set before the stripe count.

The command to set a new stripe pattern on the file system may look like this:

[root@LustreClient01 lustre]# lfs setstripe -s 4M -i 0 -c 1 /mnt/lustre

This example command sets the stripe of /mnt/lustre to 4 MB blocks starting at OST0 and spanning over one OST. If a new file is created with these settings, the following results are seen:

[root@LustreClient01 lustre]# dd if=/dev/zero of=/mnt/lustre/test1 bs=10M count=100

root@LustreClient01 lustre]# lfs df -h
UUID                  bytes     Used  Available   Use%   Mounted on
lustre-MDT0000_UUID    4.4G   214.5M       3.9G     4%   /mnt/lustre[MDT:0]
lustre-OST0000_UUID    2.0G     1.1G     830.1M    53%   /mnt/lustre[OST:0]
lustre-OST0001_UUID    2.0G    83.3M       1.8G     4%   /mnt/lustre[OST:1]
lustre-OST0002_UUID    2.0G    83.3M       1.8G     4%   /mnt/lustre[OST:2]
lustre-OST0003_UUID    2.0G    83.3M       1.8G     4%   /mnt/lustre[OST:3]
lustre-OST0004_UUID    2.0G    83.3M       1.8G     4%   /mnt/lustre[OST:4]
lustre-OST0005_UUID    2.0G    83.3M       1.8G     4%   /mnt/lustre[OST:5]

filesystem summary:   11.8G     1.5G       9.7G    12%   /mnt/lustre

In this example, the entire file was written to the first OST with a very uneven distribution of data blocks.

Continuing with this example, the file is removed and the stripe count is changed to a value of -1 to specify striping over all available OSTs:

[root@LustreClient01 lustre]# lfs setstripe -s 4M -i 0 -c -1 /mnt/lustre

Now, when a file is created, the new stripe setting evenly distributes the data over all the available OSTs:

[root@LustreClient01 lustre]# dd if=/dev/zero of=/mnt/lustre/test1 bs=10M count=100
100+0 records in
100+0 records out
1048576000 bytes (1.0 GB) copied, 20.2589 seconds, 51.8 MB/s

[root@LustreClient01 lustre]# lfs df -h
UUID                  bytes     Used  Available   Use%   Mounted on
lustre-MDT0000_UUID    4.4G   214.5M       3.9G     4%  /mnt/lustre[MDT:0]
lustre-OST0000_UUID    2.0G   251.3M       1.6G    12%  /mnt/lustre[OST:0]
lustre-OST0001_UUID    2.0G   251.3M       1.6G    12%  /mnt/lustre[OST:1]
lustre-OST0002_UUID    2.0G   251.3M       1.6G    12%  /mnt/lustre[OST:2]
lustre-OST0003_UUID    2.0G   251.3M       1.6G    12%  /mnt/lustre[OST:3]
lustre-OST0004_UUID    2.0G   247.3M       1.6G    12%  /mnt/lustre[OST:4]
lustre-OST0005_UUID    2.0G   247.3M       1.6G    12%  /mnt/lustre[OST:5]

filesystem summary:   11.8G     1.5G       9.7G    12%  /mnt/lustre

Displaying Stripe Information for a File

The lfs getstripe command can be used to display information that shows over which OSTs a file is distributed. For example, the output from the following command (showing multiple obdidx entries) indicates that the file test1 is striped over all six active OSTs in the configuration:

[root@LustreClient01 ~]# lfs getstripe /mnt/lustre/test1
OBDS:
0: lustre-OST0000_UUID ACTIVE
1: lustre-OST0001_UUID ACTIVE
2: lustre-OST0002_UUID ACTIVE
3: lustre-OST0003_UUID ACTIVE
4: lustre-OST0004_UUID ACTIVE
5: lustre-OST0005_UUID ACTIVE
/mnt/lustre/test1
     obdidx      objid     objid      group
          0          8       0x8          0
          1          4       0x4          0
          2          5       0x5          0
          3          5       0x5          0
          4          4       0x4          0
          5          2       0x2          0

In contrast, the output from the following command, which lists just a single obdidx entry, indicates that the file test2 is contained on a single OST:

[root@LustreClient01 ~]# lfs getstripe /mnt/lustre/test_2
OBDS:
0: lustre-OST0000_UUID ACTIVE
1: lustre-OST0001_UUID ACTIVE
2: lustre-OST0002_UUID ACTIVE
3: lustre-OST0003_UUID ACTIVE
4: lustre-OST0004_UUID ACTIVE
5: lustre-OST0005_UUID ACTIVE
/mnt/lustre/test_2
   obdidx      objid     objid      group
        2          8       0x8          0