Configuring Lustre File Striping

Have asked Torben to suggest subtopics for wiki

Intro...Configure striping to optimize performance ...

http://wiki.lustre.org/index.php/Striping_Guidelines

Cl400_OH - Slides 133-114

Page 13 in Torben's paper

OM Chapter 25 - Striping and IO Options

---

Using stripes

One of the main reasons for Lustre file system's performance is the striping of data blocks over multiple OSTs. The stripe count can be set on a file system, directory or file level.

To see the current stripe size, use the command lfs getstripe [file, dir, fs]. On the current system this will produce the following output:

root@LustreClient01 lustre]# lfs getstripe /mnt/lustre
OBDS:
0: lustre-OST0000_UUID ACTIVE
1: lustre-OST0001_UUID ACTIVE
2: lustre-OST0002_UUID ACTIVE
3: lustre-OST0003_UUID ACTIVE
4: lustre-OST0004_UUID ACTIVE
5: lustre-OST0005_UUID ACTIVE
/mnt/lustre
(Default) stripe_count: 2 stripe_size: 4M stripe_offset: 0

As can be seen, the default stripe count is 2 (that is, striping over two OSTs), default stripe size is 4 MB (can be set in K, M or G), and all writes start from the first OST.

Note: –When setting the stripe, the offset is set before the stripe count.

Setting a new stripe pattern on the file system can look like this:

[root@LustreClient01 lustre]# lfs setstripe /mnt/lustre 4M 0 1

This example sets the stripe of /mnt/lustre to 4 MB blocks starting at OST0 and spanning over one OST. If a new file is created with these settings, the following results are seen:

[root@LustreClient01 lustre]# dd if=/dev/zero of=/mnt/lustre/test1 bs=10M count=100
root@LustreClient01 lustre]# lfs df -h
UUID bytes Used Available Use% Mounted on
lustre-MDT0000_UUID 4.4G 214.5M 3.9G 4% /mnt/lustre[MDT:0]
lustre-OST0000_UUID 2.0G 1.1G 830.1M 53% /mnt/lustre[OST:0]
lustre-OST0001_UUID 2.0G 83.3M 1.8G 4% /mnt/lustre[OST:1]
lustre-OST0002_UUID 2.0G 83.3M 1.8G 4% /mnt/lustre[OST:2]
lustre-OST0003_UUID 2.0G 83.3M 1.8G 4% /mnt/lustre[OST:3]
lustre-OST0004_UUID 2.0G 83.3M 1.8G 4% /mnt/lustre[OST:4]
lustre-OST0005_UUID 2.0G 83.3M 1.8G 4% /mnt/lustre[OST:5]
filesystem summary: 11.8G 1.5G 9.7G 12% /mnt/lustre

As can be seen, the entire file was written to the first OST, and there is a very uneven distribution of data blocks.

Continuing with this example, the file is removed and the stripe count is changed to a value of -1, which means “stripe over all available OSTs.”

[root@LustreClient01 lustre]# lfs setstripe /mnt/lustre 4M 0 -1

Now, when a file is created, the new stripe setting evenly distributes the data over all available OSTs:

[root@LustreClient01 lustre]# dd if=/dev/zero of=/mnt/lustre/test1 bs=10M count=100
100+0 records in
100+0 records out
1048576000 bytes (1.0 GB) copied, 20.2589 seconds, 51.8 MB/s
[root@LustreClient01 lustre]# lfs df -h
UUID bytes Used Available Use% Mounted on
lustre-MDT0000_UUID 4.4G 214.5M 3.9G 4% /mnt/lustre[MDT:0]
lustre-OST0000_UUID 2.0G 251.3M 1.6G 12% /mnt/lustre[OST:0]
lustre-OST0001_UUID 2.0G 251.3M 1.6G 12% /mnt/lustre[OST:1]
lustre-OST0002_UUID 2.0G 251.3M 1.6G 12% /mnt/lustre[OST:2]
lustre-OST0003_UUID 2.0G 251.3M 1.6G 12% /mnt/lustre[OST:3]
lustre-OST0004_UUID 2.0G 247.3M 1.6G 12% /mnt/lustre[OST:4]
lustre-OST0005_UUID 2.0G 247.3M 1.6G 12% /mnt/lustre[OST:5]
filesystem summary: 11.8G 1.5G 9.7G 12% /mnt/lustre

= Determining stripe information for a file

The lfs getstripe command can be used to display information that shows over which OSTs a file is distributed. For example, the output from the following command (the multiple obdidx entries) indicates that the file test1 is striped over all six active OSTs in the configuration:

[root@LustreClient01 ~]# lfs getstripe /mnt/lustre/test1
OBDS:
0: lustre-OST0000_UUID ACTIVE
1: lustre-OST0001_UUID ACTIVE
2: lustre-OST0002_UUID ACTIVE
3: lustre-OST0003_UUID ACTIVE
4: lustre-OST0004_UUID ACTIVE
5: lustre-OST0005_UUID ACTIVE
/mnt/lustre/test1
obdidx objidobjid group
0 8 0x8 0
1 4 0x4 0
2 5 0x5 0
3 5 0x5 0
4 4 0x4 0
5 2 0x2 0

In contrast, the output from the following command, which lists just a single obdidx entry, indicates that the file test2 is contained on a single OST:

[root@LustreClient01 ~]# lfs getstripe /mnt/lustre/test_2
OBDS:
0: lustre-OST0000_UUID ACTIVE
1: lustre-OST0001_UUID ACTIVE
2: lustre-OST0002_UUID ACTIVE
3: lustre-OST0003_UUID ACTIVE
4: lustre-OST0004_UUID ACTIVE
5: lustre-OST0005_UUID ACTIVE
/mnt/lustre/test_2
obdidx objidobjid group
2 80x80