WARNING: This is the _old_ Lustre wiki, and it is in the process of being retired. The information found here is all likely to be out of date. Please search the new wiki for more up to date information.

Difference between revisions of "Configuring Lustre File Striping"

From Obsolete Lustre Wiki
Jump to navigationJump to search
(-c and -i options were swapped)
 
(32 intermediate revisions by 2 users not shown)
Line 1: Line 1:
[[Have asked Torben to suggest subtopics for wiki]]
+
<small>''(Updated: Oct 2009)''</small>
 +
__TOC__
 +
One of the main factors leading to the high performance of Lustre™ file systems is the ability to stripe data over multiple OSTs. The stripe count can be set on a file system, directory, or file level.  An example showing the use of striping is provided below.
  
Intro...Configure striping to optimize performance ...
+
For additional information, see [http://wiki.lustre.org/manual/LustreManual20_HTML/ManagingStripingFreeSpace.html#50438209_pgfId-5529 Chapter 18: ''Managing File Striping and Free Space''] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual'']
  
http://wiki.lustre.org/index.php/Striping_Guidelines
+
== Setting Up Striping ==
  
Cl400_OH - Slides 133-114
+
To see the current stripe size, use the command ''lfs getstripe [file, dir, fs]''. This command will produce output similar to the following:
 
 
Page 13 in Torben's paper
 
 
 
OM Chapter 25 - Striping and IO Options
 
-----
 
 
 
== Using stripes ==
 
One of the main reasons for Lustre file system's performance is the striping of data blocks over multiple OSTs. The stripe count can be set on a file system, directory or file level.
 
 
 
To see the current stripe size, use the command ''lfs getstripe [file, dir, fs]''. On the current system this will produce the following output:
 
  
 
<pre>
 
<pre>
Line 30: Line 22:
 
</pre>
 
</pre>
  
As can be seen, the default stripe count is 2 (that is, striping over two OSTs), default stripe size is 4 MB (can be set in K, M or G), and all writes start from the first OST.
+
In this example, the default stripe count is 2 (that is, data blocks are striped over two OSTs), the default stripe size is 4 MB (the stripe size can be set in K, M or G), and all writes start from the first OST.
  
'''''Note:''''' –When setting the stripe, the offset is set before the stripe count.
+
'''''Note:''''' When setting the stripe, the offset is set before the stripe count.
  
Setting a new stripe pattern on the file system can look like this:
+
The command to set a new stripe pattern on the file system may look like this:
  
  [root@LustreClient01 lustre]# lfs setstripe /mnt/lustre 4M 0 1
+
  [root@LustreClient01 lustre]# lfs setstripe -s 4M -i 0 -c 1 /mnt/lustre
  
This example sets the stripe of ''/mnt/lustre'' to 4 MB blocks starting at OST0 and spanning over one OST. If a new file is created with these settings, the following results are seen:
+
This example command sets the stripe of ''/mnt/lustre'' to 4 MB blocks starting at OST0 and spanning over one OST. If a new file is created with these settings, the following results are seen:
  
 
<pre>
 
<pre>
 
[root@LustreClient01 lustre]# dd if=/dev/zero of=/mnt/lustre/test1 bs=10M count=100
 
[root@LustreClient01 lustre]# dd if=/dev/zero of=/mnt/lustre/test1 bs=10M count=100
 +
 
root@LustreClient01 lustre]# lfs df -h
 
root@LustreClient01 lustre]# lfs df -h
UUID bytes Used Available Use% Mounted on
+
UUID                 bytes     Used Available   Use%   Mounted on
lustre-MDT0000_UUID 4.4G 214.5M 3.9G 4% /mnt/lustre[MDT:0]
+
lustre-MDT0000_UUID   4.4G   214.5M       3.9G     4%   /mnt/lustre[MDT:0]
lustre-OST0000_UUID 2.0G 1.1G 830.1M 53% /mnt/lustre[OST:0]
+
lustre-OST0000_UUID   2.0G     1.1G     830.1M   53%   /mnt/lustre[OST:0]
lustre-OST0001_UUID 2.0G 83.3M 1.8G 4% /mnt/lustre[OST:1]
+
lustre-OST0001_UUID   2.0G   83.3M       1.8G     4%   /mnt/lustre[OST:1]
lustre-OST0002_UUID 2.0G 83.3M 1.8G 4% /mnt/lustre[OST:2]
+
lustre-OST0002_UUID   2.0G   83.3M       1.8G     4%   /mnt/lustre[OST:2]
lustre-OST0003_UUID 2.0G 83.3M 1.8G 4% /mnt/lustre[OST:3]
+
lustre-OST0003_UUID   2.0G   83.3M       1.8G     4%   /mnt/lustre[OST:3]
lustre-OST0004_UUID 2.0G 83.3M 1.8G 4% /mnt/lustre[OST:4]
+
lustre-OST0004_UUID   2.0G   83.3M       1.8G     4%   /mnt/lustre[OST:4]
lustre-OST0005_UUID 2.0G 83.3M 1.8G 4% /mnt/lustre[OST:5]
+
lustre-OST0005_UUID   2.0G   83.3M       1.8G     4%   /mnt/lustre[OST:5]
filesystem summary: 11.8G 1.5G 9.7G 12% /mnt/lustre
+
 
 +
filesystem summary:   11.8G     1.5G       9.7G   12%   /mnt/lustre
 
</pre>
 
</pre>
  
As can be seen, the entire file was written to the first OST, and there is a very uneven distribution of data blocks.
+
In this example, the entire file was written to the first OST with a very uneven distribution of data blocks.
  
Continuing with this example, the file is removed and the stripe count is changed to a value of -1, which means “stripe over all available OSTs.”
+
Continuing with this example, the file is removed and the stripe count is changed to a value of ''-1'' to specify striping over all available OSTs:
  
  [root@LustreClient01 lustre]# lfs setstripe /mnt/lustre 4M 0 -1
+
  [root@LustreClient01 lustre]# lfs setstripe -s 4M -i 0 -c -1 /mnt/lustre
  
Now, when a file is created, the new stripe setting evenly distributes the data over all available OSTs:
+
Now, when a file is created, the new stripe setting evenly distributes the data over all the available OSTs:
  
 
<pre>
 
<pre>
Line 67: Line 61:
 
100+0 records out
 
100+0 records out
 
1048576000 bytes (1.0 GB) copied, 20.2589 seconds, 51.8 MB/s
 
1048576000 bytes (1.0 GB) copied, 20.2589 seconds, 51.8 MB/s
 +
 
[root@LustreClient01 lustre]# lfs df -h
 
[root@LustreClient01 lustre]# lfs df -h
UUID bytes Used Available Use% Mounted on
+
UUID                 bytes     Used Available   Use%   Mounted on
lustre-MDT0000_UUID 4.4G 214.5M 3.9G 4% /mnt/lustre[MDT:0]
+
lustre-MDT0000_UUID   4.4G   214.5M       3.9G     4% /mnt/lustre[MDT:0]
lustre-OST0000_UUID 2.0G 251.3M 1.6G 12% /mnt/lustre[OST:0]
+
lustre-OST0000_UUID   2.0G   251.3M       1.6G   12% /mnt/lustre[OST:0]
lustre-OST0001_UUID 2.0G 251.3M 1.6G 12% /mnt/lustre[OST:1]
+
lustre-OST0001_UUID   2.0G   251.3M       1.6G   12% /mnt/lustre[OST:1]
lustre-OST0002_UUID 2.0G 251.3M 1.6G 12% /mnt/lustre[OST:2]
+
lustre-OST0002_UUID   2.0G   251.3M       1.6G   12% /mnt/lustre[OST:2]
lustre-OST0003_UUID 2.0G 251.3M 1.6G 12% /mnt/lustre[OST:3]
+
lustre-OST0003_UUID   2.0G   251.3M       1.6G   12% /mnt/lustre[OST:3]
lustre-OST0004_UUID 2.0G 247.3M 1.6G 12% /mnt/lustre[OST:4]
+
lustre-OST0004_UUID   2.0G   247.3M       1.6G   12% /mnt/lustre[OST:4]
lustre-OST0005_UUID 2.0G 247.3M 1.6G 12% /mnt/lustre[OST:5]
+
lustre-OST0005_UUID   2.0G   247.3M       1.6G   12% /mnt/lustre[OST:5]
filesystem summary: 11.8G 1.5G 9.7G 12% /mnt/lustre
+
 
 +
filesystem summary:   11.8G     1.5G       9.7G   12% /mnt/lustre
 
</pre>
 
</pre>
  
=== Determining stripe information for a file ===
+
== Displaying Stripe Information for a File ==
The ''lfs getstripe'' command can be used to display information that shows over which OSTs a file is distributed. For example, the output from the following command (the multiple ''obdidx'' entries) indicates that the file ''test1'' is striped over all six active OSTs in the configuration:
+
 
 +
The ''lfs getstripe'' command can be used to display information that shows over which OSTs a file is distributed. For example, the output from the following command (showing multiple ''obdidx'' entries) indicates that the file ''test1'' is striped over all six active OSTs in the configuration:
  
 
<pre>
 
<pre>
Line 92: Line 89:
 
5: lustre-OST0005_UUID ACTIVE
 
5: lustre-OST0005_UUID ACTIVE
 
/mnt/lustre/test1
 
/mnt/lustre/test1
obdidx objidobjid group
+
    obdidx     objid    objid      group
0 8 0x8 0
+
          0         8       0x8         0
1 4 0x4 0
+
          1         4       0x4         0
2 5 0x5 0
+
          2         5       0x5         0
3 5 0x5 0
+
          3         5       0x5         0
4 4 0x4 0
+
          4         4       0x4         0
5 2 0x2 0
+
          5         2       0x2         0
 
</pre>
 
</pre>
  

Latest revision as of 17:10, 1 February 2012

(Updated: Oct 2009)

One of the main factors leading to the high performance of Lustre™ file systems is the ability to stripe data over multiple OSTs. The stripe count can be set on a file system, directory, or file level. An example showing the use of striping is provided below.

For additional information, see Chapter 18: Managing File Striping and Free Space in the Lustre Operations Manual

Setting Up Striping

To see the current stripe size, use the command lfs getstripe [file, dir, fs]. This command will produce output similar to the following:

root@LustreClient01 lustre]# lfs getstripe /mnt/lustre
OBDS:
0: lustre-OST0000_UUID ACTIVE
1: lustre-OST0001_UUID ACTIVE
2: lustre-OST0002_UUID ACTIVE
3: lustre-OST0003_UUID ACTIVE
4: lustre-OST0004_UUID ACTIVE
5: lustre-OST0005_UUID ACTIVE
/mnt/lustre
(Default) stripe_count: 2 stripe_size: 4M stripe_offset: 0

In this example, the default stripe count is 2 (that is, data blocks are striped over two OSTs), the default stripe size is 4 MB (the stripe size can be set in K, M or G), and all writes start from the first OST.

Note: When setting the stripe, the offset is set before the stripe count.

The command to set a new stripe pattern on the file system may look like this:

[root@LustreClient01 lustre]# lfs setstripe -s 4M -i 0 -c 1 /mnt/lustre

This example command sets the stripe of /mnt/lustre to 4 MB blocks starting at OST0 and spanning over one OST. If a new file is created with these settings, the following results are seen:

[root@LustreClient01 lustre]# dd if=/dev/zero of=/mnt/lustre/test1 bs=10M count=100

root@LustreClient01 lustre]# lfs df -h
UUID                  bytes     Used  Available   Use%   Mounted on
lustre-MDT0000_UUID    4.4G   214.5M       3.9G     4%   /mnt/lustre[MDT:0]
lustre-OST0000_UUID    2.0G     1.1G     830.1M    53%   /mnt/lustre[OST:0]
lustre-OST0001_UUID    2.0G    83.3M       1.8G     4%   /mnt/lustre[OST:1]
lustre-OST0002_UUID    2.0G    83.3M       1.8G     4%   /mnt/lustre[OST:2]
lustre-OST0003_UUID    2.0G    83.3M       1.8G     4%   /mnt/lustre[OST:3]
lustre-OST0004_UUID    2.0G    83.3M       1.8G     4%   /mnt/lustre[OST:4]
lustre-OST0005_UUID    2.0G    83.3M       1.8G     4%   /mnt/lustre[OST:5]

filesystem summary:   11.8G     1.5G       9.7G    12%   /mnt/lustre

In this example, the entire file was written to the first OST with a very uneven distribution of data blocks.

Continuing with this example, the file is removed and the stripe count is changed to a value of -1 to specify striping over all available OSTs:

[root@LustreClient01 lustre]# lfs setstripe -s 4M -i 0 -c -1 /mnt/lustre

Now, when a file is created, the new stripe setting evenly distributes the data over all the available OSTs:

[root@LustreClient01 lustre]# dd if=/dev/zero of=/mnt/lustre/test1 bs=10M count=100
100+0 records in
100+0 records out
1048576000 bytes (1.0 GB) copied, 20.2589 seconds, 51.8 MB/s

[root@LustreClient01 lustre]# lfs df -h
UUID                  bytes     Used  Available   Use%   Mounted on
lustre-MDT0000_UUID    4.4G   214.5M       3.9G     4%  /mnt/lustre[MDT:0]
lustre-OST0000_UUID    2.0G   251.3M       1.6G    12%  /mnt/lustre[OST:0]
lustre-OST0001_UUID    2.0G   251.3M       1.6G    12%  /mnt/lustre[OST:1]
lustre-OST0002_UUID    2.0G   251.3M       1.6G    12%  /mnt/lustre[OST:2]
lustre-OST0003_UUID    2.0G   251.3M       1.6G    12%  /mnt/lustre[OST:3]
lustre-OST0004_UUID    2.0G   247.3M       1.6G    12%  /mnt/lustre[OST:4]
lustre-OST0005_UUID    2.0G   247.3M       1.6G    12%  /mnt/lustre[OST:5]

filesystem summary:   11.8G     1.5G       9.7G    12%  /mnt/lustre

Displaying Stripe Information for a File

The lfs getstripe command can be used to display information that shows over which OSTs a file is distributed. For example, the output from the following command (showing multiple obdidx entries) indicates that the file test1 is striped over all six active OSTs in the configuration:

[root@LustreClient01 ~]# lfs getstripe /mnt/lustre/test1
OBDS:
0: lustre-OST0000_UUID ACTIVE
1: lustre-OST0001_UUID ACTIVE
2: lustre-OST0002_UUID ACTIVE
3: lustre-OST0003_UUID ACTIVE
4: lustre-OST0004_UUID ACTIVE
5: lustre-OST0005_UUID ACTIVE
/mnt/lustre/test1
     obdidx      objid     objid      group
          0          8       0x8          0
          1          4       0x4          0
          2          5       0x5          0
          3          5       0x5          0
          4          4       0x4          0
          5          2       0x2          0

In contrast, the output from the following command, which lists just a single obdidx entry, indicates that the file test2 is contained on a single OST:

[root@LustreClient01 ~]# lfs getstripe /mnt/lustre/test_2
OBDS:
0: lustre-OST0000_UUID ACTIVE
1: lustre-OST0001_UUID ACTIVE
2: lustre-OST0002_UUID ACTIVE
3: lustre-OST0003_UUID ACTIVE
4: lustre-OST0004_UUID ACTIVE
5: lustre-OST0005_UUID ACTIVE
/mnt/lustre/test_2
   obdidx      objid     objid      group
        2          8       0x8          0