Lustre DDN Tuning - Revision history

Sbarthel at 18:11, 22 February 2010

2010-02-22T18:11:54Z

← Older revision		Revision as of 11:11, 22 February 2010
Line 1:		Line 1:
	<small>''(Updated: Feb 2010)''</small>		<small>''(Updated: Feb 2010)''</small>
	~~----~~
	<small>'' DISCLAIMER - EXTERNAL CONTRIBUTOR CONTENT''</small>		<small>'' DISCLAIMER - EXTERNAL CONTRIBUTOR CONTENT''</small>

Sbarthel at 18:11, 22 February 2010

2010-02-22T18:11:15Z

← Older revision		Revision as of 11:11, 22 February 2010
Line 1:		Line 1:
	<small>''(Updated: Feb 2010)''</small>		<small>''(Updated: Feb 2010)''</small>
			----
	<small>'' DISCLAIMER - EXTERNAL CONTRIBUTOR CONTENT''</small>		<small>'' DISCLAIMER - EXTERNAL CONTRIBUTOR CONTENT''</small>

Sbarthel at 18:08, 22 February 2010

2010-02-22T18:08:03Z

← Older revision		Revision as of 11:08, 22 February 2010
Line 1:		Line 1:
	~~----~~		<small>''(Updated: Feb 2010)''</small>

	DISCLAIMER - EXTERNAL CONTRIBUTOR CONTENT		<small>'' DISCLAIMER - EXTERNAL CONTRIBUTOR CONTENT''</small>

	This content was submitted by an external contributor. We provide this information as a resource for the Lustre™ open-source community, but we make no representation as to the accuracy, completeness or reliability of this information.		<small>''This content was submitted by an external contributor. We provide this information as a resource for the Lustre™ open-source community, but we make no representation as to the accuracy, completeness or reliability of this information.''</small>

	----		----

Docadmin: /* Further Tuning Tips */

2010-02-18T20:00:09Z

Further Tuning Tips

← Older revision		Revision as of 13:00, 18 February 2010
Line 74:		Line 74:

	$ mke2fs -O journal_dev -b 4096 /dev/sdj [optional size]		$ mke2fs -O journal_dev -b 4096 /dev/sdj [optional size]


	----		----

Docadmin: /* Illustration - one OST per Tier */

2010-02-18T19:59:19Z

Illustration - one OST per Tier

← Older revision		Revision as of 12:59, 18 February 2010
Line 138:		Line 138:

	System verify delay: 30		System verify delay: 30

	~~=== Illustration - one OST per Tier ===~~
	~~{\| border=1 cellspacing=0~~
	\|-
	\|
	~~<pre>~~
	~~Capacity Block~~
	~~LUN Label Owner Status (Mbytes) Size Tiers Tier list~~
	~~------------------------------------------------------------------~~
	~~0 1 Ready 512 1 1~~
	~~1 1 Ready 512 1 2~~
	~~2 1 Ready 512 1 3~~
	~~3 1 Ready 512 1 4~~
	~~4 2 Ready [GHS] 1 5~~
	~~5 2 Ready [GHS] 1 6~~
	~~6 2 Critical 512 1 7~~
	~~7 2 Critical 1 8~~
	~~10 1 Cache Locked 64 512 1 1~~
	~~11 1 Cache Locked 64 512 1 2~~
	~~12 1 Cache Locked 64 512 1 3~~
	~~13 1 Cache Locked 64 512 1 4~~
	~~14 2 Ready [GHS] 64 512 1 5~~
	~~15 2 Ready [GHS] 64 512 1 6~~
	~~16 2 Critical 64 512 1 7~~
	~~17 2 Critical 64 512 1 8~~
	~~System verify extent: 16 Mbytes~~
	~~System verify delay: 30~~

	~~</pre>~~
	\|}

Docadmin: /* Further Tuning Tips */

2010-02-18T19:58:38Z

Further Tuning Tips

← Older revision		Revision as of 12:58, 18 February 2010
Line 91:		Line 91:

	For example, one OST per tier:		For example, one OST per tier:
	LUNLabel Owner Status Capacity Block Tiers Tier ~~list~~		{\| border=1 cellspacing=0
	~~(Mbytes) Size~~		\|-
	~~------------------------------------------------------------------~~-		!LUNLabel
	0 1 Ready 102400 512 1 1		!Owner
	1 1 Ready 102400 512 1 2		!Status
	2 1 Ready 102400 512 1 3		!Capacity (Mbytes)
	3 1 Ready 102400 512 1 4		!Block Size
	4 2 Ready [GHS] 102400 4096 1 5		!Tiers
	5 2 Ready [GHS] 102400 4096 1 6		!Tier
	6 2 Critical 102400 512 1 7		!List
	7 2 Critical 102400 4096 1 8		\|-
	10 1 Cache Locked 64 512 1 1		\|0\|\|1\|\|Ready\|\|102400\|\|512\|\|1\|\|1\|\|
	11 1 Ready 64 512 1 2		\|-
	12 1 Cache Locked 64 512 1 3		\|1\|\|1\|\|Ready\|\|102400\|\|512\|\|1\|\|2\|\|
	13 1 Cache Locked 64 512 1 4		\|-
	14 2 Ready [GHS] 64 512 1 5		\|2\|\|1\|\|Ready\|\|102400\|\|512\|\|1\|\|3\|\|
	15 2 Ready [GHS] 64 512 1 6		\|-
	16 2 Ready [GHS] 64 4096 1 7		\|3\|\|1\|\|Ready\|\|102400\|\|512\|\|1\|\|4\|\|
	17 2 Ready [GHS] 64 4096 1 8		\|-
			\|4\|\|2\|\|Ready [GHS]\|\|102400\|\|4096\|\|1\|\|5\|\|
			\|-
			\|5\|\|2\|\|Ready [GHS]\|\|102400\|\|4096\|\|1\|\|6\|\|
			\|-
			\|6\|\|2\|\|Critical\|\|102400\|\|512\|\|1\|\|7\|\|
			\|-
			\|7\|\|2\|\|Critical\|\|102400\|\|4096\|\|1\|\|8\|\|
			\|-
			\|10\|\|1\|\|Cache Locked\|\|64\|\|512\|\|1\|\|1\|\|
			\|-
			\|11\|\|1\|\|Ready\|\|64\|\|512\|\|1\|\|2\|\|
			\|-
			\|12\|\|1\|\|Cache Locked\|\|64\|\|512\|\|1\|\|3\|\|
			\|-
			\|13\|\|1\|\|Cache Locked\|\|64\|\|512\|\|1\|\|4\|\|
			\|-
			\|14\|\|2\|\|Ready [GHS]\|\|64\|\|512\|\|1\|\|5\|\|
			\|-
			\|15\|\|2\|\|Ready [GHS]\|\|64\|\|512\|\|1\|\|6\|\|
			\|-
			\|16\|\|2\|\|Ready [GHS]\|\|64\|\|4096\|\|1\|\|7\|\|
			\|-
			\|17\|\|2\|\|Ready [GHS]\|\|64\|\|4096\|\|1\|\|8\|\|
			\|-
			\|}
	System verify extent: 16 Mbytes		System verify extent: 16 Mbytes

	System verify delay: 30		System verify delay: 30

Docadmin: /* maxcmds */

2010-02-18T19:43:19Z

maxcmds

@@ Line 63: / Line 63: @@
 '''''Note:''''' For help determining an appropriate ''maxcmds'' value, refer to the PDF provided with the DDN firmware. This PDF lists recommended values for that specific firmware version.
-=== maxcmds ===
+===Further Tuning Tips===
-One customer experienced a 30% improvement in write performance by changing this value from the default 2 to 4. This works only with SATA-based disks and _only_ if you can guarantee that only one controller of the pair will be actually accessing the shared LUNs
+Here are some tips we have drawn from testing at a large installation:
-This information comes with a warning, as DDN support do not recommend changing this setting from the default. By increasing the value to 5, the same customer experienced some serious problems.
+* Use the full device instead of a partition (''sda'' versus ''sda1''). When using the full device, Lustre writes nicely-aligned 1 MB chunks to disk. Partitioning the disk can destroy this alignment and will noticeably impact performance.
-The DDN cli commands needed are:
+* Separate the ''ext3'' OST into two LUNs, a small LUN for the ''ext3'' journal and a big one for the "data".
-{| border=1 cellspacing=0
+* Since Lustre 1.0.4, we supply ''ext3 mkfs'' options when we create the OST like -j, -J and so on in the following manner (where ''/dev/sdj'' has been formatted before as a journal). The journal size should not be larger than 1 GB (262144 4 KB blocks) as it can consume up to this amount of RAM on the OSS node per OST.
-|}
+ $ mke2fs -O journal_dev -b 4096 /dev/sdj [optional size]
-'''S2A 9500/9550:'''
-For this hardware, a value of 16 is recommended.  The default value is 6.  The maximum value is 32 but values above 16 are not currently recommended by DDN support.
 === Illustration - one OST per Tier ===

Docadmin: /* Further Tuning tips */

2010-02-18T19:38:51Z

Further Tuning tips

← Older revision		Revision as of 12:38, 18 February 2010
Line 49:		Line 49:
	'''''Note:''''' There is no risk from an OSS/MDS node crashing, only if the DDN itself fails.		'''''Note:''''' There is no risk from an OSS/MDS node crashing, only if the DDN itself fails.

	=== ~~Further Tuning tips~~ ===		===Setting maxcmds===
	~~Experiences drawn from testing at a large installation:~~

	* separate the ~~EXT3 OST into 2 Luns~~, ~~one small lun for~~ the ~~EXT3 journal,~~ and one ~~big for~~ the ~~"data"~~		For the S2A DDN 8500 array, changing ''maxcmds'' to 4 (from the default 2) improved write performance by as much as 30 percent in a particular case. This only works with SATA-based disks and when only one controller of the pair is actually accessing the shared LUNs.
	* since Lustre 1.0.~~4 we can supply EXT3 mkfs options when we create~~ the ~~OST like -j -J etc~~.~~.. like~~ the ~~following~~ (~~where /dev/sdj~~ is ~~formatted before as a journal). The journal size should not be larger than about 1GB (262144 4kB blocks~~) ~~as it can consume up to this amount of RAM on the OSS node per OST~~.
	~~{\| border=1 cellspacing=0~~		However, this setting comes with a warning. DDN support does not recommend changing this setting from the default. By increasing the value to 5, the same setup experienced some serious problems.
	\|-
	\|		The CLI command for the DDN client is provided below (default value is 2).
	~~<pre>~~
	~~# mke2fs -O journal_dev -b 4096 /dev/sdj [optional size]in LMC {config}.sh script:~~		$ disk maxcmds=3
	$~~{LMC} --add mds --node io1 --mds iap-mds --dev /dev/sdi --mkfsoptions "-j -J device~~=~~/dev/sdj" --failover --group iap-mds~~
	~~</pre>~~		For S2A DDN 9500/9550 hardware and above, you can safely change the default from 6 to 16. Although the maximum value is 32, values higher than 16 are not currently recommended by DDN support.
	\|}
	* Very important: on the S2A ~~8500~~, ~~we have proven that we need~~ to ~~create one OST per TIER especially in Write Through (see below an illustration...) This matters if you have~~ 16 ~~Tiers : create 16 OST one Tier each instead of 8 made of 2 Tiers each~~.		'''''Note:''''' For help determining an appropriate ''maxcmds'' value, refer to the PDF provided with the DDN firmware. This PDF lists recommended values for that specific firmware version.
	* On the ~~S2A 9500 and 9550~~, ~~we measured significantly better performance with 2 tiers per LUN~~.
	* Do '''~~NOT~~''' ~~partition the DDN LUNs~~, ~~as this causes ALL IO~~ to the ~~LUNs to be misaligned by 512 bytes. The~~ DDN RAID stripes and cachelines are aligned on 1MB boundaries, and having the partition table on the LUN causes ALL 1MB writes to do a read-modify-write on an extra chunk, and ALL 1MB reads to instead read 2MB from disk into the cache. This ~~has been shown to cause a noticable performance loss.~~
	* you are not obliged to lock in cache the small luns...

	=== maxcmds ===		=== maxcmds ===

Docadmin: /* Setting Segment Size */

2010-02-18T19:36:52Z

Setting Segment Size

← Older revision		Revision as of 12:36, 18 February 2010
Line 24:		Line 24:
	For the S2A 9500 and S2A 9550 DDN storage arrays, we recommend that you use the above commands to disable readahead.		For the S2A 9500 and S2A 9550 DDN storage arrays, we recommend that you use the above commands to disable readahead.

	==Setting Segment Size==		===Setting Segment Size===
	The cache segment size noticeably affects I/O performance. Set the cache segment size differently on the MDT (which does small, random I/O) and on the OST (which does large, contiguous I/O). In customer testing, we have found the optimal values to be 64 KB for the MDT and 1 MB for the OST.		The cache segment size noticeably affects I/O performance. Set the cache segment size differently on the MDT (which does small, random I/O) and on the OST (which does large, contiguous I/O). In customer testing, we have found the optimal values to be 64 KB for the MDT and 1 MB for the OST.

Docadmin: /* Setting Readahead and MF */

2010-02-18T19:36:40Z

Setting Readahead and MF

← Older revision		Revision as of 12:36, 18 February 2010
Line 14:		Line 14:
	* S2A 9550		* S2A 9550

	== Setting Readahead and MF ==		=== Setting Readahead and MF ===
	For the S2A DDN 8500 storage array, we recommend that you disable readahead. In a 1000-client system, if each client has up to 8 read RPCs in flight, then this is 8 * 1000 * 1 MB = 8 GB of reads in flight. With a DDN cache in the range of 2 to 5 GB (depending on the model), it is unlikely that the LUN-based readahead would have ANY cache hits even if the file data were contiguous on disk (generally, file data is not contiguous). The Multiplication Factor (MF) also influences the readahead; you should disable it.		For the S2A DDN 8500 storage array, we recommend that you disable readahead. In a 1000-client system, if each client has up to 8 read RPCs in flight, then this is 8 * 1000 * 1 MB = 8 GB of reads in flight. With a DDN cache in the range of 2 to 5 GB (depending on the model), it is unlikely that the LUN-based readahead would have ANY cache hits even if the file data were contiguous on disk (generally, file data is not contiguous). The Multiplication Factor (MF) also influences the readahead; you should disable it.