Lustre Tuning - Revision history

Sbarthel: /* MDS Threads */

2010-04-30T12:57:12Z

MDS Threads

← Older revision		Revision as of 05:57, 30 April 2010
Line 45:		Line 45:
	or you reach satisfactory performance.		or you reach satisfactory performance.

	==MDS ~~Threads~~==		==MDS Service Thread Count==
	There is a similar parameter for the number of MDS service threads:		There is a similar parameter for the number of MDS service threads:

Sbarthel at 18:30, 22 February 2010

2010-02-22T18:30:58Z

← Older revision		Revision as of 11:30, 22 February 2010
Line 1:		Line 1:
			<small>''(Updated: Feb 2010)''</small>
	__TOC__		__TOC__
	Many options in Lustre™ are set by means of kernel module parameters. These parameters are contained in the ''modprobe.conf'' file (On SuSE, this may be ''modprobe.conf.local'').		Many options in Lustre™ are set by means of kernel module parameters. These parameters are contained in the ''modprobe.conf'' file (On SuSE, this may be ''modprobe.conf.local'').

Adilger: /* Number of Inodes for OST */

2010-02-09T18:26:40Z

Number of Inodes for OST

← Older revision		Revision as of 11:26, 9 February 2010
Line 95:		Line 95:

	===Number of Inodes for OST===		===Number of Inodes for OST===
	For OST file systems, it is normally advantageous to take local file system usage into account. Try and minimize the number of inodes created on each OST. This helps in reducing the format and ''e2fsck'' time, and makes more space available for data. The current default is to create one inode per 16KB of space in the OST file system, but in many environments, this is far too many inodes for the average file size. As a good rule of thumb, the OSTs should have at least:		For OST file systems, it is normally advantageous to take local file system usage into account. Try and minimize the number of inodes created on each OST, while keeping enough margin for potential variance in future usage. This helps in reducing the format and ''e2fsck'' time, and makes more space available for data. The current default is to create one inode per 16KB of space in the OST file system, but in many environments, this is far too many inodes for the average file size. As a good rule of thumb, the OSTs should have at least:

	num_ost_inodes = 4 * <num_mds_inodes> * <default_stripe_count> / <number_osts>		num_ost_inodes = 4 * <num_mds_inodes> * <default_stripe_count> / <number_osts>

	You can specify the number of inodes on the OST file systems via the ''-N<num_inodes>'' option to ''--mkfsoptions''. Alternately, if you know the average file size, then you can also specify the OST inode count for the OST file systems via ''-i <average_file_size / (number_of_stripes * 4)>''. (For example, if the average file size is 16MB and there are by default 4 stripes per file then ''--mkfsoptions='-i 1048576''' would be appropriate).		You can specify the number of inodes on the OST file systems via the ''-N<num_inodes>'' option to ''--mkfsoptions''. Alternately, if you know the average file size, then you can also specify the OST inode count for the OST file systems via ''-i <average_file_size / (number_of_stripes * 4)>''. (For example, if the average file size is 16MB and there are by default 4 stripes per file then ''--mkfsoptions='-i 1048576''' would be appropriate).

Docadmin: /* OSS Service Thread Count */

2010-01-09T00:33:21Z

OSS Service Thread Count

← Older revision		Revision as of 17:33, 8 January 2010
Line 21:		Line 21:
	* I/O completions take excessive time due to slow storage		* I/O completions take excessive time due to slow storage

	Decreasing the size of the thread pool may help:		Decreasing the size of the thread pool may help if:
	* ~~if the~~ clients are overwhelming the storage capacity		* The clients are overwhelming the storage capacity
	* ~~there~~ are lots of "slow IO" or similar messages		* There are lots of "slow I/O" or similar messages

	Increasing the number of I/O threads allows the kernel and storage to		Increasing the number of I/O threads allows the kernel and storage to
Line 34:		Line 34:
	before performance is degraded due to the high number of seeks and the OST		before performance is degraded due to the high number of seeks and the OST
	threads just waiting for I/O. In this situation, it may be advisable to decrease the		threads just waiting for I/O. In this situation, it may be advisable to decrease the
	load by decreasing the number of OST threads .		load by decreasing the number of OST threads.

	Determining the optimum number of OST threads is a process of trial and error. You		Determining the optimum number of OST threads is a process of trial and error. You

Docadmin at 20:31, 17 December 2009

2009-12-17T20:31:18Z

← Older revision		Revision as of 13:31, 17 December 2009
Line 1:		Line 1:
			__TOC__
	Many options in Lustre™ are set by means of kernel module parameters. These parameters are contained in the ''modprobe.conf'' file (On SuSE, this may be ''modprobe.conf.local'').		Many options in Lustre™ are set by means of kernel module parameters. These parameters are contained in the ''modprobe.conf'' file (On SuSE, this may be ''modprobe.conf.local'').

Docadmin: Protected "Lustre Tuning" ([edit=sysop] (indefinite) [move=sysop] (indefinite))

2009-12-16T19:09:17Z

Protected "Lustre Tuning" ([edit=sysop] (indefinite) [move=sysop] (indefinite))

← Older revision	Revision as of 12:09, 16 December 2009
(No difference)

Docadmin at 19:09, 16 December 2009

2009-12-16T19:09:09Z

← Older revision		Revision as of 12:09, 16 December 2009
Line 1:		Line 1:
	Many options in ~~Lustre~~ are set by means of kernel module parameters. These parameters are contained in the ''modprobe.conf'' file (On SuSE, this may be ''modprobe.conf.local'').		Many options in Lustre™ are set by means of kernel module parameters. These parameters are contained in the ''modprobe.conf'' file (On SuSE, this may be ''modprobe.conf.local'').

	==OSS Service Thread Count==		==OSS Service Thread Count==

Adilger: /* OSS Service Thread Count */ remove mention of ost_num_threads, and patch which is landed.

2009-12-16T11:33:25Z

OSS Service Thread Count: remove mention of ost_num_threads, and patch which is landed.

← Older revision		Revision as of 04:33, 16 December 2009
Line 18:		Line 18:
	* Several OSTs are exported from a single OSS		* Several OSTs are exported from a single OSS
	* Back-end storage is running synchronously		* Back-end storage is running synchronously
	* I/O completions take excessive time		* I/O completions take excessive time due to slow storage

	~~In such cases, a larger~~ number of I/O threads allows the kernel and storage to		Decreasing the size of the thread pool may help:
			* if the clients are overwhelming the storage capacity
			* there are lots of "slow IO" or similar messages

			Increasing the number of I/O threads allows the kernel and storage to
	aggregate many writes together for more efficient disk I/O. The OSS thread pool is		aggregate many writes together for more efficient disk I/O. The OSS thread pool is
	shared—each thread allocates approximately 1.5 MB (maximum RPC size + 0.5 MB)		shared—each thread allocates approximately 1.5 MB (maximum RPC size + 0.5 MB)
Line 29:		Line 33:
	before performance is degraded due to the high number of seeks and the OST		before performance is degraded due to the high number of seeks and the OST
	threads just waiting for I/O. In this situation, it may be advisable to decrease the		threads just waiting for I/O. In this situation, it may be advisable to decrease the
	load by decreasing the number of OST threads ~~(''ost_num_threads'' module parameter~~		load by decreasing the number of OST threads .
	~~to ost.ko module)~~.

	Determining the optimum number of OST threads is a process of trial and error. You		Determining the optimum number of OST threads is a process of trial and error. You
	may want to start with a number of OST threads equal to the number of actual disk		may want to start with a number of OST threads equal to the number of actual disk
	spindles on the node. If you use ~~RAID5+~~, subtract any dead spindles not used for		spindles on the node. If you use RAID, subtract any dead spindles not used for
	actual data (e.g., 1/3 of spindles for RAID5), and monitor the performance of clients		actual data (e.g., 1 of N of spindles for RAID5, 2 of N spindles for RAID6), and
	during usual workloads. If performance is degraded, increase the thread count and		monitor the performance of clients during usual workloads. If performance is degraded,
	see how that works until performance is degraded again or you reach satisfactory		increase the thread count and see how that works until performance is degraded again
	performance.		or you reach satisfactory performance.

	'''''Note:''''' If your disk configuration does not have writeback cache enabled and your activity is mostly writes, consider trying the patch in bug 16919 ([https://bugzilla.lustre.org/ Bugzilla]). It removes synchronous journal commit requirements, and should speed up OST writes (unless you already use fast external journal or writeback cache is enabled that mitigates the synchronousness of the journal commit).

	==MDS Threads==		==MDS Threads==

Docadmin: /* OSS Service Thread Count */

2009-12-09T23:15:46Z

OSS Service Thread Count

← Older revision		Revision as of 16:15, 9 December 2009
Line 40:		Line 40:
	performance.		performance.

	'''''Note:''''' If your disk configuration does not have writeback cache enabled and your activity is mostly writes, ~~considertrying~~ the patch in bug 16919 (Bugzilla). It removes synchronous journal commit requirements, and should speed up OST writes (unless you already use fast external journal or writeback cache is enabled that mitigates the synchronousness of the journal commit).		'''''Note:''''' If your disk configuration does not have writeback cache enabled and your activity is mostly writes, consider trying the patch in bug 16919 ([https://bugzilla.lustre.org/ Bugzilla]). It removes synchronous journal commit requirements, and should speed up OST writes (unless you already use fast external journal or writeback cache is enabled that mitigates the synchronousness of the journal commit).

	==MDS Threads==		==MDS Threads==

Docadmin: /* Options for Formatting MDS and OST */

2009-11-12T21:31:30Z

Options for Formatting MDS and OST

← Older revision		Revision as of 14:31, 12 November 2009
Line 67:		Line 67:

	=Options for Formatting MDS and OST=		=Options for Formatting MDS and OST=
	The backing file systems on an MDS and OSTs are independent of each other, so the formatting parameters for them should not be same. The size of the MDS backing file system depends solely on how many inodes you want in the total Lustre file system. It is not related to the size of the aggregate OST space.		The backing file systems on the MDS and OSTs are independent of each other, so the formatting parameters for them should not be same. The size of the MDS backing file system depends solely on how many inodes you want in the total Lustre file system. It is not related to the size of the aggregate OST space.

	==Planning for Inodes==		==Planning for Inodes==
	Every time you create a file on a Lustre file system, it consumes one inode on the MDS and one inode for each OST object that the file is striped over (normally it is based on the default stripe count option -c; but this may change on a per-file basis). In ext3/ldiskfs file systems, inodes are pre-allocated, so creating a new file does not consume any of the free blocks. However, this also means that the format-time options should be conservative as it is not possible to increase the number of inodes after the ~~filesystem~~ is formatted. But it is possible to add OSTs with additional space and inodes to the file system.		Every time you create a file on a Lustre file system, it consumes one inode on the MDS and one inode for each OST object that the file is striped over (normally it is based on the default stripe count option ''-c'', but this may change on a per-file basis). In ''ext3/ldiskfs'' file systems, inodes are pre-allocated, so creating a new file does not consume any of the free blocks. However, this also means that the format-time options should be conservative as it is not possible to increase the number of inodes after the file system is formatted. But it is possible to add OSTs with additional space and inodes to the file system.

	To be on the safe side, plan for 4KB per inode on the MDS. This is the default. For the OST, the amount of space taken by each object depends entirely upon the usage pattern of the users/applications running on the system. Lustre, by necessity, defaults to a very conservative estimate for the object size (16KB per object). You can almost always increase this for file system installations. Many Lustre file systems have average file sizes over 1MB per object.		To be on the safe side, plan for 4KB per inode on the MDS. This is the default. For the OST, the amount of space taken by each object depends entirely upon the usage pattern of the users/applications running on the system. Lustre, by necessity, defaults to a very conservative estimate for the object size (16KB per object). You can almost always increase this for file system installations. Many Lustre file systems have average file sizes over 1MB per object.

	== Sizing the MDT ==		== Sizing the MDT ==
	When calculating the MDS size, the only important factor is the average size of files to be stored in the file system. If the average file size is, for example, 5MB and you have 100TB of usable OST space, then you need at least (100TB * 1024GB/TB * 1024MB/GB / 5MB/inode) = 20 million inodes. We recommend that you have twice the minimum, that is 40 million inodes in this example. At the default 4KB per inode, this works out to only 160GB of space for the MDS.		When calculating the MDS size, the only important factor is the average size of files to be stored in the file system. If the average file size is, for example, 5MB and you have 100TB of usable OST space, then you need at least ''100TB * 1024GB/TB * 1024MB/GB / 5MB/inode = 20 million inodes''. We recommend that you have twice the minimum, that is, 40 million inodes in this example. At the default 4KB per inode, this works out to only 160GB of space for the MDS.

	Conversely, if you have a very small average file size, for example 4KB, Lustre is not very efficient. This is because you consume as much space on the MDS as you are consuming on the OSTs. This is not a very common configuration for Lustre.		Conversely, if you have a very small average file size, for example 4KB, Lustre is not very efficient. This is because you consume as much space on the MDS as you are consuming on the OSTs. This is not a very common configuration for Lustre.

	==Overriding Default Formatting Options==		==Overriding Default Formatting Options==
	To override the default formatting options for any of the Lustre backing filesystems, use the "--mkfsoptions='backing fs options'" argument to mkfs.lustre to pass formatting options to the backing mkfs. For all options to format backing ext3 and ldiskfs filesystems, see the mke2fs(8) man page; this section only discusses some Lustre-specific options.		To override the default formatting options for any of the Lustre backing filesystems, use the ''--mkfsoptions='backing fs options''' argument to ''mkfs.lustre'' to pass formatting options to the backing ''mkfs''. For all options to format backing ''ext3'' and ''ldiskfs'' filesystems, see the ''mke2fs(8)'' man page; this section only discusses some Lustre-specific options.

	===Number of Inodes for MDS===		===Number of Inodes for MDS===
	To override the inode ratio, use the option '-i <bytes per inode>' (for instance, "--mkfsoptions='-i 4096'" to create one inode per 4096 bytes of file system space). Alternately, if you are specifying some absolute number of inodes, use the '-N<number of inodes>' option. You should not specify the ’-i’ option with an inode ratio below one inode per 1024 bytes in order to avoid unintentional mistakes. Instead, use the '-N' option.		To override the inode ratio, use the option ''-i <bytes per inode>'' (for instance, ''--mkfsoptions='-i 4096''' to create one inode per 4096 bytes of file system space). Alternately, if you are specifying some absolute number of inodes, use the ''-N<number of inodes>'' option. You should not specify the ''-i'' option with an inode ratio below one inode per 1024 bytes in order to avoid unintentional mistakes. Instead, use the ''-N'' option.

	A 2TB MDS by default will have 512M inodes. The largest currently-supported file system size is 8TB, which would hold 2B inodes. With an MDS inode ratio of 1024 bytes per inode, a 2TB MDS would hold 2B inodes, and a 4TB MDS would hold 4B inodes, which is the maximum number of inodes currently supported by ext3.		A 2TB MDS by default will have 512M inodes. The largest currently-supported file system size is 8TB, which would hold 2B inodes. With an MDS inode ratio of 1024 bytes per inode, a 2TB MDS would hold 2B inodes, and a 4TB MDS would hold 4B inodes, which is the maximum number of inodes currently supported by ext3.

	===Inode Size for MDS===		===Inode Size for MDS===
	Lustre uses "large" inodes on the backing file systems in order to efficiently store Lustre metadata with each file. On the MDS, each inode is at least 512 bytes in size by default, while on the OST each inode is 256 bytes in size. Lustre (or more specifically the backing ext3 file system), also needs sufficient space left for other metadata like the journal (up to 400MB), bitmaps and directories. There are also a few regular files that Lustre uses to maintain cluster consistency.		Lustre uses "large" inodes on the backing file systems in order to efficiently store Lustre metadata with each file. On the MDS, each inode is at least 512 bytes in size by default, while on the OST each inode is 256 bytes in size. Lustre (or more specifically the backing ''ext3'' file system), also needs sufficient space left for other metadata like the journal (up to 400MB), bitmaps and directories. There are also a few regular files that Lustre uses to maintain cluster consistency.

	To specify a larger inode size, use the '-I <inodesize>' option. We do NOT recommend specifying a smaller-than-default inode size, as this can lead to serious performance problems; and you cannot change this parameter after formatting the file system. The inode ratio must always be larger than the inode size.		To specify a larger inode size, use the ''-I <inodesize>'' option. We do NOT recommend specifying a smaller-than-default inode size, as this can lead to serious performance problems; and you cannot change this parameter after formatting the file system. The inode ratio must always be larger than the inode size.

	===Number of Inodes for OST===		===Number of Inodes for OST===
	For OST file systems, it is normally advantageous to take local file system usage into account. Try and minimize the number of inodes created on each OST. This helps in reducing the format and e2fsck time, and makes more space available for data. The current default is to create one inode per 16KB of space in the OST file system, but in many environments this is far too many inodes for the average file size. As a good rule of thumb, the OSTs should have at least:		For OST file systems, it is normally advantageous to take local file system usage into account. Try and minimize the number of inodes created on each OST. This helps in reducing the format and ''e2fsck'' time, and makes more space available for data. The current default is to create one inode per 16KB of space in the OST file system, but in many environments, this is far too many inodes for the average file size. As a good rule of thumb, the OSTs should have at least:

	num_ost_inodes = 4 * <num_mds_inodes> * <default_stripe_count> / <number_osts>		num_ost_inodes = 4 * <num_mds_inodes> * <default_stripe_count> / <number_osts>

	You can specify the number of inodes on the OST file systems via the '-N<num_inodes>' option to --mkfsoptions. Alternately, if you know the average file size then you can also specify the OST inode count for the OST file systems via '-i <average_file_size / (number_of_stripes * 4)>' (For example, if the average file size is 16MB and there are by default 4 stripes per file then --mkfsoptions='-i 1048576' would be appropriate).		You can specify the number of inodes on the OST file systems via the ''-N<num_inodes>'' option to ''--mkfsoptions''. Alternately, if you know the average file size, then you can also specify the OST inode count for the OST file systems via ''-i <average_file_size / (number_of_stripes * 4)>''. (For example, if the average file size is 16MB and there are by default 4 stripes per file then ''--mkfsoptions='-i 1048576''' would be appropriate).