Large-Scale Tuning for Cray XT - Revision history

Sbarthel at 15:54, 22 February 2010

2010-02-22T15:54:48Z

← Older revision		Revision as of 08:54, 22 February 2010
Line 7:		Line 7:
	This section applies to Cray XT3 Catamount nodes and explains parameters used with the ''kptllnd'' module.		This section applies to Cray XT3 Catamount nodes and explains parameters used with the ''kptllnd'' module.

	== Network Tunables ==		====<big>Network Tunables</big>====

	With a large number of clients and servers possible on these systems, tuning various request pools becomes important, requiring changes to the ''ptllnd'' module.		With a large number of clients and servers possible on these systems, tuning various request pools becomes important, requiring changes to the ''ptllnd'' module.

Sbarthel at 15:44, 22 February 2010

2010-02-22T15:44:30Z

← Older revision		Revision as of 08:44, 22 February 2010
Line 1:		Line 1:
	~~----~~

	<small>''DISCLAIMER - EXTERNAL CONTRIBUTOR CONTENT''</small>		<small>''DISCLAIMER - EXTERNAL CONTRIBUTOR CONTENT''</small>

Sbarthel at 15:41, 22 February 2010

2010-02-22T15:41:18Z

← Older revision		Revision as of 08:41, 22 February 2010
Line 1:		Line 1:
	----		----

	DISCLAIMER - EXTERNAL CONTRIBUTOR CONTENT		<small>''DISCLAIMER - EXTERNAL CONTRIBUTOR CONTENT''</small>

	This content was submitted by an external contributor. We provide this information as a resource for the Lustre™ open-source community, but we make no representation as to the accuracy, completeness or reliability of this information.		<small>''This content was submitted by an external contributor. We provide this information as a resource for the Lustre™ open-source community, but we make no representation as to the accuracy, completeness or reliability of this information.''</small>

	----		----

Docadmin: Created page with '---- DISCLAIMER - EXTERNAL CONTRIBUTOR CONTENT This content was submitted by an external contributor. We provide this information as a resource for the Lustre™ open-source co...'

2010-02-10T02:28:52Z

Created page with '---- DISCLAIMER - EXTERNAL CONTRIBUTOR CONTENT This content was submitted by an external contributor. We provide this information as a resource for the Lustre™ open-source co...'

New page

----

DISCLAIMER - EXTERNAL CONTRIBUTOR CONTENT

This content was submitted by an external contributor. We provide this information as a resource for the Lustre™ open-source community, but we make no representation as to the accuracy, completeness or reliability of this information.

----

This section applies to Cray XT3 Catamount nodes and explains parameters used with the ''kptllnd'' module.

== Network Tunables ==

With a large number of clients and servers possible on these systems, tuning various request pools becomes important, requiring changes to the ''ptllnd'' module.
{| border="1" cellspacing=0 cellpadding="10"
! Parameter !! Description
|-
| max_nodes
| ''max_nodes'' is the maximum number of queue pairs, and, therefore, the maximum number of peers with which the LND instance can communicate. Set ''max_nodes'' to a value higher than the product of the total number of nodes and maximum processes per node.

Max nodes > (Total # Nodes) * (max_procs_per_node)

Setting ''max_nodes'' to a lower value than described causes Lustre to throw an error. Setting ''max_nodes'' to a higher value causes excess memory to be consumed.
|-
|max_procs_per_node
|''max_procs_per_node'' is the maximum number of cores (CPUs), on a single Catamount node. Portals must know this value to properly clean up various queues. LNET is not notified directly when a Catamount process aborts. The first information LNET receives is when a new Catamount process with the same Cray portals NID starts and sends a connection request. If the number of processes with that Cray portals NID exceeds the ''max_procs_per_node'' value, LNET removes the oldest one to make space for the new one.
|}

These two tunables combine to set the size of the ''ptllnd'' request buffer pool. The buffer pool must never drop an incoming message, so proper sizing is very important.

{| border="1" cellspacing=0 cellpadding="10"
|-
| Ntx
|
''Ntx'' helps to size the transmit (tx) descriptor pool. A tx descriptor is used for each send and each passive RDMA. The max number of concurrent sends == 'credits'. Passive RDMA is a response to a PUT or GET of a payload that is too big to fit in a small message buffer. For servers, this only happens on large RPCs (for instance, where a long file name is included), so the MDS could be under pressure in a large cluster. For routers, this is bounded by the number of servers. If the tx pool is exhausted, a console error message appears.
|-
|Credits
|Credits determine how many sends are in-flight at once on ''ptllnd''. Optimally, there are 8 requests in-flight per server. The default value is 128, which should be adequate for most applications.
|}