Guidelines for Setting Up a Cluster - Revision history

Sbarthel at 13:10, 22 February 2010

2010-02-22T13:10:42Z

← Older revision		Revision as of 06:10, 22 February 2010
Line 1:		Line 1:
	''(Updated: Dec 2009)''		<small>''(Updated: Dec 2009)''</small>

	Listed below are some guidelines for setting up a cluster to make it easier to manage and debug.		Listed below are some guidelines for setting up a cluster to make it easier to manage and debug.

Sbarthel at 05:26, 22 February 2010

2010-02-22T05:26:42Z

← Older revision		Revision as of 22:26, 21 February 2010
Line 1:		Line 1:
			''(Updated: Dec 2009)''

	Listed below are some guidelines for setting up a cluster to make it easier to manage and debug.		Listed below are some guidelines for setting up a cluster to make it easier to manage and debug.

Line 15:		Line 17:
	** Minimize hardware checks made by the BIOS.		** Minimize hardware checks made by the BIOS.
	** Avoid starting utilities at boot-time that ask for user input before proceeding.		** Avoid starting utilities at boot-time that ask for user input before proceeding.

	~~''(Updated 12/09)''~~

Docadmin at 18:27, 18 February 2010

2010-02-18T18:27:13Z

← Older revision		Revision as of 11:27, 18 February 2010
Line 15:		Line 15:
	** Minimize hardware checks made by the BIOS.		** Minimize hardware checks made by the BIOS.
	** Avoid starting utilities at boot-time that ask for user input before proceeding.		** Avoid starting utilities at boot-time that ask for user input before proceeding.

			''(Updated 12/09)''

Docadmin at 21:58, 1 February 2010

2010-02-01T21:58:50Z

← Older revision		Revision as of 14:58, 1 February 2010
Line 5:		Line 5:
	* '''Use a regular node naming scheme.''' A node naming scheme consisting of a short prefix combined with regularly incremented decimal node numbers (e.g., n0001, n0002, etc.) works well with an automated tool like ''pdsh''. Also, machines tend to be used for different roles in a cluster over time, so hostnames based on roles in the Lustre file system (mds, ost, etc) are not always practical. However, documenting how hostnames map to Lustre functions is useful.		* '''Use a regular node naming scheme.''' A node naming scheme consisting of a short prefix combined with regularly incremented decimal node numbers (e.g., n0001, n0002, etc.) works well with an automated tool like ''pdsh''. Also, machines tend to be used for different roles in a cluster over time, so hostnames based on roles in the Lustre file system (mds, ost, etc) are not always practical. However, documenting how hostnames map to Lustre functions is useful.
	* '''Use serial consoles.''' A serial console enables output to be logged for later retrieval in case a problem occurs. It can be provided with a useful front end like ''conman'' or ''conserver''. A front end that can send breaks to the kernel's ''sysrq'' facility over the serial console is preferable.		* '''Use serial consoles.''' A serial console enables output to be logged for later retrieval in case a problem occurs. It can be provided with a useful front end like ''conman'' or ''conserver''. A front end that can send breaks to the kernel's ''sysrq'' facility over the serial console is preferable.
	* '''Send kernel crash dumps and kernel messages to a remote system.''' Linux provides various tools, such as netdump or netconsole, to capture crash dumps remotely. See [[Debugging Tools]] for more information.		* '''Send kernel crash dumps and kernel messages to a remote system.''' Linux provides various tools, such as netdump or netconsole, to capture crash dumps remotely. See [[Diagnostic and Debugging Tools]] for more information.
	* '''Collect syslogs in one place.''' In addition to collecting logs on a per node basis, collecting syslogs in one location lets an administrator monitor a single log for errors reported to ''syslog'' from across the cluster.		* '''Collect syslogs in one place.''' In addition to collecting logs on a per node basis, collecting syslogs in one location lets an administrator monitor a single log for errors reported to ''syslog'' from across the cluster.
	* '''Set up remote power management.''' If a machine wedges, it must be possible to reboot it without physically flipping a switch. Various vendors offer serial-controlled power widgets. Power widgets that work with ''powerman'' are the most useful. Remote power management is a requirement for doing automated failover (STONITH).		* '''Set up remote power management.''' If a machine wedges, it must be possible to reboot it without physically flipping a switch. Various vendors offer serial-controlled power widgets. Power widgets that work with ''powerman'' are the most useful. Remote power management is a requirement for doing automated failover (STONITH).

Docadmin: Protected "Guidelines for Setting Up a Cluster" ([edit=sysop] (indefinite) [move=sysop] (indefinite))

2009-12-16T17:58:56Z

Protected "Guidelines for Setting Up a Cluster" ([edit=sysop] (indefinite) [move=sysop] (indefinite))

← Older revision	Revision as of 10:58, 16 December 2009
(No difference)

Docadmin at 17:58, 16 December 2009

2009-12-16T17:58:48Z

← Older revision		Revision as of 10:58, 16 December 2009
Line 1:		Line 1:
	Listed below are some guidelines for setting up a cluster to make it easier to manage and debug.		Listed below are some guidelines for setting up a cluster to make it easier to manage and debug.

	* '''Set up shared home directories.''' A shared namespace is useful for bringing up ~~Lustre~~ builds and collecting logs. The most commonly shared namespace is ''/home''.		* '''Set up shared home directories.''' A shared namespace is useful for bringing up Lustre™ builds and collecting logs. The most commonly shared namespace is ''/home''.
	* '''Use ''pdsh''.''' This parallel-distributed, multithreaded remote shell enables efficient execution of commands on multiple remote hosts in parallel.		* '''Use ''pdsh''.''' This parallel-distributed, multithreaded remote shell enables efficient execution of commands on multiple remote hosts in parallel.
	* '''Use a regular node naming scheme.''' A node naming scheme consisting of a short prefix combined with regularly incremented decimal node numbers (e.g., n0001, n0002, etc.) works well with an automated tool like ''pdsh''. Also, machines tend to be used for different roles in a cluster over time, so hostnames based on roles in the Lustre file system (mds, ost, etc) are not always practical. However, documenting how hostnames map to Lustre functions is useful.		* '''Use a regular node naming scheme.''' A node naming scheme consisting of a short prefix combined with regularly incremented decimal node numbers (e.g., n0001, n0002, etc.) works well with an automated tool like ''pdsh''. Also, machines tend to be used for different roles in a cluster over time, so hostnames based on roles in the Lustre file system (mds, ost, etc) are not always practical. However, documenting how hostnames map to Lustre functions is useful.

Docadmin at 00:47, 16 December 2009

2009-12-16T00:47:16Z

← Older revision		Revision as of 17:47, 15 December 2009
Line 5:		Line 5:
	* '''Use a regular node naming scheme.''' A node naming scheme consisting of a short prefix combined with regularly incremented decimal node numbers (e.g., n0001, n0002, etc.) works well with an automated tool like ''pdsh''. Also, machines tend to be used for different roles in a cluster over time, so hostnames based on roles in the Lustre file system (mds, ost, etc) are not always practical. However, documenting how hostnames map to Lustre functions is useful.		* '''Use a regular node naming scheme.''' A node naming scheme consisting of a short prefix combined with regularly incremented decimal node numbers (e.g., n0001, n0002, etc.) works well with an automated tool like ''pdsh''. Also, machines tend to be used for different roles in a cluster over time, so hostnames based on roles in the Lustre file system (mds, ost, etc) are not always practical. However, documenting how hostnames map to Lustre functions is useful.
	* '''Use serial consoles.''' A serial console enables output to be logged for later retrieval in case a problem occurs. It can be provided with a useful front end like ''conman'' or ''conserver''. A front end that can send breaks to the kernel's ''sysrq'' facility over the serial console is preferable.		* '''Use serial consoles.''' A serial console enables output to be logged for later retrieval in case a problem occurs. It can be provided with a useful front end like ''conman'' or ''conserver''. A front end that can send breaks to the kernel's ''sysrq'' facility over the serial console is preferable.
			* '''Send kernel crash dumps and kernel messages to a remote system.''' Linux provides various tools, such as netdump or netconsole, to capture crash dumps remotely. See [[Debugging Tools]] for more information.
	~~:In 2.6 kernels, reliable network-based consoles allow sending (nearly) all~~ kernel messages to a remote system~~, even ''oops'' messages. In 2~~.~~6.5, '~~'~~netconsole~~'' ~~is provided. In 2.6.9 and later~~, ''netdump~~'' supercedes ''~~netconsole~~''. The ''netdump'' code also supports kernel~~ crash dumps ~~over the network to another host, which can be invaluable for debugging node-crashing problems~~. [[~~Is the content in this paragraph current? Elsewhere we say netdump has been superceded by kdump.~~]]

	* '''Collect syslogs in one place.''' In addition to collecting logs on a per node basis, collecting syslogs in one location lets an administrator monitor a single log for errors reported to ''syslog'' from across the cluster.		* '''Collect syslogs in one place.''' In addition to collecting logs on a per node basis, collecting syslogs in one location lets an administrator monitor a single log for errors reported to ''syslog'' from across the cluster.

	* '''Set up remote power management.''' If a machine wedges, it must be possible to reboot it without physically flipping a switch. Various vendors offer serial-controlled power widgets. Power widgets that work with ''powerman'' are the most useful. Remote power management is a requirement for doing automated failover (STONITH).		* '''Set up remote power management.''' If a machine wedges, it must be possible to reboot it without physically flipping a switch. Various vendors offer serial-controlled power widgets. Power widgets that work with ''powerman'' are the most useful. Remote power management is a requirement for doing automated failover (STONITH).

Docadmin: moved Optimizing a Cluster for Debugging to Guidelines for Setting Up a Cluster

2009-11-19T19:48:52Z

moved Optimizing a Cluster for Debugging to Guidelines for Setting Up a Cluster

← Older revision	Revision as of 12:48, 19 November 2009
(No difference)

Docadmin at 19:47, 19 November 2009

2009-11-19T19:47:59Z

← Older revision		Revision as of 12:47, 19 November 2009
Line 1:		Line 1:
	~~[[Would this be a better name for this topic?]]~~
	~~'''[[ Guidelines for Setting Up a Cluster]]'''~~

	~~[[I'd originally planned to put this topic on the CONTRIBUTE landing page under "Debugging and testing your code..."]]~~
	~~http://wiki.lustre.org/index.php/Contribute:Contribute~~

	~~[[Now I'm wondering if it would fit better on the USE landing page, perhaps in one of the sections under "Administering Lustre...".]]~~
	~~http://wiki.lustre.org/index.php/Use:Use~~

	~~[[What do you think?]]~~


	Listed below are some guidelines for setting up a cluster to make it easier to manage and debug.		Listed below are some guidelines for setting up a cluster to make it easier to manage and debug.

	* '''Set up shared home directories.''' A shared namespace is useful for bringing up Lustre builds, collecting logs~~, using a ''blat'' command-line utility to email configuration files, etc~~. The most commonly shared namespace is ''/home''.		* '''Set up shared home directories.''' A shared namespace is useful for bringing up Lustre builds and collecting logs. The most commonly shared namespace is ''/home''.
	* '''Use ''pdsh''.''' This parallel-distributed, multithreaded remote shell enables efficient execution of commands on multiple remote hosts in parallel. ~~'''[[OK?]]'''~~		* '''Use ''pdsh''.''' This parallel-distributed, multithreaded remote shell enables efficient execution of commands on multiple remote hosts in parallel.
	* '''Use a regular node naming scheme.''' A node naming scheme consisting of a short prefix combined with regularly incremented decimal node numbers (e.g., n0001, n0002, etc.) works well with an automated tool like ''pdsh''. Also, machines tend to be used for different roles in a cluster over time, so hostnames based on roles in the Lustre file system (mds, ost, etc) are not always practical. However, documenting how hostnames map to Lustre functions is useful.		* '''Use a regular node naming scheme.''' A node naming scheme consisting of a short prefix combined with regularly incremented decimal node numbers (e.g., n0001, n0002, etc.) works well with an automated tool like ''pdsh''. Also, machines tend to be used for different roles in a cluster over time, so hostnames based on roles in the Lustre file system (mds, ost, etc) are not always practical. However, documenting how hostnames map to Lustre functions is useful.
	* '''Use serial consoles.''' A serial console enables output to be logged for later retrieval in case a problem occurs. It can be provided with a useful front end like ''conman'' or ''conserver''. A front end that can send breaks to the kernel's ''sysrq'' facility over the serial console is preferable. ~~'''[[OK?]]'''~~		* '''Use serial consoles.''' A serial console enables output to be logged for later retrieval in case a problem occurs. It can be provided with a useful front end like ''conman'' or ''conserver''. A front end that can send breaks to the kernel's ''sysrq'' facility over the serial console is preferable.

	:In 2.6 kernels, reliable network-based consoles allow sending (nearly) all kernel messages to a remote system, even ''oops'' messages. In 2.6.5, ''netconsole'' is provided. In 2.6.9 and later, ''netdump'' supercedes ''netconsole''. The ''netdump'' code also supports kernel crash dumps over the network to another host, which can be invaluable for debugging node-crashing problems. [[Is the content in this paragraph current? Elsewhere we say netdump has been superceded by kdump.]]		:In 2.6 kernels, reliable network-based consoles allow sending (nearly) all kernel messages to a remote system, even ''oops'' messages. In 2.6.5, ''netconsole'' is provided. In 2.6.9 and later, ''netdump'' supercedes ''netconsole''. The ''netdump'' code also supports kernel crash dumps over the network to another host, which can be invaluable for debugging node-crashing problems. [[Is the content in this paragraph current? Elsewhere we say netdump has been superceded by kdump.]]

	* '''Collect syslogs in one place.''' In addition to collecting logs on a per node basis, collecting syslogs in one location lets an administrator monitor a single log for errors reported to ''syslog'' from across the cluster. ~~'''[[OK?]]'''~~		* '''Collect syslogs in one place.''' In addition to collecting logs on a per node basis, collecting syslogs in one location lets an administrator monitor a single log for errors reported to ''syslog'' from across the cluster.

	* '''Set up remote power management.''' If a machine wedges, it must be possible to reboot it without physically flipping a switch. Various vendors offer serial-controlled power widgets. Power widgets that work with ''powerman'' are the most useful. Remote power management is a requirement for doing automated failover (STONITH).		* '''Set up remote power management.''' If a machine wedges, it must be possible to reboot it without physically flipping a switch. Various vendors offer serial-controlled power widgets. Power widgets that work with ''powerman'' are the most useful. Remote power management is a requirement for doing automated failover (STONITH).

	* '''Automate node provisioning.''' Although infrequently used, it's convenient to be able to reimage a node via netbooting and network software installs. ~~'''[[OK?]]'''~~		* '''Automate node provisioning.''' Although infrequently used, it's convenient to be able to reimage a node via netbooting and network software installs.

	* '''Boot quickly.''' To be able to boot quickly, do the following:		* '''Boot quickly.''' To be able to boot quickly, do the following:
	** Disable non-essential services from starting at boot-time.		** Disable non-essential services from starting at boot-time.
	** Minimize hardware checks made by the BIOS.		** Minimize hardware checks made by the BIOS.
	** Avoid starting utilities at boot-time that ask for user input before proceeding.~~'''[[OK?]]'''~~		** Avoid starting utilities at boot-time that ask for user input before proceeding.

Docadmin at 18:48, 19 November 2009

2009-11-19T18:48:32Z

← Older revision		Revision as of 11:48, 19 November 2009
Line 1:		Line 1:
			[[Would this be a better name for this topic?]]
			'''[[ Guidelines for Setting Up a Cluster]]'''

			[[I'd originally planned to put this topic on the CONTRIBUTE landing page under "Debugging and testing your code..."]]
			http://wiki.lustre.org/index.php/Contribute:Contribute

			[[Now I'm wondering if it would fit better on the USE landing page, perhaps in one of the sections under "Administering Lustre...".]]
			http://wiki.lustre.org/index.php/Use:Use

			[[What do you think?]]


	Listed below are some guidelines for setting up a cluster to make it easier to manage and debug.		Listed below are some guidelines for setting up a cluster to make it easier to manage and debug.

	* '''Set up shared home directories.''' A shared namespace is useful for bringing up Lustre builds, collecting logs, using a ''blat'' command-line utility to email configuration files, etc. The most commonly shared namespace is ''/home''.		* '''Set up shared home directories.''' A shared namespace is useful for bringing up Lustre builds, collecting logs, using a ''blat'' command-line utility to email configuration files, etc. The most commonly shared namespace is ''/home''.
	* '''Use ''pdsh''.''' This parallel-distributed, multithreaded remote shell enables efficient execution of commands on multiple remote hosts in parallel.		* '''Use ''pdsh''.''' This parallel-distributed, multithreaded remote shell enables efficient execution of commands on multiple remote hosts in parallel. '''[[OK?]]'''
	* '''Use a regular node naming scheme.''' A node naming scheme consisting of a short prefix combined with regularly incremented decimal node numbers (e.g., n0001, n0002, etc.) works well with an automated tool like ''pdsh''. Also, machines tend to be used for different roles in a cluster over time, so hostnames based on roles in the Lustre file system (mds, ost, etc) are not always practical. However, documenting how hostnames map to Lustre functions is useful.		* '''Use a regular node naming scheme.''' A node naming scheme consisting of a short prefix combined with regularly incremented decimal node numbers (e.g., n0001, n0002, etc.) works well with an automated tool like ''pdsh''. Also, machines tend to be used for different roles in a cluster over time, so hostnames based on roles in the Lustre file system (mds, ost, etc) are not always practical. However, documenting how hostnames map to Lustre functions is useful.
	* '''Use serial consoles.''' A serial console enables output to be logged for later retrieval in case a problem occurs. It can be provided with a useful front end like ''conman'' or ''conserver''. A front end that can send breaks to the kernel's ''sysrq'' facility over the serial console is preferable.		* '''Use serial consoles.''' A serial console enables output to be logged for later retrieval in case a problem occurs. It can be provided with a useful front end like ''conman'' or ''conserver''. A front end that can send breaks to the kernel's ''sysrq'' facility over the serial console is preferable. '''[[OK?]]'''

	:In 2.6 kernels, reliable network-based consoles allow sending (nearly) all kernel messages to a remote system, even ''oops'' messages. In 2.6.5, ''netconsole'' is provided. In 2.6.9 and later, ''netdump'' supercedes ''netconsole''. The ''netdump'' code also supports kernel crash dumps over the network to another host, which can be invaluable for debugging node-crashing problems. [[Is this paragraph ~~up to date~~? Elsewhere we say netdump has been superceded by kdump.]]		:In 2.6 kernels, reliable network-based consoles allow sending (nearly) all kernel messages to a remote system, even ''oops'' messages. In 2.6.5, ''netconsole'' is provided. In 2.6.9 and later, ''netdump'' supercedes ''netconsole''. The ''netdump'' code also supports kernel crash dumps over the network to another host, which can be invaluable for debugging node-crashing problems. [[Is the content in this paragraph current? Elsewhere we say netdump has been superceded by kdump.]]

	* '''Collect syslogs in one place.''' In addition to collecting logs on a per node basis, collecting syslogs in one location lets an administrator monitor a single log for errors reported to ''syslog'' from across the cluster.		* '''Collect syslogs in one place.''' In addition to collecting logs on a per node basis, collecting syslogs in one location lets an administrator monitor a single log for errors reported to ''syslog'' from across the cluster. '''[[OK?]]'''

	* '''Set up remote power management.''' If a machine wedges, it must be possible to reboot it without physically flipping a switch. Various vendors offer serial-controlled power widgets. Power widgets that work with ''powerman'' are the most useful. Remote power management is a requirement for doing automated failover (STONITH).		* '''Set up remote power management.''' If a machine wedges, it must be possible to reboot it without physically flipping a switch. Various vendors offer serial-controlled power widgets. Power widgets that work with ''powerman'' are the most useful. Remote power management is a requirement for doing automated failover (STONITH).

	* '''Automate node provisioning.''' Although infrequently used, it's convenient to be able to reimage a node via netbooting and network software installs.		* '''Automate node provisioning.''' Although infrequently used, it's convenient to be able to reimage a node via netbooting and network software installs. '''[[OK?]]'''

	* '''Boot quickly.''' To be able to boot quickly, do the following:		* '''Boot quickly.''' To be able to boot quickly, do the following:
	** Disable non-essential services from starting at boot-time.		** Disable non-essential services from starting at boot-time.
	** Minimize hardware checks made by the BIOS.		** Minimize hardware checks made by the BIOS.
	** Avoid starting utilities at boot-time that ask for user input before proceeding.		** Avoid starting utilities at boot-time that ask for user input before proceeding.'''[[OK?]]'''