Guidelines for Setting Up a Cluster

Some tips we've collected while working on clusters that can lead to a more useful debugging experience.


 * 1) Shared home directories Having a shared namespace comes in handy all the time.  Its useful for bringing up lustre builds, collecting logs, blatting configuration files, etc.  sharing /home is the least surprising.
 * 2) PDSH  pdsh is an absolute requirement. Bonus points for being able to pdsh to all nodes from any node.
 * 3) Regular naming A node naming scheme that involves a short prefix and regular incrementing decimal node numbers combines very well with automation like pdsh.  As machines tend to take on different roles as different people use the cluster, it doesn't make a lot of sense to give hostnames based on roles in the lustre universe (mds, ost, etc).
 * 4) Serial Consoles As in any data center, they're essential.  Log their output for later retrieval should the kernel go wrong.  Provide a useful front end like 'conman' or 'conserver'.  Make sure the front-end can send breaks to the kernel's sysrq facility over the serial console.  In 2.6 kernels there are also reliable network based consoles that allow sending (nearly) all of the kernel messages to a remote system, even for oops messages.  In 2.6.5 this is called "netconsole", and 2.6.9 and later this is "netdump" (which supercedes netconsole).  The "netdump" code also allows doing kernel crash dumps over the network to another host, which can be invaluable for debugging node-crashing problems.
 * 5) Collect syslogs in one place Its nice to be able to watch one log for errors that are reported to syslog across the cluster.
 * 6) Remote Power Management If a machine wedges one needs to be able to reboot it without physically flipping a switch.  Any number of vendors offer serial controlled power widgets.
 * 7) Automated Disaster Recovery Its nice to be able to reimage a node by via netbooting and network software installs.  Its a low frequency endevour, though.
 * 8) Boot Quickly
 * 9) Disable non-essential services to be started at boot-time
 * 10) Minimize hardware checks the BIOS may do
 * 11) Especially avoid things like RH's Kudzu which can ask for user input before proceeding


 * FrontPage