WARNING: This is the _old_ Lustre wiki, and it is in the process of being retired. The information found here is all likely to be out of date. Please search the new wiki for more up to date information.

Difference between revisions of "Guidelines for Setting Up a Cluster"

From Obsolete Lustre Wiki
Jump to navigationJump to search
(add netdump and netconsole mentions)
Line 4: Line 4:
 
# '''PDSH '''<br/>pdsh is an absolute requirement. Bonus points for being able to pdsh to all nodes from any node.
 
# '''PDSH '''<br/>pdsh is an absolute requirement. Bonus points for being able to pdsh to all nodes from any node.
 
# '''Regular naming'''<br/>A node naming scheme that involves a short prefix and regular incrementing decimal node numbers combines very well with automation like pdsh.  As machines tend to take on different roles as different people use the cluster, it doesn't make a lot of sense to give hostnames based on roles in the lustre universe (mds, ost, etc).
 
# '''Regular naming'''<br/>A node naming scheme that involves a short prefix and regular incrementing decimal node numbers combines very well with automation like pdsh.  As machines tend to take on different roles as different people use the cluster, it doesn't make a lot of sense to give hostnames based on roles in the lustre universe (mds, ost, etc).
# '''Serial Consoles'''<br/>As in any data center, they're essential.  Log their output for later retrieval should the kernel go wrong.  Provide a useful front end like 'conman' or 'conserver'.  Make sure the front-end can send breaks to the kernel's sysrq facility over the serial console.
+
# '''Serial Consoles'''<br/>As in any data center, they're essential.  Log their output for later retrieval should the kernel go wrong.  Provide a useful front end like 'conman' or 'conserver'.  Make sure the front-end can send breaks to the kernel's sysrq facility over the serial console.  In 2.6 kernels there are also reliable network based consoles that allow sending (nearly) all of the kernel messages to a remote system, even for oops messages.  In 2.6.5 this is called "netconsole", and 2.6.9 and later this is "netdump" (which supercedes netconsole).  The "netdump" code also allows doing kernel crash dumps over the network to another host, which can be invaluable for debugging node-crashing problems.
 
# '''Collect syslogs in one place'''<br/>Its nice to be able to watch one log for errors that are reported to syslog across the cluster.
 
# '''Collect syslogs in one place'''<br/>Its nice to be able to watch one log for errors that are reported to syslog across the cluster.
 
# '''Remote Power Management'''<br/>If a machine wedges one needs to be able to reboot it without physically flipping a switch.  Any number of vendors offer serial controlled power widgets.
 
# '''Remote Power Management'''<br/>If a machine wedges one needs to be able to reboot it without physically flipping a switch.  Any number of vendors offer serial controlled power widgets.

Revision as of 00:27, 15 June 2007

Some tips we've collected while working on clusters that can lead to a more useful debugging experience.

  1. Shared home directories
    Having a shared namespace comes in handy all the time. Its useful for bringing up lustre builds, collecting logs, blatting configuration files, etc. sharing /home is the least surprising.
  2. PDSH
    pdsh is an absolute requirement. Bonus points for being able to pdsh to all nodes from any node.
  3. Regular naming
    A node naming scheme that involves a short prefix and regular incrementing decimal node numbers combines very well with automation like pdsh. As machines tend to take on different roles as different people use the cluster, it doesn't make a lot of sense to give hostnames based on roles in the lustre universe (mds, ost, etc).
  4. Serial Consoles
    As in any data center, they're essential. Log their output for later retrieval should the kernel go wrong. Provide a useful front end like 'conman' or 'conserver'. Make sure the front-end can send breaks to the kernel's sysrq facility over the serial console. In 2.6 kernels there are also reliable network based consoles that allow sending (nearly) all of the kernel messages to a remote system, even for oops messages. In 2.6.5 this is called "netconsole", and 2.6.9 and later this is "netdump" (which supercedes netconsole). The "netdump" code also allows doing kernel crash dumps over the network to another host, which can be invaluable for debugging node-crashing problems.
  5. Collect syslogs in one place
    Its nice to be able to watch one log for errors that are reported to syslog across the cluster.
  6. Remote Power Management
    If a machine wedges one needs to be able to reboot it without physically flipping a switch. Any number of vendors offer serial controlled power widgets.
  7. Automated Disaster Recovery
    Its nice to be able to reimage a node by via netbooting and network software installs. Its a low frequency endevour, though.
  8. Boot Quickly
    1. Disable non-essential services to be started at boot-time
    2. Minimize hardware checks the BIOS may do
    3. Especially avoid things like RH's Kudzu which can ask for user input before proceeding

  • FrontPage