WARNING: This is the _old_ Lustre wiki, and it is in the process of being retired. The information found here is all likely to be out of date. Please search the new wiki for more up to date information.

Difference between revisions of "Diagnostic and Debugging Tools"

From Obsolete Lustre Wiki
Jump to navigationJump to search
Line 1: Line 1:
 
__TOC__
 
__TOC__
A variety of diagnostic and analysis tools are available to debug issues with a Lustre™ file system. Some are provided in Linux distributions, while others have been developed and are made available by the Lustre project.
+
A variety of diagnostic and analysis tools are available to debug issues with the Lustre™ software. Some of these are provided in Linux distributions, while others have been developed and are made available by the Lustre project.
  
= Lustre Debugging Tools =
+
== Lustre Debugging Tools ==
  
===In-kernel debug mechanisms ===
+
The following in-kernel debug mechanisms are incorporated into the Lustre software:
  
* debug logs: When a kernel module is first inserted, a circular debug buffer is allocated to hold substantial amount of debugging information (in megabytes or more). When the buffer fills up, it wraps around and discards the oldest information. We have added debug messages specifically for Lustre; they can be written out to this kernel log.
+
* '''Debug logs.''' A circular debug buffer holds a substantial amount of debugging information (MBs or more) during the first insertion of the kernel module. When this buffer fills up, it wraps and discards the oldest information. Lustre offers additional debug messages that can be written out to this kernel log.
* debug daemon: The debug daemon provides the facility for unlimited logging of the CDEBUG logs in Lustre.
 
* /proc/sys/lnet/debug: Contains a mask that can be used to delimit the debugging information written out to the kernel debug logs.  
 
  
===Other internal tools===
+
:The debug log holds Lustre internal logging, separate from the error messages printed to syslog or console. Entries to the Lustre debug log are controlled by the mask set by ''/proc/sys/lnet/debug''. The log defaults to 5 MB per CPU, and is a ring buffer. Newer messages overwrite older ones. The default log size
* lctl: This tool is made available by Lustre. It is very useful to filter the kernel and extract useful information.
+
* '''Debug daemon.''' The debug daemon controls logging of debug messages.
* Lustre subsystem asserts: In case of asserts, a log will be written out to /tmp/lustre_log.<timestamp>.
+
* '''''/proc/sys/lnet/debug''.''' This log contains a mask that can be used to delimit the debugging information written out to the kernel debug logs.  
* lfs: A Lustre utility that can be used to get to a Lustre file's extended attributes (among other things).
 
  
=External debugging tools for administrators and developers=
+
These tools are also provided with the Lustre software:
 +
* '''lctl.''' This tool is used to manually dump the log and post-process logs that are dumped automatically.
 +
* '''Lustre subsystem asserts.''' In case of asserts, a log writes at ''/tmp/lustre_log.<timestamp>''.
 +
* '''''lfs''.''' This Lustre utility helps get to the extended attributes of a Lustre file (among other things).
  
* strace: Allows Lustre users to trace a system call.
+
== External debugging tools for administrators and developers ==
* /var/log/messages: The directory to which fatal or serious messages are printed by the syslogd.
+
 
* Crash dumps: On some kernels, a sysrq "c" is enabled which produces a crash dump. Lustre enhances this crash dump with a log dump (the last 64K of log) to the console.
+
* '''''strace'''.'' This tool allows a system call to be traced.
* debugfs: An nteractive Ext2 filesystem debugger.
+
* '''''/var/log/messages''.''' ''syslogd'' prints fatal or serious messages at this log.
 +
* '''Crash dumps.''' On crash-dump enabled kernels, sysrq c produces a crash dump. Lustre enhances this crash dump with a log dump (the last 64 KB of the log) to the console.
 +
* '''''debugfs''.''' Interactive file system debugger.
  
 
===Logging and data collection tools===
 
===Logging and data collection tools===

Revision as of 12:09, 12 January 2010

A variety of diagnostic and analysis tools are available to debug issues with the Lustre™ software. Some of these are provided in Linux distributions, while others have been developed and are made available by the Lustre project.

Lustre Debugging Tools

The following in-kernel debug mechanisms are incorporated into the Lustre software:

  • Debug logs. A circular debug buffer holds a substantial amount of debugging information (MBs or more) during the first insertion of the kernel module. When this buffer fills up, it wraps and discards the oldest information. Lustre offers additional debug messages that can be written out to this kernel log.
The debug log holds Lustre internal logging, separate from the error messages printed to syslog or console. Entries to the Lustre debug log are controlled by the mask set by /proc/sys/lnet/debug. The log defaults to 5 MB per CPU, and is a ring buffer. Newer messages overwrite older ones. The default log size
  • Debug daemon. The debug daemon controls logging of debug messages.
  • /proc/sys/lnet/debug. This log contains a mask that can be used to delimit the debugging information written out to the kernel debug logs.

These tools are also provided with the Lustre software:

  • lctl. This tool is used to manually dump the log and post-process logs that are dumped automatically.
  • Lustre subsystem asserts. In case of asserts, a log writes at /tmp/lustre_log.<timestamp>.
  • lfs. This Lustre utility helps get to the extended attributes of a Lustre file (among other things).

External debugging tools for administrators and developers

  • strace. This tool allows a system call to be traced.
  • /var/log/messages. syslogd prints fatal or serious messages at this log.
  • Crash dumps. On crash-dump enabled kernels, sysrq c produces a crash dump. Lustre enhances this crash dump with a log dump (the last 64 KB of the log) to the console.
  • debugfs. Interactive file system debugger.

Logging and data collection tools

These logging and data collection tools can be used to collect information for debugging Lustre kernel issues.

kdump

kdump is a Linux kernel crash utility useful for debugging a system running Red Hat Enterprise Linux. For more information about kdump, see the Red Hat knowledge base article How do I configure kexec/kdump on Red Hat Enterprise Linux 5?. To download kdump, go to the Fedora Project Download site. Links are good

netdump

netdump is a crash dump utility from Red Hat that allows memory images to be dumped over a network to a central server for analysis. It is now obsolete and has been replaced by kdump. Check this with brian or cfaber.

netconsole

netconsole supports kernel-level network logging over UDP. A system requires (SysRq) allows users to collect relevant data through netconsole. For more information, see Netconsole. Content is still relevant - check that it is accurate

lctl

lctl used with the debug_kernel option dumps the lustre debugging log

Additional external debugging and analysis tools for developers

  • leak_finder.pl: An extremely useful program that helps locate memory leaks in the code.

The tools described below may be useful for debugging Lustre™ in a development environment.

Virtual Machines

A virtual machine is often used to create an isolated development and test environment.

VirtualBox

VirtualBox Open Source Edition provides enterprise-class virtualization capability for all major platforms and is available free from Sun Microsystems at Get Sun Virtual Box.

VMware Server

The VMware Server virtualization platform is available as free introductory software at Download VMware Server.

Xen

Xen is a para-virtualized environment with virtualization capabilities similar to VMware Server and Virtual Box. However, Xen allows the use of modified kernels to provide near-native performance and the ability to emulate shared storage. For more information, see Using Xen with Lustre. link to xen.org

Debuggers and Analysis Tools

kgdb

kgdb is a source-level kernel debugger that allows remote debugging using conman.

kgdb provides a special set of hooks for a Linux kernel to attach gdb from another machine over a serial console. We provide kgdb patches for some kernels like rhel4 with the Lustre patches (these are not patched in by default).

For more information, see KGDB and Using kgdb with UDP.

Also see Chapter 6. Running Programs Under gdb in the Red Hat Linux 4 Debugging with GDB guide.


NOTES - KGDB topic - ask Alex BZZZ or Robert Reid - instructions are old and not specific to Lustre - do we want to keep these around or find link to eternal site - sourceforge site has a ton of information. - VmWare instructions on this page are speific to using cdb with VmWare - but are OLD!

- For other external tools - provide pointer rather than maintain documentation on wiki. See IX4

lcrash

lcrash - Linux crash dump analyzer generic Linux tool - find link

crash

crash is used to analyze saved crash dump data.

Enter:

crash vmlinux crash_dump

For more information about using crash to analyze crash dump output, see:


NOTES See Tien's suggestion BZ 21334 www.hpc.ufl.edu/index.php/Lustre