WARNING: This is the _old_ Lustre wiki, and it is in the process of being retired. The information found here is all likely to be out of date. Please search the new wiki for more up to date information.

Difference between revisions of "Diagnostic and Debugging Tools"

From Obsolete Lustre Wiki
Jump to navigationJump to search
Line 1: Line 1:
Tools Available for Debugging & Analysis
+
==Tools Available for Debugging & Analysis==
  
 
There are several diagnostic tools available to debug Lustre; some are provided by the operating system, while others were developed and made available by the Lustre project.
 
There are several diagnostic tools available to debug Lustre; some are provided by the operating system, while others were developed and made available by the Lustre project.
  
Lustre tools -Components of same in-kernel debug mechanism
+
===Lustre tools -Components of same in-kernel debug mechanism===
  
    * debug logs: When a kernel module is first inserted, a circular debug buffer is allocated to hold substantial amount of debugging information (in megabytes or more). When the buffer fills up, it wraps around and discards the oldest information. We have added debug messages specifically for Lustre; they can be written out to this kernel log.
+
* debug logs: When a kernel module is first inserted, a circular debug buffer is allocated to hold substantial amount of debugging information (in megabytes or more). When the buffer fills up, it wraps around and discards the oldest information. We have added debug messages specifically for Lustre; they can be written out to this kernel log.
    * debug daemon: The debug daemon provides the facility for unlimited logging of the CDEBUG logs in Lustre.
+
* debug daemon: The debug daemon provides the facility for unlimited logging of the CDEBUG logs in Lustre.
    * /proc/sys/lnet/debug: Contains a mask that can be used to delimit the debugging information written out to the kernel debug logs.  
+
* /proc/sys/lnet/debug: Contains a mask that can be used to delimit the debugging information written out to the kernel debug logs.  
  
  
    * lctl: This tool is made available by Lustre. It is very useful to filter the kernel and extract useful information.
+
* lctl: This tool is made available by Lustre. It is very useful to filter the kernel and extract useful information.
    * Lustre subsystem asserts: In case of asserts, a log will be written out to /tmp/lustre_log.<timestamp>.
+
* Lustre subsystem asserts: In case of asserts, a log will be written out to /tmp/lustre_log.<timestamp>.
    * lfs: A Lustre utility that can be used to get to a Lustre file's extended attributes (among other things).  
+
* lfs: A Lustre utility that can be used to get to a Lustre file's extended attributes (among other things).  
  
  
    * leak_finder.pl: An extremely useful program that helps locate memory leaks in the code.  
+
* leak_finder.pl: An extremely useful program that helps locate memory leaks in the code.  
  
 
External tools
 
External tools
  
    * strace: Allows Lustre users to trace a system call.
+
* strace: Allows Lustre users to trace a system call.
    * /var/log/messages: The directory to which fatal or serious messages are printed by the syslogd.
+
* /var/log/messages: The directory to which fatal or serious messages are printed by the syslogd.
    * Crash dumps: On some kernels, a sysrq "c" is enabled which produces a crash dump. Lustre enhances this crash dump with a log dump (the last 64K of log) to the console.
+
* Crash dumps: On some kernels, a sysrq "c" is enabled which produces a crash dump. Lustre enhances this crash dump with a log dump (the last 64K of log) to the console.
    * debugfs: An nteractive Ext2 filesystem debugger.
+
* debugfs: An nteractive Ext2 filesystem debugger.

Revision as of 16:33, 10 January 2010

Tools Available for Debugging & Analysis

There are several diagnostic tools available to debug Lustre; some are provided by the operating system, while others were developed and made available by the Lustre project.

Lustre tools -Components of same in-kernel debug mechanism

  • debug logs: When a kernel module is first inserted, a circular debug buffer is allocated to hold substantial amount of debugging information (in megabytes or more). When the buffer fills up, it wraps around and discards the oldest information. We have added debug messages specifically for Lustre; they can be written out to this kernel log.
  • debug daemon: The debug daemon provides the facility for unlimited logging of the CDEBUG logs in Lustre.
  • /proc/sys/lnet/debug: Contains a mask that can be used to delimit the debugging information written out to the kernel debug logs.


  • lctl: This tool is made available by Lustre. It is very useful to filter the kernel and extract useful information.
  • Lustre subsystem asserts: In case of asserts, a log will be written out to /tmp/lustre_log.<timestamp>.
  • lfs: A Lustre utility that can be used to get to a Lustre file's extended attributes (among other things).


  • leak_finder.pl: An extremely useful program that helps locate memory leaks in the code.

External tools

  • strace: Allows Lustre users to trace a system call.
  • /var/log/messages: The directory to which fatal or serious messages are printed by the syslogd.
  • Crash dumps: On some kernels, a sysrq "c" is enabled which produces a crash dump. Lustre enhances this crash dump with a log dump (the last 64K of log) to the console.
  • debugfs: An nteractive Ext2 filesystem debugger.