WARNING: This is the _old_ Lustre wiki, and it is in the process of being retired. The information found here is all likely to be out of date. Please search the new wiki for more up to date information.

Difference between revisions of "Diagnostic and Debugging Tools"

From Obsolete Lustre Wiki
Jump to navigationJump to search
Line 48: Line 48:
 
Of general interest is:
 
Of general interest is:
  
* '''leak_finder.pl.''' This  program is useful for finding memory leaks in the code.
+
* '''leak_finder.pl.''' This  program is useful for finding memory leaks in the code. [[Where can this be obtained?]]
  
A ''virtual machine'' is often used to create an isolated development and test environment. Some popular virtual machines are:
+
A ''virtual machine'' is often used to create an isolated development and test environment. Some commonly-used virtual machines are:
  
 
* '''VirtualBox Open Source Edition.''' Provides enterprise-class virtualization capability for all major platforms and is available free from Sun Microsystems at [http://www.sun.com/software/products/virtualbox/get.jsp?intcmp=2945 Get Sun Virtual Box].
 
* '''VirtualBox Open Source Edition.''' Provides enterprise-class virtualization capability for all major platforms and is available free from Sun Microsystems at [http://www.sun.com/software/products/virtualbox/get.jsp?intcmp=2945 Get Sun Virtual Box].
Line 59: Line 59:
  
 
: [[Parinay is planning to "include KVM here and how lustre can be deployed on it". See BZ 21381]]
 
: [[Parinay is planning to "include KVM here and how lustre can be deployed on it". See BZ 21381]]
 
  
 
A variety of ''debuggers and analysis tools'' are available including:
 
A variety of ''debuggers and analysis tools'' are available including:
  
* '''kgdb.''' A source-level kernel debugger that allows remote debugging using ''conman''. ''kgdb'' provides a special set of hooks for a Linux kernel to attach ''gdb'' from another machine over a serial console. We provide ''kgdb'' patches for some kernels like ''rhel4'' with the Lustre patches (these are not patched in by default). [[Update needed?]]
+
* '''kgdb.''' The Linux Kernel Source Level Debugger ''kgdb'' is used in conjunction with the GNU Debugger ''gdb'' for debugging the Linux kernel.
 
 
For more information, see [[KGDB]]
 
and [[Using kgdb with UDP]].
 
 
 
Also see [http://www.linuxtopia.org/online_books/redhat_linux_debugging_with_gdb/running.html ''Chapter 6. Running Programs Under gdb''] in the ''Red Hat Linux 4 Debugging with GDB'' guide.
 
  
[[NOTES]] - KGDB topic - ask Alex BZZZ or Robert Reid - instructions are old and not specific to Lustre - do we want to keep these around or find link to eternal site - sourceforge site has a ton of information.  
+
: For more information about using ''kgdb'' with gdb, see [http://www.linuxtopia.org/online_books/redhat_linux_debugging_with_gdb/running.html ''Chapter 6. Running Programs Under gdb''] in the ''Red Hat Linux 4 Debugging with GDB'' guide.
  
1. Get patches from [ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.5/ ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.5/]
+
: sourceforge site has a ton of information.  
  
- VmWare instructions on this page are specific to using cdb with VmWare - but are OLD!
+
: [[KGDB]] and [[Using kgdb with UDP]] - [[these pages to be deleted - Robert Read says KGDB content is generic rather than Lustre specific]]
  
- For other external tools - provide pointer rather than maintain documentation on wiki. See IX4
+
: [[Does this need to be mentioned?]] We provide ''kgdb'' patches for some kernels like ''rhel4'' with the Lustre patches (these are not patched in by default). Get patches from [ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.5/ ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.5/]
  
 
* '''lcrash.'''  A utility that generates detailed kernel information and provides the ability to generate reports about system crash dumps. For more information, see [http://man-wiki.net/index.php/1:lcrash| man page]. [[Is this a good link?]]
 
* '''lcrash.'''  A utility that generates detailed kernel information and provides the ability to generate reports about system crash dumps. For more information, see [http://man-wiki.net/index.php/1:lcrash| man page]. [[Is this a good link?]]
Line 82: Line 76:
 
* '''crash.''' '''[[Red Hat Crash Utility??]]''' is used to analyze saved crash dump data when a system had panicked or locked up or appears unresponsive.  
 
* '''crash.''' '''[[Red Hat Crash Utility??]]''' is used to analyze saved crash dump data when a system had panicked or locked up or appears unresponsive.  
  
Enter:
+
:Enter:
  crash vmlinux crash_dump [[keep this?]]
+
 
 +
  crash vmlinux crash_dump  
 +
[[keep this?]]
  
: For more information about using ''crash'' to analyze crash dump output, see:
+
: For more information about using ''crash'' to analyze a crash dump, see:
  
 
: - Red Hat Magazine article [http://magazine.redhat.com/2007/08/15/a-quick-overview-of-linux-kernel-crash-dump-analysis/ ''A quick overview of Linux kernel crash dump analysis''].
 
: - Red Hat Magazine article [http://magazine.redhat.com/2007/08/15/a-quick-overview-of-linux-kernel-crash-dump-analysis/ ''A quick overview of Linux kernel crash dump analysis''].

Revision as of 12:36, 27 January 2010

A variety of diagnostic and analysis tools are available to debug issues with the Lustre™ software. Some of these are provided in Linux distributions, while others have been developed and are made available by the Lustre project.

Lustre Debugging Tools

The following in-kernel debug mechanisms are incorporated into the Lustre software:

  • Debug logs. A circular debug buffer to which Lustre internal debug messages are written (in contrast to error messages, which are printed to the syslog or console). Entries to the Lustre debug log are controlled by the mask set by /proc/sys/lnet/debug. The log size defaults to 5 MB per CPU but can be increased as a busy system will quickly overwrite 5 MB. When the buffer fills, the oldest information is discarded.
  • Debug daemon. The debug daemon controls logging of debug messages.

These tools are also provided with the Lustre software:

  • Lustre subsystem asserts. A panic-style assertion in the kernel causes Lustre to

dump the debug log to the file /tmp/lustre-log.<timestamp> where it can be retrieved after a reboot. Cliff review.

External debugging tools

Tools for administrators and developers

The tools described in this section are provided in the Linux kernel or are available at an external website.

Some general debugging tools are: Do we need to provide links to more info for these?

  • strace. This tool allows a system call to be traced.
  • /var/log/messages. syslogd prints fatal or serious messages at this log.
  • Crash dumps. On crash-dump enabled kernels, sysrq c produces a crash dump. Lustre enhances this crash dump with a log dump (the last 64 KB of the log) to the console.
  • debugfs. Interactive file system debugger.

The following logging and data collection tools can be used to collect information for debugging Lustre kernel issues:

Tools for developers

The tools described in this section may be useful for debugging Lustre™ in a development environment.

Of general interest is:

A virtual machine is often used to create an isolated development and test environment. Some commonly-used virtual machines are:

  • VirtualBox Open Source Edition. Provides enterprise-class virtualization capability for all major platforms and is available free from Sun Microsystems at Get Sun Virtual Box.
  • Xen. A para-virtualized environment with virtualization capabilities similar to VMware Server and Virtual Box. However, Xen allows the use of modified kernels to provide near-native performance and the ability to emulate shared storage. For more information, see Using Xen with Lustre or go to xen.org]
Parinay is planning to "include KVM here and how lustre can be deployed on it". See BZ 21381

A variety of debuggers and analysis tools are available including:

  • kgdb. The Linux Kernel Source Level Debugger kgdb is used in conjunction with the GNU Debugger gdb for debugging the Linux kernel.
For more information about using kgdb with gdb, see Chapter 6. Running Programs Under gdb in the Red Hat Linux 4 Debugging with GDB guide.
sourceforge site has a ton of information.
KGDB and Using kgdb with UDP - these pages to be deleted - Robert Read says KGDB content is generic rather than Lustre specific
Does this need to be mentioned? We provide kgdb patches for some kernels like rhel4 with the Lustre patches (these are not patched in by default). Get patches from ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.5/
  • lcrash. A utility that generates detailed kernel information and provides the ability to generate reports about system crash dumps. For more information, see man page. Is this a good link?
  • crash. Red Hat Crash Utility?? is used to analyze saved crash dump data when a system had panicked or locked up or appears unresponsive.
Enter:
crash vmlinux crash_dump 

keep this?

For more information about using crash to analyze a crash dump, see:
- Red Hat Magazine article A quick overview of Linux kernel crash dump analysis.
- Crash Usage: A Case Study from the white paper Red Hat Crash Utility by David Anderson.
- Kernel Trap forum entry Linux: Kernel Crash Dumps.
- White paper A Quick Overview of Linux Kernel Crash Dump Analysis.

NOTES See Tien's suggestion BZ 21334 [[1]]