WARNING: This is the _old_ Lustre wiki, and it is in the process of being retired. The information found here is all likely to be out of date. Please search the new wiki for more up to date information.
Lustre Debugging Procedures: Difference between revisions
| (66 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
<small>''(Updated: Feb 2010)''</small> | |||
__TOC__ | |||
The procedures below may be useful to administrators or developers debugging a Lustre™ file system. | The procedures below may be useful to administrators or developers debugging a Lustre™ file system. | ||
== Understanding the Lustre debug messaging format == | == Understanding the Lustre debug messaging format == | ||
Lustre debug messages are categorized by originating subsystem, message type, and location in the source code. For a list of subsystems and message types, see [http://wiki.lustre.org/manual/ | Lustre debug messages are categorized by originating subsystem, message type, and location in the source code. For a list of subsystems and message types, see [http://wiki.lustre.org/manual/LustreManual20_HTML/LustreDebugging.html#50438274_pgfId-1295747 Section 28.2.1.1: ''Lustre Debug Messages''] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual'']. | ||
'''''Note:''''' For a current list of subsystems and debug message types, see lnet/include/libcfs/libcfs.h '' | '''''Note:''''' For a current list of subsystems and debug message types, see ''lnet/include/libcfs/libcfs.h'' in the Lustre tree. | ||
The elements of a Lustre debug message are described in [http://wiki.lustre.org/manual/ | The elements of a Lustre debug message are described in [http://wiki.lustre.org/manual/LustreManual20_HTML/LustreDebugging.html#50438274_pgfId-1295842 Section 28.2.1.2: ''Format of Lustre Debug Messages''] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual'']. | ||
== Using the ''lctl'' tool to view debug messages== | == Using the ''lctl'' tool to view debug messages== | ||
The ''lctl'' tool allows debug messages to be filtered based on subsystems and message types to extract information useful for troubleshooting from a kernel debug log. For more information about the ''lctl'' tool, see [http://wiki.lustre.org/manual/ | The ''lctl'' tool allows debug messages to be filtered based on subsystems and message types to extract information useful for troubleshooting from a kernel debug log. For more information about the ''lctl'' tool, see [http://wiki.lustre.org/manual/LustreManual20_HTML/LustreDebugging.html#50438274_pgfId-1295889 Section 28.2.2: ''Using the lctl Tool to View Debug Messages''] and [http://wiki.lustre.org/manual/LustreManual20_HTML/SystemConfigurationUtilities_HTML.html#50438219_pgfId-1318224 Section 36.3: ''lctl''] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual'']. | ||
A sample ''lctl'' run is shown in [http://wiki.lustre.org/manual/ | A sample ''lctl'' run is shown in [http://wiki.lustre.org/manual/LustreManual20_HTML/LustreDebugging.html#50438274_pgfId-1295915 Section 28.2.2.1: ''Sample lctl Run''] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual'']. | ||
==== Using the ''lctl'' debug daemon option to dump the buffer to a file==== | ==== Using the ''lctl'' debug daemon option to dump the buffer to a file==== | ||
The debug_daemon option is used by ''lctl'' to control the dumping of the ''debug_kernel'' buffer to a user-specified file. For more information about the debug daemon including a list of commands, see [http://wiki.lustre.org/manual/ | The debug_daemon option is used by ''lctl'' to control the dumping of the ''debug_kernel'' buffer to a user-specified file. For more information about the debug daemon including a list of commands, see [http://wiki.lustre.org/manual/LustreManual20_HTML/LustreDebugging.html#50438274_pgfId-1295930 Section 28.2.3: ''Dumping the Buffer to a File (debug_daemon)''] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual'']. | ||
== Controlling the information written to the kernel debug log == | == Controlling the information written to the kernel debug log == | ||
Masks are provided in ''/proc/sys/lnet/subsystem_debug'' and ''/proc/sys/lnet/debug'' to be used with the ''systctl'' command to determine what information is to be written to the debug log. For more information and examples, see [http://wiki.lustre.org/manual/ | Masks are provided in ''/proc/sys/lnet/subsystem_debug'' and ''/proc/sys/lnet/debug'' to be used with the ''systctl'' command to determine what information is to be written to the debug log. For more information and examples, see [http://wiki.lustre.org/manual/LustreManual20_HTML/LustreDebugging.html#50438274_pgfId-1295954 Section 28.2.4: ''Controlling Information Written to the Kernel Debug Log''] and [http://wiki.lustre.org/manual/LustreManual20_HTML/LustreProc.html#50438271_pgfId-1290883 Section 31.3: ''Debug''] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual'']. | ||
== Using ''strace'' to trace system calls == | == Using ''strace'' to trace system calls == | ||
The ''strace'' utility | The ''strace'' utility provided with the Linux distribution enables a system call to be traced by intercepting all the system calls made by a process and recording the system call name, arguments, and return values. For more details and examples, see [http://wiki.lustre.org/manual/LustreManual20_HTML/LustreDebugging.html#50438274_pgfId-1295970 Section 28.2.5: ''Troubleshooting with strace''] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual'']. | ||
== Looking at disk content == | == Looking at disk content == | ||
In Lustre, the inodes on the metadata server contain extended attributes (EAs) that store information about file striping. EAs contain a list of all object IDs and their locations (that is, the OST that stores them). The ''lfs'' tool can be used to obtain this information for a given file via the ''getstripe'' sub-command. For more information, see see [http://wiki.lustre.org/manual/ | In Lustre, the inodes on the metadata server contain extended attributes (EAs) that store information about file striping. EAs contain a list of all object IDs and their locations (that is, the OST that stores them). The ''lfs'' tool can be used to obtain this information for a given file via the ''getstripe'' sub-command. For more information, see see [http://wiki.lustre.org/manual/LustreManual20_HTML/LustreDebugging.html#50438274_pgfId-1295983 Section 28.2.6: ''Looking at Disk Content''] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual''] ]. | ||
== Finding the Lustre UUID of an OST == | == Finding the Lustre UUID of an OST == | ||
To determine the Lustre UUID of an obdfilter disk (for example, if you mix up the cables on your OST devices, or the SCSI bus numbering suddenly changes and the SCSI devices get new names), use ''debugfs'' to get the ''last_rcvd'' file. | To determine the Lustre UUID of an obdfilter disk (for example, if you mix up the cables on your OST devices, or the SCSI bus numbering suddenly changes and the SCSI devices get new names), use ''debugfs'' to get the ''last_rcvd'' file. | ||
== Printing debug messages to the console == | == Printing debug messages to the console == | ||
| Line 52: | Line 47: | ||
</pre> | </pre> | ||
It is possible to disable '''warning''', '''error''', and '''console''' messages, though it is strongly recommended to have something like ''lctl debug_daemon'' running to capture this data to a local file system for debugging purposes. | |||
== Tracing lock traffic == | == Tracing lock traffic == | ||
Latest revision as of 08:08, 20 January 2011
(Updated: Feb 2010)
The procedures below may be useful to administrators or developers debugging a Lustre™ file system.
Understanding the Lustre debug messaging format
Lustre debug messages are categorized by originating subsystem, message type, and location in the source code. For a list of subsystems and message types, see Section 28.2.1.1: Lustre Debug Messages in the Lustre Operations Manual.
Note: For a current list of subsystems and debug message types, see lnet/include/libcfs/libcfs.h in the Lustre tree.
The elements of a Lustre debug message are described in Section 28.2.1.2: Format of Lustre Debug Messages in the Lustre Operations Manual.
Using the lctl tool to view debug messages
The lctl tool allows debug messages to be filtered based on subsystems and message types to extract information useful for troubleshooting from a kernel debug log. For more information about the lctl tool, see Section 28.2.2: Using the lctl Tool to View Debug Messages and Section 36.3: lctl in the Lustre Operations Manual.
A sample lctl run is shown in Section 28.2.2.1: Sample lctl Run in the Lustre Operations Manual.
Using the lctl debug daemon option to dump the buffer to a file
The debug_daemon option is used by lctl to control the dumping of the debug_kernel buffer to a user-specified file. For more information about the debug daemon including a list of commands, see Section 28.2.3: Dumping the Buffer to a File (debug_daemon) in the Lustre Operations Manual.
Controlling the information written to the kernel debug log
Masks are provided in /proc/sys/lnet/subsystem_debug and /proc/sys/lnet/debug to be used with the systctl command to determine what information is to be written to the debug log. For more information and examples, see Section 28.2.4: Controlling Information Written to the Kernel Debug Log and Section 31.3: Debug in the Lustre Operations Manual.
Using strace to trace system calls
The strace utility provided with the Linux distribution enables a system call to be traced by intercepting all the system calls made by a process and recording the system call name, arguments, and return values. For more details and examples, see Section 28.2.5: Troubleshooting with strace in the Lustre Operations Manual.
Looking at disk content
In Lustre, the inodes on the metadata server contain extended attributes (EAs) that store information about file striping. EAs contain a list of all object IDs and their locations (that is, the OST that stores them). The lfs tool can be used to obtain this information for a given file via the getstripe sub-command. For more information, see see Section 28.2.6: Looking at Disk Content in the Lustre Operations Manual ].
Finding the Lustre UUID of an OST
To determine the Lustre UUID of an obdfilter disk (for example, if you mix up the cables on your OST devices, or the SCSI bus numbering suddenly changes and the SCSI devices get new names), use debugfs to get the last_rcvd file.
Printing debug messages to the console
To dump debug messages to the console (/var/log/messages), set the corresponding debug mask in the printk flag:
sysctl -w lnet.printk=-1
This slows down the system dramatically, so it is also possible to selectively enable or disable this capability for particular flags:
sysctl -w lnet.printk=+vfstrace sysctl -w lnet.printk=-vfstrace
It is possible to disable warning, error, and console messages, though it is strongly recommended to have something like lctl debug_daemon running to capture this data to a local file system for debugging purposes.
Tracing lock traffic
Lustre has a specific debug type category for tracing lock traffic. Use:
lctl> filter all_types lctl> show dlmtrace lctl> debug_kernel [filename]
