Lustre Debugging Procedures

The procedures below may be useful to administrators or developers debugging a Lustre™ file system.

Understanding the Lustre debug messaging format

Lustre debug messages are categorized by originating subsystem, message type, and location in the source code. For a list of subsystems and message types, see 23.1 Lustre Debug Messages in the Lustre Operations Manual.

Note: For a current list of subsystems and debug message types, see lnet/include/libcfs/libcfs.h Keep this note?

The elements of a Lustre debug message are described in 23.1.1 Format of Lustre Debug Messages in the Lustre Operations Manual.

Using the lctl tool to view debug messages

The lctl tool allows debug messages to be filtered based on subsystems and message types to extract information useful for troubleshooting from a kernel debug log. For more information, see 23.2.3 The lctl Tool in the Lustre Operations Manual.

A sample lctl run is shown in 23.2.7 Sample lctl Run in the Lustre Operations Manual.

Using the lctl debug daemon option to dump the buffer to a file

The debug_daemon option is used by lctl to control the dumping of the debug_kernel buffer to a user-specified file. For more information about the debug daemon including a list of commands, see 23.2.1 Debug Daemon Option to lctl in the Lustre Operations Manual.

Controlling the information written to the kernel debug log

Masks are provided in /proc/sys/lnet/subsystem_debug and /proc/sys/lnet/debug to be used with the systctl command to determine what information is to be written to the debug log. For more information and examples, see 23.2.2 Controlling the Kernel Debug Log in the Lustre Operations Manual.

Using strace to trace system calls

The strace utility made available by the operating system ...provided with the Linux distribution?. ...enables a system call to be traced by intercepting all the systems calls made by a process and record the system call name, arguments, and return values. For more details and examples, see 23.3 Troubleshooting with strace in the Lustre Operations Manual.

Looking at disk content

In Lustre, the inodes on the metadata server contain extended attributes (EAs) that store information about file striping. EAs contain a list of all object IDs and their locations (that is, the OST that stores them). The lfs tool can be used to obtain this information for a given file via the getstripe sub-command. For more information, see see 23.4 Looking at Disk Content in the Lustre Operations Manual.

Finding the Lustre UUID of an OST

To determine the Lustre UUID of an obdfilter disk (for example, if you mix up the cables on your OST devices, or the SCSI bus numbering suddenly changes and the SCSI devices get new names), use debugfs to get the last_rcvd file.

Tcpdump

Obsolete? Lustre provides a modified version of tcpdump that can be used to decode the complete Lustre message packet. [NOTE: It is currently out-of-date and unusable] The tool has more developed support to read packets from clients to OSTs; the functionality to decode the packets between clients and MDSs is not well developed yet.

Printing to /var/log/messages

To dump debug messages to the console set the corresponding debug mask in the printk flag:

sysctl -w lnet.printk=-1

This slows down the system dramatically. It is also possible to selectively enable or disable this for particular flags:

sysctl -w lnet.printk=+vfstrace
sysctl -w lnet.printk=-vfstrace

In newer versions of Lustre (1.4.12, 1.6.5) it is even possible to disable warning, error, and console messages, though it is strongly recommended to have something like lctl debug_daemon running to capture this data to a local filesystem for debugging purposes.

Tracing Lock Traffic

We have built a special debug type category for tracing lock traffic:

lctl> filter all_types lctl> show dlmtrace lctl> debug_kernel [filename]

NOTES Put at end - used only in extreme cases when we are helping them trace a bad problem but useful for admin to know.