Lustre Debugging Procedures

The procedures below may be useful to administrators or developers debugging a Lustre™ file system.

Understanding the Lustre debug messaging format

Lustre debug messages are categorized by originating subsystem, message type, and location in the source code. For a list of subsystems and message types, see 23.1 Lustre Debug Messages in the Lustre Operations Manual.

Note: For a current list of subsystems and debug message types, see lnet/include/libcfs/libcfs.h Keep this note?

The elements of a Lustre debug message are described in 23.1.1 Format of Lustre Debug Messages in the Lustre Operations Manual.

Using the lctl tool to view debug messages

The lctl tool allows debug messages to be filtered based on subsystems and message types to extract information useful for troubleshooting from a kernel debug log. For more information, see 23.2.3 The lctl Tool in the Lustre Operations Manual.

A sample lctl run is shown in 23.2.7 Sample lctl Run in the Lustre Operations Manual.

Using the debug daemon option to lctl

The debug_daemon option is used by lctl to control the dumping of the debug_kernel buffer to a user-specified file. For more information about the debug daemon including a list of commands, see 23.2.1 Debug Daemon Option to lctl in the Lustre Operations Manual.

Controlling the Kernel Debug Log

The amount of information printed to the kernel debug logs can be controlled by masks in /proc/sys/lnet/subsystem_debug and /proc/sys/lnet/debug. The subsystem_debug mask is for controlling the subsystems (e.g. obdfilter, net, portals, OSC, etc) and the debug mask is for controlling the debug types that would be written out to the log (e.g. info, error, trace, alloc, etc).

• sysctl -w lnet.debug=0 - Turns off Lustre debugging completely
• sysctl -w lnet.debug=-1 - Enables full debugging
• sysctl -w lnet.debug=net - Turns on logging of messages related to network communications.
• sysctl -w lnet.debug=+net - Turns on logging of messages related to network communications, in addition to existing debug flags.
• sysctl -w lnet.debug=-net - Turns off logging of messages related to network communications, leaving debug flags alone.

The different options are listed in lnet/include/libcfs/libcfs.h.

Use of strace for Troubleshooting

strace is a system call trace utility made available by the operating system. It can be used to trace a system call, intercept all the systems calls made by a process, and record the system call name, arguments, and return values. This tool is useful when trying to troubleshoot a failed system call.

The strace can be invoked for a system call as shown below:

$strace <system-call> <args>

A system call might sometimes fork child processes, we need to use the -f option in such case to trace the child processes also:

$strace -f <system-call> <args>

We can redirect the strace output to a file that maybe examined at a later time:

$strace -o <filename> <system-call> <args>

We can use the -ff option along with -o to cause the trace output to be saved in filename.pid, where pid is the process id of the process being traced.

Use -ttt to timestamp all lines in the strace output, so they can be correlated to operations in the lustre kernel debug log.

If the debugging is done in User mode Linux, it's a good idea to save the traces on the host machine by using the command shown below; here we assume that the hostfs is mounted on /r:

$strace -o /r/tmp/vi.strace

Looking at Disk Content

In Lustre, the information about objects that make a file is stored in the extended attributes of the file inode. This will contain a list of all the object ids and their locations (that is, which OST device stores them). Lustre has a tool called the lfs that can be used to obtain this information for a given file name via the getstripe subcommand. A corresponding lfs setstripe command can be used to specify the striping attributes of a new file or directory.

The lfs getstripe utility is written in C; it takes a Lustre filename as input and lists all the objects that form a part of this file. So, to obtain this information for the file /mnt/lustre/frog in Lustre filesystem, we would do:

$lfs getstripe /mnt/lustre/frog $ OBDs: 0 : OSC_localhost_UUID 1 : OSC_localhost_2_UUID 2 : OSC_localhost_3_UUID obdidx objid 0 17 1 4

The debugfs tool is provided by the e2fsprogs package; it can be used for interactive debugging of an ext3/ldiskfs filesystem. It can either be used just to check the status, or it can be used to modify some information in the filesystem, maybe to correct the filesystem state. In Lustre, all the objects that belong to a file are stored in an underlying ldiskfs filesystem on the OST's. The filesystem uses the object id's as the file names. So, once the object id's are known, debugfs can be used to obtain the attributes of all the objects from the different OST's. A sample run for the /mnt/lustre/frog file used in the example above is shown here:

$ debugfs -c /tmp/ost1 debugfs: cd O debugfs: cd 0 /* for files in group 0 */ debugfs: cd d<objid % 32> debugfs: stat <objid> /* for getattr on object */ debugfs: quit ## Suppose object id is 36, then follow the steps below: $ debugfs /tmp/ost1 debugfs: cd O debugfs: cd 0 debugfs: cd d4 /* objid % 32 */ debugfs: stat 36 /* for getattr on obj 4*/ debugfs: dump 36 /tmp/obj.36 /* dump contents of obj 4 */ debugfs: quit

More help on debugfs can be obtained from the man pages.

Finding the lustre UUID of an OST

To find the Lustre UUID of an obdfilter disk (in the event, perhaps, you mix up the cables on your OST devices or the SCSI bus numbering suddenly changes and your SCSI devices get new names), use debugfs to get the last_rcvd file debugfs -R "dump last_rcvd /tmp/last_rcvf"

Tcpdump

Obsolete? Lustre provides a modified version of tcpdump that can be used to decode the complete Lustre message packet. [NOTE: It is currently out-of-date and unusable] The tool has more developed support to read packets from clients to OSTs; the functionality to decode the packets between clients and MDSs is not well developed yet.

Printing to /var/log/messages

To dump debug messages to the console set the corresponding debug mask in the printk flag:

sysctl -w lnet.printk=-1

This slows down the system dramatically. It is also possible to selectively enable or disable this for particular flags:

sysctl -w lnet.printk=+vfstrace
sysctl -w lnet.printk=-vfstrace

In newer versions of Lustre (1.4.12, 1.6.5) it is even possible to disable warning, error, and console messages, though it is strongly recommended to have something like lctl debug_daemon running to capture this data to a local filesystem for debugging purposes.

Tracing Lock Traffic

We have built a special debug type category for tracing lock traffic:

lctl> filter all_types lctl> show dlmtrace lctl> debug_kernel [filename]

NOTES Put at end - used only in extreme cases when we are helping them trace a bad problem but useful for admin to know.