Lustre Debugging for Developers
(Updated: Jan 2011)
The procedures below may be useful to developers debugging Lustre code.
Adding Debugging to the Source Code
The debug infrastructure provides a number of macros that can be used in Lustre™ source code to aid in debugging or reporting serious errors.
To use these macros, you will need to set the DEBUG_SUBSYSTEM variable at the top of the file to as shown below:
#define DEBUG_SUBSYSTEM S_PORTALS
A list of available macros with descriptions is provided in see Section 24.2.8: Adding Debugging to the Lustre Source Code in the Lustre Operations Manual.
Accessing a ptlrpc Request History
Each service maintains a request history, which can be useful for first occurrence troubleshooting. Prlrpc is an RPC protocol layered on LNET. This protocol deals with stateful servers and has semantics and built in support for recovery. Prlrpc is a subsystem in the Lustre Debug Messages section.
Finding memory leaks using leak_finder.pl
Memory leaks can occur in code when memory has been allocated and then not freed once it is no longer required. The leak_finder.pl program provides a way to find memory leaks.
Using LWT Tracing
Lustre offers a lightweight tracing facility called LWT. It prints fixed size requests into a buffer and is much faster than LDEBUG. The LWT tracking facility can be used to debug difficult problems. Use of LWT tracing requires rebuilding Lustre code.
Dumped LWT trace-based records contain:
- Current CPU
- Process counter
- Pointer to file
- Pointer to line in the file
- Four void * pointers
An lctl command can be used to dump the logs to files.