WARNING: This is the _old_ Lustre wiki, and it is in the process of being retired. The information found here is all likely to be out of date. Please search the new wiki for more up to date information.

Lustre Debugging for Developers: Difference between revisions

From Obsolete Lustre Wiki
Jump to navigationJump to search
 
(29 intermediate revisions by 2 users not shown)
Line 1: Line 1:
[[Intro...]]
<small>''(Updated: Jan 2011)''</small>
__TOC__
The procedures below may be useful to developers debugging Lustre code.
 
== Adding Debugging to the Source Code ==
== Adding Debugging to the Source Code ==
The debug infrastructure provides a number of macros that can be used in Lustre™ source code to aid in debugging or reporting serious errors.  
The debug infrastructure provides a number of macros that can be used in Lustre™ source code to aid in debugging or reporting serious errors.  


To use these macros, you will need to set the ''DEBUG_SUBSYSTEM'' variable at the top of the file to '''[[do what??]]''' as shown below:
To use these macros, you will need to set the ''DEBUG_SUBSYSTEM'' variable at the top of the file to as shown below:


  #define DEBUG_SUBSYSTEM S_PORTALS
  #define DEBUG_SUBSYSTEM S_PORTALS


A list of available macros with descriptions is provided in see [http://wiki.lustre.org/manual/LustreManual18_HTML/LustreDebugging.html#50532482_pgfId-1291518 Section 23.2.8: ''Adding Debugging to the Lustre Source Code''] in the [[Lustre Documentation|''Lustre Operations Manual'']].
A list of available macros with descriptions is provided in see [http://wiki.lustre.org/manual/LustreManual20_HTML/LustreDebugging.html#50438274_pgfId-1296027 Section 28.3.1: ''Adding Debugging to the Lustre Source Code''] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual''].
 
== Ptlrpc Request History ...Requesting a service history using ''prlrpc''? ==
Each service maintains a request history, which can be useful for first occurrence troubleshooting.


[[Is ptlrpc an acronym?]]
== Accessing a ''ptlrpc'' Request History ==
Each service maintains a request history, which can be useful for first occurrence troubleshooting. ''Prlrpc'' is an RPC protocol layered on LNET. This protocol deals with stateful servers and has semantics and built in support for recovery. ''Prlrpc'' is a subsystem in the Lustre Debug Messages section.


[[PTLRPC An RPC protocol layered on LNET. This protocol deals with stateful servers
For more information about how to use ''prlrpc'', see [http://wiki.lustre.org/manual/LustreManual20_HTML/LustreDebugging.html#50438274_pgfId-1296100 Section 28.3.2: ''Accessing a Ptlrpc Request History''] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual''].
and has exactly-once semantics and built in support for recovery.]]


[[prlrpc is listed as a subsystem in the Lustre Debug Messages section.]]
=Finding memory leaks using ''leak_finder.pl''=


For more information about how to use ''prlrpc'', see [http://wiki.lustre.org/manual/LustreManual18_HTML/LustreDebugging.html#50532482_pgfId-1291584 Section 23.5: ''Ptlrpc Request History''] in the [[Lustre Documentation|''Lustre Operations Manual'']].
Memory leaks can occur in code when memory has been allocated and then not freed once it is no longer required. The ''leak_finder.pl'' program provides a way to find memory leaks.


== LWT Tracing ==
For details, see [http://wiki.lustre.org/manual/LustreManual20_HTML/LustreDebugging.html#50438274_pgfId-1296154 Section 28.3.3: ''Finding Memory Leaks Using leak_finder.pl''] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual''].
A lightweight tracing facility called LWT prints fixed size requests into a buffer and is faster than LDEBUG.


The records that are dumped contain:
=Using LWT Tracing=
Lustre offers a lightweight tracing facility called LWT. It prints fixed size requests into a buffer and is much faster than LDEBUG. The LWT tracking facility can be used to debug difficult problems. Use of LWT tracing requires rebuilding LNET code.


Dumped LWT trace-based records contain:
* Current CPU
* Current CPU
* Process counter
* Process counter
* Pointer to file
* Pointer to file
* Pointer to line in the file
* Pointer to line in the file
* Four void * pointers
* Four void * pointers  
 
An ''lctl'' command dumps the logs to files.
 
This tracking facility has been used successfully to debug difficult problems.
 
[[Is this included with Lustre?]]
 
=Finding memory leaks=
 
Memory leaks can occur in code when memory has been allocated and then not freed once it is no longer required. The ''leak_finder.pl'' program provides a way to find memory leaks.  To use this program, follow these steps:
 
1. Turn on debugging to make sure all malloc and free entries are collected by entering:
 
sysctl -w lnet.debug=+malloc
 
2. Use ''lctl'' to dump the log into a user specified log file as shown in [[previous section]].
 
3. Run the leak finder on the contents of the log file by entering:
 
perl leak_finder.pl <ascii-logname>
 
The output is similar to:
 
:<pre>
; malloced  8bytes  at a3116744 called pathcopy
; (lprocfs_status.c:lprocfs_add_vars:80)
; freed      8bytes  at a3116744 called pathcopy
; (lprocfs_status.c:lprocfs_add_vars:80)
</pre>
 
The ''leak_finder.pl'' tool also displays any leaks that were found found:


Leak:32bytes allocated at a23a8fc  (service.c:ptlrpc_init_svc:144,debug file line 241)
An ''lctl'' command can be used to dump the logs to files.

Latest revision as of 07:18, 20 January 2011

(Updated: Jan 2011)

The procedures below may be useful to developers debugging Lustre code.

Adding Debugging to the Source Code

The debug infrastructure provides a number of macros that can be used in Lustre™ source code to aid in debugging or reporting serious errors.

To use these macros, you will need to set the DEBUG_SUBSYSTEM variable at the top of the file to as shown below: 

#define DEBUG_SUBSYSTEM S_PORTALS

A list of available macros with descriptions is provided in see Section 28.3.1: Adding Debugging to the Lustre Source Code in the Lustre Operations Manual.

Accessing a ptlrpc Request History

Each service maintains a request history, which can be useful for first occurrence troubleshooting. Prlrpc is an RPC protocol layered on LNET. This protocol deals with stateful servers and has semantics and built in support for recovery. Prlrpc is a subsystem in the Lustre Debug Messages section.

For more information about how to use prlrpc, see Section 28.3.2: Accessing a Ptlrpc Request History in the Lustre Operations Manual.

Finding memory leaks using leak_finder.pl

Memory leaks can occur in code when memory has been allocated and then not freed once it is no longer required. The leak_finder.pl program provides a way to find memory leaks.

For details, see Section 28.3.3: Finding Memory Leaks Using leak_finder.pl in the Lustre Operations Manual.

Using LWT Tracing

Lustre offers a lightweight tracing facility called LWT. It prints fixed size requests into a buffer and is much faster than LDEBUG. The LWT tracking facility can be used to debug difficult problems. Use of LWT tracing requires rebuilding LNET code.

Dumped LWT trace-based records contain:

  • Current CPU
  • Process counter
  • Pointer to file
  • Pointer to line in the file
  • Four void * pointers

An lctl command can be used to dump the logs to files.

Retrieved from "http://wiki.old.lustre.org/index.php?title=Lustre_Debugging_for_Developers&oldid=12173"