WARNING: This is the _old_ Lustre wiki, and it is in the process of being retired. The information found here is all likely to be out of date. Please search the new wiki for more up to date information.

Lustre Debugging for Developers: Difference between revisions

From Obsolete Lustre Wiki
Jump to navigationJump to search
Line 20: Line 20:


For more information about how to use ''prlrpc'', see [http://wiki.lustre.org/manual/LustreManual18_HTML/LustreDebugging.html#50532482_pgfId-1291584 Section 23.5: ''Ptlrpc Request History''] in the [[Lustre Documentation|''Lustre Operations Manual'']].
For more information about how to use ''prlrpc'', see [http://wiki.lustre.org/manual/LustreManual18_HTML/LustreDebugging.html#50532482_pgfId-1291584 Section 23.5: ''Ptlrpc Request History''] in the [[Lustre Documentation|''Lustre Operations Manual'']].
[[ ...omit below this line - included in OM...]]
Ptlrpc history works as follows:
# request_in_callback() adds the new request to the service's request history.
# When a request buffer becomes idle add '''it''' [[the contents of the request buffer?]] to the service's request buffer history list.
# Cull buffers from the service's request buffer history if '''it''' has grown above "req_buffer_history_max" and remove '''its''' [[reqs]] from the service's request history.
The request history is accessed and controlled using the [[following /proc files under the service directory]]:
{|
| || ''req_buffer_history_len'' ||  Number of request buffers currently in the history
|-
| || ''req_buffer_history_max''||  Maximum number of request buffers to keep
|-
| || ''req_history''|| [[Request history]]
|}
Note that requests in the history include "live" requests that are actually being handled. Each line in ''req_history'' looks like:
<seq>:<target NID>:<client ID>:<xid>:<length>:<phase> <svc specific>
Where:
{|
| || ''seq'' ||  Request sequence number
|-
| || ''target NID'' ||  Destination NID of the incoming request
|-
| || ''client ID'' ||  client's PID and NID
|-
| || ''xid'' ||  rq_xid
|-
| || ''length'' ||  size of the request message
|-
| || ''phase:'' ||
|-
| || &nbsp;&nbsp;New || Waiting to be handled or couldn't be unpacked
|-
| || &nbsp;&nbsp;Interpret || Unpacked and being handled
|-
| || &nbsp;&nbsp;Complete || Handled
|-
| || ''svc&nbsp;specific'' || [[Service-specific request printout.]]  Currently the only service that does this is the OST, which prints the opcode if the message has been unpacked successfully.
|}


== LWT Tracing ==
== LWT Tracing ==

Revision as of 10:56, 27 January 2010

Intro...

Adding Debugging to the Source Code

The debug infrastructure provides a number of macros that can be used in Lustre™ source code to aid in debugging or reporting serious errors.

To use these macros, you will need to set the DEBUG_SUBSYSTEM variable at the top of the file to do what?? as shown below: 

#define DEBUG_SUBSYSTEM S_PORTALS

A list of available macros with descriptions is provided in see Section 23.2.8: Adding Debugging to the Lustre Source Code in the Lustre Operations Manual.

Ptlrpc Request History ...Requesting a service history using prlrpc?

Each service maintains a request history, which can be useful for first occurrence troubleshooting.

Is ptlrpc an acronym?

[[PTLRPC An RPC protocol layered on LNET. This protocol deals with stateful servers and has exactly-once semantics and built in support for recovery.]]

prlrpc is listed as a subsystem in the Lustre Debug Messages section.

For more information about how to use prlrpc, see Section 23.5: Ptlrpc Request History in the Lustre Operations Manual.

LWT Tracing

A lightweight tracing facility called LWT prints fixed size requests into a buffer and is faster than LDEBUG.

The records that are dumped contain:

  • Current CPU
  • Process counter
  • Pointer to file
  • Pointer to line in the file
  • Four void * pointers

An lctl command dumps the logs to files.

This tracking facility has been used successfully to debug difficult problems.

Is this included with Lustre?

Finding memory leaks

Memory leaks can occur in code when memory has been allocated and then not freed once it is no longer required. The leak_finder.pl program provides a way to find memory leaks. To use this program, follow these steps:

1. Turn on debugging to make sure all malloc and free entries are collected by entering: 

sysctl -w lnet.debug=+malloc

2. Use lctl to dump the log into a user specified log file as shown in previous section.

3. Run the leak finder on the contents of the log file by entering: 

perl leak_finder.pl <ascii-logname>

The output is similar to: 



malloced   8bytes   at a3116744 called pathcopy


(lprocfs_status.c


lprocfs_add_vars:80)


freed      8bytes   at a3116744 called pathcopy


(lprocfs_status.c


lprocfs_add_vars:80)



The leak_finder.pl tool also displays any leaks that were found found:



Leak:32bytes allocated at a23a8fc  (service.c:ptlrpc_init_svc:144,debug file line 241)



Retrieved from "http://wiki.old.lustre.org/index.php?title=Lustre_Debugging_for_Developers&oldid=10705"