Lustre Debugging Procedures - Revision history

Sbarthel: /* Looking at disk content */

2011-01-20T15:08:28Z

Looking at disk content

← Older revision		Revision as of 08:08, 20 January 2011
Line 28:		Line 28:
	== Looking at disk content ==		== Looking at disk content ==

	In Lustre, the inodes on the metadata server contain extended attributes (EAs) that store information about file striping. EAs contain a list of all object IDs and their locations (that is, the OST that stores them). The ''lfs'' tool can be used to obtain this information for a given file via the ''getstripe'' sub-command. For more information, see see [http://wiki.lustre.org/manual/~~LustreManual18_HTML~~/LustreDebugging.html#~~50651266_pgfId~~-~~1291546~~ Section 24.4: ''Looking at Disk Content''] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual''] ].		In Lustre, the inodes on the metadata server contain extended attributes (EAs) that store information about file striping. EAs contain a list of all object IDs and their locations (that is, the OST that stores them). The ''lfs'' tool can be used to obtain this information for a given file via the ''getstripe'' sub-command. For more information, see see [http://wiki.lustre.org/manual/LustreManual20_HTML/LustreDebugging.html#50438274_pgfId-1295983 Section 28.2.6: ''Looking at Disk Content''] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual''] ].

	== Finding the Lustre UUID of an OST ==		== Finding the Lustre UUID of an OST ==

Sbarthel: /* Using strace to trace system calls */

2011-01-20T15:07:47Z

Using strace to trace system calls

← Older revision		Revision as of 08:07, 20 January 2011
Line 24:		Line 24:

	== Using ''strace'' to trace system calls ==		== Using ''strace'' to trace system calls ==
	The ''strace'' utility provided with the Linux distribution enables a system call to be traced by intercepting all the system calls made by a process and recording the system call name, arguments, and return values. For more details and examples, see [http://wiki.lustre.org/manual/~~LustreManual18_HTML~~/LustreDebugging.html#~~50651266_pgfId~~-~~1291533~~ Section 24.3: ''Troubleshooting with strace''] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual''].		The ''strace'' utility provided with the Linux distribution enables a system call to be traced by intercepting all the system calls made by a process and recording the system call name, arguments, and return values. For more details and examples, see [http://wiki.lustre.org/manual/LustreManual20_HTML/LustreDebugging.html#50438274_pgfId-1295970 Section 28.2.5: ''Troubleshooting with strace''] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual''].

	== Looking at disk content ==		== Looking at disk content ==

Sbarthel: /* Looking at disk content */

2011-01-20T15:07:08Z

Looking at disk content

← Older revision		Revision as of 08:07, 20 January 2011
Line 28:		Line 28:
	== Looking at disk content ==		== Looking at disk content ==

	In Lustre, the inodes on the metadata server contain extended attributes (EAs) that store information about file striping. EAs contain a list of all object IDs and their locations (that is, the OST that stores them). The ''lfs'' tool can be used to obtain this information for a given file via the ''getstripe'' sub-command. For more information, see see [http://wiki.lustre.org/manual/LustreManual18_HTML/LustreDebugging.html#50651266_pgfId-1291546 Section 24.4: ''Looking at Disk Content''] in the [~~[Lustre Documentation\|~~''Lustre Operations Manual'']].		In Lustre, the inodes on the metadata server contain extended attributes (EAs) that store information about file striping. EAs contain a list of all object IDs and their locations (that is, the OST that stores them). The ''lfs'' tool can be used to obtain this information for a given file via the ''getstripe'' sub-command. For more information, see see [http://wiki.lustre.org/manual/LustreManual18_HTML/LustreDebugging.html#50651266_pgfId-1291546 Section 24.4: ''Looking at Disk Content''] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual''] ].

	== Finding the Lustre UUID of an OST ==		== Finding the Lustre UUID of an OST ==

Sbarthel: /* Using strace to trace system calls */

2011-01-20T15:06:52Z

Using strace to trace system calls

← Older revision		Revision as of 08:06, 20 January 2011
Line 24:		Line 24:

	== Using ''strace'' to trace system calls ==		== Using ''strace'' to trace system calls ==
	The ''strace'' utility provided with the Linux distribution enables a system call to be traced by intercepting all the system calls made by a process and recording the system call name, arguments, and return values. For more details and examples, see [http://wiki.lustre.org/manual/LustreManual18_HTML/LustreDebugging.html#50651266_pgfId-1291533 Section 24.3: ''Troubleshooting with strace''] in the [~~[Lustre Documentation\|~~''Lustre Operations Manual'']].		The ''strace'' utility provided with the Linux distribution enables a system call to be traced by intercepting all the system calls made by a process and recording the system call name, arguments, and return values. For more details and examples, see [http://wiki.lustre.org/manual/LustreManual18_HTML/LustreDebugging.html#50651266_pgfId-1291533 Section 24.3: ''Troubleshooting with strace''] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual''].

	== Looking at disk content ==		== Looking at disk content ==

Sbarthel: /* Controlling the information written to the kernel debug log */

2011-01-20T15:06:22Z

Controlling the information written to the kernel debug log

← Older revision		Revision as of 08:06, 20 January 2011
Line 21:		Line 21:

	== Controlling the information written to the kernel debug log ==		== Controlling the information written to the kernel debug log ==
	Masks are provided in ''/proc/sys/lnet/subsystem_debug'' and ''/proc/sys/lnet/debug'' to be used with the ''systctl'' command to determine what information is to be written to the debug log. For more information and examples, see [http://wiki.lustre.org/manual/LustreManual20_HTML/LustreDebugging.html#50438274_pgfId-1295954 Section 28.2.4: ''Controlling Information Written to the Kernel Debug Log''] and [http://wiki.lustre.org/manual/~~LustreManual18_HTML~~/LustreProc.html#~~50651263_pgfId~~-1290883 Section 21.3: ''Debug ~~Support~~''] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual''].		Masks are provided in ''/proc/sys/lnet/subsystem_debug'' and ''/proc/sys/lnet/debug'' to be used with the ''systctl'' command to determine what information is to be written to the debug log. For more information and examples, see [http://wiki.lustre.org/manual/LustreManual20_HTML/LustreDebugging.html#50438274_pgfId-1295954 Section 28.2.4: ''Controlling Information Written to the Kernel Debug Log''] and [http://wiki.lustre.org/manual/LustreManual20_HTML/LustreProc.html#50438271_pgfId-1290883 Section 31.3: ''Debug''] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual''].

	== Using ''strace'' to trace system calls ==		== Using ''strace'' to trace system calls ==

Sbarthel: /* Controlling the information written to the kernel debug log */

2011-01-20T15:03:24Z

Controlling the information written to the kernel debug log

← Older revision		Revision as of 08:03, 20 January 2011
Line 21:		Line 21:

	== Controlling the information written to the kernel debug log ==		== Controlling the information written to the kernel debug log ==
	Masks are provided in ''/proc/sys/lnet/subsystem_debug'' and ''/proc/sys/lnet/debug'' to be used with the ''systctl'' command to determine what information is to be written to the debug log. For more information and examples, see [http://wiki.lustre.org/manual/~~LustreManual18_HTML~~/LustreDebugging.html#~~50651266_pgfId~~-~~1291441~~ Section 24.2.2: ''Controlling the Kernel Debug Log''] and [http://wiki.lustre.org/manual/LustreManual18_HTML/LustreProc.html#50651263_pgfId-1290883 Section 21.3: ''Debug Support''] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual''].		Masks are provided in ''/proc/sys/lnet/subsystem_debug'' and ''/proc/sys/lnet/debug'' to be used with the ''systctl'' command to determine what information is to be written to the debug log. For more information and examples, see [http://wiki.lustre.org/manual/LustreManual20_HTML/LustreDebugging.html#50438274_pgfId-1295954 Section 28.2.4: ''Controlling Information Written to the Kernel Debug Log''] and [http://wiki.lustre.org/manual/LustreManual18_HTML/LustreProc.html#50651263_pgfId-1290883 Section 21.3: ''Debug Support''] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual''].

	== Using ''strace'' to trace system calls ==		== Using ''strace'' to trace system calls ==

Sbarthel: /* Controlling the information written to the kernel debug log */

2011-01-20T15:02:34Z

Controlling the information written to the kernel debug log

← Older revision		Revision as of 08:02, 20 January 2011
Line 21:		Line 21:

	== Controlling the information written to the kernel debug log ==		== Controlling the information written to the kernel debug log ==
	Masks are provided in ''/proc/sys/lnet/subsystem_debug'' and ''/proc/sys/lnet/debug'' to be used with the ''systctl'' command to determine what information is to be written to the debug log. For more information and examples, see [http://wiki.lustre.org/manual/LustreManual18_HTML/LustreDebugging.html#50651266_pgfId-1291441 Section 24.2.2: ''Controlling the Kernel Debug Log''] and [http://wiki.lustre.org/manual/LustreManual18_HTML/LustreProc.html#50651263_pgfId-1290883 Section 21.3: ''Debug Support''] in the [~~[Lustre Documentation\|~~''Lustre Operations Manual'']].		Masks are provided in ''/proc/sys/lnet/subsystem_debug'' and ''/proc/sys/lnet/debug'' to be used with the ''systctl'' command to determine what information is to be written to the debug log. For more information and examples, see [http://wiki.lustre.org/manual/LustreManual18_HTML/LustreDebugging.html#50651266_pgfId-1291441 Section 24.2.2: ''Controlling the Kernel Debug Log''] and [http://wiki.lustre.org/manual/LustreManual18_HTML/LustreProc.html#50651263_pgfId-1290883 Section 21.3: ''Debug Support''] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual''].

	== Using ''strace'' to trace system calls ==		== Using ''strace'' to trace system calls ==

Sbarthel: /* Using the lctl debug daemon option to dump the buffer to a file */

2011-01-20T15:01:53Z

Using the lctl debug daemon option to dump the buffer to a file

← Older revision		Revision as of 08:01, 20 January 2011
Line 18:		Line 18:

	==== Using the ''lctl'' debug daemon option to dump the buffer to a file====		==== Using the ''lctl'' debug daemon option to dump the buffer to a file====
	The debug_daemon option is used by ''lctl'' to control the dumping of the ''debug_kernel'' buffer to a user-specified file. For more information about the debug daemon including a list of commands, see [http://wiki.lustre.org/manual/LustreManual20_HTML/LustreDebugging.html#50438274_pgfId-1295930 Section 28.2.3: ''~~Debug Daemon Option~~ to ~~lctl~~''] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual''].		The debug_daemon option is used by ''lctl'' to control the dumping of the ''debug_kernel'' buffer to a user-specified file. For more information about the debug daemon including a list of commands, see [http://wiki.lustre.org/manual/LustreManual20_HTML/LustreDebugging.html#50438274_pgfId-1295930 Section 28.2.3: ''Dumping the Buffer to a File (debug_daemon)''] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual''].

	== Controlling the information written to the kernel debug log ==		== Controlling the information written to the kernel debug log ==

Sbarthel: /* Using the lctl debug daemon option to dump the buffer to a file */

2011-01-20T15:01:23Z

Using the lctl debug daemon option to dump the buffer to a file

← Older revision		Revision as of 08:01, 20 January 2011
Line 18:		Line 18:

	==== Using the ''lctl'' debug daemon option to dump the buffer to a file====		==== Using the ''lctl'' debug daemon option to dump the buffer to a file====
	The debug_daemon option is used by ''lctl'' to control the dumping of the ''debug_kernel'' buffer to a user-specified file. For more information about the debug daemon including a list of commands, see [http://wiki.lustre.org/manual/~~LustreManual18_HTML~~/LustreDebugging.html#~~50651266_pgfId~~-~~1291420~~ Section 24.2.1: ''Debug Daemon Option to lctl''] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual''].		The debug_daemon option is used by ''lctl'' to control the dumping of the ''debug_kernel'' buffer to a user-specified file. For more information about the debug daemon including a list of commands, see [http://wiki.lustre.org/manual/LustreManual20_HTML/LustreDebugging.html#50438274_pgfId-1295930 Section 28.2.3: ''Debug Daemon Option to lctl''] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual''].

	== Controlling the information written to the kernel debug log ==		== Controlling the information written to the kernel debug log ==

Sbarthel: /* Using the lctl tool to view debug messages */

2011-01-20T15:00:19Z

Using the lctl tool to view debug messages

← Older revision		Revision as of 08:00, 20 January 2011
Line 15:		Line 15:
	The ''lctl'' tool allows debug messages to be filtered based on subsystems and message types to extract information useful for troubleshooting from a kernel debug log. For more information about the ''lctl'' tool, see [http://wiki.lustre.org/manual/LustreManual20_HTML/LustreDebugging.html#50438274_pgfId-1295889 Section 28.2.2: ''Using the lctl Tool to View Debug Messages''] and [http://wiki.lustre.org/manual/LustreManual20_HTML/SystemConfigurationUtilities_HTML.html#50438219_pgfId-1318224 Section 36.3: ''lctl''] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual''].		The ''lctl'' tool allows debug messages to be filtered based on subsystems and message types to extract information useful for troubleshooting from a kernel debug log. For more information about the ''lctl'' tool, see [http://wiki.lustre.org/manual/LustreManual20_HTML/LustreDebugging.html#50438274_pgfId-1295889 Section 28.2.2: ''Using the lctl Tool to View Debug Messages''] and [http://wiki.lustre.org/manual/LustreManual20_HTML/SystemConfigurationUtilities_HTML.html#50438219_pgfId-1318224 Section 36.3: ''lctl''] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual''].

	A sample ''lctl'' run is shown in [http://wiki.lustre.org/manual/~~LustreManual18_HTML~~/LustreDebugging.html#~~50651266_pgfId~~-~~1291503~~ Section 24.2.7: ''Sample lctl Run''] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual''].		A sample ''lctl'' run is shown in [http://wiki.lustre.org/manual/LustreManual20_HTML/LustreDebugging.html#50438274_pgfId-1295915 Section 28.2.2.1: ''Sample lctl Run''] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual''].

	==== Using the ''lctl'' debug daemon option to dump the buffer to a file====		==== Using the ''lctl'' debug daemon option to dump the buffer to a file====