Using Red Hat Cluster Manager with Lustre - Revision history

Docadmin: /* Preliminary Notes */

2010-12-20T16:13:35Z

Preliminary Notes

← Older revision		Revision as of 09:13, 20 December 2010
Line 19:		Line 19:
	It is assumed that two Lustre server nodes share a number of Lustre targets. Each Lustre node provides a number of Lustre targets and, in case of a failure, the active/non-failed node takes over the Lustre targets of the failed nodes and makes them available to the Lustre clients.		It is assumed that two Lustre server nodes share a number of Lustre targets. Each Lustre node provides a number of Lustre targets and, in case of a failure, the active/non-failed node takes over the Lustre targets of the failed nodes and makes them available to the Lustre clients.

	Furthermore, to make sure the Lustre targets are mounted only on one of Lustre server nodes at a time, ~~we have implemented~~ STONITH fencing. This requires a way to make sure the failed node is shut down in case of a failure. In the examples below, it is assumed that the Lustre server nodes are equipped with a service processor allowing to shut down a failed node using IPMI. For other methods of fencing, refer to the RedHat Cluster documentation.		Furthermore, to make sure the Lustre targets are mounted only on one of Lustre server nodes at a time, STONITH fencing is implemented. This requires a way to make sure the failed node is shut down in case of a failure. In the examples below, it is assumed that the Lustre server nodes are equipped with a service processor allowing to shut down a failed node using IPMI. For other methods of fencing, refer to the RedHat Cluster documentation.

	== Setting Up RedHat Cluster ==		== Setting Up RedHat Cluster ==

Docadmin: /* Preliminary Notes */

2010-12-20T16:12:18Z

Preliminary Notes

← Older revision		Revision as of 09:12, 20 December 2010
Line 13:		Line 13:
	== Preliminary Notes ==		== Preliminary Notes ==

	This document is based on the RedHat Cluster version 2.0, which is part of RedHat Enterprise Linux version 5.5. For other versions or RHEL-based distributions the syntax or methods to ~~setup~~ and run RedHat Cluster may differ.		This document is based on the RedHat Cluster version 2.0, which is part of RedHat Enterprise Linux version 5.5. For other versions or RHEL-based distributions, the syntax or methods to set up and run RedHat Cluster may differ.

	In comparison with other HA solutions RedHat Cluster as in RHEL 5.5 is an old HA solution. We recommend using other HA solutions like Pacemaker, if possible.		In comparison with other HA solutions, RedHat Cluster as in RHEL 5.5 is an old HA solution. We recommend using other HA solutions like Pacemaker, if possible.

	It is assumed that two Lustre server nodes share a number of Lustre targets. Each Lustre node provides a number of Lustre targets and in case of a failure the ~~not~~ failed node takes over the Lustre targets of the failed nodes and makes them available to the Lustre clients.		It is assumed that two Lustre server nodes share a number of Lustre targets. Each Lustre node provides a number of Lustre targets and, in case of a failure, the active/non-failed node takes over the Lustre targets of the failed nodes and makes them available to the Lustre clients.

	Furthermore, to make sure the Lustre targets are mounted only on one of ~~the~~ Lustre server nodes at a time we ~~implement~~ STONITH fencing. This requires a way to make sure the failed node is shut down in case of a failure. In the examples ~~shown in this article~~ it is assumed that the Lustre server nodes are equipped with a service processor allowing to shut down a failed node using IPMI. For other methods of fencing refer to the RedHat Cluster documentation.		Furthermore, to make sure the Lustre targets are mounted only on one of Lustre server nodes at a time, we have implemented STONITH fencing. This requires a way to make sure the failed node is shut down in case of a failure. In the examples below, it is assumed that the Lustre server nodes are equipped with a service processor allowing to shut down a failed node using IPMI. For other methods of fencing, refer to the RedHat Cluster documentation.

	== Setting Up RedHat Cluster ==		== Setting Up RedHat Cluster ==

Docadmin: /* Preliminary Notes */

2010-12-20T16:09:53Z

Preliminary Notes

← Older revision		Revision as of 09:09, 20 December 2010
Line 15:		Line 15:
	This document is based on the RedHat Cluster version 2.0, which is part of RedHat Enterprise Linux version 5.5. For other versions or RHEL-based distributions the syntax or methods to setup and run RedHat Cluster may differ.		This document is based on the RedHat Cluster version 2.0, which is part of RedHat Enterprise Linux version 5.5. For other versions or RHEL-based distributions the syntax or methods to setup and run RedHat Cluster may differ.

	In comparison with other HA solutions RedHat Cluster as in RHEL 5.5 is an old HA solution. ~~It is recommended to use~~ other HA solutions like ~~pacemaker~~, if possible.		In comparison with other HA solutions RedHat Cluster as in RHEL 5.5 is an old HA solution. We recommend using other HA solutions like Pacemaker, if possible.

	It is assumed that two Lustre server nodes share a number of Lustre targets. Each ~~of the~~ Lustre ~~nodes provide~~ a number of Lustre targets and in case of a failure the not failed node takes over the Lustre targets of the failed nodes and makes them available to the Lustre clients.		It is assumed that two Lustre server nodes share a number of Lustre targets. Each Lustre node provides a number of Lustre targets and in case of a failure the not failed node takes over the Lustre targets of the failed nodes and makes them available to the Lustre clients.

	Furthermore, to make sure the Lustre targets are mounted only on one of the Lustre server nodes at a time we implement STONITH fencing. This requires a way to make sure the failed node is shut down in case of a failure. In the examples shown in this article it is assumed that the Lustre server nodes are equipped with a service processor allowing to shut down a failed node using IPMI. For other methods of fencing refer to the RedHat Cluster documentation.		Furthermore, to make sure the Lustre targets are mounted only on one of the Lustre server nodes at a time we implement STONITH fencing. This requires a way to make sure the failed node is shut down in case of a failure. In the examples shown in this article it is assumed that the Lustre server nodes are equipped with a service processor allowing to shut down a failed node using IPMI. For other methods of fencing refer to the RedHat Cluster documentation.

Sven at 14:37, 14 December 2010

2010-12-14T14:37:57Z

← Older revision		Revision as of 07:37, 14 December 2010
Line 13:		Line 13:
	== Preliminary Notes ==		== Preliminary Notes ==

	This document is based on the RedHat Cluster version 2.0, which is part of RedHat Enterprise Linux version 5.5. For other ~~Versions~~ or RHEL-based distributions the syntax or methods to setup and run RedHat Cluster may differ.		This document is based on the RedHat Cluster version 2.0, which is part of RedHat Enterprise Linux version 5.5. For other versions or RHEL-based distributions the syntax or methods to setup and run RedHat Cluster may differ.

	In comparison with other HA solutions RedHat Cluster as in RHEL 5.5 is ~~a pretty~~ old HA solution. It is recommended to use other HA solutions like pacemaker, if possible.		In comparison with other HA solutions RedHat Cluster as in RHEL 5.5 is an old HA solution. It is recommended to use other HA solutions like pacemaker, if possible.

	It is assumed that two Lustre server nodes share a number of Lustre targets. Each of the Lustre nodes provide a number of Lustre targets and in case of a failure the not failed node takes over the Lustre targets of the failed nodes and makes them available to the Lustre clients.		It is assumed that two Lustre server nodes share a number of Lustre targets. Each of the Lustre nodes provide a number of Lustre targets and in case of a failure the not failed node takes over the Lustre targets of the failed nodes and makes them available to the Lustre clients.

	Furthermore, to make sure the Lustre targets are mounted only on one of the Lustre server nodes at a time we implement STONITH fencing. This requires a way to make sure the failed node is shut down in case of a failure. In the examples shown in this article it is assumed that the Lustre server nodes are equipped with a service processor allowing to shut down a failed node using IPMI.		Furthermore, to make sure the Lustre targets are mounted only on one of the Lustre server nodes at a time we implement STONITH fencing. This requires a way to make sure the failed node is shut down in case of a failure. In the examples shown in this article it is assumed that the Lustre server nodes are equipped with a service processor allowing to shut down a failed node using IPMI. For other methods of fencing refer to the RedHat Cluster documentation.

	== Setting Up RedHat Cluster ==		== Setting Up RedHat Cluster ==

			Setting up RedHat Cluster consists of three steps:
			* setup ''openais'',
			* configure the cluster and,
			* start the RedHat cluster services

	==== Setting Up the ''openais'' Communication Stack ====		==== Setting Up the ''openais'' Communication Stack ====
Line 49:		Line 54:
	ringnumber: 0		ringnumber: 0
	bindnetaddr: 10.0.0.0		bindnetaddr: 10.0.0.0
	mcastaddr: 226.94.1.1		mcastaddr: 226.94.1.1
	mcastport: 5405		mcastport: 5405
	}		}
	}		}
	</pre>		</pre>

	''Openais'' uses the option ''bindnetaddr'' to determine which interface is to be used for cluster communication. ~~The~~ example above ~~assumes~~ one of the node’s interfaces is configured on the network 10.0.0.0. The value of the option is calculated from the IP address AND the network mask for the interface (IP & MASK) so the final bits of the address are cleared. Thus the configuration file is independent of any node and can be copied to all nodes.		''Openais'' uses the option ''bindnetaddr'' to determine which interface is to be used for cluster communication. In the example shown above, it is assumed that one of the node’s interfaces is configured on the network 10.0.0.0. The value of the option is calculated from the IP address AND the network mask for the interface (IP & MASK) so the final bits of the address are cleared. Thus the configuration file is independent of any node and can be copied to all nodes.

	'''2. Create an AIS key		'''2. Create an AIS key
Line 67:		Line 72:
	== Installing RedHat Cluster ==		== Installing RedHat Cluster ==

	The minimum installation of RedHat Cluster consists of the Cluster Manager package ''cman'' and the Resource Group Manager package ''rgmanager''. The ''cman'' package can be found in the RHEL repository. The ''rgmanager'' package is part of the Cluster repository. It can be found on the RHEL DVD in the Cluster sub-directory and may need to be added to the yum configuration manually. With yum configured correctly RedHat Cluster can be installed using:		The minimum installation of RedHat Cluster consists of the Cluster Manager package ''cman'' and the Resource Group Manager package ''rgmanager''. The ''cman'' package can be found in the RHEL repository. The ''rgmanager'' package is part of the Cluster repository. It can be found on the RHEL DVD/ISO image in the ''Cluster'' sub-directory and may need to be added to the yum configuration manually. With yum configured correctly RedHat Cluster can be installed using:
	<pre>		<pre>
	yum install cman rgmanager		yum install cman rgmanager
Line 73:		Line 78:
	If yum is not set up correctly, the rpm packages and their dependencies need to be installed manually.		If yum is not set up correctly, the rpm packages and their dependencies need to be installed manually.

	== Installing the ~~Lustre Resource Skript~~ ==		== Installing the Lustrefs resource script ==

	The ''rgmanager'' package includes a number of resource scripts (''/usr/share/cluster'') which are used to integrate resources like network interfaces or file systems with ''rgmanager''. Unfortunately, there is no resource script for Lustre included.		The ''rgmanager'' package includes a number of resource scripts (''/usr/share/cluster'') which are used to integrate resources like network interfaces or file systems with ''rgmanager''. Unfortunately, there is no resource script for Lustre included.
Line 84:		Line 89:
	Make sure the script is executable.		Make sure the script is executable.

	== Configure ~~RedHat~~ Cluster ==		== Configure your Cluster ==

	RedHat Cluster uses ''/etc/cluster/cluster.conf'' as central configuration file. This file is in XML format. The complete schema of the XML file can be found at http://sources.redhat.com/cluster/doc/cluster_schema_rhel5.html.		RedHat Cluster uses ''/etc/cluster/cluster.conf'' as central configuration file. This file is in XML format. The complete schema of the XML file can be found at http://sources.redhat.com/cluster/doc/cluster_schema_rhel5.html.
Line 92:		Line 97:
	<?xml version="1.0" ?>		<?xml version="1.0" ?>
	<cluster config_version="1" name="Lustre">		<cluster config_version="1" name="Lustre">

	...		...
	</cluster>		</cluster>

Sven: /* Start RedHat Cluster */

2010-12-14T14:13:16Z

Start RedHat Cluster

@@ Line 260: / Line 260: @@
   service:lustre_1                   lustre1                            started
   service:lustre_2                   lustre2                            started
 </pre>

Sven: /* Useful tools to use with RedHat Cluster */

2010-12-14T14:01:32Z

Useful tools to use with RedHat Cluster

← Older revision		Revision as of 07:01, 14 December 2010
Line 262:		Line 262:
	</pre>		</pre>

	== ~~Useful~~ tools to use with RedHat Cluster ==		== Other tools to use with RedHat Cluster ==

	RedHat cluster is a complex system of programs and services. There are a number of tools available to interact and/or make working with RedHat Cluster easier. In this section a number of these tools are presented. For more details read the man pages.		RedHat cluster is a complex system of programs and services. There are a number of tools available to interact and/or make working with RedHat Cluster easier. In this section a number of these tools are presented. For more details read the man pages.

Sven: /* Useful tools to use with RedHat Cluster */

2010-12-14T13:58:32Z

Useful tools to use with RedHat Cluster

← Older revision		Revision as of 06:58, 14 December 2010
Line 267:		Line 267:

	; cman_tool : can be used to manage the cman subsystem. It can be used to add or remove nodes to a cluster configuration		; cman_tool : can be used to manage the cman subsystem. It can be used to add or remove nodes to a cluster configuration
	; ccs_tool :		; ccs_tool : may be used to update the configuration of the running cluster
	; clustat :		; clustat : show the status of the cluster and if and where services are currently running
	; clusvcadm :		; clusvcadm : can be used to enable, disable or relocate services in a cluster
	; system-config-cluster :		; system-config-cluster : a graphical user interface for cluster configuration

Sven: /* Useful tools to use with RedHat Cluster */

2010-12-14T13:32:53Z

Useful tools to use with RedHat Cluster

← Older revision		Revision as of 06:32, 14 December 2010
Line 266:		Line 266:
	RedHat cluster is a complex system of programs and services. There are a number of tools available to interact and/or make working with RedHat Cluster easier. In this section a number of these tools are presented. For more details read the man pages.		RedHat cluster is a complex system of programs and services. There are a number of tools available to interact and/or make working with RedHat Cluster easier. In this section a number of these tools are presented. For more details read the man pages.

	~~=== cman_tool ===~~		; cman_tool : can be used to manage the cman subsystem. It can be used to add or remove nodes to a cluster configuration
			; ccs_tool :
	cman_tool can be used to manage the cman subsystem. It can be used to add or remove nodes to a cluster configuration		; clustat :
			; clusvcadm :
	~~===~~ ccs_tool ~~===~~		; system-config-cluster :

	~~===~~ clustat ~~===~~

	~~===~~ clusvcadm ~~===~~

	~~===~~ system-config-cluster ~~===~~

Sven: /* cman_tool */

2010-12-14T13:29:36Z

cman_tool

← Older revision		Revision as of 06:29, 14 December 2010
Line 267:		Line 267:

	=== cman_tool ===		=== cman_tool ===

			cman_tool can be used to manage the cman subsystem. It can be used to add or remove nodes to a cluster configuration

	=== ccs_tool ===		=== ccs_tool ===

Sven: /* Start RedHat Cluster */

2010-12-14T12:14:55Z

Start RedHat Cluster

@@ Line 237: / Line 237: @@
 ==== ''rgmanager'' service ====
-==== verifying RedHat Cluster ====
+To verify the state of the cluster run ''clustat'' again. With the above configuration the output should look like this:
 == Useful tools to use with RedHat Cluster ==