WARNING: This is the _old_ Lustre wiki, and it is in the process of being retired. The information found here is all likely to be out of date. Please search the new wiki for more up to date information.

Difference between revisions of "Building and Installing Lustre from Source Code"

From Obsolete Lustre Wiki
Jump to navigationJump to search
 
(59 intermediate revisions by 2 users not shown)
Line 1: Line 1:
If you are using a Linux kernel that is unsupported by Lustre, you will need to build and install Lustre from source code. This involves several steps:
+
<small>''(Updated: Feb 2010)''</small>
 +
__TOC__
 +
If you need to build a customized Lustre™ server kernel or are using a Linux kernel that has not been tested with the version of Lustre you are installing, you may need to build and install Lustre from source code. This involves several steps:
 
* Patching the core kernel
 
* Patching the core kernel
 
* Configuring the kernel to work with Lustre
 
* Configuring the kernel to work with Lustre
 
* Creating Lustre and kernel RPMs from source code.  
 
* Creating Lustre and kernel RPMs from source code.  
  
If you are installing Lustre on a supported Linux kernel, you will find it easier to install Lustre from packaged binaries (RPMs). For more information about this installation method, see [[Installing Lustre from Downloaded RPMs]].
+
Please note that the Lustre/kernel configurations available at the [http://www.oracle.com/technetwork/indexes/downloads/sun-az-index-095901.html#L Lustre download site] have been extensively tested and verified with Lustre. The recommended method for installing Lustre servers is to use these prebuilt binary packages (RPMs). For more information about this installation method, see [[Installing Lustre from Downloaded RPMs]].
 +
 
 +
For a list of available RPMs for Lustre 1.8.x and the latest version of Lustre 1.6 (1.6.7.2), see the [[Lustre_Release_Information#Lustre_Support_Matrix|Lustre Test Matrix]].
  
 
'''''Caution:'''''  Lustre contains kernel modifications which interact with storage devices and may introduce security issues and data loss if not installed, configured and administered correctly. Before installing Lustre, be cautious and back up ''ALL'' data.
 
'''''Caution:'''''  Lustre contains kernel modifications which interact with storage devices and may introduce security issues and data loss if not installed, configured and administered correctly. Before installing Lustre, be cautious and back up ''ALL'' data.
Line 11: Line 15:
 
modules (typically, the drivers) must be linked against the Linux kernel. The LNET
 
modules (typically, the drivers) must be linked against the Linux kernel. The LNET
 
modules in Lustre also need these references. To meet these requirements, a specific
 
modules in Lustre also need these references. To meet these requirements, a specific
process must be followed to install and recompile Lustre. See "Installing Lustre with
+
process must be followed to install and recompile Lustre. See [http://wiki.lustre.org/manual/LustreManual20_HTML/InstallingLustrefrSourceCode.html#50438210_pgfId-1299622 Section 29.4: ''Installing Lustre with a Third-Party Network Stack''] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual''] for an example showing how to install Lustre 1.6.6 using the Myricom MX 1.2.7 driver. The same process can be used for other third party network stacks.
a Third-Party Network Stack" in the [http://manual.lustre.org/index.php?title=Main_Page#Lustre_Operations_Manual Lustre Operations Manual] for an example showing how to install Lustre 1.6.6 using the Myricom MX 1.2.7 driver. The same process can be used for other third party network stacks.
 
  
 
== Patching the Kernel ==
 
== Patching the Kernel ==
Line 33: Line 36:
 
kernels, you have to get the required packages to successfully install Quilt.
 
kernels, you have to get the required packages to successfully install Quilt.
 
If you cannot locate a Quilt package or fulfill its dependencies, you can build Quilt
 
If you cannot locate a Quilt package or fulfill its dependencies, you can build Quilt
from a tarball, available at [http://savannah.nongnu.org/projects/quilt].
+
from a tarball, available at the [http://savannah.nongnu.org/projects/quilt Quilt project website].
  
 
For additional information on using Quilt, including its commands, see the [http://www.suse.de/~agruen/quilt.pdf Introduction to Quilt] and the [http://linux.die.net/man/1/quilt quilt(1) man page].
 
For additional information on using Quilt, including its commands, see the [http://www.suse.de/~agruen/quilt.pdf Introduction to Quilt] and the [http://linux.die.net/man/1/quilt quilt(1) man page].
  
 
===Get the Lustre Source and Unpatched Kernel===
 
===Get the Lustre Source and Unpatched Kernel===
The Lustre Group supports several Linux unpatched kernels for use with Lustre and provides a series of patches for each one. The Lustre patches are maintained in the ''kernel_patch'' directory bundled with the Lustre source code. The unpatched kernels are also available for download.
+
The Lustre Engineering Team has targeted several Linux kernels for use with Lustre servers (MDS/OSS) and provides a series of patches for each one. The Lustre patches are maintained in the ''kernel_patch'' directory bundled with the Lustre source code.
 +
 
 +
'''''Note:''''' Each patch series has been tailored to a specific kernel version, and may or may not apply cleanly to other versions of the kernel.
  
 
To obtain the Lustre source and unpatched kernel:
 
To obtain the Lustre source and unpatched kernel:
Line 45: Line 50:
 
For more information on these prerequisites, see [[Preparing to Install Lustre]].
 
For more information on these prerequisites, see [[Preparing to Install Lustre]].
  
2. ''Get the Lustre source code.'' On the [http://www.sun.com/software/products/lustre/get.jsp Lustre download site], select the
+
2. ''Download the Lustre source code.'' On the [http://www.oracle.com/technetwork/indexes/downloads/sun-az-index-095901.html#L Lustre download site], select a version of Lustre to download and then select ''Source'' as the platform.
Lustre version you want and then select ''Source'' as the platform. The files required to install Lustre from source code (unpatched kernels, Lustre
 
source and ''e2fsprogs'') are listed.
 
  
3. ''Download the Lustre source code'' (''lustre-<ver>.tar.gz'').
+
4. ''Download the unpatched kernel.''  
  
4. ''Download the unpatched kernel you want to use.''
+
For convenience, Sun maintains an archive of unpatched kernel sources at [http://downloads.lustre.org/public/kernels/ http://downloads.lustre.org/public/kernels/].  
If you do not know the kernel’s filename, check the [[which_patch]] file.
 
  
: a. In the Lustre source file, navigate to the [[which_patch]] file (lustre/kernel_patches/which_patch) and get the filename of the kernel you want to use. The which_patch file lists the kernels supported in this release.
+
5. ''To save time later, download ''e2fsprogs'' now.
  
: b. Download the selected kernel from the location where you downloaded the Lustre source in Step 2.
+
The source code for Sun's Lustre-enabled ''e2fsprogs'' distribution can be found at [http://downloads.lustre.org/public/tools/e2fsprogs/ http://downloads.lustre.org/public/tools/e2fsprogs/]
 
 
5. ''To save time later, download the e2fsprogs tarball'' (''e2fsprogs-<ver>tar.gz'').
 
  
 
===Patch the Kernel===
 
===Patch the Kernel===
Line 65: Line 65:
 
1.6.5.1.
 
1.6.5.1.
  
1. ''Unpack the Lustre source and kernel to separate source trees.'' Lustre source and the unpatched kernel were previously downloaded in [[#Get the Lustre Source and Unpatched Kernel|Get the Lustre Source and Unpatched Kernel]].
+
1. ''Unpack the Lustre source and kernel to separate source trees.''  
  
 
: a. ''Unpack the Lustre source.'' For this procedure, we assume that the resulting source tree is in ''/tmp/lustre-1.6.5.1''.
 
: a. ''Unpack the Lustre source.'' For this procedure, we assume that the resulting source tree is in ''/tmp/lustre-1.6.5.1''.
Line 78: Line 78:
  
 
4. Set up the necessary symlinks between the kernel patches and the Lustre
 
4. Set up the necessary symlinks between the kernel patches and the Lustre
source. This example assumes that the Lustre source files are unpacked under ''/tmp/lustre-1.6.5.1'' and you have chosen the ''2.6-rhel5.series'' file). Run:
+
source. This example assumes that the Lustre source files are unpacked under ''/tmp/lustre-1.6.5.1'' and you have chosen the ''2.6-rhel5.series'' file. Run:
  
 
<pre>
 
<pre>
 
$ cd /tmp/kernels/linux-2.6.18
 
$ cd /tmp/kernels/linux-2.6.18
 
$ rm -f patches series
 
$ rm -f patches series
$ ln -s /tmp/lustre-1.6.5.1/lustre/kernel_patches/series/2.6-\
+
$ ln -s /tmp/lustre-1.6.5.1/lustre/kernel_patches/series/2.6-rhel5.series ./series
rhel5.series ./series
 
 
$ ln -s /tmp/lustre-1.6.5.1/lustre/kernel_patches/patches .
 
$ ln -s /tmp/lustre-1.6.5.1/lustre/kernel_patches/patches .
 
</pre>
 
</pre>
Line 143: Line 142:
 
</pre>
 
</pre>
  
'''''Note:''''' If the steps to create the RPMs fail, contact Lustre Support by opening a bug in [http://bugzilla.lustre.org Bugzilla].
+
'''''Note:''''' If the steps to create the RPMs fail, contact Lustre Support by reporting a bug (see [[Reporting Bugs]]).
  
'''''Note:''''' Lustre supports several features and packages that extend the core functionality of Lustre. These features/packages can be enabled at the build time by issuing appropriate arguments to the configure command. For a list of supported features and packages, run ./configure –help in the Lustre source tree. The configs/ directory of the kernel source contains the config files matching each the kernel version. Copy one to .config at the root of the kernel tree.
+
'''''Note:''''' Lustre supports several features and packages that extend the core functionality of Lustre. These features/packages can be enabled at the build time by issuing appropriate arguments to the ''configure'' command. For a list of supported features and packages, run ''./configure –help'' in the Lustre source tree. The ''configs/'' directory of the kernel source contains the config files matching each the kernel version. Copy one to ''.config'' at the root of the kernel tree.
  
 
3. ''Create the kernel package.'' Navigate to the kernel source directory and run:
 
3. ''Create the kernel package.'' Navigate to the kernel source directory and run:
Line 157: Line 156:
 
'''''Note:''''' Step 3 is only valid for RedHat and SuSE kernels. If you are using a stock Linux kernel, you need to get a script to create the kernel RPM.
 
'''''Note:''''' Step 3 is only valid for RedHat and SuSE kernels. If you are using a stock Linux kernel, you need to get a script to create the kernel RPM.
  
4. ''Install the Lustre packages.'' Some Lustre packages are installed on servers (MDS and OSSs), and others are installed on Lustre clients.[[7footnote]] For guidance on where to install specific packages, see [[TABLE 3-1]].  
+
4. ''Install the Lustre packages.'' Some Lustre packages are installed on servers (MDS and OSSs), and others are installed on Lustre clients. For guidance on where to install specific packages, see [[Lustre Packages]].  
  
 
'''''Note:'''''  Running the patched server kernel on the clients is optional. It is not necessary unless the clients will be used for multiple purposes, for example, to run as a client and an OST.
 
'''''Note:'''''  Running the patched server kernel on the clients is optional. It is not necessary unless the clients will be used for multiple purposes, for example, to run as a client and an OST.
Line 165: Line 164:
 
:a. ''Install the kernel, modules and ldiskfs packages.'' Navigate to the directory where the RPMs are stored, and use the ''rpm -ivh'' command to install the kernel, module and ''ldiskfs'' packages.
 
:a. ''Install the kernel, modules and ldiskfs packages.'' Navigate to the directory where the RPMs are stored, and use the ''rpm -ivh'' command to install the kernel, module and ''ldiskfs'' packages.
  
<pre>
+
: <pre>$ rpm -ivh kernel-lustre-smp-<ver> \
$ rpm -ivh kernel-lustre-smp-<ver> \
+
;kernel-ib-<ver> \
kernel-ib-<ver> \
+
;lustre-modules-<ver> \
lustre-modules-<ver> \
+
;lustre-ldiskfs-<ver>
lustre-ldiskfs-<ver>
 
 
</pre>
 
</pre>
  
 
: b. ''Install the utilities/userspace packages.'' Use the ''rpm -ivh'' command to install the utilities packages. For example:
 
: b. ''Install the utilities/userspace packages.'' Use the ''rpm -ivh'' command to install the utilities packages. For example:
  
: (pre> $ rpm -ivh lustre-<ver>
+
: <pre>$ rpm -ivh lustre-<ver>
 
</pre>
 
</pre>
  
 
: c. ''Install the'' e2fsprogs ''package.'' Make sure the ''e2fsprogs'' package downloaded in Step 5 is unpacked, and use the ''rpm -i'' command to install it. For example:
 
: c. ''Install the'' e2fsprogs ''package.'' Make sure the ''e2fsprogs'' package downloaded in Step 5 is unpacked, and use the ''rpm -i'' command to install it. For example:
  
$ rpm -i e2fsprogs-<ver>
+
: <pre>$ rpm -i e2fsprogs-<ver>
 +
</pre>
  
 
If you want to add any optional packages to your Lustre file system, install them
 
If you want to add any optional packages to your Lustre file system, install them
Line 189: Line 188:
  
 
: a. ''If you applied the patched kernel to any clients, reboot them.'' Unpatched clients do not need to be rebooted.
 
: a. ''If you applied the patched kernel to any clients, reboot them.'' Unpatched clients do not need to be rebooted.
: b. ''Reboot the servers.'' Once all the machines have rebooted, the next steps are to configure Lustre Networking (LNET) and the Lustre file system. See [[Configuring Lustre]].
+
: b. ''Reboot the servers.'' Once all the machines have rebooted, the next steps are to configure Lustre Networking (LNET) and the Lustre file system. See [[Configuring the Lustre File System]].

Latest revision as of 11:59, 20 January 2011

(Updated: Feb 2010)

If you need to build a customized Lustre™ server kernel or are using a Linux kernel that has not been tested with the version of Lustre you are installing, you may need to build and install Lustre from source code. This involves several steps:

  • Patching the core kernel
  • Configuring the kernel to work with Lustre
  • Creating Lustre and kernel RPMs from source code.

Please note that the Lustre/kernel configurations available at the Lustre download site have been extensively tested and verified with Lustre. The recommended method for installing Lustre servers is to use these prebuilt binary packages (RPMs). For more information about this installation method, see Installing Lustre from Downloaded RPMs.

For a list of available RPMs for Lustre 1.8.x and the latest version of Lustre 1.6 (1.6.7.2), see the Lustre Test Matrix.

Caution: Lustre contains kernel modifications which interact with storage devices and may introduce security issues and data loss if not installed, configured and administered correctly. Before installing Lustre, be cautious and back up ALL data.

Note: When using third-party network hardware with Lustre, the third-party modules (typically, the drivers) must be linked against the Linux kernel. The LNET modules in Lustre also need these references. To meet these requirements, a specific process must be followed to install and recompile Lustre. See Section 29.4: Installing Lustre with a Third-Party Network Stack in the Lustre Operations Manual for an example showing how to install Lustre 1.6.6 using the Myricom MX 1.2.7 driver. The same process can be used for other third party network stacks.

Patching the Kernel

If you are using non-standard hardware, plan to apply a Lustre patch, or have another reason not to use packaged Lustre binaries, you have to apply several Lustre patches to the core kernel and run the Lustre configure script against the kernel.

Introducing the Quilt Utility

To simplify the process of applying Lustre patches to the kernel, we recommend that you use the Quilt utility.

Quilt manages a stack of patches on a single source tree. A series file lists the patch files and the order in which they are applied. Patches are applied, incrementally, on the base tree and all preceding patches. You can:

  • Apply patches from the stack (quilt push)
  • Remove patches from the stack (quilt pop)
  • Query the contents of the series file (quilt series), the contents of the stack (quilt applied, quilt previous, quilt top), and the patches that are not applied at a particular moment (quilt next, quilt unapplied).
  • Edit and refresh (update) patches with Quilt, as well as revert inadvertent changes, and fork or clone patches and show the diffs before and after work.

A variety of Quilt packages (RPMs, SRPMs and tarballs) are available from various sources. Use the most recent version you can find. Quilt depends on several other utilities, e.g., the coreutils RPM available only in RedHat 9. For other RedHat kernels, you have to get the required packages to successfully install Quilt. If you cannot locate a Quilt package or fulfill its dependencies, you can build Quilt from a tarball, available at the Quilt project website.

For additional information on using Quilt, including its commands, see the Introduction to Quilt and the quilt(1) man page.

Get the Lustre Source and Unpatched Kernel

The Lustre Engineering Team has targeted several Linux kernels for use with Lustre servers (MDS/OSS) and provides a series of patches for each one. The Lustre patches are maintained in the kernel_patch directory bundled with the Lustre source code.

Note: Each patch series has been tailored to a specific kernel version, and may or may not apply cleanly to other versions of the kernel.

To obtain the Lustre source and unpatched kernel:

1. Verify that all of the Lustre installation requirements have been met. For more information on these prerequisites, see Preparing to Install Lustre.

2. Download the Lustre source code. On the Lustre download site, select a version of Lustre to download and then select Source as the platform.

4. Download the unpatched kernel.

For convenience, Sun maintains an archive of unpatched kernel sources at http://downloads.lustre.org/public/kernels/.

5. To save time later, download e2fsprogs now.

The source code for Sun's Lustre-enabled e2fsprogs distribution can be found at http://downloads.lustre.org/public/tools/e2fsprogs/

Patch the Kernel

This procedure describes how to use Quilt to apply the Lustre patches to the kernel. To illustrate the steps in this procedure, a RHEL 5 kernel is patched for Lustre 1.6.5.1.

1. Unpack the Lustre source and kernel to separate source trees.

a. Unpack the Lustre source. For this procedure, we assume that the resulting source tree is in /tmp/lustre-1.6.5.1.
b. Unpack the kernel. For this procedure, we assume that the resulting source tree (also known as the destination tree) is in /tmp/kernels/linux-2.6.18.

2. Select a config file for your kernel, located in the kernel_configs directory (lustre/kernel_patches/kernel_config). The kernel_config directory contains the .config files, which are named to indicate the kernel and architecture with which they are associated. For example, the configuration file for the 2.6.18 kernel shipped with RHEL 5 (suitable for i686 SMP systems) is kernel-2.6.18-2.6-rhel5-i686-smp.config.

3. Select the series file for your kernel, located in the series directory (lustre/kernel_patches/series). The series file contains the patches that need to be applied to the kernel.

4. Set up the necessary symlinks between the kernel patches and the Lustre source. This example assumes that the Lustre source files are unpacked under /tmp/lustre-1.6.5.1 and you have chosen the 2.6-rhel5.series file. Run:

$ cd /tmp/kernels/linux-2.6.18
$ rm -f patches series
$ ln -s /tmp/lustre-1.6.5.1/lustre/kernel_patches/series/2.6-rhel5.series ./series
$ ln -s /tmp/lustre-1.6.5.1/lustre/kernel_patches/patches .

5. Use Quilt to apply the patches in the selected series file to the unpatched kernel. Run:

$ cd /tmp/kernels/linux-2.6.18
$ quilt push -av

The patched destination tree acts as a base Linux source tree for Lustre.

Create and Install the Lustre Packages

After patching the kernel, configure it to work with Lustre, create the Lustre packages (RPMs) and install them.

1. Configure the patched kernel to run with Lustre. Run:

$ cd <path to kernel tree>
$ cp /boot/config-‘uname -r‘ .config
$ make oldconfig || make menuconfig
$ make include/asm
$ make include/linux/version.h
$ make SUBDIRS=scripts
$ make include/linux/utsrelease.h

2. Run the Lustre configure script against the patched kernel and create the Lustre packages.

$ cd <path to lustre source tree>
$ ./configure --with-linux=<path to kernel tree>
$ make rpms

This creates a set of .rpms in /usr/src/redhat/RPMS/<arch> with an appended date-stamp. The SuSE path is /usr/src/packages.

Note: You do not need to run the Lustre configure script against an unpatched kernel.

Example set of RPMs:

lustre-1.6.5.1-\
2.6.18_53.xx.xx.el5_lustre.1.6.5.1.custom_20081021.i686.rpm
lustre-debuginfo-1.6.5.1-\
2.6.18_53.xx.xx.el5_lustre.1.6.5.1.custom_20081021.i686.rpm
lustre-modules-1.6.5.1-\
2.6.18_53.xx.xxel5_lustre.1.6.5.1.custom_20081021.i686.rpm
lustre-source-1.6.5.1-\
2.6.18_53.xx.xx.el5_lustre.1.6.5.1.custom_20081021.i686.rpm

Note: If the steps to create the RPMs fail, contact Lustre Support by reporting a bug (see Reporting Bugs).

Note: Lustre supports several features and packages that extend the core functionality of Lustre. These features/packages can be enabled at the build time by issuing appropriate arguments to the configure command. For a list of supported features and packages, run ./configure –help in the Lustre source tree. The configs/ directory of the kernel source contains the config files matching each the kernel version. Copy one to .config at the root of the kernel tree.

3. Create the kernel package. Navigate to the kernel source directory and run:

$ make rpm

Example result:

kernel-2.6.95.0.3.EL_lustre.1.6.5.1custom-1.i686.rpm

Note: Step 3 is only valid for RedHat and SuSE kernels. If you are using a stock Linux kernel, you need to get a script to create the kernel RPM.

4. Install the Lustre packages. Some Lustre packages are installed on servers (MDS and OSSs), and others are installed on Lustre clients. For guidance on where to install specific packages, see Lustre Packages.

Note: Running the patched server kernel on the clients is optional. It is not necessary unless the clients will be used for multiple purposes, for example, to run as a client and an OST.

Lustre packages should be installed in this order:

a. Install the kernel, modules and ldiskfs packages. Navigate to the directory where the RPMs are stored, and use the rpm -ivh command to install the kernel, module and ldiskfs packages.
$ rpm -ivh kernel-lustre-smp-<ver> \
kernel-ib-<ver> \
lustre-modules-<ver> \
lustre-ldiskfs-<ver>
b. Install the utilities/userspace packages. Use the rpm -ivh command to install the utilities packages. For example:
$ rpm -ivh lustre-<ver>
c. Install the e2fsprogs package. Make sure the e2fsprogs package downloaded in Step 5 is unpacked, and use the rpm -i command to install it. For example:
$ rpm -i e2fsprogs-<ver>

If you want to add any optional packages to your Lustre file system, install them now.

5. Verify that the boot loader (grub.conf or lilo.conf) has been updated to load the patched kernel.

6. Reboot the patched clients and the servers.

a. If you applied the patched kernel to any clients, reboot them. Unpatched clients do not need to be rebooted.
b. Reboot the servers. Once all the machines have rebooted, the next steps are to configure Lustre Networking (LNET) and the Lustre file system. See Configuring the Lustre File System.