WARNING: This is the _old_ Lustre wiki, and it is in the process of being retired. The information found here is all likely to be out of date. Please search the new wiki for more up to date information.

Difference between revisions of "Applying Lustre Patches to a Kernel"

From Obsolete Lustre Wiki
Jump to navigationJump to search
Line 22: Line 22:
 
  ....
 
  ....
  
 +
 +
 +
 +
== Applying Lustre Kernel Patches to a Tree ==
 +
 +
After you have checked out the Lustre source (see [[Accessing Lustre Code]], follow these steps to apply the appropriate Lustre kernel patches to your tree:
 +
 +
1. ''Select a series file.'' Look in the file ''lustre/kernel_patches/which_patch''. This shows:
 +
* What series files apply against what kernel sources
 +
* For each series file there is only kernel source that is supported
 +
* the architecture is mentioned
 +
 +
2. ''Choose a config file.'' The ".config" files corresponding to supported kernels are in the directory ''lustre/kernel_patches/kernel_configs''.
 +
 +
The name of each config file begins with the series name. For example, the config file ''2.6-rhel5.config'' is a UML config file for the vanilla 2.4.20 series.[[Are the numbers correct in this sentence?]]
 +
 +
3. Unpack a kernel source tree somewhere (they can be found at http://downloads.lustre.org/public/kernels/). This could involve ''tar -xf linux-2.6.18-128.1.1-el5.tar.bz2''. We'll pretend that the resulting source tree is in '''''/tmp/kernels/linux-2.6.18-128.1.1''''' and we call this the destination tree.
  
 
== Introduction to the Quilt package ==
 
== Introduction to the Quilt package ==
Line 33: Line 50:
 
* Inadvertent changes can be reverted and patches forked or cloned. Diffs allow before and after change comparisons.
 
* Inadvertent changes can be reverted and patches forked or cloned. Diffs allow before and after change comparisons.
  
''Quilt'' is included in most Linux distributions, [[or can be installed afterward using yum, apt-get, etc.]] [[(OK to replace with something like this?)]] It can also be downloaded from the [http://savannah.nongnu.org/projects/quilt Quilt Project Site].
+
''Quilt'' is included in most Linux distributions, [[or can be installed afterward using yum, apt-get, etc.]] [[(OK to replace with something like this?)]] It can also be downloaded from the [http://savannah.nongnu.org/projects/quilt Quilt Project Site].=== Applying the patches ===
 
 
== Quick Instructions for Applying Existing Patches to a Tree ==
 
 
 
=== Choosing the kernel series, config files ===
 
 
 
1. Check out Lustre from CVS or take a tar ball.
 
 
 
2. Choose a series file. Look in the file ''lustre/kernel_patches/which_patch''. This shows:
 
* What series files apply against what kernel sources
 
* For each series file there is only kernel source that is supported
 
* the architecture is mentioned
 
 
 
3. Choose a config file. Supported kernel ".config" files are in the ''lustre/kernel_patches/kernel_configs''
 
* The file name of a config file starts with the series name: e.g. ''2.6-rhel5.config'' is a UML config file for the vanilla 2.4.20 series.
 
 
 
4. Unpack a kernel source tree somewhere (they can be found at http://downloads.lustre.org/public/kernels/). This could involve ''tar -xf linux-2.6.18-128.1.1-el5.tar.bz2''. We'll pretend that the resulting source tree is in '''''/tmp/kernels/linux-2.6.18-128.1.1''''' and we call this the destination tree.
 
 
 
=== Applying the patches ===
 
  
 
1. Now you need use ''quilt'' to setup the series to use for your kernel. This means adding two symlinks to your linux source tree:
 
1. Now you need use ''quilt'' to setup the series to use for your kernel. This means adding two symlinks to your linux source tree:

Revision as of 15:18, 15 September 2009

This page describes how to apply Lustre kernel patches to a tree, how to set up a package supplied with the Lustre code called Quilt to use to manage changes to patches, and how to modify an existing kernel patch or contribute a new kernel patch.

Overview of Lustre Patches

To support Lustre development and functionality, some changes must be made to the core Linux kernel. These changes are organized in a set of kernel patches kept in the Lustre CVS repository in the directory lustre/kernel_patches/patches/.

The patches to be applied depend on the kernel that is to be used. A series file is created in lustre/kernel_patches/series/ for each supported kernel to define and control the patches to be used for that kernel.

For example, the file lustre/kernel_patches/series/rh-2.4.20 lists all the patches that must be applied to a Red Hat 2.4.20 kernel to build a Lustre compatible kernel. An excerpt from the current rh-2.4.20 series is shown below:

mcore-2.4.20-8.patch
dsp.patch
dev_read_only_2.4.20-rh.patch
exports_2.4.20-rh-hp.patch
kmem_cache_validate_2.4.20-rh.patch
lustre_version.patch
vfs_intent-2.4.20-rh.patch
invalidate_show-2.4.20-rh.patch
iod-rmap-exports-2.4.20.patch
export-truncate.patch
....
....
....



Applying Lustre Kernel Patches to a Tree

After you have checked out the Lustre source (see Accessing Lustre Code, follow these steps to apply the appropriate Lustre kernel patches to your tree:

1. Select a series file. Look in the file lustre/kernel_patches/which_patch. This shows:

  • What series files apply against what kernel sources
  • For each series file there is only kernel source that is supported
  • the architecture is mentioned

2. Choose a config file. The ".config" files corresponding to supported kernels are in the directory lustre/kernel_patches/kernel_configs.

The name of each config file begins with the series name. For example, the config file 2.6-rhel5.config is a UML config file for the vanilla 2.4.20 series.Are the numbers correct in this sentence?

3. Unpack a kernel source tree somewhere (they can be found at http://downloads.lustre.org/public/kernels/). This could involve tar -xf linux-2.6.18-128.1.1-el5.tar.bz2. We'll pretend that the resulting source tree is in /tmp/kernels/linux-2.6.18-128.1.1 and we call this the destination tree.

Introduction to the Quilt package

A package called Quilt package has been developed by Andreas Gruenbacher to facilitate managing many patches on a single source tree. A general overview of how this works is as follows:

  • A series file lists an ordered collection of patches.
  • The patches in the series form a stack.
  • Quilt can be used to push and pop the patches.
  • When the stack is managed with Quilt, patches can be edited and refreshed (updated).
  • Inadvertent changes can be reverted and patches forked or cloned. Diffs allow before and after change comparisons.

Quilt is included in most Linux distributions, or can be installed afterward using yum, apt-get, etc. (OK to replace with something like this?) It can also be downloaded from the Quilt Project Site.=== Applying the patches ===

1. Now you need use quilt to setup the series to use for your kernel. This means adding two symlinks to your linux source tree:

a. symlink series -> ../lustre/kernel_patches/series/2.6-rhel5.series

b. symlink patches -> ../lustre/kernel_patches/patches

# cd /usr/src/linux-2.6.18-128.1.1
# ln -s ../lustre/kernel_patches/series/2.6-rhel5.series series
# ln -s ../lustre/kernel_patches/patches patches

2. Now apply all the patches listed in the series file that quilt added a symlink for in the kernel source tree:

# cd /usr/src/linux-2.6.18-128.1.1
# quilt push -av

3. If the right series file was chosen and the patches were up to date the destination Linux tree should now be able to act as a base Linux source tree for Lustre.

User Notes

Lustre Kernel Release Maintenance

Typically a kernel developer will fix one kernel and introduce changes to the series for this kernel. Other developers in the QA or release engineering team need to fix up the other kernels. It is critical that the procedure below is followed in detail to avoid enormous time wastage.

Naming of Complete Patches

  • Patches should be named <patchname><kernel-version>.patch ( for example vfs_intent-2.4.20-rh.patch) . We store these in a subdirectory patches of lustre/kernel_patches/
  • class_obd.c should have a:
    • LUSTRE_MIN_VERSION == Smallest acceptable Lustre-version.
    • LUSTRE_MAX_VERSION == Highest acceptable Lustre-version.
    • This allows a script to list obsolete complete patches and clean them up.
  • which_patch should have one entry per line kernel-memnonic series so that the lbuild script can plough through it and build kernels.

When updating patches, it is best to keep new versions of the patches as similar to the old ones as possible, so that "cvs diff" will clearly show the changes that have been made to the patch. Also, it is best to keep the different versions of each patch as close as possible, so "diff -u foo-rhel4.patch foo-sles10.patch" will show as small a diff as possible and we can verify that each of the patches contain the fixes that have been applied to the others.

To make this easier, options can be passed to quilt via $HOME/.quiltrc:

export QUILT_DIFF_OPTS="-upa"
export QUILT_NO_DIFF_TIMESTAMPS=1

Maintenance: Series Files

  • A series file lists patches that are part of the series.
  • By Lustre convention, a series file supports one kernel, not two or more kernels.
  • The number of patches used by all the different series needs to be kept to a minimum.

Bug fixes involving kernel patches

Peter finds the solution to a bug, he needs to changes a patch that affects fs/ext3/iopen.c. Peter works on a private branch. He proceeds as follows:

  • He looks in the series file and sees that iopen-2.6-rhel5 is the name of the patch
  • Pop to that patch: quilt pop iopen-2.6-rhel5
  • Add the fix
  • quilt refresh -o
    • this updates the patch in the lustre source
    • -o starts a new session, gendiff shows diffs made in a session
  • quilt gendiff > peters-fix.diff
    • to show what was done in this session to fix the bug
  • quilt push -a
    • put all other patches back in
  • Test the code

Peter now assigns a bug to Alex asking him to apply peters-fix.diff to all iopen.c in all series. Alex proceeds similarly with a collection of kernel sources calling quilt refresh for each iopen patch file he needs to fix.

Alex now files an issue on the QA team to test all series that have the affected iopen patches.

Kernel upgrades

Suppose Linus sends us a new kernel, 2.6.25 and we have patches for 2.6.24.

  • Update which_patch
  • Start pushing patches, suppose one foo-2.6.20.patch fails (it had been around since 2.6.20 probably):
    • If the patch is shared with another series fork it:
    • quilt fork foo-2.6.25.patch
    • quilt push -f
      • This forces the patch in
      • fix conflicts
    • quilt refresh

You can back out a forced patch with quilt pop -R -f.

Making Changes to the Patches

  • Push the patches up to including the one that needs changes.
  • Edit the source file that needs changing.
  • Call quilt refresh.
  • Verify that the rest of the series applies by calling quilt push.

Adding a New File to the Kernel

  • Make sure the top patch is the patch you want. (quilt push {patch_name} or quilt pop {patch_name})
  • Edit the New file
  • Call quilt add [-p {patch_name}] {file_name} to add the file to one of your patches, If the top patch is just the patch you want to add your new file in.the option -p {patch_name} can be omitted.

Making Changes in Another Source File

  • Make sure the top patch is the patch you want. (quilt push {patch_name} or quilt pop {patch_name})
  • Call quilt add {that_source_name} to add that source file to the top patch
  • Make changes in the source file
  • Call quilt refresh to refresh the patch.

Adding a Patch that Someone Else Gave You

To Add a Patch into a Series:

  • Ideally a patch can be added to the end of the series. This is most easily done with quilt add. After the patch is imported it still needs to be applied and refreshed with quilt push and quilt refresh. Remember to cvs_add the patch with -ko so that tags in the context of the diff aren't changed by CVS, rendering the patch unusable.
  • If introducing a new patch use quilt new, editing, and quilt refresh.

General Guidelines

  1. Try to limit the scope of changes in a patch file to a group of related changes.

CVS Directory Layout

  1. patches/ - Contains all the patch files themselves. We should have a patch per functional change.
  2. series/ - The text files that patch-utils use to define the ordering of patches that are applied to a tree. We have a series file for each kernel tree variant that requires wildly different patches (architecture differences, stock vs. redhat, etc).

Changing Patches

It may be necessary to study Andrew's documentation also. Hopefully the instructions below are complete.