WARNING: This is the _old_ Lustre wiki, and it is in the process of being retired. The information found here is all likely to be out of date. Please search the new wiki for more up to date information.

Upgrading to a New Version of Lustre: Difference between revisions

From Obsolete Lustre Wiki
Jump to navigationJump to search
 
(117 intermediate revisions by 2 users not shown)
Line 1: Line 1:
[[Use the procedures in this chapter to upgrade Lustre 1.6.x to Lustre 1.8.0.]]
<small>''(Updated: Dec 2009)''</small>


'''''Note:''''' In Lustre version 1.6 and later, the file system name (''--fsname parameter'') is limited to 8 characters so that it fits on the disk label.
__TOC__
This section provides information about supported upgrades, upgrade paths, and interoperability of nodes with different versions of Lustre™ installed. It also describes procedures for upgrading your Lustre file system to a new version of Lustre. For information about available releases, see [[Lustre_Release_Information|Lustre Release Information]].
 
== Supported Upgrades ==


== Supported Upgrades and Downgrades==
For Lustre 1.8.x, the following upgrades are supported:
For Lustre 1.8.x, the following upgrades are supported:
* Lustre 1.6.x to Lustre 1.8.0.
* Lustre 1.6.x (latest version) to Lustre 1.8.x (latest version).
* One minor version to the next (for example, 1.8.0 > 1.8.x).
* Lustre 1.8.x (any minor version) to Lustre 1.8.x (latest version).
 
=== Lustre Component Interoperability ===
 
Lustre interoperability enables 1.8.x servers ("new" servers) to work with 1.6.x clients ("old" clients), 1.6.x servers ("old" servers) to work with 1.8.x clients ("new" clients), and "mixed" environments with both 1.6.x and 1.8.x servers. For
example, half of each OSS failover pair could be upgraded to enable a quick reversion to 1.6 by powering down the 1.8 servers.
 
The table below describes the interoperability between clients, OSTs, and MDTs with different versions of Lustre installed.
 
{| border="1" cellspacing=0 cellpadding="10"
! Lustre Component !! Interoperability with Other Lustre Components
|-
| rowspan="1" valign="top" | Clients
| &#149; Old, live clients can communicate with old, new, or a mix of servers.<br>&#149; Old clients can start up using old, new, or a mix of servers.<br>&#149; New clients can start up using old, new, or a mix of servers.<br>'''''Note:''''' Old clients cannot mount a file system that was created by a new MDT.
|-
| rowspan="1" valign="top" | OSTs
| &#149; Old OSTs can communicate with new clients/MDTs.<br>&#149; New OSTs can only be started after the MGS has been started (typically after the MDT has been upgraded).
|-
| rowspan="1" valign="top" | MDTs
| &#149; An old MDT can communicate with new clients.<br>&#149; A new co-located MGS/MDT can be started at any point.<br>&#149; A new non-co-located MDT can be started after the MGS starts.
|}
 
== Upgrading Lustre 1.6.x to Lustre 1.8.x ==
Two Lustre upgrade paths are supported to meet the upgrade requirements of different Lustre environments.
 
* ''Entire file system upgrade'' - All servers and clients are shut down and upgraded at the same time. See [[#Performing a Complete File System Upgrade|Performing a Complete File System Upgrade]].
* ''Rolling upgrade'' - Individual servers (or their failover partners) and clients are upgraded one at a time and restarted, so that the file system never goes down.  See [[#Performing a Rolling Upgrade|Performing a Rolling Upgrade]].
 
'''''Note:''''' If you upgrade some Lustre components to 1.8.x but not others (such as running 1.8 clients in a file system with 1.6 OSTs), and run a mixed environment, you may see one or more warnings similar to this:
 
<pre>
LustreError: 3877:0:(socklnd_cb.c:2228:ksocknal_recv_hello())
Unknown protocol version (2.x expected) from 192.168.2.43
</pre>
 
This warning is given when the 1.6 and 1.8 components use different protocols. It
can be safely ignored because the Lustre components negotiate a common protocol.
In this example, the 1.8 clients fall back to use the 1.6 protocol with the 1.6 OSTs.
 
=== Performing a Complete File System Upgrade ===
This procedure describes a complete file system upgrade in which 1.8.x Lustre packages are installed on multiple 1.6.x servers and clients, requiring a file system shut down. If you want to upgrade one Lustre component at a time and avoid the shutdown, see [[#Performing a Rolling Upgrade|Performing a Rolling Upgrade]].
 
'''''Tip:''''' In a Lustre upgrade, the package install and file system unmount steps are reversible; you can do either step first. To minimize downtime, this procedure first performs the 1.8.x package installation, and then unmounts the file system.
 
1. ''Make a complete, restorable file system backup before upgrading Lustre.''
 
2. ''Install the 1.8.x packages on the Lustre servers and/or clients.'' Some or all servers can be upgraded. Some or all clients can be upgraded. For help determining where to install a specific package, see [[Lustre Packages]].
 
:a. ''Install the kernel, modules and ldiskfs packages.'' For example:
 
:<pre>
;$ rpm -ivh
;kernel-lustre-smp-<ver> \
;kernel-ib-<ver> \
;lustre-modules-<ver> \
;lustre-ldiskfs-<ver>
</pre>
 
:b. ''Upgrade the utilities/userspace packages.'' For example:
 
:<pre>
;$ rpm -Uvh lustre-<ver>
</pre>
 
:c. ''If a new e2fsprogs package is available, upgrade it.'' For example:
 
:<pre>
;$ rpm -Uvh e2fsprogs-<ver>
</pre>
 
:There may or may not be a new ''e2fsprogs'' package with a Lustre upgrade. The ''e2fsprogs'' release schedule is independent of Lustre releases.
 
:d. (Optional) ''If you want to add optional packages to your Lustre system, install them now.''
 
3. ''Shut down the file system.'' Shut down the components in this order: clients, then the MDT, then OSTs. Unmounting a block device causes Lustre to be shut down on that node.
 
:a. Unmount the clients. On each client node, run:
 
:<pre>
;umount <mount point>
</pre>
 
:b. Unmount the MDT. On the MDS node, run:
 
:<pre>
;umount <mount point>
</pre>
 
:c. Unmount the OSTs (be sure to unmount all OSTs). On each OSS node, run:
 
:<pre>
;umount <mount point>
</pre>
 
4. ''Unload the old Lustre modules'' by either:
* Rebooting the node
:- OR -
* Removing the Lustre modules manually. Run ''lustre_rmmod'' several times and use ''lsmod'' to check the currently loaded modules.
 
5. ''Start the upgraded file system.'' Start the components in this order: OSTs, then the MDT, then clients.
:a. ''Mount the OSTs'' (be sure to mount all OSTs). On each OSS node, run:


For Lustre 1.8.0, downgrades in the same ranges are supported.
:<pre>
* If you upgrade from Lustre 1.6.x > 1.8.0, you can downgrade to version 1.6.x.
;mount -t lustre <block device name> <mount point>
* If you upgrade from one minor version to the next (for example Lustre 1.8.0 > 1.8.x), you can downgrade to an earlier minor version (e.g., version 1.8.x). [[Is this correctly stated?]]
</pre>


'''''Caution:''''' A fresh installation of Lustre 1.8.x is not guaranteed to be downgradable to an earlier Lustre version.
:b. ''Mount the MDT.'' On the MDS node, run:


For supported upgrade paths for earlier releases, see [[Upgrade_to_1.6|Upgrading from 1.4.6 and later to 1.6]].
:<pre>
;mount -t lustre <block device name> <mount point>
</pre>


== Upgrade procedures ==
:c. ''Mount the file system on the clients.'' On each client node, run:


[[Which should be included here and which referred to in the manual (Chapter 13)?]]
:<pre>
;mount -t lustre <MGS node>&#58;/<fsname> <mount point>
</pre>


Upgrading from Lustre 1.6.x to Lustre 1.8.0
If you have a problem upgrading Lustre, contact us by [[Reporting Bugs|submitting a bug]] to our bug tracker [https://bugzilla.lustre.org/query.cgi Bugzilla].
* Prerequisites to Upgrading Lustre (see questions below)
* Starting Clients
* Upgrading a Single File System
* Upgrading Multiple File Systems with a Shared MGS
Upgrading Lustre 1.8.0 to the Next Minor Version


Downgrading from Lustre 1.8.0 to Lustre 1.6.x
=== Performing a Rolling Upgrade ===
* Downgrade Requirements
This procedure describes a rolling upgrade in which one Lustre component (server or client) is upgraded and restarted at a time while the file system is running. If you want to upgrade the complete Lustre file system or multiple components at a time, requiring a file system shutdown, see [[#Performing a Complete File System Upgrade|Performing a Complete File System Upgrade]].
* Downgrading a File System


----


:* [[Upgrading from 1.6.x to 1.8.x]]
'''''Note:''''' If the Lustre component to be upgraded is an OSS in a failover pair, follow these special upgrade steps to minimize downtime:


1. ''Fail over the server to its peer server'', so the file system remains available.


:* [[Use:Change_Log_1.8|Change Log 1.8]]
2. ''Install the Lustre 1.8.x packages on the idle server.''
:* [[Upgrade_to_1.6|Upgrading from 1.4.6 and later to 1.6]]
:* [[Change_Log_1.6|Change Log 1.6]]


3. ''Unload the old Lustre modules on the idle server'' by either:
:* Rebooting the node.
:- OR -
:* Removing the Lustre modules manually by running the ''lustre_rmmod'' command several times and checking the currently loaded modules with the ''lsmod'' command.


== Prerequisites to Upgrading Lustre==
4. ''Fail back services to the now upgraded server.''


From "Upgrading from Lustre 1.6.x to Lustre 1.8.0"
5. ''Repeat Steps 1 to 4 on the peer server.'' This limits the outage (per OSS) to a single server for as long as it takes to fail over.
[[Are these procedures that must be completed before the upgrade is started?]] [[Do they apply to any upgrade path?]]


Remember the following points before upgrading Lustre.
----
To perform a rolling upgrade:


The MDT must be upgraded before the OSTs are upgraded.
1. ''Make a complete, restorable file system backup before upgrading Lustre.''


1. Shut down lconf failover.
2. ''Install the 1.8.x packages on the Lustre component (server or client).'' For help determining where to install a specific package, see [[Lustre Packages]].


2. Install the new modules.
:a. ''Install the kernel, modules and ''ldiskfs'' packages.'' For example:


3. Run tunefs.lustre.
:<pre>
;$ rpm -ivh
;kernel-lustre-smp-<ver> \
;kernel-ib-<ver> \
;lustre-modules-<ver> \
;lustre-ldiskfs-<ver>
</pre>


4. Mount startup.
:b. ''Upgrade the utilities/userspace packages.'' For example:


A Lustre upgrade can be done across a failover pair.
:<pre>
;$ rpm -Uvh lustre-<ver>
</pre>


1. On the backup server, install the new modules.
:c. ''If a new ''e2fsprogs'' package is available, upgrade it.'' For example:
2. Shut down lconf failover.


3. On the new server, run tunefs.lustre.
:<pre>
;$ rpm -Uvh e2fsprogs-<ver>
</pre>


4. On the new server, mount startup.
There may or may not be a new ''e2fsprogs'' package with a Lustre upgrade. The ''e2fsprogs'' release schedule is independent of Lustre releases.


5. On the primary server, install the new modules.
:d. (''Optional) If you want to add optional packages to your Lustre system, install them now.''


==Lustre Failover and Rolling Upgrades in OM. (OM 1-17)==
3. ''Unload the old Lustre modules'' by either:
:* Rebooting the node
:- OR -
:* Removing the Lustre modules manually. Run ''lustre_rmmod'' several times and use ''lsmod'' to check the currently-loaded modules.


[[Include a page with this content or refer to OM?]]
4. ''If the upgraded component is a server, fail back services to the new server.''


Lustre offers a robust, application-transparent failover mechanism that delivers call completion. This failover mechanism, in conjunction with software that offers interoperability between versions, is used to support rolling upgrades of file system software on active clusters.
If you have a problem upgrading Lustre, contact us via the [https://bugzilla.lustre.org/query.cgi Bugzilla] bug tracker.


The Lustre recovery feature allows servers to be upgraded without taking down the system. The server is simply taken offline, upgraded and restarted (or failed over to a standby server with the new software). All active jobs continue to run without failures, they merely experience a delay.
== Upgrading Lustre 1.8.x to the Next Minor Version ==


Lustre MDSs are configured as an active/passive pair, while OSSs are typically deployed in an active/active configuration that provides redundancy without extra overhead, as shown in FIGURE 1-8. Often the standby MDS is the active MDS for another Lustre file system, so no nodes are idle in the cluster.
To upgrade Lustre 1.8.x to the next minor version, for example, Lustre 1.8.0.1 > 1.8.x,
follow these procedures:
* To upgrade the complete file system or multiple file system components at the same time, requiring a file system shutdown, see [[#Performing_a_Complete_File_System_Upgrade|Performing a Complete File System Upgrade]]
* To upgrade one Lustre component (server or client) at a time, while the file system is running, see [[#Performing_a_Rolling_Upgrade|Performing a Rolling Upgrade]]


FIGURE 1-8 Lustre failover configurations for OSSs and MDSs
== Downgrading from Lustre 1.8.x ==
For Lustre 1.8.x, the following downgrades are supported:
* If you upgraded from Lustre 1.6.x > 1.8.x, you can downgrade to version 1.6.x.
* If you upgraded from one minor version to the next (for example, Lustre 1.8.0 > 1.8.x), you can downgrade to the earlier minor version.


Although a file system checking tool (lfsck) is provided for disaster recovery, journaling and sophisticated protocols re-synchronize the cluster within seconds, without the need for a lengthy fsck. Lustre version interoperability between successive minor versions is guaranteed. As a result, the Lustre failover capability is used regularly to upgrade the software without cluster downtime.
For a procedure to downgrade from Lustre 1.8.x to Lustre 1.6.x, see [http://wiki.lustre.org/manual/LustreManual18_HTML/UpgradingLustre.html#50651255_38760 Section 13.5: ''Downgrading from Lustre 1.8.x to Lustre 1.6.x''] in the [http://wiki.lustre.org/manual/LustreManual18_HTML/index.html ''Lustre Operations Manual''].


Note – Lustre does not provide redundancy for data; it depends exclusively on redundancy of backing storage devices. The backing OST storage should be RAID 5 or, preferably, RAID 6 storage. MDT storage should be RAID 1 or RAID 0+1.
'''''Caution:''''' A new installation of Lustre 1.8.x is not guaranteed to be downgradable to an earlier Lustre version.

Latest revision as of 10:23, 20 January 2011

(Updated: Dec 2009)

This section provides information about supported upgrades, upgrade paths, and interoperability of nodes with different versions of Lustre™ installed. It also describes procedures for upgrading your Lustre file system to a new version of Lustre. For information about available releases, see Lustre Release Information.

Supported Upgrades

For Lustre 1.8.x, the following upgrades are supported:

  • Lustre 1.6.x (latest version) to Lustre 1.8.x (latest version).
  • Lustre 1.8.x (any minor version) to Lustre 1.8.x (latest version).

Lustre Component Interoperability

Lustre interoperability enables 1.8.x servers ("new" servers) to work with 1.6.x clients ("old" clients), 1.6.x servers ("old" servers) to work with 1.8.x clients ("new" clients), and "mixed" environments with both 1.6.x and 1.8.x servers. For example, half of each OSS failover pair could be upgraded to enable a quick reversion to 1.6 by powering down the 1.8 servers.

The table below describes the interoperability between clients, OSTs, and MDTs with different versions of Lustre installed.

Lustre Component Interoperability with Other Lustre Components
Clients &#149; Old, live clients can communicate with old, new, or a mix of servers.
&#149; Old clients can start up using old, new, or a mix of servers.
&#149; New clients can start up using old, new, or a mix of servers.
Note: Old clients cannot mount a file system that was created by a new MDT.
OSTs &#149; Old OSTs can communicate with new clients/MDTs.
&#149; New OSTs can only be started after the MGS has been started (typically after the MDT has been upgraded).
MDTs &#149; An old MDT can communicate with new clients.
&#149; A new co-located MGS/MDT can be started at any point.
&#149; A new non-co-located MDT can be started after the MGS starts.

Upgrading Lustre 1.6.x to Lustre 1.8.x

Two Lustre upgrade paths are supported to meet the upgrade requirements of different Lustre environments.

Note: If you upgrade some Lustre components to 1.8.x but not others (such as running 1.8 clients in a file system with 1.6 OSTs), and run a mixed environment, you may see one or more warnings similar to this: 

LustreError: 3877:0:(socklnd_cb.c:2228:ksocknal_recv_hello())
Unknown protocol version (2.x expected) from 192.168.2.43

This warning is given when the 1.6 and 1.8 components use different protocols. It can be safely ignored because the Lustre components negotiate a common protocol. In this example, the 1.8 clients fall back to use the 1.6 protocol with the 1.6 OSTs.

Performing a Complete File System Upgrade

This procedure describes a complete file system upgrade in which 1.8.x Lustre packages are installed on multiple 1.6.x servers and clients, requiring a file system shut down. If you want to upgrade one Lustre component at a time and avoid the shutdown, see Performing a Rolling Upgrade.

Tip: In a Lustre upgrade, the package install and file system unmount steps are reversible; you can do either step first. To minimize downtime, this procedure first performs the 1.8.x package installation, and then unmounts the file system.

1. Make a complete, restorable file system backup before upgrading Lustre.

2. Install the 1.8.x packages on the Lustre servers and/or clients. Some or all servers can be upgraded. Some or all clients can be upgraded. For help determining where to install a specific package, see Lustre Packages.

a. Install the kernel, modules and ldiskfs packages. For example:


$ rpm -ivh


kernel-lustre-smp-<ver> \


kernel-ib-<ver> \


lustre-modules-<ver> \


lustre-ldiskfs-<ver>



b. Upgrade the utilities/userspace packages. For example:




$ rpm -Uvh lustre-<ver>



c. If a new e2fsprogs package is available, upgrade it. For example:




$ rpm -Uvh e2fsprogs-<ver>



There may or may not be a new e2fsprogs package with a Lustre upgrade. The e2fsprogs release schedule is independent of Lustre releases.


d. (Optional) If you want to add optional packages to your Lustre system, install them now.


3. Shut down the file system. Shut down the components in this order: clients, then the MDT, then OSTs. Unmounting a block device causes Lustre to be shut down on that node.



a. Unmount the clients. On each client node, run:




umount <mount point>



b. Unmount the MDT. On the MDS node, run:




umount <mount point>



c. Unmount the OSTs (be sure to unmount all OSTs). On each OSS node, run:




umount <mount point>



4. Unload the old Lustre modules by either:



  • Rebooting the node


- OR -


  • Removing the Lustre modules manually. Run lustre_rmmod several times and use lsmod to check the currently loaded modules.


5. Start the upgraded file system. Start the components in this order: OSTs, then the MDT, then clients.



a. Mount the OSTs (be sure to mount all OSTs). On each OSS node, run:




mount -t lustre <block device name> <mount point>



b. Mount the MDT. On the MDS node, run:




mount -t lustre <block device name> <mount point>



c. Mount the file system on the clients. On each client node, run:




mount -t lustre <MGS node>:/<fsname> <mount point>



If you have a problem upgrading Lustre, contact us by submitting a bug to our bug tracker Bugzilla.



Performing a Rolling Upgrade


This procedure describes a rolling upgrade in which one Lustre component (server or client) is upgraded and restarted at a time while the file system is running. If you want to upgrade the complete Lustre file system or multiple components at a time, requiring a file system shutdown, see Performing a Complete File System Upgrade.





Note: If the Lustre component to be upgraded is an OSS in a failover pair, follow these special upgrade steps to minimize downtime:

1. Fail over the server to its peer server, so the file system remains available.

2. Install the Lustre 1.8.x packages on the idle server.

3. Unload the old Lustre modules on the idle server by either:



  • Rebooting the node.


- OR -

  • Removing the Lustre modules manually by running the lustre_rmmod command several times and checking the currently loaded modules with the lsmod command.


4. Fail back services to the now upgraded server.

5. Repeat Steps 1 to 4 on the peer server. This limits the outage (per OSS) to a single server for as long as it takes to fail over.





To perform a rolling upgrade:

1. Make a complete, restorable file system backup before upgrading Lustre.

2. Install the 1.8.x packages on the Lustre component (server or client). For help determining where to install a specific package, see Lustre Packages.



a. Install the kernel, modules and ldiskfs packages. For example:




$ rpm -ivh


kernel-lustre-smp-<ver> \


kernel-ib-<ver> \


lustre-modules-<ver> \


lustre-ldiskfs-<ver>



b. Upgrade the utilities/userspace packages. For example:




$ rpm -Uvh lustre-<ver>



c. If a new e2fsprogs package is available, upgrade it. For example:




$ rpm -Uvh e2fsprogs-<ver>



There may or may not be a new e2fsprogs package with a Lustre upgrade. The e2fsprogs release schedule is independent of Lustre releases.



d. (Optional) If you want to add optional packages to your Lustre system, install them now.


3. Unload the old Lustre modules by either:



  • Rebooting the node


- OR -

  • Removing the Lustre modules manually. Run lustre_rmmod several times and use lsmod to check the currently-loaded modules.


4. If the upgraded component is a server, fail back services to the new server.

If you have a problem upgrading Lustre, contact us via the Bugzilla bug tracker.



Upgrading Lustre 1.8.x to the Next Minor Version


To upgrade Lustre 1.8.x to the next minor version, for example, Lustre 1.8.0.1 > 1.8.x,
follow these procedures:




Downgrading from Lustre 1.8.x


For Lustre 1.8.x, the following downgrades are supported:



  • If you upgraded from Lustre 1.6.x > 1.8.x, you can downgrade to version 1.6.x.
    • 

  • If you upgraded from one minor version to the next (for example, Lustre 1.8.0 > 1.8.x), you can downgrade to the earlier minor version.


For a procedure to downgrade from Lustre 1.8.x to Lustre 1.6.x, see Section 13.5: Downgrading from Lustre 1.8.x to Lustre 1.6.x in the Lustre Operations Manual.

Caution: A new installation of Lustre 1.8.x is not guaranteed to be downgradable to an earlier Lustre version.



Retrieved from "http://wiki.old.lustre.org/index.php?title=Upgrading_to_a_New_Version_of_Lustre&oldid=12219"