WARNING: This is the _old_ Lustre wiki, and it is in the process of being retired. The information found here is all likely to be out of date. Please search the new wiki for more up to date information.

Lustre 2.0: Difference between revisions

From Obsolete Lustre Wiki
Jump to navigationJump to search
No edit summary
 
(41 intermediate revisions by 3 users not shown)
Line 1: Line 1:
Lustre 2.0 will introduce several significant new features and improved system functionality. This page provides descriptions of these features and lists the benefits offered by upgrading to Lustre 2.0. For the latest information on when Lustre 2.0 is expected to be released, see the [http://wiki.lustre.org/index.php?title=Lustre_Roadmap Lustre Roadmap].
<small>''(Updated: Aug 2010)''</small>
__TOC__
This page offers resources for the Lustre™ 2.0/2.x release family.  


==Changelogs==
==Lustre 2.0 Matrix==


Changelogs record events that change the filesystem namespace or file metadata. Events such as file creation, deletion, renaming, attribute changes, etc. are recorded with the target and parent file identifiers (FIDs), the name of the target, and a timestamp. These records can be used for a variety of purposes:
Lustre 2.0 will support the following Linux distributions and architectures.


* Record recent changes to feed into an archiving system.
{| border=1 cellpadding=0
* Use changelog entries to exactly replicate changes in a filesystem mirror.
* Set up "watch scripts" that take action on certain events or directories. Changelog record are persistent (on disk) until explicitly cleared by the user.  The are guaranteed to accurately reflect on-disk changes in the event of a server failure.
* Maintain a rough audit trail (file/directory changes with timestamps, but no user information).
 
These are sample changelog entries:
 
2 02MKDIR 4298396676 0x0 t=[0x200000405:0x15f9:0x0] p=[0x13:0x15e5a7a3:0x0] pics
3 01CREAT 4298402264 0x0 t=[0x200000405:0x15fa:0x0] p=[0x200000405:0x15f9:0x0] chloe.jpg
4 06UNLNK 4298404466 0x0 t=[0x200000405:0x15fa:0x0] p=[0x200000405:0x15f9:0x0] chloe.jpg
5 07RMDIR 4298405394 0x0 t=[0x200000405:0x15f9:0x0] p=[0x13:0x15e5a7a3:0x0] pics
 
The record types are:
 
{| class="wikitable"
| <small>'''MARK''' || internal record keeping</small>
|-
| <small>'''CREAT''' || regular file creation</small>
|-
| <small>'''MKDIR''' || directory creation</small>
|-
| <small>'''HLINK''' || hardlink</small>
|-
|-
| <small>'''SLINK''' || softlink</small>
!Component
!Linux Distribution
!Architecture
|-
|-
| <small>'''MKNOD''' || other file creation</small>
|rowspan="3"|<small>'''Server'''</small>
|<small>OEL 5.4</small>
|<small>x86_64</small>
|-
|-
| <small>'''UNLNK''' || regular file removal</small>
|<small>RHEL 5.4</small>
|<small>x86_64</small>
|-
|-
| <small>'''RMDIR''' || directory removal</small>
|style="background:#E8E8E8"|<small>Dropped: SLES, vanilla (2.6.22)</small>
|style="background:#E8E8E8"|<small>Dropped: i686</small>
|-
|-
| <small>'''RNMFM''' || rename, original</small>
|rowspan="6"|<small>'''Patchless Client'''</small>
|<small>OEL 5.4</small>
|<small>x86_64</small>
|-
|-
| <small>'''RNMTO''' || rename, final</small>
|<small>RHEL 5</small>
|<small>x86_64</small>
|-
|-
| <small>'''OPEN'''  || file opened for write</small>
|<small>SLES 10, 11</small>
|<small>x86_64</small>
|-
|-
| <small>'''CLOSE''' || file closed for write</small>
|<small>Fedora 11 (2.6.29) [New]</small>
|<small>i686</small>
|-
|-
| <small>'''IOCTL''' || ioctl on file or dir</small>
|style="background:#E8E8E8"|<small>Dropped: vanilla (2.6.22)</small>
|-
|
| <small>'''TRUNC''' || regular file truncated</small>
|-
| <small>'''SATTR''' || attribute change</small>
|-
| <small>'''XATTR''' || extended attribute change</small>
|-
| <small>'''UNKNW''' || unknown op</small>
|-
|-
|}
|}


FID-to-full-pathname and pathname-to-FID functions are also included to map target and parent FIDs into the filesystem namespace.
==Lustre Interoperability - 1.8 to 2.0==
 
<big>Why should I upgrade to Lustre 2.0 to get it?</big>
 
Changelogs offer these benefits:
 
* File/directory change notification
* Event notification
* Filesystem replication
* File backup policy decisions
* Audit trail
 
<big>Additional Resources</big>
 
For more information about changelogs see:
 
*
*
*
 
==Replication==
 
The replication feature makes a (qualified) replica of a Lustre filesystem on another filesystem target. The target may be another Lustre filesystem or any other filesystem. This features differs from existing backup/replication/synchronization systems primarily in that it is designed to avoid walking the namespace tree, which for very large filesystems becomes unreasonably time-consuming. Replication is based on server changelogs, and uses the information in those logs to determine which directory and file operations to execute on the replicated system. The replicated filesystem is an exact copy of the namespace of the original system as of a given point in time. However, the replicated filesystem it is '''not''' a snapshot of the original filesystem in that its contents may differ from the contents of the original filesystem. File contents of the replica will be the contents of the file at the time the data transfer of that file took place.  
 
<big>Why should I upgrade to Lustre 2.0 to get it?</big>
 
Replication offers this benefit:
 
* Namespace-coherent duplication of large filesystems without walking the filesystem.
 
<big>Additional Resources</big>
 
For more information on replication, see:
 
*
*
*
 
==HSM==
 
The HSM feature provides several mechanisms to interface with an external HSM system. External components include the policy engine, and file storage, retrieval, and removal methods. The external components are expected to run in userspace. Internal components include Lustre metadata extensions, and a distributed coordinator/agent architecture to call the file storage methods. Policy engine input and feedback is expected to occur primarily though the changelog. In its initial implementation, the HSM feature uses HPSS for the external components.
 
<big>Why should I upgrade to Lustre 2.0 to get it?</big>
 
HSM offers these benefits:


* Cost-effective filesystem expansion
Lustre 2.0 will be built on an architectural code base which is different than the one used with prior releases. New 2.0 users will not be affected by these changes when installing Lustre. Initially, existing Lustre users will need to follow a slightly different procedure to upgrade to 2.0. See [[Lustre_Interoperability_-_Upgrading_From_1.8_to_2.0|Upgrading from 1.8 to 2.0]].
* Potential for backup policies in the policy engine


<big>Additional Resources</big>
==Lustre 2.0/2.x Features==


For more information on HSM, see:
Lustre 2.0 will introduce several significant new features and improved system functionality. This page previews these features and benefits offered by upgrading to Lustre 2.0, and also describes features targeted at later 2.x releases. See [[Lustre_2.0_Features|Lustre 2.0 Features]]


*
==Lustre 2.0 Release Milestones==
*
The Lustre 2.0 team will be publishing interim Alpha and Beta RPMs throughout the testing and stabilization process. The team plans to publish a milestone release every 4-6 weeks until Lustre 2.0 GA is achieved. The first Alpha milestone was announced at LUG 2009 on April 16, 2009. Please see the [[Lustre_2.0_Release_Milestone_Status|Lustre 2.0 Release Milestone Status]] page for related testing documents for this and future Alpha and Beta milestones.
*

Latest revision as of 09:38, 24 August 2010

(Updated: Aug 2010)

This page offers resources for the Lustre™ 2.0/2.x release family.

Lustre 2.0 Matrix

Lustre 2.0 will support the following Linux distributions and architectures.

Component Linux Distribution Architecture
Server OEL 5.4 x86_64
RHEL 5.4 x86_64
Dropped: SLES, vanilla (2.6.22) Dropped: i686
Patchless Client OEL 5.4 x86_64
RHEL 5 x86_64
SLES 10, 11 x86_64
Fedora 11 (2.6.29) [New] i686
Dropped: vanilla (2.6.22)

Lustre Interoperability - 1.8 to 2.0

Lustre 2.0 will be built on an architectural code base which is different than the one used with prior releases. New 2.0 users will not be affected by these changes when installing Lustre. Initially, existing Lustre users will need to follow a slightly different procedure to upgrade to 2.0. See Upgrading from 1.8 to 2.0.

Lustre 2.0/2.x Features

Lustre 2.0 will introduce several significant new features and improved system functionality. This page previews these features and benefits offered by upgrading to Lustre 2.0, and also describes features targeted at later 2.x releases. See Lustre 2.0 Features

Lustre 2.0 Release Milestones

The Lustre 2.0 team will be publishing interim Alpha and Beta RPMs throughout the testing and stabilization process. The team plans to publish a milestone release every 4-6 weeks until Lustre 2.0 GA is achieved. The first Alpha milestone was announced at LUG 2009 on April 16, 2009. Please see the Lustre 2.0 Release Milestone Status page for related testing documents for this and future Alpha and Beta milestones.

Retrieved from "http://wiki.old.lustre.org/index.php?title=Lustre_2.0&oldid=11832"