Obsolete Lustre Wiki - User contributions [en]

File:821-2076-10.pdf

2011-01-20T19:31:01Z

Sbarthel: uploaded a new version of "File:821-2076-10.pdf": Lustre 2.0 Operations Manual (Rev 2), released January 2011

Lustre 2.0 manual

File:821-2076-10.pdf

2011-01-20T19:17:45Z

Sbarthel: uploaded a new version of "File:821-2076-10.pdf": Lustre 2.0 Operations Manual (Rev 2), released January 2011

Lustre 2.0 manual

Accessing Lustre Code

2011-01-20T19:10:19Z

Sbarthel: /* Naming conventions */

''(Updated: Jan 2010)''

'''''NOTICE:''''' The transition from CVS to Git took place on Monday, December 14. For more information about the transition, see the [[Git Transition Notice]]. For details about how to migrate to Git, see [[Migrating to Git]].

----

We welcome and encourage contributions to the development and testing of a more robust, feature-rich Lustre™. You can obtain the latest bleeding-edge Lustre source code by anonymous Git access.

git clone git://git.lustre.org/prime/lustre.git

'''''Note:''''' For more information about using Git, including tutorials and guides to help you get started, see the [http://git-scm.com/documentation Git documentation] page. ''For descriptions of the commands you are most likely to need, see the Commands section at the bottom of this page.''

See [[Contribute]] for more information about developing, testing, and submitting a patch to the Lustre code.

'''''Note:''''' If you have questions or experience problems, send email to the [mailto:lustre-wiki-feedback@sun.com Admins].

For more information about Git, see the [http://git-scm.com/ Git home]

=== Naming conventions ===

Stable development branches are named b''{major}''_''{minor}'' (for example, b1_6 and b1_8). Even-numbered minor releases are considered stable releases. Odd-numbered minor releases correspond to alpha and beta releases and will sometimes be given v''{major}''_''{minor}''_''{patch}'' tags to provide a point of reference for internal and external testing.

A release branch is created an official release to isolate it from further development and named b_release_''{major}''_''{minor}''_''{patch}'' (for example, b_release_1_8_0). A final release gets a tag in the form v''{major}''_''{minor}''_''{patch}'' (for example, v1_8_0 or v1_6_7_1).

Work for the next upcoming version is done on the ''master'' branch.

The Lustre [[Subsystem Map]] describes each of the subsystems in the Lustre code.

Architecture Descriptions

2011-01-20T19:09:28Z

Sbarthel:

''(Updated: Jan 2010)''

The architecture descriptions listed below provide information about Lustre architecture and design and are intended to help users better understand the conceptual framework of the Lustre file system.

'''''Note:''''' ''These documents reflect the state of design of a Lustre feature at a particular point in time. They many contain information that is incomplete or obsolete and may not reflect the current architecture, features and functionality of Lustre.''

[[Architecture - Adaptive Timeouts - Use Cases|''Adaptive Timeouts - Use Cases'' ]] (Network RPC timeouts based on server and network loading)

[[Architecture - Backup|''Backup'' ]] (File system backup)

[[Architecture - Caching OSS|''Caching OSS'' ]] (Caching on object storage servers)

[[Architecture - Changelogs|''Changelogs'' ]] (Per-server logs of data or metadata changes)

[[Architecture - Changelogs 1.6|''Changelogs 1.6'' ]] (Used to facilitate efficient replication of large Lustre 1.6 filesystems)

[[Architecture - Client Cleanup|''Client Cleanup'' ]] (Use cases, business drivers, models to consider, implementation contraints)

[[Architecture - Clustered Metadata|''Clustered Metadata'' ]] (Clustered metadata server capability)

[[Architecture - Commit on Share|''Commit on Share'' ]] (Better recover-ability in an environment where clients miss reconnect window)

[[Architecture - CROW|''CROW'' ]] (Create On Write optimizes create performance by deferring OSS object creation)

[[Architecture - CTDB with Lustre|''CTDB with Lustre'' ]] (Cluster implementation of the TDB database with Lustre provides a solution for windows pCIFS)

[[Architecture - Cuts|''Cuts'' ]] (Technique for recovering file system metadata stored on file server clusters)

[[Architecture - DMU OSD|''DMU OSD'' ]] (An implementation of the Object Storage Device API for a Data Management Unit)

[[Architecture - End-to-end Checksumming|''End-to-end Checksumming'' ]] (Lustre network checksumming)

[[Architecture - Epochs|''Epochs'' ]] (Used to merge distributed data and meta-data updates in a redundant cluster configuration)

[[Architecture - External File Locking|''External File Locking'' ]] (File range lock and whole-file lock capabilities)

[[Architecture - FIDs on OST|''FIDs on OST'' ]] (File identifiers used to identify objects on an object storage target)

[[Architecture - Fileset|''Fileset'' ]] (An efficient representation of a group of file identifiers (FIDs))

[[Architecture - Flash Cache|''Flash Cache'' ]] (Very fast read-only flash storage)

[[Architecture - Free Space Management|''Free Space Management'' ]] (Managing free space for stripe allocation)

[[Architecture - GNS|''GNS'' ]] (Global namespace for a distributed file system)

[[Architecture - HSM|''HSM'' ]] (Hierarchical storage management)

[[Architecture - HSM and Cache|''HSM and Cache'' ]] (Reuse of components by Lustre features that involve migration of file system objects)

[[Architecture - HSM Migration|''HSM Migration'' ]] (Use cases and high-level architecture for migrating files between Lustre and a HSM system)

[[Architecture - Interoperability fids zfs|''Interoperability fids zfs'' ]] (Client, server, network, storage interoperability during migration to clusters based on file identifiers and the ZFS file system)

[[Architecture - Interoperability 1.6 1.8 2.0|''Interoperability 1.6 / 1.8 / 2.0'' ]] (interoperability definitions and QAS summary)

[[Architecture - IO system|''IO system'' ]] (Client IO and server I/O request handling)

[[Architecture - Libcfs|''Libcfs'' ]] (Portable runtime environment for process management and debugging support)

[[Architecture - Llog over OSD|''Llog over OSD'' ]] (Re-implement llog API to use OSD device as backend device)

[[Architecture - LRE Images|''LRE Images'' ]] (Provide development and training Lustre software environments based on supported environments for Lustre)

[[Architecture - Lustre Logging API|''Lustre Logging API'' ]] (Requirements and detailed description)

[[Architecture - MDS striping format|''MDS striping format'' ]] (Striping extended attributes, striping formats, striping APIs)

[[Architecture - MDS-on-DMU|''MDS-on-DMU'' ]] (Metadata server on the ZFS Data Management Unit - use cases, features and functional behavior)

[[Architecture - Metadata API|''Metadata API'' ]] (A set of methods used by the Lustre file system driver to access and manipulate metadata)

[[Architecture - Migration (1)|''Migration (1)'' ]] (Overview of development path for migration capabilities)
''
[[Architecture - Migration (2)|Migration (2)'' ]] (Use cases, quality attribute scenarios, and implementation details)

[[Architecture - MPI IO and NetCDF|''MPI IO and NetCDF'' ]] (Message Passing Interface I/O and network Common Data Form libraries - Lustre ADIO driver improvements and internal optimization)

[[Architecture - MPI LND|''MPI LND'' ]] (Link to paper ''Lustre Networking over MPI'')

[[Architecture - Multiple Interfaces For LNET|''Multiple Interfaces For LNET'' ]] (Use cases and configuration management for Lustre networking)

[[Architecture - Network Request Scheduler|''Network Request Scheduler'' ]] (Requirements for network request scheduler to manage incoming RPC requests on a server)

[[Architecture - New Metadata API|''New Metadata API'' ]] (Proposal and use cases)

[[Architecture - Open by fid|''Open by fid'' ]] (Returns a file descriptor based on a file ID - implementation choices, design description, use cases)

[[Architecture - OSS-on-DMU|''OSS-on-DMU'' ]](Object storage server on the ZFS Data Management Unit)

[[Architecture - PAG|''PAG'' ]] (Process Authentication Groups - Use of Linux keyring, setuid in Lustre, Kerberos credential, use cases)

[[Architecture - Pools of targets|''Pools of targets'' ]] (Use cases, command line definitions of pools of OSTs, implementation constraints)

[[Architecture - Profiling Tools for IO|''Profiling Tools for IO'' ]] (Profiling system based on Ganglia)

[[Architecture - Proxy Cache|''Proxy Cache'' ]] (Caching and aggregation used to reduce load on backend server and provide better throughput and latency to clients)

[[Architecture - Punch and Extent Migration|''Punch and Extent Migration'' ]] (Prototypes for ''punch'' and ''migrate'' functionality)

[[Architecture - Punch and Extent Migration Requirements|''Punch and Extent Migration Requirements'']] (Punch functionality use cases)

[[Architecture - Recovery Failures|''Recovery Failures'' ]] (Recovery terminology, architectures, and use cases)

[[Architecture - Request Redirection|''Request Redirection'' ]] (Allows target OST to redirect client requests to other servers)

[[Architecture - Scalable Pinger|''Scalable Pinger'' ]] (Provides peer health information to Lustre clients and servers)

[[Architecture - Security|''Security'' ]] (Detailed description of the security architecture for Lustre)

[[Architecture - Server Network Striping|''Server Network Striping'' ]] (Description of Lustre-level striping of file data over multiple object servers with redundancy)

[[Architecture - Simple Space Balance Migration|''Simple Space Balance Migration'' ]] (A subset of full data migration limited to migrating files that are not currently in use)

[[Architecture - Simplified Interoperation|''Simplified Interoperation'' ]] (Controlled server shutdown simplifies inter-operation on server upgrades)

[[Architecture - Space Manager|''Space Manager'' ]] (Manages file system free space)

[[Architecture - Sub Tree Locks|''Sub Tree Locks'' ]] (A lock on a directory that protects a namespace (or part of a namespace) rooted at that directory)

[[Architecture - User Level Access|''User Level Access'' ]] (LNET userspace API driver that exports the LNET API to userspace)

[[Architecture - User Level OSS|''User Level OSS'' ]] (Functionality related to Lustre client or OSS failure)

[[Architecture - Userspace Servers|''Userspace Servers'' ]] (Requirements for capability to run a Lustre server in user space)

[[Architecture - Version Based Recovery|''Version Based Recovery'' ]] (A recovery mechanism allowing clients to recover outside of a strict order or later in time - requirements, use cases, quality attribute scenarios, implementation details)

[[Architecture - Wide Striping|''Wide Striping'' ]] (Mechanism to encode striping information compactly to efficiently support striping of files across many devices)

[[Architecture - Wire Level Protocol|''Wire Level Protocol'' ]] (Wire formats used by Lustre - messages, wire and record packet structures, recovery protocol)

[[Architecture - Write Back Cache|''Write Back Cache'' ]] (Allows client meta-data operations to be delayed and batched)

[[Architecture - ZFS for Lustre|''ZFS for Lustre'' ]] (Architecture and requirements related to Lustre servers using the ZFS Data Management Unit)

[[Architecture - ZFS large dnodes|''ZFS large dnodes'' ]] (Increased dnode size to allow more data in the inode)

[[Architecture - ZFS TinyZAP|''ZFS TinyZAP'' ]] (A compact ZFS Attribute Processor format that allows arbitrary values to be stored)

Architecture Descriptions

2011-01-20T19:08:44Z

Sbarthel:

''(Updated: Jan 2010)''

The architecture descriptions listed below provide information about Lustre architecture and design and are intended to help users better understand the conceptual framework of the Lustre file system.

'''''Note:''''' ''These documents reflect the state of design of a Lustre feature at a particular point in time. They many contain information that is incomplete or obsolete and may not reflect the current architecture, features and functionality of Lustre.''

[[Architecture - Adaptive Timeouts - Use Cases|''Adaptive Timeouts - Use Cases'' ]] (Network RPC timeouts based on server and network loading)

[[Architecture - Backup|''Backup'' ]] (File system backup)

[[Architecture - Caching OSS|''Caching OSS'' ]] (Caching on object storage servers)

[[Architecture - Changelogs|''Changelogs'' ]] (Per-server logs of data or metadata changes)

[[Architecture - Changelogs 1.6|''Changelogs 1.6'' ]] (Used to facilitate efficient replication of large Lustre 1.6 filesystems)

[[Architecture - Client Cleanup|''Client Cleanup'' ]] (Use cases, business drivers, models to consider, implementation contraints)

[[Architecture - Clustered Metadata|''Clustered Metadata'' ]] (Clustered metadata server capability)

[[Architecture - Commit on Share|''Commit on Share'' ]] (Better recover-ability in an environment where clients miss reconnect window)

[[Architecture - CROW|''CROW'' ]] (Create On Write optimizes create performance by deferring OSS object creation)

[[Architecture - CTDB with Lustre|''CTDB with Lustre'' ]] (Cluster implementation of the TDB database with Lustre provides a solution for windows pCIFS)

[[Architecture - Cuts|''Cuts'' ]] (Technique for recovering file system metadata stored on file server clusters)

[[Architecture - DMU OSD|''DMU OSD'' ]] (An implementation of the Object Storage Device API for a Data Management Unit)

[[Architecture - End-to-end Checksumming|''End-to-end Checksumming'' ]] (Lustre network checksumming)

[[Architecture - Epochs|''Epochs'' ]] (Used to merge distributed data and meta-data updates in a redundant cluster configuration)

[[Architecture - External File Locking|''External File Locking'' ]] (File range lock and whole-file lock capabilities)

[[Architecture - FIDs on OST|''FIDs on OST'' ]] (File identifiers used to identify objects on an object storage target)

[[Architecture - Fileset|''Fileset'' ]] (An efficient representation of a group of file identifiers (FIDs))

[[Architecture - Flash Cache|''Flash Cache'' ]] (Very fast read-only flash storage)

[[Architecture - Free Space Management|''Free Space Management'' ]] (Managing free space for stripe allocation)

[[Architecture - GNS|''GNS'' ]] (Global namespace for a distributed file system)

[[Architecture - HSM|''HSM'' ]] (Hierarchical storage management)

[[Architecture - HSM and Cache|''HSM and Cache'' ]] (Reuse of components by Lustre features that involve migration of file system objects)

[[Architecture - HSM Migration|''HSM Migration'' ]] (Use cases and high-level architecture for migrating files between Lustre and a HSM system)

[[Architecture - Interoperability fids zfs|''Interoperability fids zfs'' ]] (Client, server, network, storage interoperability during migration to clusters based on file identifiers and the ZFS file system)

[[Architecture - Interoperability 1.6 / 1.8 / 2.0|''Interoperability 1.6 1.8 2.0'' ]] (interoperability definitions and QAS summary)

[[Architecture - IO system|''IO system'' ]] (Client IO and server I/O request handling)

[[Architecture - Libcfs|''Libcfs'' ]] (Portable runtime environment for process management and debugging support)

[[Architecture - Llog over OSD|''Llog over OSD'' ]] (Re-implement llog API to use OSD device as backend device)

[[Architecture - LRE Images|''LRE Images'' ]] (Provide development and training Lustre software environments based on supported environments for Lustre)

[[Architecture - Lustre Logging API|''Lustre Logging API'' ]] (Requirements and detailed description)

[[Architecture - MDS striping format|''MDS striping format'' ]] (Striping extended attributes, striping formats, striping APIs)

[[Architecture - MDS-on-DMU|''MDS-on-DMU'' ]] (Metadata server on the ZFS Data Management Unit - use cases, features and functional behavior)

[[Architecture - Metadata API|''Metadata API'' ]] (A set of methods used by the Lustre file system driver to access and manipulate metadata)

[[Architecture - Migration (1)|''Migration (1)'' ]] (Overview of development path for migration capabilities)
''
[[Architecture - Migration (2)|Migration (2)'' ]] (Use cases, quality attribute scenarios, and implementation details)

[[Architecture - MPI IO and NetCDF|''MPI IO and NetCDF'' ]] (Message Passing Interface I/O and network Common Data Form libraries - Lustre ADIO driver improvements and internal optimization)

[[Architecture - MPI LND|''MPI LND'' ]] (Link to paper ''Lustre Networking over MPI'')

[[Architecture - Multiple Interfaces For LNET|''Multiple Interfaces For LNET'' ]] (Use cases and configuration management for Lustre networking)

[[Architecture - Network Request Scheduler|''Network Request Scheduler'' ]] (Requirements for network request scheduler to manage incoming RPC requests on a server)

[[Architecture - New Metadata API|''New Metadata API'' ]] (Proposal and use cases)

[[Architecture - Open by fid|''Open by fid'' ]] (Returns a file descriptor based on a file ID - implementation choices, design description, use cases)

[[Architecture - OSS-on-DMU|''OSS-on-DMU'' ]](Object storage server on the ZFS Data Management Unit)

[[Architecture - PAG|''PAG'' ]] (Process Authentication Groups - Use of Linux keyring, setuid in Lustre, Kerberos credential, use cases)

[[Architecture - Pools of targets|''Pools of targets'' ]] (Use cases, command line definitions of pools of OSTs, implementation constraints)

[[Architecture - Profiling Tools for IO|''Profiling Tools for IO'' ]] (Profiling system based on Ganglia)

[[Architecture - Proxy Cache|''Proxy Cache'' ]] (Caching and aggregation used to reduce load on backend server and provide better throughput and latency to clients)

[[Architecture - Punch and Extent Migration|''Punch and Extent Migration'' ]] (Prototypes for ''punch'' and ''migrate'' functionality)

[[Architecture - Punch and Extent Migration Requirements|''Punch and Extent Migration Requirements'']] (Punch functionality use cases)

[[Architecture - Recovery Failures|''Recovery Failures'' ]] (Recovery terminology, architectures, and use cases)

[[Architecture - Request Redirection|''Request Redirection'' ]] (Allows target OST to redirect client requests to other servers)

[[Architecture - Scalable Pinger|''Scalable Pinger'' ]] (Provides peer health information to Lustre clients and servers)

[[Architecture - Security|''Security'' ]] (Detailed description of the security architecture for Lustre)

[[Architecture - Server Network Striping|''Server Network Striping'' ]] (Description of Lustre-level striping of file data over multiple object servers with redundancy)

[[Architecture - Simple Space Balance Migration|''Simple Space Balance Migration'' ]] (A subset of full data migration limited to migrating files that are not currently in use)

[[Architecture - Simplified Interoperation|''Simplified Interoperation'' ]] (Controlled server shutdown simplifies inter-operation on server upgrades)

[[Architecture - Space Manager|''Space Manager'' ]] (Manages file system free space)

[[Architecture - Sub Tree Locks|''Sub Tree Locks'' ]] (A lock on a directory that protects a namespace (or part of a namespace) rooted at that directory)

[[Architecture - User Level Access|''User Level Access'' ]] (LNET userspace API driver that exports the LNET API to userspace)

[[Architecture - User Level OSS|''User Level OSS'' ]] (Functionality related to Lustre client or OSS failure)

[[Architecture - Userspace Servers|''Userspace Servers'' ]] (Requirements for capability to run a Lustre server in user space)

[[Architecture - Version Based Recovery|''Version Based Recovery'' ]] (A recovery mechanism allowing clients to recover outside of a strict order or later in time - requirements, use cases, quality attribute scenarios, implementation details)

[[Architecture - Wide Striping|''Wide Striping'' ]] (Mechanism to encode striping information compactly to efficiently support striping of files across many devices)

[[Architecture - Wire Level Protocol|''Wire Level Protocol'' ]] (Wire formats used by Lustre - messages, wire and record packet structures, recovery protocol)

[[Architecture - Write Back Cache|''Write Back Cache'' ]] (Allows client meta-data operations to be delayed and batched)

[[Architecture - ZFS for Lustre|''ZFS for Lustre'' ]] (Architecture and requirements related to Lustre servers using the ZFS Data Management Unit)

[[Architecture - ZFS large dnodes|''ZFS large dnodes'' ]] (Increased dnode size to allow more data in the inode)

[[Architecture - ZFS TinyZAP|''ZFS TinyZAP'' ]] (A compact ZFS Attribute Processor format that allows arbitrary values to be stored)

Architecture Descriptions

2011-01-20T19:07:56Z

Sbarthel:

''(Updated: Jan 2010)''

The architecture descriptions listed below provide information about Lustre architecture and design and are intended to help users better understand the conceptual framework of the Lustre file system.

'''''Note:''''' ''These documents reflect the state of design of a Lustre feature at a particular point in time. They many contain information that is incomplete or obsolete and may not reflect the current architecture, features and functionality of Lustre.''

[[Architecture - Adaptive Timeouts - Use Cases|''Adaptive Timeouts - Use Cases'' ]] (Network RPC timeouts based on server and network loading)

[[Architecture - ''Backup|Backup'' ]] (File system backup)

[[Architecture - Caching OSS|''Caching OSS'' ]] (Caching on object storage servers)

[[Architecture - Changelogs|''Changelogs'' ]] (Per-server logs of data or metadata changes)

[[Architecture - Changelogs 1.6|''Changelogs 1.6'' ]] (Used to facilitate efficient replication of large Lustre 1.6 filesystems)

[[Architecture - Client Cleanup|''Client Cleanup'' ]] (Use cases, business drivers, models to consider, implementation contraints)

[[Architecture - Clustered Metadata|''Clustered Metadata'' ]] (Clustered metadata server capability)

[[Architecture - Commit on Share|''Commit on Share'' ]] (Better recover-ability in an environment where clients miss reconnect window)

[[Architecture - CROW|''CROW'' ]] (Create On Write optimizes create performance by deferring OSS object creation)

[[Architecture - CTDB with Lustre|''CTDB with Lustre'' ]] (Cluster implementation of the TDB database with Lustre provides a solution for windows pCIFS)

[[Architecture - Cuts|''Cuts'' ]] (Technique for recovering file system metadata stored on file server clusters)

[[Architecture - DMU OSD|''DMU OSD'' ]] (An implementation of the Object Storage Device API for a Data Management Unit)

[[Architecture - End-to-end Checksumming|''End-to-end Checksumming'' ]] (Lustre network checksumming)

[[Architecture - Epochs|''Epochs'' ]] (Used to merge distributed data and meta-data updates in a redundant cluster configuration)

[[Architecture - External File Locking|''External File Locking'' ]] (File range lock and whole-file lock capabilities)

[[Architecture - FIDs on OST|''FIDs on OST'' ]] (File identifiers used to identify objects on an object storage target)

[[Architecture - Fileset|''Fileset'' ]] (An efficient representation of a group of file identifiers (FIDs))

[[Architecture - Flash Cache|''Flash Cache'' ]] (Very fast read-only flash storage)

[[Architecture - Free Space Management|''Free Space Management'' ]] (Managing free space for stripe allocation)

[[Architecture - GNS|''GNS'' ]] (Global namespace for a distributed file system)

[[Architecture - HSM|''HSM'' ]] (Hierarchical storage management)

[[Architecture - HSM and Cache|''HSM and Cache'' ]] (Reuse of components by Lustre features that involve migration of file system objects)

[[Architecture - HSM Migration|''HSM Migration'' ]] (Use cases and high-level architecture for migrating files between Lustre and a HSM system)

[[Architecture - Interoperability fids zfs|''Interoperability fids zfs'' ]] (Client, server, network, storage interoperability during migration to clusters based on file identifiers and the ZFS file system)

[[Architecture - Interoperability 1.6 1.8 2.0|''Interoperability 1.6 1.8 2.0'' ]] (interoperability definitions and QAS summary)

[[Architecture - IO system|''IO system'' ]] (Client IO and server I/O request handling)

[[Architecture - Libcfs|''Libcfs'' ]] (Portable runtime environment for process management and debugging support)

[[Architecture - Llog over OSD|''Llog over OSD'' ]] (Re-implement llog API to use OSD device as backend device)

[[Architecture - LRE Images|''LRE Images'' ]] (Provide development and training Lustre software environments based on supported environments for Lustre)

[[Architecture - Lustre Logging API|''Lustre Logging API'' ]] (Requirements and detailed description)

[[Architecture - MDS striping format|''MDS striping format'' ]] (Striping extended attributes, striping formats, striping APIs)

[[Architecture - MDS-on-DMU|''MDS-on-DMU'' ]] (Metadata server on the ZFS Data Management Unit - use cases, features and functional behavior)

[[Architecture - Metadata API|''Metadata API'' ]] (A set of methods used by the Lustre file system driver to access and manipulate metadata)

[[Architecture - Migration (1)|''Migration (1)'' ]] (Overview of development path for migration capabilities)
''
[[Architecture - Migration (2)|Migration (2)'' ]] (Use cases, quality attribute scenarios, and implementation details)

[[Architecture - MPI IO and NetCDF|''MPI IO and NetCDF'' ]] (Message Passing Interface I/O and network Common Data Form libraries - Lustre ADIO driver improvements and internal optimization)

[[Architecture - MPI LND|''MPI LND'' ]] (Link to paper ''Lustre Networking over MPI'')

[[Architecture - Multiple Interfaces For LNET|''Multiple Interfaces For LNET'' ]] (Use cases and configuration management for Lustre networking)

[[Architecture - Network Request Scheduler|''Network Request Scheduler'' ]] (Requirements for network request scheduler to manage incoming RPC requests on a server)

[[Architecture - New Metadata API|''New Metadata API'' ]] (Proposal and use cases)

[[Architecture - Open by fid|''Open by fid'' ]] (Returns a file descriptor based on a file ID - implementation choices, design description, use cases)

[[Architecture - OSS-on-DMU|''OSS-on-DMU'' ]](Object storage server on the ZFS Data Management Unit)

[[Architecture - PAG|''PAG'' ]] (Process Authentication Groups - Use of Linux keyring, setuid in Lustre, Kerberos credential, use cases)

[[Architecture - Pools of targets|''Pools of targets'' ]] (Use cases, command line definitions of pools of OSTs, implementation constraints)

[[Architecture - Profiling Tools for IO|''Profiling Tools for IO'' ]] (Profiling system based on Ganglia)

[[Architecture - Proxy Cache|''Proxy Cache'' ]] (Caching and aggregation used to reduce load on backend server and provide better throughput and latency to clients)

[[Architecture - Punch and Extent Migration|''Punch and Extent Migration'' ]] (Prototypes for ''punch'' and ''migrate'' functionality)

[[Architecture - Punch and Extent Migration Requirements|''Punch and Extent Migration Requirements'']] (Punch functionality use cases)

[[Architecture - Recovery Failures|''Recovery Failures'' ]] (Recovery terminology, architectures, and use cases)

[[Architecture - Request Redirection|''Request Redirection'' ]] (Allows target OST to redirect client requests to other servers)

[[Architecture - Scalable Pinger|''Scalable Pinger'' ]] (Provides peer health information to Lustre clients and servers)

[[Architecture - Security|''Security'' ]] (Detailed description of the security architecture for Lustre)

[[Architecture - Server Network Striping|''Server Network Striping'' ]] (Description of Lustre-level striping of file data over multiple object servers with redundancy)

[[Architecture - Simple Space Balance Migration|''Simple Space Balance Migration'' ]] (A subset of full data migration limited to migrating files that are not currently in use)

[[Architecture - Simplified Interoperation|''Simplified Interoperation'' ]] (Controlled server shutdown simplifies inter-operation on server upgrades)

[[Architecture - Space Manager|''Space Manager'' ]] (Manages file system free space)

[[Architecture - Sub Tree Locks|''Sub Tree Locks'' ]] (A lock on a directory that protects a namespace (or part of a namespace) rooted at that directory)

[[Architecture - User Level Access|''User Level Access'' ]] (LNET userspace API driver that exports the LNET API to userspace)

[[Architecture - User Level OSS|''User Level OSS'' ]] (Functionality related to Lustre client or OSS failure)

[[Architecture - Userspace Servers|''Userspace Servers'' ]] (Requirements for capability to run a Lustre server in user space)

[[Architecture - Version Based Recovery|''Version Based Recovery'' ]] (A recovery mechanism allowing clients to recover outside of a strict order or later in time - requirements, use cases, quality attribute scenarios, implementation details)

[[Architecture - Wide Striping|''Wide Striping'' ]] (Mechanism to encode striping information compactly to efficiently support striping of files across many devices)

[[Architecture - Wire Level Protocol|''Wire Level Protocol'' ]] (Wire formats used by Lustre - messages, wire and record packet structures, recovery protocol)

[[Architecture - Write Back Cache|''Write Back Cache'' ]] (Allows client meta-data operations to be delayed and batched)

[[Architecture - ZFS for Lustre|''ZFS for Lustre'' ]] (Architecture and requirements related to Lustre servers using the ZFS Data Management Unit)

[[Architecture - ZFS large dnodes|''ZFS large dnodes'' ]] (Increased dnode size to allow more data in the inode)

[[Architecture - ZFS TinyZAP|''ZFS TinyZAP'' ]] (A compact ZFS Attribute Processor format that allows arbitrary values to be stored)

Building and Installing Lustre from Source Code

2011-01-20T18:59:21Z

Sbarthel:

''(Updated: Feb 2010)''
__TOC__
If you need to build a customized Lustre™ server kernel or are using a Linux kernel that has not been tested with the version of Lustre you are installing, you may need to build and install Lustre from source code. This involves several steps:
* Patching the core kernel
* Configuring the kernel to work with Lustre
* Creating Lustre and kernel RPMs from source code.

Please note that the Lustre/kernel configurations available at the [http://www.oracle.com/technetwork/indexes/downloads/sun-az-index-095901.html#L Lustre download site] have been extensively tested and verified with Lustre. The recommended method for installing Lustre servers is to use these prebuilt binary packages (RPMs). For more information about this installation method, see [[Installing Lustre from Downloaded RPMs]].

For a list of available RPMs for Lustre 1.8.x and the latest version of Lustre 1.6 (1.6.7.2), see the [[Lustre_Release_Information#Lustre_Support_Matrix|Lustre Test Matrix]].

'''''Caution:''''' Lustre contains kernel modifications which interact with storage devices and may introduce security issues and data loss if not installed, configured and administered correctly. Before installing Lustre, be cautious and back up ''ALL'' data.

'''''Note:''''' When using third-party network hardware with Lustre, the third-party
modules (typically, the drivers) must be linked against the Linux kernel. The LNET
modules in Lustre also need these references. To meet these requirements, a specific
process must be followed to install and recompile Lustre. See [http://wiki.lustre.org/manual/LustreManual20_HTML/InstallingLustrefrSourceCode.html#50438210_pgfId-1299622 Section 29.4: ''Installing Lustre with a Third-Party Network Stack''] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual''] for an example showing how to install Lustre 1.6.6 using the Myricom MX 1.2.7 driver. The same process can be used for other third party network stacks.

== Patching the Kernel ==
If you are using non-standard hardware, plan to apply a Lustre patch, or have
another reason not to use packaged Lustre binaries, you have to apply several Lustre
patches to the core kernel and run the Lustre configure script against the kernel.

===Introducing the Quilt Utility===
To simplify the process of applying Lustre patches to the kernel, we recommend that you use the Quilt utility.

Quilt manages a stack of patches on a single source tree. A series file lists the patch files and the order in which they are applied. Patches are applied, incrementally, on the base tree and all preceding patches. You can:
* Apply patches from the stack (''quilt push'')
* Remove patches from the stack (''quilt pop'')
* Query the contents of the series file (''quilt series''), the contents of the stack (''quilt applied'', ''quilt previous'', ''quilt top''), and the patches that are not applied at a particular moment (''quilt next'', ''quilt unapplied'').
* Edit and refresh (''update'') patches with Quilt, as well as revert inadvertent changes, and fork or clone patches and show the diffs before and after work.

A variety of Quilt packages (RPMs, SRPMs and tarballs) are available from various
sources. Use the most recent version you can find. Quilt depends on several other
utilities, e.g., the ''coreutils'' RPM available only in RedHat 9. For other RedHat
kernels, you have to get the required packages to successfully install Quilt.
If you cannot locate a Quilt package or fulfill its dependencies, you can build Quilt
from a tarball, available at the [http://savannah.nongnu.org/projects/quilt Quilt project website].

For additional information on using Quilt, including its commands, see the [http://www.suse.de/~agruen/quilt.pdf Introduction to Quilt] and the [http://linux.die.net/man/1/quilt quilt(1) man page].

===Get the Lustre Source and Unpatched Kernel===
The Lustre Engineering Team has targeted several Linux kernels for use with Lustre servers (MDS/OSS) and provides a series of patches for each one. The Lustre patches are maintained in the ''kernel_patch'' directory bundled with the Lustre source code.

'''''Note:''''' Each patch series has been tailored to a specific kernel version, and may or may not apply cleanly to other versions of the kernel.

To obtain the Lustre source and unpatched kernel:

1. ''Verify that all of the Lustre installation requirements have been met.''
For more information on these prerequisites, see [[Preparing to Install Lustre]].

2. ''Download the Lustre source code.'' On the [http://www.oracle.com/technetwork/indexes/downloads/sun-az-index-095901.html#L Lustre download site], select a version of Lustre to download and then select ''Source'' as the platform.

4. ''Download the unpatched kernel.''

For convenience, Sun maintains an archive of unpatched kernel sources at [http://downloads.lustre.org/public/kernels/ http://downloads.lustre.org/public/kernels/].

5. ''To save time later, download ''e2fsprogs'' now.

The source code for Sun's Lustre-enabled ''e2fsprogs'' distribution can be found at [http://downloads.lustre.org/public/tools/e2fsprogs/ http://downloads.lustre.org/public/tools/e2fsprogs/]

===Patch the Kernel===
This procedure describes how to use Quilt to apply the Lustre patches to the kernel.
To illustrate the steps in this procedure, a RHEL 5 kernel is patched for Lustre
1.6.5.1.

1. ''Unpack the Lustre source and kernel to separate source trees.''

: a. ''Unpack the Lustre source.'' For this procedure, we assume that the resulting source tree is in ''/tmp/lustre-1.6.5.1''.

: b. ''Unpack the kernel.'' For this procedure, we assume that the resulting source tree (also known as the destination tree) is in ''/tmp/kernels/linux-2.6.18''.

2. Select a config file for your kernel, located in the ''kernel_configs'' directory
(''lustre/kernel_patches/kernel_config''). The ''kernel_config'' directory contains the ''.config'' files, which are named to indicate the kernel and architecture with which they are associated. For example, the configuration file for the 2.6.18 kernel shipped with RHEL 5 (suitable for i686 SMP systems) is ''kernel-2.6.18-2.6-rhel5-i686-smp.config''.

3. Select the ''series'' file for your kernel, located in the ''series'' directory
(''lustre/kernel_patches/series''). The ''series'' file contains the patches that need to be applied to the kernel.

4. Set up the necessary symlinks between the kernel patches and the Lustre
source. This example assumes that the Lustre source files are unpacked under ''/tmp/lustre-1.6.5.1'' and you have chosen the ''2.6-rhel5.series'' file. Run:

<pre>
$ cd /tmp/kernels/linux-2.6.18
$ rm -f patches series
$ ln -s /tmp/lustre-1.6.5.1/lustre/kernel_patches/series/2.6-rhel5.series ./series
$ ln -s /tmp/lustre-1.6.5.1/lustre/kernel_patches/patches .
</pre>

5. Use Quilt to apply the patches in the selected ''series'' file to the unpatched
kernel. Run:

<pre>
$ cd /tmp/kernels/linux-2.6.18
$ quilt push -av
</pre>

The patched destination tree acts as a base Linux source tree for Lustre.

===Create and Install the Lustre Packages===

After patching the kernel, configure it to work with Lustre, create the Lustre
packages (RPMs) and install them.

1. Configure the patched kernel to run with Lustre. Run:

<pre>
$ cd <path to kernel tree>
$ cp /boot/config-‘uname -r‘ .config
$ make oldconfig || make menuconfig
$ make include/asm
$ make include/linux/version.h
$ make SUBDIRS=scripts
$ make include/linux/utsrelease.h
</pre>

2. Run the Lustre configure script against the patched kernel and create the
Lustre packages.

<pre>
$ cd <path to lustre source tree>
$ ./configure --with-linux=<path to kernel tree>
$ make rpms
</pre>

This creates a set of ''.rpms'' in ''/usr/src/redhat/RPMS/<arch>'' with an
appended date-stamp. The SuSE path is ''/usr/src/packages''.

'''''Note:''''' You do not need to run the Lustre configure script against an unpatched
kernel.

Example set of RPMs:

<pre>
lustre-1.6.5.1-\
2.6.18_53.xx.xx.el5_lustre.1.6.5.1.custom_20081021.i686.rpm
lustre-debuginfo-1.6.5.1-\
2.6.18_53.xx.xx.el5_lustre.1.6.5.1.custom_20081021.i686.rpm
lustre-modules-1.6.5.1-\
2.6.18_53.xx.xxel5_lustre.1.6.5.1.custom_20081021.i686.rpm
lustre-source-1.6.5.1-\
2.6.18_53.xx.xx.el5_lustre.1.6.5.1.custom_20081021.i686.rpm
</pre>

'''''Note:''''' If the steps to create the RPMs fail, contact Lustre Support by reporting a bug (see [[Reporting Bugs]]).

'''''Note:''''' Lustre supports several features and packages that extend the core functionality of Lustre. These features/packages can be enabled at the build time by issuing appropriate arguments to the ''configure'' command. For a list of supported features and packages, run ''./configure –help'' in the Lustre source tree. The ''configs/'' directory of the kernel source contains the config files matching each the kernel version. Copy one to ''.config'' at the root of the kernel tree.

3. ''Create the kernel package.'' Navigate to the kernel source directory and run:

$ make rpm

Example result:

kernel-2.6.95.0.3.EL_lustre.1.6.5.1custom-1.i686.rpm

'''''Note:''''' Step 3 is only valid for RedHat and SuSE kernels. If you are using a stock Linux kernel, you need to get a script to create the kernel RPM.

4. ''Install the Lustre packages.'' Some Lustre packages are installed on servers (MDS and OSSs), and others are installed on Lustre clients. For guidance on where to install specific packages, see [[Lustre Packages]].

'''''Note:''''' Running the patched server kernel on the clients is optional. It is not necessary unless the clients will be used for multiple purposes, for example, to run as a client and an OST.

Lustre packages should be installed in this order:

:a. ''Install the kernel, modules and ldiskfs packages.'' Navigate to the directory where the RPMs are stored, and use the ''rpm -ivh'' command to install the kernel, module and ''ldiskfs'' packages.

: <pre>$ rpm -ivh kernel-lustre-smp-<ver> \
;kernel-ib-<ver> \
;lustre-modules-<ver> \
;lustre-ldiskfs-<ver>
</pre>

: b. ''Install the utilities/userspace packages.'' Use the ''rpm -ivh'' command to install the utilities packages. For example:

: <pre>$ rpm -ivh lustre-<ver>
</pre>

: c. ''Install the'' e2fsprogs ''package.'' Make sure the ''e2fsprogs'' package downloaded in Step 5 is unpacked, and use the ''rpm -i'' command to install it. For example:

: <pre>$ rpm -i e2fsprogs-<ver>
</pre>

If you want to add any optional packages to your Lustre file system, install them
now.

5. ''Verify that the boot loader (''grub.conf ''or'' lilo.conf'') has been updated to load the patched kernel.''

6. ''Reboot the patched clients and the servers.''

: a. ''If you applied the patched kernel to any clients, reboot them.'' Unpatched clients do not need to be rebooted.
: b. ''Reboot the servers.'' Once all the machines have rebooted, the next steps are to configure Lustre Networking (LNET) and the Lustre file system. See [[Configuring the Lustre File System]].

Change Log 2.0

2011-01-20T18:58:01Z

Sbarthel:

''(Updated: Aug 2010)''

= Welcome to Lustre 2.0.0 =

This release represents a departure from the previous trains that were closely related to one another. As such, we have no previous release to show you the change from. Future 2.x releases will show the change from this or subsequent releases. All that said, you can find out more details from one of the following sources:

[http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre 2.0 Operations Manual'']

[http://wiki.lustre.org/images/6/60/821-2077-10.pdf ''Lustre 2.0.0 Release Notes'']

'''Support for networks:'''
* socklnd - any kernel supported by Lustre,
* qswlnd - Qsnet kernel modules 5.20 and later,
* openiblnd - IbGold 1.8.2,
* o2iblnd - OFED 1.1, 1.2.0, 1.2.5, 1.3, and 1.4.1
* viblnd - Voltaire ibhost 3.4.5 and later,
* ciblnd - Topspin 3.2.0,
* iiblnd - Infiniserv 3.3 + PathBits patch,
* gmlnd - GM 2.1.22 and later,
* mxlnd - MX 1.2.10 or later,
* ptllnd - Portals 3.3 / UNICOS/lc 1.5.x, 2.0.x

'''Server support for kernels:'''
* 2.6.18-164.11.1.el5 (RHEL 5)
* 2.6.18-164.11.1.0.1.el5 (OEL 5)

'''Client support for unpatched kernels:''' see [http://wiki.lustre.org/index.php?title=Patchless_Client "Patchless Client"]
* 2.6.18-164.11.1.el5 (RHEL 5),
* 2.6.18-164.11.1.0.1.el5 (OEL 5)
* 2.6.16.60-0.42.8 (SLES 10),
* 2.6.27.19-5 (SLES11)
* 2.6.29.4-167.fc11 (FC11)

'''Recommended e2fsprogs version:'''
* 1.41.10-sun2

Configuring InfiniBand Connectivity

2011-01-20T18:57:04Z

Sbarthel: /* Configuring LNET as an OFED Infiniband Network */

''(Updated: Dec 2009)''
__TOC__
To configure Infiniband connectivity, follow the procedures appropriate for your system below.

== Configuring LNET as an OFED Infiniband Network ==

To configure LNET as an OFED InfiniBand network, enter:

<pre>
options lnet networks="o2ib3(ib3)"
</pre>
The node specified is on ''o2ib'' network ''3'' using HCA ''ib3''.

For information about other LNET options, see [http://wiki.lustre.org/manual/LustreManual20_HTML/ConfigurationFilesModuleParameters.html#50438293_pgfId-1293321 35.2.1: ''LNET Options''] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual''].

== Building and Configuring Infiniband Support for Lustre™ ==

The distributed kernels do not yet include 3rd-party Infiniband modules. As a result,
our Lustre™ packages can not include IB network drivers for Lustre. However,
we do distribute the source code. You will need to build your Infiniband software
stack against the supplied kernel and then build new Lustre packages.

====Voltaire====

To build Lustre with Voltaire Infiniband sources, add the following as an argument to the configure script:
--with-vib=<path-tovoltaire-sources>

To configure Lustre, use:
--nettype vib --nid <IPoIB address>

'''''Note:''''' In Lustre v1.4.5, the Voltaire IB module (''kvibnal'') will not work on the Altix system. This is due to hardware differences in the Altix system.

====OpenIB generation 1 / Mellanox Gold====

To build Lustre with OpenIB Infiniband sources, add the following as an argument to the configure script:
--with-openib=<path_to_openib sources>

To configure Lustre, use:
--nettype openib --nid <IPoIB address>

====Silverstorm====

A Silverstorm driver for Lustre is available.

To build Silverstorm with Lustre, configure Lustre with:
--with-iib=<path to silverstorm sources>

====OpenIB 1.0====
An OpenIB 1.0 driver for Lustre is available.

Configuring Lustre File Striping

2011-01-20T18:56:13Z

Sbarthel:

''(Updated: Oct 2009)''
__TOC__
One of the main factors leading to the high performance of Lustre™ file systems is the ability to stripe data over multiple OSTs. The stripe count can be set on a file system, directory, or file level. An example showing the use of striping is provided below.

For additional information, see [http://wiki.lustre.org/manual/LustreManual20_HTML/ManagingStripingFreeSpace.html#50438209_pgfId-5529 Chapter 18: ''Managing File Striping and Free Space''] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual'']

== Setting Up Striping ==

To see the current stripe size, use the command ''lfs getstripe [file, dir, fs]''. This command will produce output similar to the following:

<pre>
root@LustreClient01 lustre]# lfs getstripe /mnt/lustre
OBDS:
0: lustre-OST0000_UUID ACTIVE
1: lustre-OST0001_UUID ACTIVE
2: lustre-OST0002_UUID ACTIVE
3: lustre-OST0003_UUID ACTIVE
4: lustre-OST0004_UUID ACTIVE
5: lustre-OST0005_UUID ACTIVE
/mnt/lustre
(Default) stripe_count: 2 stripe_size: 4M stripe_offset: 0
</pre>

In this example, the default stripe count is 2 (that is, data blocks are striped over two OSTs), the default stripe size is 4 MB (the stripe size can be set in K, M or G), and all writes start from the first OST.

'''''Note:''''' When setting the stripe, the offset is set before the stripe count.

The command to set a new stripe pattern on the file system may look like this:

[root@LustreClient01 lustre]# lfs setstripe -s 4M -c 0 -i 1 /mnt/lustre

This example command sets the stripe of ''/mnt/lustre'' to 4 MB blocks starting at OST0 and spanning over one OST. If a new file is created with these settings, the following results are seen:

<pre>
[root@LustreClient01 lustre]# dd if=/dev/zero of=/mnt/lustre/test1 bs=10M count=100

root@LustreClient01 lustre]# lfs df -h
UUID bytes Used Available Use% Mounted on
lustre-MDT0000_UUID 4.4G 214.5M 3.9G 4% /mnt/lustre[MDT:0]
lustre-OST0000_UUID 2.0G 1.1G 830.1M 53% /mnt/lustre[OST:0]
lustre-OST0001_UUID 2.0G 83.3M 1.8G 4% /mnt/lustre[OST:1]
lustre-OST0002_UUID 2.0G 83.3M 1.8G 4% /mnt/lustre[OST:2]
lustre-OST0003_UUID 2.0G 83.3M 1.8G 4% /mnt/lustre[OST:3]
lustre-OST0004_UUID 2.0G 83.3M 1.8G 4% /mnt/lustre[OST:4]
lustre-OST0005_UUID 2.0G 83.3M 1.8G 4% /mnt/lustre[OST:5]

filesystem summary: 11.8G 1.5G 9.7G 12% /mnt/lustre
</pre>

In this example, the entire file was written to the first OST with a very uneven distribution of data blocks.

Continuing with this example, the file is removed and the stripe count is changed to a value of ''-1'' to specify striping over all available OSTs:

[root@LustreClient01 lustre]# lfs setstripe -s 4M -c 0 -i -1 /mnt/lustre

Now, when a file is created, the new stripe setting evenly distributes the data over all the available OSTs:

<pre>
[root@LustreClient01 lustre]# dd if=/dev/zero of=/mnt/lustre/test1 bs=10M count=100
100+0 records in
100+0 records out
1048576000 bytes (1.0 GB) copied, 20.2589 seconds, 51.8 MB/s

[root@LustreClient01 lustre]# lfs df -h
UUID bytes Used Available Use% Mounted on
lustre-MDT0000_UUID 4.4G 214.5M 3.9G 4% /mnt/lustre[MDT:0]
lustre-OST0000_UUID 2.0G 251.3M 1.6G 12% /mnt/lustre[OST:0]
lustre-OST0001_UUID 2.0G 251.3M 1.6G 12% /mnt/lustre[OST:1]
lustre-OST0002_UUID 2.0G 251.3M 1.6G 12% /mnt/lustre[OST:2]
lustre-OST0003_UUID 2.0G 251.3M 1.6G 12% /mnt/lustre[OST:3]
lustre-OST0004_UUID 2.0G 247.3M 1.6G 12% /mnt/lustre[OST:4]
lustre-OST0005_UUID 2.0G 247.3M 1.6G 12% /mnt/lustre[OST:5]

filesystem summary: 11.8G 1.5G 9.7G 12% /mnt/lustre
</pre>

== Displaying Stripe Information for a File ==

The ''lfs getstripe'' command can be used to display information that shows over which OSTs a file is distributed. For example, the output from the following command (showing multiple ''obdidx'' entries) indicates that the file ''test1'' is striped over all six active OSTs in the configuration:

<pre>
[root@LustreClient01 ~]# lfs getstripe /mnt/lustre/test1
OBDS:
0: lustre-OST0000_UUID ACTIVE
1: lustre-OST0001_UUID ACTIVE
2: lustre-OST0002_UUID ACTIVE
3: lustre-OST0003_UUID ACTIVE
4: lustre-OST0004_UUID ACTIVE
5: lustre-OST0005_UUID ACTIVE
/mnt/lustre/test1
obdidx objid objid group
0 8 0x8 0
1 4 0x4 0
2 5 0x5 0
3 5 0x5 0
4 4 0x4 0
5 2 0x2 0
</pre>

In contrast, the output from the following command, which lists just a single ''obdidx'' entry, indicates that the file ''test2'' is contained on a single OST:

<pre>
[root@LustreClient01 ~]# lfs getstripe /mnt/lustre/test_2
OBDS:
0: lustre-OST0000_UUID ACTIVE
1: lustre-OST0001_UUID ACTIVE
2: lustre-OST0002_UUID ACTIVE
3: lustre-OST0003_UUID ACTIVE
4: lustre-OST0004_UUID ACTIVE
5: lustre-OST0005_UUID ACTIVE
/mnt/lustre/test_2
obdidx objid objid group
2 8 0x8 0
</pre>

Configuring Lustre for Failover

2011-01-20T18:55:18Z

Sbarthel:

''(Updated: Feb 2010)''

If you plan to enable failover server functionality with Lustre™ (either on an OSS or MDS), you must add high-availability (HA) software to your cluster software. You can use any HA software package with Lustre.

For more information about failover in a Lustre file system, see [http://wiki.lustre.org/manual/LustreManual20_HTML/UnderstandingFailover.html#50438253_pgfId-1304327‎ Chapter 3: ''Understanding Failover in Lustre''] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual''].

For a user-contributed guide to configuring Lustre failover with Red Hat Cluster Manager, see [[Using Red Hat Cluster Manager with Lustre]].

For a user-contributed guide to configuring Lustre failover with Pacemaker, see [[Using Pacemaker with Lustre]].

'''''Note:''' Using Red Hat Cluster Manager or Pacemaker with Lustre is not officially supported at this time.''

Configuring the Lustre File System

2011-01-20T18:53:06Z

Sbarthel:

''(Updated: Sep 2009)''
__TOC__
This page describes how to configure a simple Lustre™ file system comprised of a combined MGS/MDT, an OST and a client. The administrative utilities provided with Lustre, however, can be used to set up systems with many different configurations.

'''''Note:''''' We recommend that you use dotted-quad (dot-decimal) notation for IP
addresses (IPv4) rather than host names. This aids in reading debug logs and helps
when debugging configurations with multiple interfaces.

== Configuring the Lustre File System ==
This section contains a procedure for configuring the Lustre File System. For an example showing configuration of a Lustre installation comprising a combined MGS/MDT, an OST and a client see the [[Lustre Configuration Example]].

To configure Lustre Networking (LNET) and the Lustre file
system, complete these steps:

1. ''Define the module options for Lustre networking (LNET)'' by adding this line
to the ''/etc/modprobe.conf'' file. The ''modprobe.conf'' file is a Linux file that specifies which parts of the kernel are loaded.

:<pre>options lnet networks=<network interfaces that LNET can use></pre>

:This step restricts LNET to using only the specified network interfaces and prevents LNET from using all network interfaces.

:As an alternative to modifying the ''modprobe.conf'' file, you can modify the ''modprobe.local'' file or the configuration files in the ''modprobe.d'' directory.

:'''''Note:''''' For details on configuring networking and LNET, see [http://wiki.lustre.org/manual/LustreManual20_HTML/UnderstandingLustreNetworking.html#50438191_pgfId-1289854 Chapter 2: ''Understanding Lustre Networking (LNET'')] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual''].

2. ''(Optional) Prepare the block devices to be used as OSTs or MDTs.'' Depending on the hardware used in the MDS and OSS nodes, you may want to set up a hardware or software RAID to increase the reliability of the Lustre system. For more details on how to set up a hardware or software RAID, see the documentation for your RAID controller or see [http://wiki.lustre.org/manual/LustreManual20_HTML/ConfiguringStorage.html#50438208_pgfId-1289851 Chapter 6: ''Configuring Storage on a Lustre File System''] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual''].

3. ''Create a combined MGS/MDT file system on the block device.'' On the MDS
node, run:

:<pre>mkfs.lustre --fsname=<fsname> --mgs --mdt <block device name>
</pre>

:The default file system name (''fsname'') is ''lustre''.

:'''''Note:''''' If you plan to generate multiple file systems, the MGS should be on its own dedicated block device.

4. ''Mount the combined MGS/MDT file system on the block device.'' On the MDS
node, run:
:<pre>mount -t lustre <block device name> <mount point>
</pre>

5. ''Create the OST.'' On the OSS node, run:

:<pre>mkfs.lustre --ost --fsname=<fsname> --mgsnode=<NID> <block device name>
</pre>

:You can have as many OSTs per OSS as the hardware or drivers allow.

:You should use only one OST per block device. Optionally, you can create an OST which uses the raw block device and does not require partitioning.

:'''''Note:''''' If the block device has more than 8 TB of storage, it must be partitioned due to the ext3 file system limitation. Lustre can support block devices with multiple partitions, but they are not recommended because bottlenecks may result.

6. ''Mount the OST.'' On the OSS node where the OST was created, run:

:<pre>mount -t lustre <block device name> <mount point>
</pre>

:'''''Note:''''' ''To create additional OSTs, repeat Steps 5 and 6.''

7. ''Create the client (mount the file system on the client).'' On the client node, run:

:<pre>
;mount -t lustre <MGS node>:/<fsname> <mount point>
</pre>

:'''''Note:''''' To create additional clients, repeat Step 7]].

8. ''Verify that the file system started and is working'' by running the UNIX
commands ''df'', ''dd'' and ''ls'' on the client node.

:a. ''Run the ''df'' command.''

:<pre>
;[root@client1 /] df -h
</pre>

:b. ''Run the ''dd'' command.''

:<pre>
;[root@client1 /] cd /lustre
;[root@client1 /lustre] dd if=/dev/zero of=/lustre/zero.dat bs=4M count=2
</pre>

c. ''Run the ''ls'' command.''

:<pre>
;[root@client1 /lustre] ls -lsah
</pre>

If you have a problem mounting the file system, check the syslogs for errors.

== Lustre Configuration Example ==

For an example illustrating the configuration steps described in the previous section for
a Lustre installation comprising a combined MGS/MDT, an OST and a client, see [[Lustre Configuration Example]].

== Lustre Configuration Utilities ==

Once the Lustre file system is configured, it is ready for use. If additional configuration is necessary, see [[Lustre System Configuration Utilities]].

Configuring the Lustre File System

2011-01-20T18:52:13Z

Sbarthel:

''(Updated: Sep 2009)''
__TOC__
This page describes how to configure a simple Lustre™ file system comprised of a combined MGS/MDT, an OST and a client. The administrative utilities provided with Lustre, however, can be used to set up systems with many different configurations.

'''''Note:''''' We recommend that you use dotted-quad (dot-decimal) notation for IP
addresses (IPv4) rather than host names. This aids in reading debug logs and helps
when debugging configurations with multiple interfaces.

== Configuring the Lustre File System ==
This section contains a procedure for configuring the Lustre File System. For an example showing configuration of a Lustre installation comprising a combined MGS/MDT, an OST and a client see the [[Lustre Configuration Example]].

To configure Lustre Networking (LNET) and the Lustre file
system, complete these steps:

1. ''Define the module options for Lustre networking (LNET)'' by adding this line
to the ''/etc/modprobe.conf'' file. The ''modprobe.conf'' file is a Linux file that specifies which parts of the kernel are loaded.

:<pre>options lnet networks=<network interfaces that LNET can use></pre>

:This step restricts LNET to using only the specified network interfaces and prevents LNET from using all network interfaces.

:As an alternative to modifying the ''modprobe.conf'' file, you can modify the ''modprobe.local'' file or the configuration files in the ''modprobe.d'' directory.

:'''''Note:''''' For details on configuring networking and LNET, see [http://wiki.lustre.org/manual/LustreManual20_HTML/UnderstandingLustreNetworking.html#50438191_pgfId-1289854 Chapter 2: ''Understanding Lustre Networking (LNET'')] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual''].

2. ''(Optional) Prepare the block devices to be used as OSTs or MDTs.'' Depending on the hardware used in the MDS and OSS nodes, you may want to set up a hardware or software RAID to increase the reliability of the Lustre system. For more details on how to set up a hardware or software RAID, see the documentation for your RAID controller or see [http://wiki.lustre.org/manual/LustreManual20_HTML/ConfiguringStorage.html#50438208_pgfId-1289851 Chapter 6: Configuring Storage on a Lustre File System] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual''].

3. ''Create a combined MGS/MDT file system on the block device.'' On the MDS
node, run:

:<pre>mkfs.lustre --fsname=<fsname> --mgs --mdt <block device name>
</pre>

:The default file system name (''fsname'') is ''lustre''.

:'''''Note:''''' If you plan to generate multiple file systems, the MGS should be on its own dedicated block device.

4. ''Mount the combined MGS/MDT file system on the block device.'' On the MDS
node, run:
:<pre>mount -t lustre <block device name> <mount point>
</pre>

5. ''Create the OST.'' On the OSS node, run:

:<pre>mkfs.lustre --ost --fsname=<fsname> --mgsnode=<NID> <block device name>
</pre>

:You can have as many OSTs per OSS as the hardware or drivers allow.

:You should use only one OST per block device. Optionally, you can create an OST which uses the raw block device and does not require partitioning.

:'''''Note:''''' If the block device has more than 8 TB of storage, it must be partitioned due to the ext3 file system limitation. Lustre can support block devices with multiple partitions, but they are not recommended because bottlenecks may result.

6. ''Mount the OST.'' On the OSS node where the OST was created, run:

:<pre>mount -t lustre <block device name> <mount point>
</pre>

:'''''Note:''''' ''To create additional OSTs, repeat Steps 5 and 6.''

7. ''Create the client (mount the file system on the client).'' On the client node, run:

:<pre>
;mount -t lustre <MGS node>:/<fsname> <mount point>
</pre>

:'''''Note:''''' To create additional clients, repeat Step 7]].

8. ''Verify that the file system started and is working'' by running the UNIX
commands ''df'', ''dd'' and ''ls'' on the client node.

:a. ''Run the ''df'' command.''

:<pre>
;[root@client1 /] df -h
</pre>

:b. ''Run the ''dd'' command.''

:<pre>
;[root@client1 /] cd /lustre
;[root@client1 /lustre] dd if=/dev/zero of=/lustre/zero.dat bs=4M count=2
</pre>

c. ''Run the ''ls'' command.''

:<pre>
;[root@client1 /lustre] ls -lsah
</pre>

If you have a problem mounting the file system, check the syslogs for errors.

== Lustre Configuration Example ==

For an example illustrating the configuration steps described in the previous section for
a Lustre installation comprising a combined MGS/MDT, an OST and a client, see [[Lustre Configuration Example]].

== Lustre Configuration Utilities ==

Once the Lustre file system is configured, it is ready for use. If additional configuration is necessary, see [[Lustre System Configuration Utilities]].

Configuring the Lustre File System

2011-01-20T18:51:25Z

Sbarthel: /* Configuring the Lustre File System */

''(Updated: Sep 2009)''
__TOC__
This page describes how to configure a simple Lustre™ file system comprised of a combined MGS/MDT, an OST and a client. The administrative utilities provided with Lustre, however, can be used to set up systems with many different configurations.

'''''Note:''''' We recommend that you use dotted-quad (dot-decimal) notation for IP
addresses (IPv4) rather than host names. This aids in reading debug logs and helps
when debugging configurations with multiple interfaces.

== Configuring the Lustre File System ==
This section contains a procedure for configuring the Lustre File System. For an example showing configuration of a Lustre installation comprising a combined MGS/MDT, an OST and a client see the [[Lustre Configuration Example]].

To configure Lustre Networking (LNET) and the Lustre file
system, complete these steps:

1. ''Define the module options for Lustre networking (LNET)'' by adding this line
to the ''/etc/modprobe.conf'' file. The ''modprobe.conf'' file is a Linux file that specifies which parts of the kernel are loaded.

:<pre>options lnet networks=<network interfaces that LNET can use></pre>

:This step restricts LNET to using only the specified network interfaces and prevents LNET from using all network interfaces.

:As an alternative to modifying the ''modprobe.conf'' file, you can modify the ''modprobe.local'' file or the configuration files in the ''modprobe.d'' directory.

:'''''Note:''''' For details on configuring networking and LNET, see [http://wiki.lustre.org/manual/LustreManual20_HTML/UnderstandingLustreNetworking.html#50438191_pgfId-1289854 Chapter 2: ''Understanding Lustre Networking (LNET'')] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual''].

2. ''(Optional) Prepare the block devices to be used as OSTs or MDTs.'' Depending on the hardware used in the MDS and OSS nodes, you may want to set up a hardware or software RAID to increase the reliability of the Lustre system. For more details on how to set up a hardware or software RAID, see the documentation for your RAID controller or see [http://wiki.lustre.org/manual/LustreManual20_HTML/ConfiguringStorage.html#50438208_pgfId-1289851 Chapter 6: Configuring Storage on a Lustre File System] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html Lustre Operations Manual].

3. ''Create a combined MGS/MDT file system on the block device.'' On the MDS
node, run:

:<pre>mkfs.lustre --fsname=<fsname> --mgs --mdt <block device name>
</pre>

:The default file system name (''fsname'') is ''lustre''.

:'''''Note:''''' If you plan to generate multiple file systems, the MGS should be on its own dedicated block device.

4. ''Mount the combined MGS/MDT file system on the block device.'' On the MDS
node, run:
:<pre>mount -t lustre <block device name> <mount point>
</pre>

5. ''Create the OST.'' On the OSS node, run:

:<pre>mkfs.lustre --ost --fsname=<fsname> --mgsnode=<NID> <block device name>
</pre>

:You can have as many OSTs per OSS as the hardware or drivers allow.

:You should use only one OST per block device. Optionally, you can create an OST which uses the raw block device and does not require partitioning.

:'''''Note:''''' If the block device has more than 8 TB of storage, it must be partitioned due to the ext3 file system limitation. Lustre can support block devices with multiple partitions, but they are not recommended because bottlenecks may result.

6. ''Mount the OST.'' On the OSS node where the OST was created, run:

:<pre>mount -t lustre <block device name> <mount point>
</pre>

:'''''Note:''''' ''To create additional OSTs, repeat Steps 5 and 6.''

7. ''Create the client (mount the file system on the client).'' On the client node, run:

:<pre>
;mount -t lustre <MGS node>:/<fsname> <mount point>
</pre>

:'''''Note:''''' To create additional clients, repeat Step 7]].

8. ''Verify that the file system started and is working'' by running the UNIX
commands ''df'', ''dd'' and ''ls'' on the client node.

:a. ''Run the ''df'' command.''

:<pre>
;[root@client1 /] df -h
</pre>

:b. ''Run the ''dd'' command.''

:<pre>
;[root@client1 /] cd /lustre
;[root@client1 /lustre] dd if=/dev/zero of=/lustre/zero.dat bs=4M count=2
</pre>

c. ''Run the ''ls'' command.''

:<pre>
;[root@client1 /lustre] ls -lsah
</pre>

If you have a problem mounting the file system, check the syslogs for errors.

== Lustre Configuration Example ==

For an example illustrating the configuration steps described in the previous section for
a Lustre installation comprising a combined MGS/MDT, an OST and a client, see [[Lustre Configuration Example]].

== Lustre Configuration Utilities ==

Once the Lustre file system is configured, it is ready for use. If additional configuration is necessary, see [[Lustre System Configuration Utilities]].

Configuring the Lustre File System

2011-01-20T18:47:53Z

Sbarthel: /* Configuring the Lustre File System */

''(Updated: Sep 2009)''
__TOC__
This page describes how to configure a simple Lustre™ file system comprised of a combined MGS/MDT, an OST and a client. The administrative utilities provided with Lustre, however, can be used to set up systems with many different configurations.

'''''Note:''''' We recommend that you use dotted-quad (dot-decimal) notation for IP
addresses (IPv4) rather than host names. This aids in reading debug logs and helps
when debugging configurations with multiple interfaces.

== Configuring the Lustre File System ==
This section contains a procedure for configuring the Lustre File System. For an example showing configuration of a Lustre installation comprising a combined MGS/MDT, an OST and a client see the [[Lustre Configuration Example]].

To configure Lustre Networking (LNET) and the Lustre file
system, complete these steps:

1. ''Define the module options for Lustre networking (LNET)'' by adding this line
to the ''/etc/modprobe.conf'' file. The ''modprobe.conf'' file is a Linux file that specifies which parts of the kernel are loaded.

:<pre>options lnet networks=<network interfaces that LNET can use></pre>

:This step restricts LNET to using only the specified network interfaces and prevents LNET from using all network interfaces.

:As an alternative to modifying the ''modprobe.conf'' file, you can modify the ''modprobe.local'' file or the configuration files in the ''modprobe.d'' directory.

:'''''Note:''''' For details on configuring networking and LNET, see [http://wiki.lustre.org/manual/LustreManual20_HTML/UnderstandingLustreNetworking.html#50438191_pgfId-1289854 Chapter 2: Understanding Lustre Networking (LNET)] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html Lustre Operations Manual].

2. ''(Optional) Prepare the block devices to be used as OSTs or MDTs.'' Depending on the hardware used in the MDS and OSS nodes, you may want to set up a hardware or software RAID to increase the reliability of the Lustre system. For more details on how to set up a hardware or software RAID, see the documentation for your RAID controller or see [http://wiki.lustre.org/manual/LustreManual20_HTML/ConfiguringStorage.html#50438208_pgfId-1289851 Chapter 6: Configuring Storage on a Lustre File System] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html Lustre Operations Manual].

3. ''Create a combined MGS/MDT file system on the block device.'' On the MDS
node, run:

:<pre>mkfs.lustre --fsname=<fsname> --mgs --mdt <block device name>
</pre>

:The default file system name (''fsname'') is ''lustre''.

:'''''Note:''''' If you plan to generate multiple file systems, the MGS should be on its own dedicated block device.

4. ''Mount the combined MGS/MDT file system on the block device.'' On the MDS
node, run:
:<pre>mount -t lustre <block device name> <mount point>
</pre>

5. ''Create the OST.'' On the OSS node, run:

:<pre>mkfs.lustre --ost --fsname=<fsname> --mgsnode=<NID> <block device name>
</pre>

:You can have as many OSTs per OSS as the hardware or drivers allow.

:You should use only one OST per block device. Optionally, you can create an OST which uses the raw block device and does not require partitioning.

:'''''Note:''''' If the block device has more than 8 TB of storage, it must be partitioned due to the ext3 file system limitation. Lustre can support block devices with multiple partitions, but they are not recommended because bottlenecks may result.

6. ''Mount the OST.'' On the OSS node where the OST was created, run:

:<pre>mount -t lustre <block device name> <mount point>
</pre>

:'''''Note:''''' ''To create additional OSTs, repeat Steps 5 and 6.''

7. ''Create the client (mount the file system on the client).'' On the client node, run:

:<pre>
;mount -t lustre <MGS node>:/<fsname> <mount point>
</pre>

:'''''Note:''''' To create additional clients, repeat Step 7]].

8. ''Verify that the file system started and is working'' by running the UNIX
commands ''df'', ''dd'' and ''ls'' on the client node.

:a. ''Run the ''df'' command.''

:<pre>
;[root@client1 /] df -h
</pre>

:b. ''Run the ''dd'' command.''

:<pre>
;[root@client1 /] cd /lustre
;[root@client1 /lustre] dd if=/dev/zero of=/lustre/zero.dat bs=4M count=2
</pre>

c. ''Run the ''ls'' command.''

:<pre>
;[root@client1 /lustre] ls -lsah
</pre>

If you have a problem mounting the file system, check the syslogs for errors.

== Lustre Configuration Example ==

For an example illustrating the configuration steps described in the previous section for
a Lustre installation comprising a combined MGS/MDT, an OST and a client, see [[Lustre Configuration Example]].

== Lustre Configuration Utilities ==

Once the Lustre file system is configured, it is ready for use. If additional configuration is necessary, see [[Lustre System Configuration Utilities]].

Download:Download

2011-01-20T18:44:47Z

Sbarthel:

Lustre™ is a scalable, secure, highly-available cluster file system. It is designed, developed and maintained by Oracle Corporation.
[[Learn|Learn More]]

Official production releases and pre-release versions of Lustre software are available for download. Official releases offer new features and enhancements, and have undergone thorough test cyles. They are available at the Oracle [https://cds.sun.com/is-bin/INTERSHOP.enfinity/WFS/CDS-CDS_SMI-Site/en_US/-/USD/ViewProductDetail-Start?ProductRef=LUSTRE-184-G-F@CDS-CDS_SMI download] site. Pre-release versions of Lustre are still being coded or are undergoing release testing. They are available for checkout from the Lustre source repository.

If you are ready to get a production-level release of Lustre or ready to try a pre-release version, download it here.

<div class="categoryLeft">

<big>Official Releases</big>

The latest official release of Lustre software is always available from Oracle Corporation, along with earlier production versions. To download an official release of Lustre, visit the Oracle <ins>[https://cds.sun.com/is-bin/INTERSHOP.enfinity/WFS/CDS-CDS_SMI-Site/en_US/-/USD/ViewProductDetail-Start?ProductRef=LUSTRE-184-G-F@CDS-CDS_SMI download]</ins> site.

Currently, all Lustre l.8.x and 2.0.0 versions are available for download. To determine which Lustre release supports the features and environment you want, see the <ins>[[Lustre_Release_Information#Lustre_Support_Matrix|Lustre Test Matrix]]</ins>.

A [https://www.sun.com/software/products/lustre/datasheet.pdf <ins>datasheet</ins> for Lustre 1.8] is also available.

* [https://cds.sun.com/is-bin/INTERSHOP.enfinity/WFS/CDS-CDS_SMI-Site/en_US/-/USD/ViewProductDetail-Start?ProductRef=LUSTRE-184-G-F@CDS-CDS_SMI Get Lustre from Oracle]

</div>
<div class="categoryRight">

<big>Pre-Release Versions</big>

As an open-source product, we encourage contributions to develop and test Lustre by trying out pre-release versions of the software. To obtain Lustre code from the source repository, you must have the Git version control system installed.

* [[Accessing_Lustre_Code|Get Lustre from Git]]
</div>

Finding a Project

2011-01-20T18:43:00Z

Sbarthel:

''(Updated: Nov 2009)''
__TOC__
This page describes how to [[#Finding a Bug to Fix|find a bug to fix]], [[#Selecting a Project to Enhance Lustre|select a project to enhance Lustre]] or [[#Helping with Lustre Testing|help with Lustre testing]]. Lustre defects and features or to-do items are logged in the Bugzilla bug tracking system.

You can also also contact the [[Lustre_Mailing_Lists|Lustre Development mailing list]] (often referred to as [mailto:lustre-devel@lists.lustre.org lustre-devel]) to discuss ideas for projects that match your skills and interests. Note that Lustre encompasses a number of development areas, including user tools, documentation, disk filesystems, networking, kernel integration, etc., so you can almost always find a project that is interesting and challenging.

Having a specific problem to fix requires an understanding of the flow of operations and how that maps to specific code. It gives you a concrete goal that provides a context for investigating the code, rather than just reading vaguely through the vast Lustre code base.

Once you have selected a project, contact [mailto:lustre-devel@lists.lustre.org lustre-devel] to discuss the best approach to take and to keep others aware of what you are working on.

== Finding a Bug to Fix ==
Fixing bugs in Lustre is a good way to become familiar with the Lustre code if you've not worked on it before. Some ways to find a bug you'd like to work on are:

* ''Search [https://bugzilla.lustre.org/buglist.cgi?query_format=advanced&product=Lustre&keywords_type=anywords&keywords=easy+needs-test&bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&order=Reuse+same+sort+as+last+time Bugzilla] key words for bugs designated "easy" bugs.'' Some Lustre developers use this keyword to indicate that a bug could be fixed by someone without in-depth familiarity with the Lustre code.
* ''Search [https://bugzilla.lustre.org/query.cgi Bugzilla] for very old Lustre bugs.'' These are typically non-critical bugs that are not dependent on a release timeline. They can vary widely in complexity. In particular, doing an empty Bugzilla query and looking at the first 100 items (sorted by bug number) shows a lot of bugs that are either relatively hard to reproduce, not generally visible to users, or "nice-to-have" features that no customer has specifically prioritized to be fixed.

== Selecting a Project to Enhance Lustre ==
If you'd like to take on a project to enhance or add a new feature to Lustre, consider one of these options:

* Pick a project from the [[Lustre Project List]]. For guidance in selecting or proceeding with a project, contact [mailto:lustre-devel@lists.lustre.org lustre-devel].

* Ask for a project on [mailto:lustre-devel@lists.lustre.org lustre-devel]. This mailing list is read by many of the Lustre developers and is a good place for questions, ideas, feedback.

* ''Assist with keeping Lustre up-to-date with recent kernel changes.'' Porting Lustre to newer kernel versions is an ongoing effort, given the large number of vendor and upstream kernel releases. For some changes, a simple fix to the Lustre code will be required, while for others, a good understanding of the Linux kernel and how Lustre interfaces with it is needed.
* ''Propose a new feature that can be developed as a separate module on top of Lustre.'' Be sure to get feedback on your proposal by contacting [mailto:lustre-devel@lists.lustre.org lustre-devel] before you get started.

== Helping with Lustre Testing ==
Testing Lustre under a variety of workloads is always of interest. The more unusual the IO pattern used by a benchmark, application, or testing tool, the more likely it is to find something of interest.

To find out how you can contribute to the testing of upcoming Lustre releases, see [[Lustre_Test_Plans|Lustre Test Plans]].

GSS / Kerberos

2011-01-20T18:41:13Z

Sbarthel: /* Secure MGC - MGS connection */

'''Note:''' Only the HEAD branch supports GSS/Kerberos functionality. It is subject to changes at any time, and backward compatibility is NOT guaranteed.

= Kerberos Lustre Setup =

== Security Flavor ==
A security flavor is a string to describe what kind authentication and data transformation be performed upon a PTLRPC connection. It covers both RPC message and BULK data.

The support flavors are described in following table:

{|border=1 cellspacing=0
|bgcolor=#E6E6E6| Base Flavor||bgcolor= #E6E6E6|Authentication||bgcolor=#E6E6E6|RPC Message Protection||bgcolor=#E6E6E6|Bulk Data Protection||bgcolor=#E6E6E6|Notes
|-
|'''''null'''''||N/A ||N/A ||N/A '''[*]''' ||Almost no performance overhead. The on-wire rpc format is compatible with old versions (1.4.x, 1.6.x, 1.8.x).
|-
|'''''plain'''''||N/A ||N/A ||checksum||(obsolete)
|-
|'''''krb5n'''''||GSS/Kerberos5 ||null||checksum (adler32)||No protection of rpc message, adler32 checksum protection of bulk data, light performance overhead.
|-
|'''''krb5a'''''||GSS/Kerberos5 ||partly integrity (krb5)||checksum (adler32)||Only header of rpc message is integrity protected, adler32 checksum protection of bulk data, more performance overhead compare to krb5n.
|-
|'''''krb5i'''''||GSS/Kerberos5 ||integrity (krb5)||integrity (krb5)||transformation algorithm is determined by actual Kerberos algorithms in use; Heavy performance penalty.
|-
|'''''krb5p'''''||GSS/Kerberos5 ||privacy (krb5)||privacy (krb5)||transformation privacy protection algorithm is determined by actual Kerberos algorithms in use; The heaviest performance penalty.
|}

'''[*]''' In Lustre 1.4 and 1.6 it is possible to enable bulk data checksumming to provide integrity checking using CRC32. In 1.6.5 this is expected to be the default behaviour, using the Adler32 mechanism by default (lower CPU overhead than CRC32).

In the future, we may want to support customize flavor to some extend. For example, allow set different flavors for RPC message and bulk data.

== Kerberos Setup ==
=== Distribution ===
* We only support MIT Kerberos 5, version from 1.3.x to latest 1.6.x.

=== Configuration ===
1. Configure client nodes:
*For each client node, create a lustre_root principal and generate keytab.
kadmin> addprinc -randkey lustre_root/client_host.domain@REALM
kadmin> ktadd -e aes128-cts:normal lustre_root/client_host.domain@REALM
*Install the keytab on the client node.

2. Configure MDS node:
*For each MDS node, create a lustre_mds principal and generate keytab.
kadmin> addprinc -randkey lustre_mds/mds_host.domain@REALM
kadmin> ktadd -e aes128-cts:normal lustre_mds/mds_host.domain@REALM
*Install the keytab on the MDS node.

3. Configure OSS node:
*For each OSS node, create a lustre_oss principal and generate keytab.
kadmin> addprinc -randkey lustre_oss/oss_host.domain@REALM
kadmin> ktadd -e aes128-cts:normal lustre_oss/oss_host.domain@REALM
*Install the keytab on the OSS node.

NOTES:
*The ''host.domain'' should be the FQDN in your network, otherwise server might not recognize any GSS request.

*As an alternative of the client keytab, if you want to save the trouble of assigning unique keytab for each client node, you can create a general lustre_root principal and its keytab, and install the same keytab on as many client nodes as you want. '''But be aware that in this way one compromised client means all clients are insecure'''.
kadmin> addprinc -randkey lustre_root@REALM
kadmin> ktadd -e aes128-cts:normal lustre_root@REALM

*To merge keytab files, you need the tool '''''ktutil''''', for more details please refers to manual of ktutil.

*Lustre support following ''enctypes'' for MIT Kerberos 5 version 1.4 or higher:
**''des-cbc-md5''
**''des3-hmac-sha1''
**''aes128-cts''
**''aes256-cts''

*For MIT Kerberos 1.3.x, only ''des-cbc-md5'' works because a known issue between libgssapi and Kerberos library.

== Required packages ==
Every node should have follow packages installed:
* '''''libgssapi''''' version 0.10 or higher. Some newer Linux distributions already come with it. If not, build & install from source: http://www.citi.umich.edu/projects/nfsv4/linux/libgssapi/libgssapi-0.11.tar.gz
* '''''keyutils'''''

== Kernel & Environment ==
* System wide configuration:
On Each node (MDT, OST, Client) following line should be added into /etc/fstab to be automatically mounted
nfsd /proc/fs/nfsd nfsd defaults 0 0
Each MDT and Client node add following line into /etc/request-key.conf:
create lgssc * * /usr/sbin/lgss_keyring %o %k %t %d %c %u %g %T %P %S
Note you might need to replace '''/usr/sbin/lgss_keyring''' in above line to the actual path to lgss_keyring binary in your setting.

* Networking:
If you are using network which is '''NOT''' TCP or Infiniband (e.g. Quadrics Elan, Myrinet, etc), you need configure a '''''/etc/lustre/nid2hostname''''' on '''each''' server node (MDT & OST), which is a simple script to translate NID into hostname. Following is sample on a Elan cluster:

#!/bin/bash
set -x
exec 2>/tmp/$(basename $0).debug

# convert a NID for a LND to a hostname, for GSS for example

# called with thre arguments: lnd netid nid
# $lnd will be string "QSWLND", "GMLND", etc.
# $netid will be number in hex string format, like "0x16", etc.
# $nid has the same format as $netid
# output the corresponding hostname, or error message leaded by a '@' for error logging.

lnd=$1
netid=$2
nid=$3

# uppercase the hex
nid=$(echo $nid | tr '[abcdef]' '[ABCDEF]')
# and convert to decimal
nid=$(echo -e "ibase=16\n${nid/#0x}" | bc)
case $lnd in
QSWLND) # simply stick "mtn" on the front
echo "mtn$nid"
;;
*) echo "@unknown LND: $lnd"
;;
esac

== Build Lustre ==
Enable GSS during configuration:

./configure --enable-gss --other-options

== Running ==
=== GSS Daemons ===
Make sure start the daemon process '''lsvcgssd''' on each OST and MDT node before starting Lustre. The command syntax is:
lsvcgssd [-f] [-v]
* ''-f'': running at foreground instead of as daemon, thus output error/warning messages to front console instead of system log.
* ''-v'': increase verbosity by 1. The default is 0, maximum is 4.

=== Setting Security Flavors ===
Note: If nothing specified, by default all RPC connections will use '''''null'''''.

On MGS there's a persistent sptlrpc rule database, by specifying set of rules you can change security flavors between nodes. A rule is in form of:
<spec>=<flavor>
Rules can be manipulated on MGS node. To add a rule:
mgs> lctl conf_param <spec>=<flavor>
If there a existing rule of <spec> part, it will overwritten.

To delete a rule:
mgs> lctl conf_param -d <spec>

Current rule set could be obtained by:
msg> cat /proc/fs/lustre/mgs/<mgs-name>/live/<fs-name> | grep "srpc.flavor"

'''Note''':
* Rules have persistent storage on MGS, so it applied across re-mount.
* It doesn't matter in which order you add a set of rules, lustre keep rules in certain order or priority.
* After you changed a rule, usually it will take the system within 1 minutes to apply the new rules to all nodes, depend on system load.
* Before you change a rule, make sure affected nodes are ready for the new security flavor. E.g. you changed flavor from '''''null''''' to '''''krb5p''''' but GSS/Kerberos env is not properly configured on affected nodes, those nodes might be evicted because they can't communicate with others.
* You can also specify rules via device on-disk parameters, by mke2fs.lustre or tune2fs.lustre. The syntax is the same, and the rule only applied to connections to this specific target (MDT/OST).

=== Rules Syntax & Examples ===
The general syntax is:
<target>.srpc.flavor.<network>[.<direction>]=flavor

* <target>: could be filesystem name, or specific MDT/OST device name. For example, ''lustre'', ''lustre-MDT0000'', ''lustre-OST0001'', etc.
* <network>: LNET network name of the RPC initiator. For example, ''tcp0'', ''elan1'', ''o2ib0''.
* <direction>: could be one of ''cli2mdt'', ''cli2ost'', ''mdt2mdt'', ''mdt2ost''. In most cases you don't need to specify <direction> part.

Examples:
* Apply ''krb5i'' on '''ALL''' connections:
mgs> lctl conf_param lustre.srpc.flavor.default=krb5i

* Nodes in network ''tcp0'' use ''krb5p''; All other nodes use ''null''
mgs> lctl conf_param lustre.srpc.flavor.tcp0=krb5p
mgs> lctl conf_param lustre.srpc.flavor.default=null

* Nodes in network ''tcp0'' use ''krb5p''; Nodes in ''elan1'' use ''plain''; Amount other nodes, clients use ''krb5i'' to MDT/OST, MDT use ''null'' to other MDTs, MDT use ''plain'' to OSTs.
mgs> lctl conf_param lustre.srpc.flavor.tcp0=krb5p
mgs> lctl conf_param lustre.srpc.flavor.elan1=plain
mgs> lctl conf_param lustre.srpc.flavor.default.cli2mdt=krb5i
mgs> lctl conf_param lustre.srpc.flavor.default.cli2ost=krb5i
mgs> lctl conf_param lustre.srpc.flavor.default.mdt2mdt=null
mgs> lctl conf_param lustre.srpc.flavor.default.mdt2ost=plain

=== Authenticate Normal Users ===
On client nodes, a non-root user need '''''kinit''''' before accessing Lustre, just like other Kerberized applications.
* Required by kerberos, the user's principal (''username@REALM'') should be added into KDC.
* Client and MDT nodes should have the same user database, i.e. the user name and uid/gid translation.
A use could destroy the established security contexts before logout, by "lfs flushctx":

lfs flushctx [-k]

Here "-k" means also destroy the on-disk kerberos credential cache, equals to "kdestroy", otherwise it only destroy established contexts in Lustre kernel memory.

== Secure MGC - MGS connection ==
Each node can specify what flavor to use to connect to MGS, by option '''''mgssec=flavor''''' upon mounting a target device or client. By default ''null'' is chosen. Once a flavor is chosen, it can't be changed until umount.

Because each node has only one connection to MGS, so if there's more than one target device or client on a single node, all the "mgssec=" specification must be the same. Or simply missing option "mgssec=" means "using currently chosen flavor.

By default, MGS accept RPCs with any flavor. But sysad can configure MGS to only accept certain flavor from certain network. The syntax is similar but with target as a special "_mgs":
mgs> lctl conf_param _mgs.srpc.flavor.<network>=flavor
'''NOTE:''' apply inappropriate flavor may lead to a node never be able to communicate with MGS until restart. So use it carefully.

== Cross-Realms Authentication ==
Due to idmap functionality is missing, we don't support cross-realm authentication currently.

HPC Software Workshop and Seminars - Regensburg Germany 2009

2011-01-20T18:39:53Z

Sbarthel: /* Lustre Advanced Seminar and Open Storage Track Agenda and Slides */

''(Updated: Sep 2009)''

The three-day [http://hpcworkshop.com HPC Software Workshop] offered three tracks with Sun and customer presentations around Grid Engine, Open Storage (including Lustre™ and SAM-QFS), and software Development Tools, including Sun Studio and Sun HPC ClusterTools.

* The '''Grid Engine''' track was the Grid Engine community meeting for 2009. We discussed our progress since the last year, how our community members are using Grid Engine today, and where we are planning to go in the future.

* Lustre talks in the '''Open Storage''' track provided participants with the opportunity to learn new technical information, acquire best practices, and share knowledge about Lustre technology. Other topics in the '''Open Storage''' track included Sun Open Storage and other Sun open file systems, including ZFS and SAM-QFS.

* The '''Development Tools''' track covered the end-to-end process of developing and deploying applications and services in HPC and cloud computing environments, including Sun HPC ClusterTools and Sun Studio software. Other topics in this track included advanced language research for HPC and cloud computing technologies.

Additionally, three advanced seminars were held as adjuncts to the HPC Software Workshop.

* A special '''Lustre Advanced Technical Seminar''' was offered with senior Lustre engineers. They discussed several areas of the product they are working on, and took questions and provided answers about tuning, configuration, and debugging.

* An '''HPC Parallel Programming Seminar''' described aspects of parallel computing and provided an introduction to the basics of the MPI and OpenMP programming models.

* The '''Grid Engine Advanced Seminar''' was presented to Sun Grid Engine administrators wanting to learn about administration facilities, expert configurations and best practices for complex use case scenarios. The class was conducted by Sun Grid Engine experts and engineers.

====<big>Lustre Advanced Seminar and Open Storage Track Agenda and Slides</big>====

The HPC Software Workshop featured an advanced Lustre seminar and numerous Open Storage Track sessions on select Lustre topics.

----
<big>'''September 7 - Monday'''</big>

The Lustre Advanced Technical Seminar covered topics of interest to Lustre users, and featured the following presentations from our senior-level developers:

[[Media:Shpc-2009-lombardi-lustre-practice.pdf|''Lustre in Practice'']] - Johann Lombardi, Sun

[[Media:Shpc-2009-lombardi-lustre-protocol.pdf|''Some Protocol Basics'']] - Johann Lombardi, Sun

[[Media:Shpc-2009-adilger-workshop.pdf|''Lustre Deep Dive'']] - Andreas Dilger, Sun

----
<big>'''September 8 - Tuesday'''</big>

[[Media:Tuesday_HPC_Workshop_Lustre_Roadmap.pdf|''Lustre Roadmap'']] - Dan Ferber, Sun

[[Media:Tuesday_lustre_automotive.pdf|''Lustre in Automotive'']] - Daniel Kobras, science + computing ag

[[Media:Tuesday_shpc-2009-zfs.pdf|''Lustre, ZFS & End-to-End Data Integrity'']] - Andreas Dilger, Sun

[[Media:Tuesday_CERNLustreEvaluations-SunHPCWorkshop08SEP2009.pdf|''Lustre Investigations at CERN'']] - Arne Wiebalck, CERN

[[Media:Tuesday_shpc-2009-cmd.pdf|''Lustre Clustered Metadata'']] - Andreas Dilger, Sun

[[Media:Tuesday_lustre_and_physics_HPC09.pdf|''Lustre, SGE & Physics Code'']] - Rajmund Krivec, J. Stefan Institute

----
<big>'''September 9 - Wednesday'''</big>

[[Media:Wednesday_Snowbird_Regensburg_TKP.pdf|''Sun Lustre Storage System'']] - Torben Kling-Petersen, Sun

[[Media:Wednesday_Lustre_FZJ.pdf|''Lustre at FZ Juelich - Status and Goals'']] - Otto Büchner, Forschungszentrum, Juelich

[[Media:Wednesday_HPC_Workshop_Lustre_Performance_Sep_2009_Rev2.pdf|''Sample Lustre Performance Data'']] - Dan Ferber, Sun

[[Media:Wednesday_shpc-2009-benchmarking.pdf|''Lustre Performance Tips and Tricks'']] - Torben Kling-Petersen, Sun

[[Media:Wednesday_SAM-QFS.HPC.PUB.pdf|''Data Management with Shared QFS and Storage Archive Manager'']] - Harriet Coverston, Sun

[[Media:Wednesday_Tools.pdf|''Lustre Management & Monitoring Tools'']] - Sven Trautman, Sun

----
<big>'''September 10 - Thursday'''</big>

[[Media:Thursday_Quota.pdf|''Lustre Quota: Current and Future'']] - Johann Lombardi, Sun

[[Media:Thursday_HPC_Workshop_Lustre_Community_Sep_2009_Rev2.pdf|''Lustre Community'']] - Dan Ferber, Sun

HPC Software Workshop and Seminars - Regensburg Germany 2009

2011-01-20T18:32:01Z

Sbarthel: /* Lustre Advanced Seminar and Open Storage Track Agenda and Slides */

''(Updated: Sep 2009)''

The three-day [http://hpcworkshop.com HPC Software Workshop] offered three tracks with Sun and customer presentations around Grid Engine, Open Storage (including Lustre™ and SAM-QFS), and software Development Tools, including Sun Studio and Sun HPC ClusterTools.

* The '''Grid Engine''' track was the Grid Engine community meeting for 2009. We discussed our progress since the last year, how our community members are using Grid Engine today, and where we are planning to go in the future.

* Lustre talks in the '''Open Storage''' track provided participants with the opportunity to learn new technical information, acquire best practices, and share knowledge about Lustre technology. Other topics in the '''Open Storage''' track included Sun Open Storage and other Sun open file systems, including ZFS and SAM-QFS.

* The '''Development Tools''' track covered the end-to-end process of developing and deploying applications and services in HPC and cloud computing environments, including Sun HPC ClusterTools and Sun Studio software. Other topics in this track included advanced language research for HPC and cloud computing technologies.

Additionally, three advanced seminars were held as adjuncts to the HPC Software Workshop.

* A special '''Lustre Advanced Technical Seminar''' was offered with senior Lustre engineers. They discussed several areas of the product they are working on, and took questions and provided answers about tuning, configuration, and debugging.

* An '''HPC Parallel Programming Seminar''' described aspects of parallel computing and provided an introduction to the basics of the MPI and OpenMP programming models.

* The '''Grid Engine Advanced Seminar''' was presented to Sun Grid Engine administrators wanting to learn about administration facilities, expert configurations and best practices for complex use case scenarios. The class was conducted by Sun Grid Engine experts and engineers.

====<big>Lustre Advanced Seminar and Open Storage Track Agenda and Slides</big>====

The HPC Software Workshop featured an advanced Lustre seminar and numerous Open Storage Track sessions on select Lustre topics.

----
<big>'''September 7 - Monday'''</big>

The Lustre Advanced Technical Seminar covered topics of interest to Lustre users, and featured the following presentations from our senior-level developers:

[[Media:Shpc-2009-lombardi-lustre-practice.pdf|''Lustre in Practice'']] - Johann Lombardi, Sun

'''''Lustre: Some Protocol Basics''''' - Johann Lombardi, Sun [[Media:Shpc-2009-lombardi-lustre-protocol.pdf|''- Slides'']]

'''''Lustre Deep Dive''''' - Andreas Dilger, Sun [[Media:Shpc-2009-adilger-workshop.pdf|''- Slides'']]

----
<big>'''September 8 - Tuesday'''</big>

'''''Lustre Roadmap''''' - Dan Ferber, Sun [[Media:Tuesday_HPC_Workshop_Lustre_Roadmap.pdf|''- Slides'']]

'''''Lustre in Automotive''''' - Daniel Kobras, science + computing ag [[Media:Tuesday_lustre_automotive.pdf|''- Slides'']]

'''''Lustre, ZFS & End-to-End Data Integrity''''' - Andreas Dilger, Sun [[Media:Tuesday_shpc-2009-zfs.pdf|''- Slides'']]

'''''Lustre Investigations at CERN''''' - Arne Wiebalck, CERN [[Media:Tuesday_CERNLustreEvaluations-SunHPCWorkshop08SEP2009.pdf|''- Slides'']]

'''''Lustre Clustered Metadata''''' - Andreas Dilger, Sun [[Media:Tuesday_shpc-2009-cmd.pdf|''Slides'']]

'''''Lustre, SGE & Physics Codes''''' - Rajmund Krivec, J. Stefan Institute [[Media:Tuesday_lustre_and_physics_HPC09.pdf|''- Slides'']]

----
<big>'''September 9 - Wednesday'''</big>

'''''Sun Lustre Storage System''''' - Torben Kling-Petersen, Sun [[Media:Wednesday_Snowbird_Regensburg_TKP.pdf|''- Slides'']]

'''''Lustre at FZ Juelich - Status and Goals''''' - Otto Büchner, Forschungszentrum, Juelich [[Media:Wednesday_Lustre_FZJ.pdf|''- Slides'']]

'''''Sample Lustre Performance Data''''' - Dan Ferber, Sun [[Media:Wednesday_HPC_Workshop_Lustre_Performance_Sep_2009_Rev2.pdf|''- Slides'']]

'''''Lustre Performance Tips and Tricks''''' - Torben Kling-Petersen, Sun [[Media:Wednesday_shpc-2009-benchmarking.pdf|''- Slides'']]

'''''Data Management with Shared QFS and Storage Archive Manager (SAM)''''' - Harriet Coverston, Sun [[Media:Wednesday_SAM-QFS.HPC.PUB.pdf|''- Slides'']]

'''''Lustre Management & Monitoring Tools''''' - Sven Trautman, Sun [[Media:Wednesday_Tools.pdf|''- Slides'']]

----
<big>'''September 10 - Thursday'''</big>

'''''Lustre Quota: Current and Future''''' - Johann Lombardi, Sun [[Media:Thursday_Quota.pdf|''- Slides'']]

'''''Lustre Community''''' - Dan Ferber, Sun [[Media:Thursday_HPC_Workshop_Lustre_Community_Sep_2009_Rev2.pdf|''- Slides'']]

Installing Lustre from Downloaded RPMs

2011-01-20T18:29:03Z

Sbarthel:

''(Updated: Jan 2010)''

This procedure describes how to install Lustre™ from the RPM packages. For additional information, see [http://wiki.lustre.org/manual/LustreManual20_HTML/InstallingLustre.html#50438261_pgfId-1292575 Section 8: ''Installing the Lustre Software''] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual''].

'''''Note:''''' In all Lustre installations, the server kernel that runs on an MDS, MGS or OSS must be patched. However, running a patched kernel on a Lustre client is optional and only required if the client will be used for multiple purposes, such as running as both a client and an OST.

'''''Caution:''''' Lustre contains kernel modifications that interact with storage devices and may introduce security issues and data loss if not installed, configured or
administered properly. Before installing Lustre, back up ''ALL'' data.

Complete this procedure to install Lustre from RPMs:

1. ''Verify that all of the Lustre installation requirements have been met.'' For more information on these prerequisites, see [[Preparing to Install Lustre]].

2. ''Download the Lustre RPMs''.

:a. ''On the [http://www.oracle.com/technetwork/indexes/downloads/sun-az-index-095901.html#L Lustre download site], select your platform. The files required'' to install Lustre (kernels, modules and utilities RPMs) will be listed.
:b. ''Download the required files''. Use the Sun Download Manager (SDM) or download the files individually.

3. ''Install the Lustre packages.''
Some Lustre packages are installed on servers (MDS and OSSs), and others are installed on Lustre clients. Lustre packages must be installed in a specific order.

:a. ''For each Lustre package, determine if it needs to be installed on servers and/or clients''. See [[Lustre Packages]] to determine where to install a specific package. Depending on your platform, not all of the listed files need to be installed.

::'''''Caution:''''' For a non-production Lustre environment or for testing, a Lustre client and server can run on the same machine. However, for best performance in a production environment, Lustre must be installed on dedicated systems. Performance and other issues can occur when an MDS or OSS and a client are running on the same machine. The MDS and MGS can run on the same machine.

:b. ''Install the kernel, modules and ''ldiskfs'' packages.'' Use the ''rpm -ivh'' command to install the kernel, module and ldiskfs packages. For example:

:<pre>
;$ rpm -ivh kernel-lustre-smp-<ver> kernel-ib-<ver> lustre-modules-<ver> lustre-ldiskfs-<ver>
</pre>

:c. ''Install the utilities/userspace packages.''Use the ''rpm -ivh'' command to install the utilities packages. For example:

:<pre>
;$ rpm -ivh lustre-<ver>
</pre>

:d. ''Install the'' e2fsprogs ''package.'' Use the ''rpm -i'' command to install the ''e2fsprogs'' package. For example:

:<pre>
;$ rpm -i e2fsprogs-<ver>
</pre>

:'''''Note:''''' If a version of ''e2fsprogs'' is already installed on your Linux system, install the Lustre-specific version of ''e2fsprogs'' using ''rpm -U'' to update the installed ''e2fsprogs'' package. For example:

:<pre>
;$ rpm -U e2fsprogs-<ver>
</pre>

:The ''rpm'' command options ''--force'' or ''--nodeps'' are not required to install or update ''e2fsprogs''. We specifically recommend that you not use these options.

:e. ''(Optional) If you want to add any optional packages to your Lustre file system, install them now.'' Optional packages include file system creation and repair tools, debugging tools, test programs and scripts, Linux kernel and Lustre source code, and other packages. A complete list of optional packages for your platform is provided on the [http://www.oracle.com/technetwork/indexes/downloads/sun-az-index-095901.html#L Lustre download site].

4. ''Verify that the boot loader ''(grub.conf ''or'' lilo.conf) ''has been updated to load the patched kernel.''

5. ''Reboot the servers.''

Proceed to [[Configuring the Lustre File System]] to configure Lustre Networking (LNET) and the Lustre file system.

Learn:Learn

2011-01-20T18:27:09Z

Sbarthel:

Access resources such as information about current and upcoming Lustre features, a roadmap, publications about Lustre and training opportunities.

<div class="categoryLeft">

<big>Interoperability, Features and Roadmap</big>

These resources detail Lustre releases and plans.

* <ins>[[Lustre_Release_Information|Lustre Release Information]]</ins> lists current Lustre releases and supported kernels and networks for these releases.

* <ins>[[Lustre_1.8|Lustre 1.8]]</ins> describes features and benefits offered by upgrading to Lustre 1.8.

* <ins>[[Lustre_2.0|Lustre 2.0]]</ins> describes features being developed for the next-generation release of Lustre.

</div>

<div class="categoryRight">

<big>Publications</big>

Use these resources to learn about Lustre.

* <ins>[[Lustre_Community_Events, Conferences and Meetings|Lustre Presentations]]</ins> by Lustre engineers at Lustre User Group meetings, conferences, Lustre all-hands meetings, and other Lustre-related events.

* <ins>[[Lustre Publications]]</ins> from a variety of sources, including links to videos and podcasts, white papers and blueprints.

* <ins>[http://wiki.lustre.org/manual/LustreManual20_HTML/index.html Lustre Operations Manual]</ins>, the primary source for Lustre user documentation.

</div>

<div class="categoryLeft">

<big>Lustre Projects and Future Features</big>

To learn about active Lustre projects that span releases or features that are not in Lustre 1.8 or Lustre 2.0, see <ins>[[Lustre Projects]]</ins>.

</div>

<div class="categoryRight">

<big>Lustre Internals</big>

* See the <ins>[http://wiki.lustre.org/lid/subsystem-map/subsystem-map.html Lustre Internals Documentation]</ins> for detailed descriptions of the Lustre codebase in an easily accessible format.

* The LID includes <ins>[http://wiki.lustre.org/lid/doxygen.api/modules.html Doxygen API information]</ins> for various Lustre modules.

</div>

<div class="categoryLeft">

<big>Training</big>

Lustre training is available from Oracle University at locations in the <ins>[http://www.sun.com/training/catalog/courses/CL-400.xml United States]</ins>, <ins>[http://uk.sun.com/training/catalog/courses/CL-400.xml Great Britain]</ins>, <ins>[https://www.suntrainingcatalogue.com/eduserv/client/loadCourse.do?coId=de_DE_CL-400&l=de_DE Germany], and [http://au.sun.com/training/catalog/courses/CL-400.xml APAC countries]</ins>.
* <ins>[http://www.sun.com/training/catalog/courses/CL-400.xml Administering Lustre Based Clusters (CL-400)]</ins>

</div>

Learn:Learn

2011-01-20T18:26:44Z

Sbarthel:

Access resources such as information about current and upcoming Lustre features, a roadmap, publications about Lustre and training opportunities.

<div class="categoryLeft">

<big>Interoperability, Features and Roadmap</big>

These resources detail Lustre releases and plans.

* <ins>[[Lustre_Release_Information|Lustre Release Information]]</ins> lists current Lustre releases and supported kernels and networks for these releases.

* <ins>[[Lustre_1.8|Lustre 1.8]]</ins> describes features and benefits offered by upgrading to Lustre 1.8.

* <ins>[[Lustre_2.0|Lustre 2.0]]</ins> describes features being developed for the next-generation release of Lustre.

</div>

<div class="categoryRight">

<big>Publications</big>

Use these resources to learn about Lustre.

* <ins>[[Lustre_Community_Events, Conferences and Meetings|Lustre Presentations]]</ins> by Lustre engineers at Lustre User Group meetings, conferences, Lustre all-hands meetings, and other Lustre-related events.

* <ins>[[Lustre Publications]]</ins> from a variety of sources, including links to videos and podcasts, white papers and blueprints.

* <ins>[http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual'']</ins>, the primary source for Lustre user documentation.

</div>

<div class="categoryLeft">

<big>Lustre Projects and Future Features</big>

To learn about active Lustre projects that span releases or features that are not in Lustre 1.8 or Lustre 2.0, see <ins>[[Lustre Projects]]</ins>.

</div>

<div class="categoryRight">

<big>Lustre Internals</big>

* See the <ins>[http://wiki.lustre.org/lid/subsystem-map/subsystem-map.html Lustre Internals Documentation]</ins> for detailed descriptions of the Lustre codebase in an easily accessible format.

* The LID includes <ins>[http://wiki.lustre.org/lid/doxygen.api/modules.html Doxygen API information]</ins> for various Lustre modules.

</div>

<div class="categoryLeft">

<big>Training</big>

Lustre training is available from Oracle University at locations in the <ins>[http://www.sun.com/training/catalog/courses/CL-400.xml United States]</ins>, <ins>[http://uk.sun.com/training/catalog/courses/CL-400.xml Great Britain]</ins>, <ins>[https://www.suntrainingcatalogue.com/eduserv/client/loadCourse.do?coId=de_DE_CL-400&l=de_DE Germany], and [http://au.sun.com/training/catalog/courses/CL-400.xml APAC countries]</ins>.
* <ins>[http://www.sun.com/training/catalog/courses/CL-400.xml Administering Lustre Based Clusters (CL-400)]</ins>

</div>

Lustre 1.8

2011-01-20T18:24:55Z

Sbarthel: /* Version-Based Recovery */

''(Updated: Jan 2010)''
__TOC__
Lustre™ 1.8.0.1 introduces several robust, new features and improved system functionality. This page provides feature descriptions and lists the benefits offered by upgrading to the Lustre 1.8 release branch. The change log and release notes are [[Change_Log_1.8|here]].

==Adaptive Timeouts==

The adaptive timeouts feature (enabled, by default) causes Lustre to use an adaptive mechanism to set RPC timeouts, so users no longer have to tune the obd_timeout value. RPC service time histories are tracked on all servers for each service, and estimates for future RPCs are reported back to clients. Clients use these service time estimates along with their own observations of the network delays to set future RPC timeout values.

If server request processing slows down, its estimates increase and the clients allow more time for RPC completion before retrying. If RPCs queued up on the server approach their timeouts, the server sends early replies to the client, telling it to allow more time. Conversely, as the load on the server is reduced, the RPC timeout values decrease, allowing faster client detection of non-responsive servers and faster attempts to reconnect to a server's failover partner.

<big>Why should I upgrade to Lustre 1.8 to get it?</big>

Adaptive timeouts offers these benefits:

* Simplified management for small and large clusters.
* Automatically adjusts RPC timeouts as network conditions and server load changes.
* Reduces server recovery time in cases where the server load is low at time of failure.

<big>Additional Resources</big>

For more information about adaptive timeouts, see:

* [[Architecture - Adaptive_Timeouts_-_Use_Cases|Architecture page - Adaptive timeouts (use cases)]]
* [[Media:Adaptive-timeouts-hld.pdf|HLD - Adaptive RPC timeouts]]

==Client Interoperability==

The client interoperability feature enables Lustre 1.8 clients to work with a new network protocol that will be introduced in Lustre 2.0. This feature allows transparent client, server, network and storage interoperability when migrating from 1.6 architecture-based clusters to clusters with 2.0 architecture-based servers. Lustre 1.8.x clients will interoperate with Lustre 2.0 servers.

<big>Why should I upgrade to Lustre 1.8 to get it?</big>

Client interoperability offers this benefit:

* When Lustre 2.x is released, Lustre 1.8.x users will be able to upgrade to 2.x servers while the Lustre filesystem is up and running. This transparent upgrade feature will enable users to upgrade their servers to Lustre 2.x and reboot them without disturbing applications using the filesystem on clients. It will no longer be necessary to unmount clients from the filesystem to upgrade servers to the new software. After the 2.x upgrade, Lustre 2.x servers will interoperate with 1.8.x clients.

<big>Additional Resources</big>

For more information on client interoperability, see:

* [[Architecture - Interoperability_fids_zfs|Architecture page - Interoperability FIDs and ZFS]]
* [[Media:Interop_disk_fidea.pdf|HLD - Interoperability at the Server Side]]
* [[Media:Sptlrpc_interop-hld.pdf|HLD - Sptlrpc interoperability]]
* [[Media:Interop-client-recov-dld.pdf|DLD - Interoperable Client Recovery]]
* [[Media:Sptlrpc_interop-dld.pdf|DLD - Sptlrpc interoperability]]

==OSS Read Cache==

The OSS read cache feature provides read-only caching of data on an OSS. It uses a regular Linux pagecache to store the data. OSS read cache improves Lustre performance when several clients access the same data set, and the data fits the OSS cache (which can occupy most of the available memory). The overhead of OSS read cache is very low on modern CPUs, and cache misses do not negatively impact performance compared to Lustre releases before OSS read cache was available.

<big>Why should I upgrade to Lustre 1.8 to get it?</big>

OSS read cache can improve Lustre performance, and offers these benefits:

* Allows OSTs to cache read data more frequently
* Improves repeated reads to match network speeds instead of disk speeds
* Provides the building block for OST write cache (small write aggregation)

<big>Additional Resources</big>

For more information on OSS read cache, see:

* [[Architecture - Caching_OSS|Architecture page - Caching OSS]]

==OST Pools==

The OST pools feature allows the administrator to name a group of OSTs for file striping purposes. For instance, a group of local OSTs could be defined for faster access; a group of higher-performance OSTs could be defined for specific applications; a group of non-RAID OSTs could be defined for scratch files; or groups of OSTs could be defined for particular users.

Pools are defined by the system administrator, using regular Lustre tools (lctl). Pool usage is specified and stored along with other striping information
(e.g., stripe count, stripe size) for directories or individual files (lfs
setstripe or llapi_create_file()). Traditional automated OST selection
optimizations (QOS) occur within a pool (e.g., free-space leveling within
the pool). OSTs can be added or removed from a pool at any time (and existing
files always remain in place and available.)

OST pools characteristics include:

* An OST can be associated with multiple pools
* No ordering of OSTs is implied or defined within a pool
* OST membership in a pool can change over time
* a directory can default to a specific pool and new files/subdirectories created therein will use that pool

'''NOTE:''' In its current implementation, the OST pools feature does not implement an automated policy or restrict users from creating files in any of the pools; it must be managed directly by administrator/user. It is a building block for policy-managed storage.

<big>Why should I upgrade to Lustre 1.8 to get it?</big>

OST pools offers these benefits:

* Allows sets of OSTs to be managed via named groups
* Pools can separate heterogeneous OSTs within the same filesystem
** Fast vs. slow disks
** Local network vs. remote network (e.g. WAN)
** RAID 1 vs. RAID5 backing storage, etc.
** Specific OSTs for users/groups/applications (by directory)
* Easier disk usage policy implementation for administrators
* Hardware can be more closely optimized for particular usage patterns
* Human-readable stripe mappings

<big>Additional Resources</big>

For more information on OST pools, see:

* [[Architecture - Pools_of_targets|Architecture page - OST pools]]
* [[Media:OstPools-DLD.pdf|DLD - OST Pools]]
* [[Media:Ostpools-large-scale_testplan.pdf|Test plan - OST pools]]

==Version-Based Recovery==

Version-based Recovery (VBR) improves the robustness of client recovery operations and allows Lustre to recover, even if multiple clients fail at the same time as the server. With VBR, recovery is more flexible; not all clients are evicted if some miss recovery, and a missed client may try to recover after the server recovery window.

<big>Why should I upgrade to Lustre 1.8 to get it?</big>

VBR functionality in Lustre 1.8 allows more flexible recovery after a failure. Previous Lustre releases enforced a strict, in-order replay condition that required all clients to reconnect during the recovery period. If a client was missing and the recovery period timed out, then the remaining clients were evicted. With VBR, conditional out-of-order replay is allowed. VBR uses versions to detect conflicting transactions. If an object's version matches what is expected, the transaction is replayed. If there is a version mis-match, clients attempting to modify the object are stopped. Recovery continues even if some clients do not reconnect (the missed clients can try to recover later). With VBR, Lustre clients may successfully recover in a wider variety of failure scenarios.

VBR offers these benefits:

* Improves the robustness of client recovery operations
* Allows Lustre recovery to continue even if multiple clients fail at the same time as the server
* Provides a building block for disconnected client operations

<big>Additional Resources</big>

For more information on VBR, see:

* [http://wiki.lustre.org/manual/LustreManual20_HTML/LustreRecovery.html#50438268_pgfId-1287769 Section 30.4: ''Version-based Recovery''] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual''].
* [[Architecture - Version_Based_Recovery|Architecture page - VBR]]
* [[Media:20080612165106%21Version_base_recovery-hld.pdf|HLD - VBR]]
* [[Media:Version_recovery.pdf|DLD - VBR]]
* [[Media:VBR_phase2_large_scale_testplan.pdf|Test plan - VBR]]

Lustre 2.0 Features

2011-01-20T18:23:01Z

Sbarthel:

''(Updated: Nov 2009)''
__TOC__
[[Lustre_2.0|Lustre 2.0]] release introduced several significant new features and improved system functionality. This page provides descriptions of these features and lists the benefits offered by upgrading to the Lustre 2.0 release family.

=Lustre 2.0.0=

The initial [[Lustre 2.0]] release (known as 2.0.0) offers these features:

===Changelogs===

Changelogs record events that change the filesystem namespace or file metadata. Events such as file creation, deletion, renaming, attribute changes, etc. are recorded with the target and parent file identifiers (FIDs), the name of the target, and a timestamp. These records can be used for a variety of purposes:

* Record recent changes to feed into an archiving system.
* Use changelog entries to exactly replicate changes in a filesystem mirror.
* Set up "watch scripts" that take action on certain events or directories. Changelog record are persistent (on disk) until explicitly cleared by the user. The are guaranteed to accurately reflect on-disk changes in the event of a server failure.
* Maintain a rough audit trail (file/directory changes with timestamps, but no user information).

These are sample changelog entries:

2 02MKDIR 4298396676 0x0 t=[0x200000405:0x15f9:0x0] p=[0x13:0x15e5a7a3:0x0] pics
3 01CREAT 4298402264 0x0 t=[0x200000405:0x15fa:0x0] p=[0x200000405:0x15f9:0x0] chloe.jpg
4 06UNLNK 4298404466 0x0 t=[0x200000405:0x15fa:0x0] p=[0x200000405:0x15f9:0x0] chloe.jpg
5 07RMDIR 4298405394 0x0 t=[0x200000405:0x15f9:0x0] p=[0x13:0x15e5a7a3:0x0] pics

The record types are:

{| border=1 cellpadding=0
!Record Type
!Description
|-
|MARK|||internal recordkeeping
|-
|CREAT|||regular file creation
|-
|MKDIR|||directory creation
|-
|HLINK|||hardlink
|-
|SLINK|||softlink
|-
|MKNOD|||other file creation
|-
|UNLNK|||regular file removal
|-
|RMDIR|||directory removal
|-
|RNMFM|||rename, original
|-
|RNMTO|||rename, final
|-
|IOCTL|||ioctl on file or directory
|-
|TRUNC|||regular file truncated
|-
|SATTR|||attribute change
|-
|XATTR|||extended attribute change
|-
|UNKNW|||unknown op
|}

FID-to-full-pathname and pathname-to-FID functions are also included to map target and parent FIDs into the filesystem namespace.

<big>Why should I upgrade to Lustre 2.0.0 to get it?</big>

Changelogs offer these benefits:

* File/directory change notification
* Event notification
* Filesystem replication
* File backup policy decisions
* Audit trail

<big>Additional Resources</big>

For more information about changelogs, see:

* [http://wiki.lustre.org/manual/LustreManual20_HTML/LustreMonitoring.html#50438273_pgfId-1296751 Section 12.1: Changelogs - Lustre 2.0 manual]

===Commit on Share===

The Commit on Share (COS) feature prevents missing clients from causing cascading evictions of other clients. If some clients miss the recovery window, remaining clients are not evicted.

When an MDS starts up and enters recovery mode after a failover or service restart, clients begin to reconnect and replay their uncommitted transactions. If one or more clients miss the recovery window, this may cause other clients to abort their transactions or be evicted. The transactions of evicted clients cannot be applied and are aborted. This causes a cascade effect as transactions dependent on the aborted ones fail and so on. COS addresses this problem by eliminating dependent transactions. With no dependent, uncommitted transactions to apply, the clients replay their requests independently without the risk of being evicted.

<big>Why should I upgrade to Lustre 2.0.0 to get it?</big>

COS offers these benefits:

* Allows clients to always be able to recover, regardless of whether other clients have failed.
* Reduces recovery problems when multiple node failures occur

<big>Additional Resources</big>

For more information on COS, see:

* [http://wiki.lustre.org/manual/LustreManual20_HTML/LustreRecovery.html#50438268_pgfId-1292073 Section 30.5: Commit on Share - Lustre 2.0 manual]
* [[Media:COS_TestPlan.pdf|COS Test Plan]]

===lustre_rsync===

The lustre_rsync feature provides namespace and data replication to an external (remote) backup system without having to scan the file system for inode changes and modification times. Lustre metadata changelogs are used to record file system changes and determine which directory and file operations to execute on the replicated system. The lustre_rsync feature differs from existing backup/replication/synchronization systems because it avoids full file system scans, which can be unreasonably time-consuming for very large file systems. Also, the lustre_rsync process can be resumed from where it left off, so the replicated file system is fully synchronized when operation completes. Lustre_rsync may be bi-directional for distinct directories.

The replicated system may be another Lustre file system or any other file system. The replica is an exact copy of the namespace of the original file system at a given point in time. However, the replicated file system is '''not''' a snapshot of the source file system in that its contents may differ from the original file system's contents. On the replicated file system, a file's contents will be the data in the file at the time the file transfer occurred.

<big>Why should I upgrade to Lustre 2.0.0 to get it?</big>

Lustre_rsync offers these benefits:

* Namespace-coherent duplication of large file systems without scanning the complete file system
* Functionality is safe when run repeatedly or run after an aborted attempt
* Synchronization facility to switch the role of source and target file systems
* In the case of recovery, the feature provides for reverse replication

<big>Additional Resources</big>

For more information on replication, see:

* [http://wiki.lustre.org/manual/LustreManual20_HTML/SystemConfigurationUtilities_HTML.html#50438219_pgfId-1317225 Section 36.13: Lustre_rsync - Lustre 2.0 manual]

Lustre 2.0 Features

2011-01-20T18:19:23Z

Sbarthel:

''(Updated: Nov 2009)''
__TOC__
[[Lustre_2.0|Lustre 2.0]] release introduced several significant new features and improved system functionality. This page provides descriptions of these features and lists the benefits offered by upgrading to the Lustre 2.0 release family.

=Lustre 2.0.0=

The initial [[Lustre 2.0]] release (known as 2.0.0) offers these features:

===Changelogs===

Changelogs record events that change the filesystem namespace or file metadata. Events such as file creation, deletion, renaming, attribute changes, etc. are recorded with the target and parent file identifiers (FIDs), the name of the target, and a timestamp. These records can be used for a variety of purposes:

* Record recent changes to feed into an archiving system.
* Use changelog entries to exactly replicate changes in a filesystem mirror.
* Set up "watch scripts" that take action on certain events or directories. Changelog record are persistent (on disk) until explicitly cleared by the user. The are guaranteed to accurately reflect on-disk changes in the event of a server failure.
* Maintain a rough audit trail (file/directory changes with timestamps, but no user information).

These are sample changelog entries:

2 02MKDIR 4298396676 0x0 t=[0x200000405:0x15f9:0x0] p=[0x13:0x15e5a7a3:0x0] pics
3 01CREAT 4298402264 0x0 t=[0x200000405:0x15fa:0x0] p=[0x200000405:0x15f9:0x0] chloe.jpg
4 06UNLNK 4298404466 0x0 t=[0x200000405:0x15fa:0x0] p=[0x200000405:0x15f9:0x0] chloe.jpg
5 07RMDIR 4298405394 0x0 t=[0x200000405:0x15f9:0x0] p=[0x13:0x15e5a7a3:0x0] pics

The record types are:

{| border=1 cellpadding=0
!Record Type
!Description
|-
|MARK|||internal recordkeeping
|-
|CREAT|||regular file creation
|-
|MKDIR|||directory creation
|-
|HLINK|||hardlink
|-
|SLINK|||softlink
|-
|MKNOD|||other file creation
|-
|UNLNK|||regular file removal
|-
|RMDIR|||directory removal
|-
|RNMFM|||rename, original
|-
|RNMTO|||rename, final
|-
|IOCTL|||ioctl on file or directory
|-
|TRUNC|||regular file truncated
|-
|SATTR|||attribute change
|-
|XATTR|||extended attribute change
|-
|UNKNW|||unknown op
|}

FID-to-full-pathname and pathname-to-FID functions are also included to map target and parent FIDs into the filesystem namespace.

<big>Why should I upgrade to Lustre 2.0.0 to get it?</big>

Changelogs offer these benefits:

* File/directory change notification
* Event notification
* Filesystem replication
* File backup policy decisions
* Audit trail

<big>Additional Resources</big>

For more information about changelogs, see:

* [http://wiki.lustre.org/manual/LustreManual20_HTML/LustreMonitoring.html#50618497_pgfId-1296751 Changelogs Topic - Lustre 2.0 manual]

===Commit on Share===

The Commit on Share (COS) feature prevents missing clients from causing cascading evictions of other clients. If some clients miss the recovery window, remaining clients are not evicted.

When an MDS starts up and enters recovery mode after a failover or service restart, clients begin to reconnect and replay their uncommitted transactions. If one or more clients miss the recovery window, this may cause other clients to abort their transactions or be evicted. The transactions of evicted clients cannot be applied and are aborted. This causes a cascade effect as transactions dependent on the aborted ones fail and so on. COS addresses this problem by eliminating dependent transactions. With no dependent, uncommitted transactions to apply, the clients replay their requests independently without the risk of being evicted.

<big>Why should I upgrade to Lustre 2.0.0 to get it?</big>

COS offers these benefits:

* Allows clients to always be able to recover, regardless of whether other clients have failed.
* Reduces recovery problems when multiple node failures occur

<big>Additional Resources</big>

For more information on COS, see:

* [http://wiki.lustre.org/manual/LustreManual20_HTML/LustreRecovery.html#50618492_pgfId-1292073 Commit on Share Topic - Lustre 2.0 manual]
* [[Media:COS_TestPlan.pdf|COS Test Plan]]

===lustre_rsync===

The lustre_rsync feature provides namespace and data replication to an external (remote) backup system without having to scan the file system for inode changes and modification times. Lustre metadata changelogs are used to record file system changes and determine which directory and file operations to execute on the replicated system. The lustre_rsync feature differs from existing backup/replication/synchronization systems because it avoids full file system scans, which can be unreasonably time-consuming for very large file systems. Also, the lustre_rsync process can be resumed from where it left off, so the replicated file system is fully synchronized when operation completes. Lustre_rsync may be bi-directional for distinct directories.

The replicated system may be another Lustre file system or any other file system. The replica is an exact copy of the namespace of the original file system at a given point in time. However, the replicated file system is '''not''' a snapshot of the source file system in that its contents may differ from the original file system's contents. On the replicated file system, a file's contents will be the data in the file at the time the file transfer occurred.

<big>Why should I upgrade to Lustre 2.0.0 to get it?</big>

Lustre_rsync offers these benefits:

* Namespace-coherent duplication of large file systems without scanning the complete file system
* Functionality is safe when run repeatedly or run after an aborted attempt
* Synchronization facility to switch the role of source and target file systems
* In the case of recovery, the feature provides for reverse replication

<big>Additional Resources</big>

For more information on replication, see:

* [http://wiki.lustre.org/manual/LustreManual20_HTML/BackupAndRestore.html#50618423_pgfId-1293842 Lustre_rsync Topic - Lustre 2.0 manual]

Lustre 2.0 Release Milestone Status

2011-01-20T18:18:37Z

Sbarthel: /* Lustre 2.0 Beta-1 April 15, 2010 */

''(Updated: Apr 2010)''
__TOC__
For the the upcoming Lustre™ 2.0 GA release, each alpha or beta milestone achieved is summarized below, including the focus of the milestone, testing results, a link to that release of the Lustre software, and links to additional documentation.

You can get to the latest milestone release from: http://downloads.lustre.org/public/lustre/v2.0/latest/

==Lustre 2.0 Beta-1 April 15, 2010==

The focus of the first Beta release (Beta-1) of 2.0 was to continue improve stability of Lustre while landing fixes to HEAD and completing additional bug fixes. 116 total fixes were landed during this cycle.

This release was tested on RHEL5/x86_64 and OEL5/x86_64 with both IB and TCP connectivity. Known failures are documented in the HEAD Daily Testing Document available from lustre.org.

The Beta-1 release of Lustre 2.0 can be downloaded from http://downloads.lustre.org/public/lustre/v2.0/beta/Lustre_2.0_Beta1/

The following documentation is available related to the Beta-1 release:
* [[Media:Lustre2.0Beata1Summary.pdf|Lustre 2.0 Beta-1 Summary]] includes: goals, timelines, fixes landed and milestone outcomes
* [[Media:HEADDailyTestingResults-Beta1.pdf|Lustre 2.0 Beta-1 Daily HEAD Testing Results]] includes: test pass/fail status and bugs found in each daily testing run for the duration of the Beta-1 Milestone
* Lustre 2.0 Operations Manual includes: installation, configuration, administration, troubleshooting and reference information - [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html HTML] and [http://wiki.lustre.org/images/3/35/821-2076-10.pdf PDF].

==Lustre 2.0 Alpha-7 February 10, 2010==

The focus of the seventh Alpha release (Alpha-7) of 2.0 was to continue to improve stability of Lustre through landing of fixes to the main trunk of the Lustre code base. 157 total fixes were landed during this cycle.

This release was tested on RHEL5/x86_64 and OEL5/x86_64 with both IB and TCP connectivity. Known failures are documented in the HEAD Daily Testing Document available from lustre.org.

The Alpha-7 release of Lustre 2.0 can be downloaded from http://downloads.lustre.org/public/lustre/v2.0/alpha/Lustre_2.0_Alpha7/

The following documentation is available related to the Alpha-7 release:
* [[Media:Lustre2.0Alpha7Summary-2.pdf|Lustre 2.0 Alpha-7 Summary]] includes: goals, timelines, fixes landed and milestone outcomes
* [[Media:HeadDailyTesting-Alpha-7.pdf|Lustre 2.0 Alpha-7 Daily HEAD Testing Results]] includes: test pass/fail status and bugs found in each daily testing run for the duration of the Alpha-7 Milestone

==Lustre 2.0 Alpha-6 December 2, 2009==

The focus of the sixth Alpha release (Alpha-6) of 2.0 was to continue to improve the stability of Lustre while landing fixes to HEAD and completing additional bug fixes. 104 total fixes were landed during this cycle.

This release was tested on RHEL5/x86_64 servers with both IB and TCP connectivity. Known failures are documented in the HEAD Daily Testing Document available [[Media:HEADDailyTestingResults-Alpha6.pdf|here]].

The Alpha-6 release of Lustre 2.0 can be downloaded from http://downloads.lustre.org/public/lustre/v2.0/alpha/Lustre_2.0_Alpha6/

The following documentation is available related to the Alpha-6 release:
* [[Media:Lustre2.0Alpha6Summary.pdf|Lustre 2.0 Alpha-6 Summary]] includes: goals, timelines, fixes landed and milestone outcomes
* [[Media:HEADDailyTestingResults-Alpha6.pdf|Lustre 2.0 Alpha-6 Daily HEAD Testing Results]] includes: test pass/fail status and bugs found in each daily testing run for the duration of the Alpha-6 Milestone

==Lustre 2.0 Alpha-5 October 14, 2009==

The focus of the fifth Alpha release (Alpha-5) of 2.0 was to continue to improve the stability of Lustre while landing fixes to HEAD and completing additional bug fixes. 155 total fixes were landed during this cycle.

This release was tested on RHEL5/x86_64 with both IB and TCP connectivity. Known failures are documented in the HEAD Daily Testing Document available [[Media:HEADDailyTestingResults-Alpha5.pdf|here]].

The Alpha-5 release of Lustre 2.0 can be downloaded from http://downloads.lustre.org/public/lustre/v2.0/alpha/Lustre_2.0_Alpha5/

The following documentation is available related to the Alpha-5 release:
*[[Media:Lustre2.0Alpha5Summary.pdf|Lustre 2.0 Alpha-5 Summary]] includes: goals, timelines, fixes landed and milestone outcomes
*[[Media:HEADDailyTestingResults-Alpha5.pdf|Lustre 2.0 Alpha-5 Daily HEAD Testing Results]] includes: test pass/fail status and bugs found in each daily testing run for the duration of the Alpha-5 Milestone

==Lustre 2.0 Alpha-4 July 1, 2009==

The focus of the fourth Alpha release (Alpha-4) of 2.0 was to continue to improve stability of Lustre while landing fixes to HEAD and completing additional bug fixes. 120 total fixes were landed during this cycle.

This release was tested on RHEL5/x86_64 and SLES 10/x86_64 with both IB and TCP connectivity. Known failures are documented in the HEAD Daily Testing Document available [[Media:HEADDailyTestingResults-Alpha4.pdf|here]].

This Alpha-4 release of Lustre 2.0 can be downloaded from http://downloads.lustre.org/public/lustre/v2.0/alpha/Lustre_2.0_Alpha4/

The following documentation is available related to the Alpha-4 release:
*[[Media:Lustre2.0Alpha4Summary.pdf|Lustre 2.0 Alpha-4 Summary]] includes: goals, timelines, fixes landed and milestone outcomes
*[[Media:HEADDailyTestingResults-Alpha4.pdf|Lustre 2.0 Alpha-4 Daily HEAD Testing Results]] includes: test pass/fail status and bugs found in each daily testing run for the duration of the Alpha-4 Milestone

==Lustre 2.0 Alpha-3 June 10, 2009==

The focus of the third Alpha release (Alpha-3) of 2.0 was to continue to improve stability of Lustre while landing queued fixes to HEAD and completing additional bug fixes. 95 total fixes were landed during this cycle. Notable check-ins during the period:
*Final patches for Size on MDS (SOM) preview were checked-in (Bug 1028)
*OFED 1.4.1 support was landed

This release was tested on RHEL5/x86_64 and SLES 10/x86_64 with both IB and TCP connectivity. Known failures are documented in the HEAD Daily Testing Document available [[Media:HEADDailyTestingResults-Alpha3.pdf|here]]. Additionally, the backlog of HEAD patches that were queued for landing once HEAD was stabilized has been worked through and patches have been landed.

This Alpha-3 release of Lustre 2.0 can be downloaded from http://downloads.lustre.org/public/lustre/v2.0/alpha/Lustre_2.0_Alpha3/

The following documentation is available related to the Alpha-3 release:
*[[Media:Lustre2.0Alpha3Summary.pdf|Lustre 2.0 Alpha-3 Summary]] includes: goals, timelines, fixes landed and milestone outcomes
*[[Media:HEADDailyTestingResults-Alpha3.pdf|Lustre 2.0 Alpha-3 Daily HEAD Testing Results]] includes: test pass/fail status and bugs found in each daily testing run for the duration of the Alpha-3 Milestone

==Lustre 2.0 Alpha-2 May 12, 2009==

The focus of the second Alpha release (Alpha-2) of 2.0 was to continue improve stability of Lustre while landing queued fixes to HEAD and completing additional bug fixes. 85 total fixes were landed during this cycle.

This release was tested on RHEL5/x86_64 and SLES 10/x86_64 with both IB and TCP connectivity. Known failures are documented in the HEAD Daily Testing Document available [[Media:HEADDailyTestingResults-Alpha2.pdf|here]]. Additionally, IOR and simul runs were executed iteratively on a modest cluster (approx 75 clients) and run for 18 hours successfully with iSCSI attached HW in file per process mode. IOR and Simul were tested on a larger direct attach cluster and IOR has run successfully for 4 hours.

This Alpha-2 release of Lustre 2.0 can be downloaded from http://downloads.lustre.org/public/lustre/v2.0/alpha/Lustre_2.0_Alpha2/

The following documentation is available related to the Alpha-2 release:
*[[Media:Lustre2.0Alpha2Summary.pdf|Lustre 2.0 Alpha-2 Summary]] includes: goals, timelines, fixes landed and milestone outcomes
*[[Media:HEADDailyTestingResults-Alpha2.pdf|Lustre 2.0 Alpha-2 Daily HEAD Testing Results]] includes: test pass/fail status and bugs found in each daily testing run for the duration of the Alpha-2 Milestone

==Lustre 2.0 Alpha April 16, 2009==

The goal of the first Alpha release of Lustre 2.0 was to demonstrate basic stability of Lustre on a single platform/distro. For this release, RHEL5/x86_64 was selected because it is the most often downloaded distribution from lustre.org.

To achieve our goal, daily ''acc-sm'' runs were executed on small test clusters and bug fixing priority focused on fixing bugs that prevented clean ''acc-sm'' runs. Additionally, IOR and simul runs were executed on a modest cluster (approx 75 clients) to verify basic stability. During this Alpha phase, only bug fixes advancing stability were landed. All other HEAD landing requests were held and deferred to a later landing date post Alpha.

This first alpha release of Lustre 2.0 can be downloaded from http://downloads.lustre.org/public/lustre/v2.0/alpha/Lustre_2.0_Alpha/

The following documentation is available related to the Alpha-1 release:
*[[Media:Lustre2.0AlphaSummary.pdf|Lustre 2.0 Alpha Summary]] includes: goals, timelines, fixes landed and milestone outcomes
*[[Media:Lustre-20-alpha-test-plan.pdf|Lustre 2.0 Alpha Test Plan]]
*[[Media:HEADDailyTestingResults.pdf|Daily HEAD Testing Results]] includes: test pass/fail status and bugs found in each testing run

Lustre All-Hands Meeting 12/08

2011-01-20T18:17:03Z

Sbarthel:

Once a year, the Lustre™ Engineering team gathers to discuss new features under development and testing efforts. This week-long event is known as the Lustre all-hands meeting. The Development presentations made at the December 2008 all-hands meeting are available here:

* [[media:Simplified_InteropRecovery.pdf|''Simplified Interoperability Recovery'']] - Huang Hua
* [[media:RecoveryTalk_2009.pdf|''Recovery Overview'']] - Robert Read
* [[media:Quotas-TOI.pdf|''Quotas-TOI'']] - Yong Fan
* [[media:QualityInitiativeTalk.pdf|''Quality Initiative Talk'']] - Robert Read
* [[media:OST_Pools.pdf|''OST Pools'']] - Nathan Rutman
* [[media:OST_Migration_RAID1_SNS.pdf|''OST Migration RAID1 SNS'']] - Andreas Dilger
* [[media:NRS.pdf|''Lustre NRS Simulation'']] - Yingjin Qian, Wang Di
* [[media:LustreInterop_1_8.pdf|''Lustre Interoperability 1.8'']] - Huang Hua
* [[media:HDFS.pdf|''HDFS'']] - Wang Di
* [[media:GIT_Overview.pdf|''GIT Overview'']] - Robert Read
* [[media:COS-TOI.pdf|''COS-TOI'']] - Alexander Zarochentsev
* [[media:CLIO-TOI.pdf|''CLIO-TOI'']] - Nikita Danilov
* [[media:CLIO-TOI-notes.pdf|''CLIO-TOI-notes'']] - Nikita Danilov
* [[media:CLIO.pdf|''CLIO'']] - Nikita Danilov

Lustre All-Hands Meeting 3/08

2011-01-20T18:15:33Z

Sbarthel:

Once a year, the Lustre™ Engineering team gathers to discuss new features under development and testing efforts. This week-long event is known as the Lustre all-hands meeting. The Development and QE presentations made at the March 2008 all-hands meeting are available here:

== Development ==
* [[Media:RMG-process-0308.pdf|''RMG Processes'']] - Andreas Dilger
* [[Media:Eeb-launch-0308.pdf|''Lustre Development Strategy'']] - Eric Barton
* [[Media:Mdt.pdf|''HEAD MDS'']] - Nikita Danilov
* [[Media:Lnet-0308.pdf|''LNET'']] - Issac Huang

== QE ==
* [[Media:Jd.day1-0308.pdf|''Day 1'']] - [[Media:Jd.day2-0308.pdf|''Day 2'']] - JD
* [[Media:Lustre-release%26weekly-testing-030.pdf|''Lustre Release & weekly testing'']] - Jian Yu
* [[Media:Buildsys-0308.pdf|''Build System'']] - Yibin Wang
* [[Media:Head-testing-0308.pdf''|HEAD testing'']] - Zheng Chen
* [[Media:Latest.b1_6-0308.pdf|''b1.6 testing'']] - Peng Ye
* [[Media:Perf.testing-0308.pdf|''Performance testing'']] - Jack Chen

Lustre Center of Excellence at Oak Ridge National Laboratory

2011-01-20T18:14:04Z

Sbarthel:

''(Updated: Dec 2009)''
__TOC__
The Lustre™ Center of Excellence (LCE) at ORNL advances the state of Lustre for use in large scale HPC environments. As you'll see from the projects below, the ORNL LCE focuses on both systems and applications aspects of ensuring that Lustre meets the needs of DOE and the HPC community in general.

==Events==

The LCE sponsors events to encourage HPC community involvement
in analyzing IO and storage requirements and identifying ways for
Lustre to address these requirements.

===LCE Summit - February 2008===
*[[Media:LCE_Summit_Summary_Draft_March_14_2008.pdf|''LCE Summit Meeting Summary'']]
*[[Media:LCESummitSlides.pdf|''LCE Summit Slides and Notes'']]

===LCE Application I/O Workshop - April 16, 2008===
*[[Media:April2008ApplicationIOWorkshop.pdf|''Application IO Workshop, Agenda and Notes'']]
*[[Media:Lustre_workshop_WangDi.pdf|''Lustre and Application IO - Wang Di's Slides'']]
*[[Media:Lustre_workshop_Oleg.pdf|''Lustre and Application IO - Oleg Drokin's Slides'']]

===Lustre Scalability Workshop - Feb 10 & 11, 2009, ORNL===
*[[Media:Notes_on_SW1_Notes.pdf|''Scalability Workshop Notes'']]
*[[Media:LustreScalabilityWP_Updated.pdf|''Scalability White Paper'']]
*[[Media:Shipman_Feb_lustre_scalability.pdf|''Galen Shipman's presentation'']]
*[[Media:Eric-Barton_-_Lustre-Multi_PF_Roadmap-090130.pdf|''Eric Barton's presentation'']]

===Lustre Scalability Workshop - May 19 & 20, 2009, ORNL===
The May Lustre Scalability Workshop at ORNL was focused on long term (2015) HPC IO and storage requirements and on presentations on the IO objectives of the DOD HPCS program and how Lustre will achieve them.
*[[Media:Dawson_Lustre_Workshop_May_2009.pdf|''Lustre Scalability Workshop, Initial Gap Response'' - John Dawson]]
*[[Media:Carrier_2009-05-19_ORNL_LCE_HPCS.pdf|''HPCS IO'' - John Carrier]]
*[[Media:Newman_May_Lustre_Workshop.pdf|''What is HPCS and How Does it Impact IO'' - Henry Newman]]
*[[Media:Shipman_May_lustre_scalability_workshop.pdf|''2015 Parallel File System Requirements'' - Galen Shipman]]
*[[Media:Dilger_Lustre_HPCS_May_Workshop.pdf|''Lustre HPCS Design Overview'' - Andreas Dilger]]

===Scalability Workshop Follow Up===
*[[Media:Lustre_Scalability_Workshop.pdf‎ |''Scalability Gap Response'']] dated October 2009 is the final version of Sun's response to the scalability gaps identified and discussed during the LCE Scalability Workshops in February and May of 2009.

==White Papers==

LCE personnel have written a variety of papers on High Performance IO and potential Lustre features. Links to these documents are below.

*[https://www.sun.com/offers/details/Peta-Scale_wp.xml ''Lustre Scalability - An Oak Ridge National Laboratory/Lustre Center of Excellence Paper'']

*[[Media:Lce_pop_submitted.pdf|''Improving I/O Performance in POP (Parallel Ocean Program)'']]

* Scalability Improvements
**[[Media:Lustre_Enhancement_Report_Interval_Trees.pdf|''Using Interval Tree to scale extent locks'']]
**[[Media:Lustre_Enhancement_Report_UUID_Hash_Tables.pdf|''Implement hash tables to scale export lookups'']]
**[[Media:A_Novel_Network_Request_Scheduler_for_a_Large_Scale_Storage_System.pdf|''A Novel Network Request Scheduler for a Large Scale Storage System'']]

*Lustre ADIO Driver Enhancements Whitepaper
**[[Media:Lustre_ADIO_Driver_Whitepaper_0926.pdf|''Lustre ADIO Driver Enhancements'']]

*FMS Application IO Analysis
**[[Media:FMS_Investigation_Report_%280915%29.pdf|''FMS Application IO Performance Analysis'']]

==Presentations==

See the ORNL [http://wiki.lustre.org/index.php/Lustre_User_Group slides and video] from their presentation at LUG 2009.

==Press Articles==

* [http://www.nccs.gov/2009/06/30/ornl-hosts-lustre-part-ii/ ''ORNL Hosts Lustre Part II'']

==Lustre Internals Manual==
LCE and ORNL have written a Lustre filesystem internals document the describes the internal operation of Lustre.
*[[Media:Understanding_Lustre_Filesystem_Internals.pdf|''Understanding Lustre Filesystem Internals'']]

== Archives of Older Material ==

This material may be out of date, and is preserved here for archive purposes.

*[[Media:Peta-Scale_wp.pdf|''2008 Paper on IO with the Lustre File System at ORNL'']]

 
For more information about computing at Oak Ridge, click [http://www.nccs.gov/ here].

Lustre Center of Excellence at Oak Ridge National Laboratory

2011-01-20T18:11:45Z

Sbarthel:

''(Updated: Dec 2009)''
__TOC__
The Lustre™ Center of Excellence (LCE) at ORNL advances the state of Lustre for use in large scale HPC environments. As you'll see from the projects below, the ORNL LCE focuses on both systems and applications aspects of ensuring that Lustre meets the needs of DOE and the HPC community in general.

==Events==

The LCE sponsors events to encourage HPC community involvement
in analyzing IO and storage requirements and identifying ways for
Lustre to address these requirements.

===LCE Summit - February 2008===
*[[Media:LCE_Summit_Summary_Draft_March_14_2008.pdf|''LCE Summit Meeting Summary'']]
*[[Media:LCESummitSlides.pdf|''LCE Summit Slides and Notes'']]

===LCE Application I/O Workshop - April 16, 2008===
*[[Media:April2008ApplicationIOWorkshop.pdf|''Application IO Workshop, Agenda and Notes'']]
*[[Media:Lustre_workshop_WangDi.pdf|''Lustre and Application IO - Wang Di's Slides'']]
*[[Media:Lustre_workshop_Oleg.pdf|''Lustre and Application IO - Oleg Drokin's Slides'']]

===Lustre Scalability Workshop - Feb 10 & 11, 2009, ORNL===
*[[Media:Notes_on_SW1_Notes.pdf|''Scalability Workshop Notes'']]
*[[Media:LustreScalabilityWP_Updated.pdf|''Scalability White Paper'']]
*[[Media:Shipman_Feb_lustre_scalability.pdf|''Galen Shipman's presentation'']]
*[[Media:Eric-Barton_-_Lustre-Multi_PF_Roadmap-090130.pdf|''Eric Barton's presentation'']]

===Lustre Scalability Workshop - May 19 & 20, 2009, ORNL===
The May Lustre Scalability Workshop at ORNL was focused on long term (2015) HPC IO and storage requirements and on presentations on the IO objectives of the DOD HPCS program and how Lustre will achieve them.
*[[Media:Dawson_Lustre_Workshop_May_2009.pdf|''Lustre Scalability Workshop, Initial Gap Response'' - John Dawson]]
*[[Media:Carrier_2009-05-19_ORNL_LCE_HPCS.pdf|''HPCS IO'' - John Carrier]]
*[[Media:Newman_May_Lustre_Workshop.pdf|''What is HPCS and How Does it Impact IO'' - Henry Newman]]
*[[Media:Shipman_May_lustre_scalability_workshop.pdf|''2015 Parallel File System Requirements'' - Galen Shipman]]
*[[Media:Dilger_Lustre_HPCS_May_Workshop.pdf|''Lustre HPCS Design Overview'' - Andreas Dilger]]

===Scalability Workshop Follow Up===
*[[Media:Lustre_Scalability_Workshop.pdf‎ |''Scalability Gap Response'']] dated October 2009 is the final version of Sun's response to the scalability gaps identified and discussed during the LCE Scalability Workshops in February and May of 2009.

==White Papers==

LCE personnel have written a variety of papers on High Performance IO and potential Lustre features. Links to these documents are below.

*[https://www.sun.com/offers/details/Peta-Scale_wp.xml ''Lustre Scalability - An Oak Ridge National Laboratory/Lustre Center of Excellence Paper'']

*[[Media:Lce_pop_submitted.pdf|''Improving I/O Performance in POP (Parallel Ocean Program)'']]

* Scalability Improvements
**[[Media:Lustre_Enhancement_Report_Interval_Trees.pdf|''Using Interval Tree to scale extent locks'']]
**[[Media:Lustre_Enhancement_Report_UUID_Hash_Tables.pdf|''Implement hash tables to scale export lookups'']]
**[[Media:A_Novel_Network_Request_Scheduler_for_a_Large_Scale_Storage_System.pdf|''A Novel Network Request Scheduler for a Large Scale Storage System'']]

*Lustre ADIO Driver Enhancements Whitepaper
**[[Media:Lustre_ADIO_Driver_Whitepaper_0926.pdf|''Lustre ADIO Driver Enhancements'']]

*FMS Application IO Analysis
**[[Media:FMS_Investigation_Report_%280915%29.pdf|''FMS Application IO Performance Analysis'']]

==Presentations==

See the ORNL [http://wiki.lustre.org/index.php/Lustre_User_Group slides and video] from their presentation at LUG 2009.

==Press Articles==

* [http://www.nccs.gov/2009/06/30/ornl-hosts-lustre-part-ii/ ORNL Hosts Lustre Part II]

==Lustre Internals Manual==
LCE and ORNL have written a Lustre filesystem internals document the describes the internal operation of Lustre.
*[[Media:Understanding_Lustre_Filesystem_Internals.pdf|Understanding Lustre Filesystem Internals]]

== Archives of Older Material ==

This material may be out of date, and is preserved here for archive purposes.

*[[Media:Peta-Scale_wp.pdf| 2008 Paper on IO with the Lustre File System at ORNL]]

 
For more information about computing at Oak Ridge, click [http://www.nccs.gov/ here].

Lustre Center of Excellence at Oak Ridge National Laboratory

2011-01-20T18:09:12Z

Sbarthel: /* Archives of Older Material */

''(Updated: Dec 2009)''
__TOC__
The Lustre™ Center of Excellence (LCE) at ORNL advances the state of Lustre for use in large scale HPC environments. As you'll see from the projects below, the ORNL LCE focuses on both systems and applications aspects of ensuring that Lustre meets the needs of DOE and the HPC community in general.

==Events==

The LCE sponsors events to encourage HPC community involvement
in analyzing IO and storage requirements and identifying ways for
Lustre to address these requirements.

===LCE Summit - February 2008===
*[[Media:LCE_Summit_Summary_Draft_March_14_2008.pdf|LCE Summit Meeting Summary]]
*[[Media:LCESummitSlides.pdf|LCE Summit Slides and Notes]]

===LCE Application I/O Workshop - April 16, 2008===
*[[Media:April2008ApplicationIOWorkshop.pdf|Application IO Workshop, Agenda and Notes]]
*[[Media:Lustre_workshop_WangDi.pdf|Lustre and Application IO - Wang Di's Slides]]
*[[Media:Lustre_workshop_Oleg.pdf|Lustre and Application IO - Oleg Drokin's Slides]]

===Lustre Scalability Workshop - Feb 10 & 11, 2009, ORNL===
*[[Media:Notes_on_SW1_Notes.pdf|Scalability Workshop Notes]]
*[[Media:LustreScalabilityWP_Updated.pdf|Scalability White Paper]]
*[[Media:Shipman_Feb_lustre_scalability.pdf|Galen Shipman's presentation]]
*[[Media:Eric-Barton_-_Lustre-Multi_PF_Roadmap-090130.pdf|Eric Barton's presentation]]

===Lustre Scalability Workshop - May 19 & 20, 2009, ORNL===
The May Lustre Scalability Workshop at ORNL was focused on long term (2015) HPC IO and storage requirements and on presentations on the IO objectives of the DOD HPCS program and how Lustre will achieve them.
*[[Media:Dawson_Lustre_Workshop_May_2009.pdf|Lustre Scalability Workshop, Initial Gap Response - John Dawson]]
*[[Media:Carrier_2009-05-19_ORNL_LCE_HPCS.pdf|HPCS IO - John Carrier]]
*[[Media:Newman_May_Lustre_Workshop.pdf|What is HPCS and How Does it Impact IO - Henry Newman]]
*[[Media:Shipman_May_lustre_scalability_workshop.pdf|2015 Parallel File System Requirements - Galen Shipman]]
*[[Media:Dilger_Lustre_HPCS_May_Workshop.pdf|Lustre HPCS Design Overview - Andreas Dilger]]

===Scalability Workshop Follow Up===
*[[Media:Lustre_Scalability_Workshop.pdf‎ |Scalability Gap Response]] dated October 2009 is the final version of Sun's response to the scalability gaps identified and discussed during the LCE Scalability Workshops in February and May of 2009.

==White Papers==

LCE personnel have written a variety of papers on High Performance IO and potential Lustre features. Links to these documents are below.

*[https://www.sun.com/offers/details/Peta-Scale_wp.xml Lustre Scalability - An Oak Ridge National Laboratory/Lustre Center of Excellence Paper]

*[[Media:Lce_pop_submitted.pdf|Improving I/O Performance in POP (Parallel Ocean Program)]]

* Scalability Improvements
**[[Media:Lustre_Enhancement_Report_Interval_Trees.pdf|Using Interval Tree to scale extent locks]]
**[[Media:Lustre_Enhancement_Report_UUID_Hash_Tables.pdf|Implement hash tables to scale export lookups]]
**[[Media:A_Novel_Network_Request_Scheduler_for_a_Large_Scale_Storage_System.pdf|A Novel Network Request Scheduler for a Large Scale Storage System]]

*Lustre ADIO Driver Enhancements Whitepaper
**[[Media:Lustre_ADIO_Driver_Whitepaper_0926.pdf|Lustre ADIO Driver Enhancements]]

*FMS Application IO Analysis
**[[Media:FMS_Investigation_Report_%280915%29.pdf|FMS Application IO Performance Analysis]]

==Presentations==

See the ORNL [http://wiki.lustre.org/index.php/Lustre_User_Group slides and video] from their presentation at LUG 2009.

==Press Articles==

* [http://www.nccs.gov/2009/06/30/ornl-hosts-lustre-part-ii/ ORNL Hosts Lustre Part II]

==Lustre Internals Manual==
LCE and ORNL have written a Lustre filesystem internals document the describes the internal operation of Lustre.
*[[Media:Understanding_Lustre_Filesystem_Internals.pdf|Understanding Lustre Filesystem Internals]]

== Archives of Older Material ==

This material may be out of date, and is preserved here for archive purposes.

*[[Media:Peta-Scale_wp.pdf| 2008 Paper on IO with the Lustre File System at ORNL]]

 
For more information about computing at Oak Ridge, click [http://www.nccs.gov/ here].

Lustre Customers

2011-01-20T18:08:16Z

Sbarthel:

''(Updated: Nov 2009)''

Below are listed a handful of typical Lustre™ customers. For a look at other customers, see contributions by participants in [[GetInvolved:Get_Involved|Lustre User Group]] meetings.

<ul>
<li>[[Media:DIF.pdf|CEA]]
<li>[http://www.sun.com/customers/software/chevron.xml Chevron]
<li>[http://www.sun.com/customers/software/framestore.xml Framestore]
<li>[[Media:StephenSimms.pdf|Indiana University]]
<li>[http://www.nccs.gov/2009/06/30/ornl-hosts-lustre-part-ii/ Oak Ridge National Laboratory]
<li>[http://hpc.pnl.gov/projects/active-storage/papers/lci-evanfelix.pdf Pacific Northwest National Laboratory]
<li>[[Media:SNL_LUG_2009.pdf|Sandia National Laboratory]]
<li>[http://www.sun.com/customers/servers/tacc.xml Texas Advanced Computing Center]
<li>[http://www.sun.com/customers/servers/tokyo_tech1.xml Tokyo Institute of Technology]
</ul>

Lustre Design Documents

2011-01-20T18:06:24Z

Sbarthel:

The Lustre™ team has prepared several design documents as part of the [[Lustre_HPCS_Activities|DARPA High Productivity Computing Systems (HPCS) program]]:

*[[Media:JKD_Wiki_V1_2009_08_07_Lustre_HPCS_Overview.pdf|''Lustre HPCS Design Overview'']]
*[[Media:FSCK_Design-2009-06-15-09.pdf|''Filesystem Integrity Check Design'']]
*[[Media:End-to-End-Integrity-2009-06-15.pdf|''End to End Data Integrity Design'']]
*[[Media:Channel Bonding_06_15_09.pdf|''LNET Channel Bonding Design'']]
*[[Media:Rebuild_performance-2009-06-15.pdf|''Rebuild Performance Design'']]
*[[Media:HPCS_CMD_06_15_09.pdf|''Clustered Metadata Design'']]

In November 2009, the senior Lustre engineers met in conjunction with the SC'09 event in Portland, Oregon, to discuss designs for upcoming features and implementation status of current development projects. The following project reviews and design presentations took place at this meeting:

* [[Media:SC09-kDMU-Code.pdf|''Kernel DMU Project Review'']] - Alex Zhuravlev
* [[Media:SC09-SOM-Code.pdf|''Size On MDS Project Review'']] - Vitaly Fertman
* [[Media:SC09-FID-on-OST.pdf|''FIDs on OST Design Presentation'']] - Mike Pershin
* [[Media:SC09-HSM-Code.pdf|''HSM Project Review'']] - Nathan Rutman
* [[Media:SC09-SMP-Scaling.pdf|''Server SMP Scaling Project Review'']] - Liang Zhen
* [[Media:SC09-Quota-Code.pdf|''Quota Project Review'']] - Johann Lombardi
* [[Media:SC09-Imperative-Recovery.pdf|''Imperative Recovery Design Presentation'']] - Nicolas
* [[Media:SC09-ZFS-End-to-End-Integrity.pdf|''ZFS, Lustre, End-to-end Data Integrity BOF'']] - Andreas Dilger
* [[Media:SC09-LOV-OSC.pdf|''LOV/OSC/llog Redesign Presentation'']] - Alex Zhuravlev
* [[Media:SC09-CMD-Code.pdf|''Clustered Metadata Project Review'']] - Tom Wang
* [[Media:SC09-Simplified-Interop.pdf|''Simplified Interop (CSS) Design Presentation'']] - Eric Mei

The [[Lustre Design Document Archive]] contains older architecture and design documents.

Lustre Documentation

2011-01-20T18:04:12Z

Sbarthel:

__TOC__
The ''Lustre Operations Manual'' is the primary source of documentation about the Lustre product. The manual covers the Lustre product and provides users with installation, configuration, tuning, monitoring and troubleshooting information.

Different versions of the ''Lustre Operations Manual'' support different versions of Lustre software. To determine which version of the ''Lustre Operations Manual'' supports the Lustre software you are using, refer to the table below. The latest version of the Operations Manual is available in both HTML and PDF formats.

{| border="1" cellspacing=0 cellpadding="10"
! Manual Part No. !! Supported Lustre Version !! Release Date !! HTML !! PDF
|-
| 821-2076-10'''
| '''2.0'''
| January 2011
| [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html Lustre Manual 2.0]
| [[Media:821-2076-10.pdf|Lustre 2.0 Manual]]
|-
| 821-2077-10'''
| '''2.0'''
| July 2010
| n/a
| [[Media:821-2077-10.pdf|Lustre 2.0 Release Notes]]
|-
| 821-0035-12'''
| '''1.8'''
| December 2010
| [http://wiki.lustre.org/manual/LustreManual18_HTML/index.html Lustre Manual 1.8]
| [[Media:821-0035-12.pdf|Lustre 1.8 Manual]]
|-
| 820-3681-12'''
| '''1.6'''
| June 2010
| [http://wiki.lustre.org/manual/LustreManual16_HTML/index.html Lustre Manual 1.6]
| [[Media:820-3681_v1_18.pdf|Lustre 1.6 Manual]]
|}

Lustre Fall Workshop 10/2010

2011-01-20T18:03:04Z

Sbarthel:

During the week of October 25-28, 2010, the Lustre team held a workshop in Beijing. Below are the presentations offered during this time.

Project Overviews:

Feature Overviews:
*[[Media:Beijing-2010.2-ZFS_overview_3.1_Dilger.pdf|''ZFS Overview'']] - Andreas Dilger
*[[Media:HSM_api_Green.pdf|''HSM Overview'']] - Oleg Drokin
*[[Media:quota_on_osd.pdf|''Quotas Overview'']] - Alex Zhuravlev

Lustre HPCS Activities

2011-01-20T18:02:04Z

Sbarthel:

''(Updated: Aug 2009)''

The DARPA High Productivity Computing Systems (HPCS) project is a program created to ensure that US Government agencies continue to have access to the advanced high-performance computing technologies needed to fulfill their missions. The program objectives include achieving extremely high IO performance and file system scale and reliability. [http://www.cray.com Cray Inc.] has partnered with Sun to use Lustre™ to achieve these goals.

General information on the HPCS program is available at:

*http://www.highproductivity.org/

*http://www.darpa.mil/ipto/programs/hpcs/hpcs.asp

John Carrier of Cray, Inc. presented an overview of the HPCS IO project at the Lustre User Group meeting in April of 2008. The slides and video of John Carrier's presentation can be found here:

*[[Media:LUG08_Cray_HPCS.pdf|''DARPA HPCS Project Slides'']] and *[http://video.google.com/videoplay?docid=-5970693206965456534&hl=en ''Video''] - John Carrier, Cray Inc.

There were three presentations on the HPCS IO project at the Scalability Workshop held by the [[Lustre Center of Excellence at Oak Ridge National Laboratory]] in May, 2009.

*[[Media:Carrier_2009-05-19_ORNL_LCE_HPCS.pdf|''HPCS IO'' - John Carrier, Cray Inc.]]
*[[Media:Newman_May_Lustre_Workshop.pdf|''What is HPCS and How Does it Impact IO'' - Henry Newman, Instrumental Inc.]]
*[[Media:Dilger_Lustre_HPCS_May_Workshop.pdf|''Lustre HPCS Design Overview'' - Andreas Dilger, Sun Microsystems]]

The Lustre team has prepared several design documents as part of the HPCS program. These designs are:

*[[Media:JKD_Wiki_V1_2009_08_07_Lustre_HPCS_Overview.pdf|''Lustre HPCS Design Overview'']]
*[[Media:FSCK_Design-2009-06-15-09.pdf|''Filesystem Integrity Check Design'']]
*[[Media:End-to-End-Integrity-2009-06-15.pdf|''End to End Data Integrity Design'']]
*[[Media:Channel Bonding_06_15_09.pdf|''LNet Channel Bonding Design'']]
*[[Media:Rebuild_performance-2009-06-15.pdf|''Rebuild Performance Design'']]
*[[Media:HPCS_CMD_06_15_09.pdf|''Clustered Metadata Design'']]

Lustre HPCS Activities

2011-01-20T18:01:43Z

Sbarthel:

''(Updated: Aug 2009)''

The DARPA High Productivity Computing Systems (HPCS) project is a program created to ensure that US Government agencies continue to have access to the advanced high-performance computing technologies needed to fulfill their missions. The program objectives include achieving extremely high IO performance and file system scale and reliability. [http://www.cray.com Cray Inc.] has partnered with Sun to use Lustre™ to achieve these goals.

General information on the HPCS program is available at:

*http://www.highproductivity.org/

*http://www.darpa.mil/ipto/programs/hpcs/hpcs.asp

John Carrier of Cray, Inc. presented an overview of the HPCS IO project at the Lustre User Group meeting in April of 2008. The slides and video of John Carrier's presentation can be found here:

*[[Media:LUG08_Cray_HPCS.pdf|''DARPA HPCS Project Slides'']] and *[http://video.google.com/videoplay?docid=-5970693206965456534&hl=en Video] - John Carrier, Cray Inc.

There were three presentations on the HPCS IO project at the Scalability Workshop held by the [[Lustre Center of Excellence at Oak Ridge National Laboratory]] in May, 2009.

*[[Media:Carrier_2009-05-19_ORNL_LCE_HPCS.pdf|''HPCS IO'' - John Carrier, Cray Inc.]]
*[[Media:Newman_May_Lustre_Workshop.pdf|''What is HPCS and How Does it Impact IO'' - Henry Newman, Instrumental Inc.]]
*[[Media:Dilger_Lustre_HPCS_May_Workshop.pdf|''Lustre HPCS Design Overview'' - Andreas Dilger, Sun Microsystems]]

The Lustre team has prepared several design documents as part of the HPCS program. These designs are:

*[[Media:JKD_Wiki_V1_2009_08_07_Lustre_HPCS_Overview.pdf|''Lustre HPCS Design Overview'']]
*[[Media:FSCK_Design-2009-06-15-09.pdf|''Filesystem Integrity Check Design'']]
*[[Media:End-to-End-Integrity-2009-06-15.pdf|''End to End Data Integrity Design'']]
*[[Media:Channel Bonding_06_15_09.pdf|''LNet Channel Bonding Design'']]
*[[Media:Rebuild_performance-2009-06-15.pdf|''Rebuild Performance Design'']]
*[[Media:HPCS_CMD_06_15_09.pdf|''Clustered Metadata Design'']]

Lustre Packages

2011-01-20T17:58:11Z

Sbarthel:

''(Updated: Feb 2010)''

The table below describes each Lustre™ package and indicates about where each package is to be installed.

{| border=1 cellpadding=0
|-
!width="220"|Lustre Package
!width="300"|Description
!width="160"|Install on servers
!width="160"|Install on clients
|-
|Lustre kernel RPMs|| || ||
|-
|   kernel-lustre-<ver>
|Lustre-patched kernel package for RHEL 5 (i686, ia64 and x86_64) platform.
|align="center"|X||
|-
|-
|   kernel-lustre-smp-<ver>
|Lustre-patched kernel package for SuSE Server 10 (x86_64) platform.
|align="center"|X||
|-
|   kernel-lustre-bigsmp-<ver>
|Lustre-patched kernel package for SuSE Server 10 (i686) platform.
|align="center"|X||
|-
|   kernel-ib-<ver>
|Lustre OFED package. Install if the network interconnect is InfiniBand (IB).
|align="center"|X
|align="center"|X
|-
|-
|
   kernel-lustre-default-<ver>
   kernel-lustre-default-base-<ver>
   kernel-lustre-default-extra-<ver>
|Lustre-patched kernel package for SuSE Server 11 (i686 and x86_64) platform.
|align="center"|X||
|-
|Lustre module RPMs|| || ||
|-
|   lustre-modules-<ver>
|Lustre modules for the patched kernel.
|align="center"|X||
|-
|   lustre-client-modules-<ver>
|Lustre modules for patchless clients.
|
|align="center"|X
|-
|Lustre utilities|| || ||
|-
|   lustre-<ver>
|Lustre utilities package. This includes userspace utilities to configure and run Lustre.
|align="center"|X||
|-
|   lustre-ldiskfs-<ver>
|Lustre-patched backing file system kernel module package for the ext3 file system.
|align="center"|X||
|-
|   e2fsprogs-<ver>
|Utilities package used to maintain the ext3 backing file system.
|align="center"|X||
|-
|   lustre-client-<ver>
|Lustre utilities for patchless clients.
|
|align="center"|X
|}
For packages used to patch a client kernel (optional), see [http://wiki.lustre.org/manual/LustreManual20_HTML/InstallingLustre.html#50438261_pgfId-1294006 Section 8.1: ''Preparing to Install the Lustre Software''] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual''].

Lustre Projects

2011-01-20T17:57:03Z

Sbarthel:

This page lists Lustre™ projects that may span multiple releases or are features in development that have not yet been assigned to a release. You can find links to feature descriptions for current and future releases of Lustre on the [[Learn]] page.

Projects:
<ul>
<li>[[Metadata Performance Project]]
<li>[[Running Hadoop with Lustre]]
<li>[[Lustre HPCS Activities]]
<li> [[ZFS and Lustre]]
<li> [[Windows Native Client]]
</ul>

Features:
<ul>
<li>[[Clustered Metadata]]
</ul>

Lustre System Configuration Utilities

2011-01-20T17:47:20Z

Sbarthel:

''(Updated: Jan 2010)''

A set of system configuration utilities are provided with Lustre. These utilities include:

: '''mkfs.lustre''' - Used to format a disk for a Lustre service.

: '''tunefs.lustre''' - Used to modify configuration information on a Lustre target disk.

: '''lctl''' - Used to directly control Lustre via an ''ioctl'' interface, allowing various configuration, maintenance and debugging features to be accessed.

: '''mount.lustre''' - Used to start a Lustre client or target service.

For examples using these utilities, see:
* [[Completing Basic Administrative Tasks]] - many of these procedures use one or more system configuration utilities. For example, in [http://wiki.lustre.org/manual/LustreManual20_HTML/LustreOperations.html#50438194_pgfId-1307131 ''Running Multiple Lustre File Systems''], ''mkfs.lustre'' is used to create a second file system and to set up a Lustre installation with two file systems.
* [[Configuring the Lustre File System]] - ''mkfs.lustre'' and ''tunefs.lustre'' are used to configure a Lustre file system after installation.
* [[Managing OSTs]] and related subtopics - ''lctl'' is used to show the status of an OST, to deactivate and activate an OST, to set weighting for free space, and to create a new pool and add an OST.

For more information about the Lustre system configuration utilities, see [http://wiki.lustre.org/manual/LustreManual20_HTML/SystemConfigurationUtilities_HTML.html#50438219_pgfId-5529 Chapter 36: ''System Configuration Utilities''] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual'']. This chapter also describes other system configuration utilities, as well as utilities to manage large clusters, perform application profiling, and test and debug Lustre.

For additional information and examples, see the following chapters in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual'']:

* [http://wiki.lustre.org/manual/LustreManual20_HTML/ConfiguringLustre.html#50438267_pgfId-1290860 Chapter 10: ''Configuring Lustre'']

* [http://wiki.lustre.org/manual/LustreManual20_HTML/ConfiguringLustre.html#50438267_pgfId-1290956 Section 10.1.1: ''Simple Lustre Configuration Example'']

* [http://wiki.lustre.org/manual/LustreManual20_HTML/ConfiguringLustre.html#50438267_pgfId-1293213 Section 10.2: ''Additional Configuration Options'']

* [http://wiki.lustre.org/manual/LustreManual20_HTML/LustreMonitoring.html#50438273_pgfId-5529 Chapter 12: ''Lustre Monitoring'']

* [http://wiki.lustre.org/manual/LustreManual20_HTML/LustreOperations.html#50438194_pgfId-1289959 Chapter 13: ''Lustre Operations'']

* [http://wiki.lustre.org/manual/LustreManual20_HTML/UserUtilities_HTML.html#50438206_pgfId-5529 Chapter 32: ''User Utilities'']

Lustre User Group 2006

2011-01-20T17:45:16Z

Sbarthel:

__TOC__
The fourth annual Lustre User Group Spring meeting was held April 18-20, 2006 on Hilton Head Island, SC.

The Lustre User Group was founded by Pacific Northwest National Laboratory and Lawrence Livermore National Laboratory to promote the use of the open source Lustre file system for HPC and scalable I/O. The annual meeting is an opportunity for current and prospective Lustre users to share and develop best practices, provide direct feedback to the Lustre development team, and receive an update on our plans for the coming year.

--------------------------------------------------------------------------------

==LUG 2006 Conference Schedule==
The target attendees for this meeting are users or administrators of a Lustre file system, and partners/resellers with Lustre offerings. Developers or researchers working with Lustre are also welcome, but should note that there is little content specifically for developers.

To allow for requested topics and submitted presentations, the agenda has several open slots. Please let us know what you would like to see in the program, and we'll do our best to accommodate the most popular requests.

If you would like to make a presentation at the meeting, please send a short proposal to lustre2006-submissions@clusterfs.com. This e-mail address is being protected from spam bots, you need JavaScript enabled to view it. Due to limited availability, please limit your presentation to topics of general interest such as best practices, lessons learned, research projects based on Lustre, or similar. Please do not submit sales presentations, and please submit before March 24.

The last two half-day sessions will provide a structured, organized forum for identifying and clarifying Lustre's future requirements. The goals are to gather input from customers and partners, identify conflicting requirements, and prioritize the rest. This replaces the half-day group discussions of the previous two LUG meetings.

While everyone is welcome to attend the first 1.5 days of the meeting, we must ask that organizations designate no more than two representatives for the Lustre Requirements Forum. These individuals are historically the liaisons from strategic Lustre partners and users who can speak to their organizations' requirements. To help us get an accurate count, please indicate on the registration form whether you will attend the Lustre Requirements Forum.

===Tuesday April 18, 2006===

9:00-9:30 a.m. Welcome & Introduction
Dr.Peter Braam - President & CEO

9:30-10:00 a.m. [[Media:Lug_06_cfs_business_overview.pdf|Business Overview]]
Jeffrey Denworth - Director of Sales

10:00-10:30 a.m. [[Media:Lug-supportpresentation-060418.pdf|Support & QA Update]]
Peter Bojanic - Director of Engineering

10:30-11:00 a.m. Break

11:00-12:00 a.m. Lustre now and future:
architecture & roadmap

12:00-1:00 p.m. Lunch

1:00-1:30 p.m. Lustre now and future:
architecture & roadmap

1:30-2:00 p.m. Experiences with Lustre in a Seismic Processing Environment
Chevron Corporation

2:00-2:30 p.m. [[Media:Lug06-osu.pdf|Evaluating Lustre on DDR & SDR Infiniband]]
Dr.DK Panda - Ohio State University

3:00-3:30 p.m. Break

3:30-3:45 p.m. [[Media:Lug06-ncsa.pdf|Lustre - WAN Update]]
Chris Cribbs - NCSA

3:45-4:10 p.m. CEA TERA10 100GB/s + Lustre Deployment
Stephane Thiell - CEA DAM

4:10-4:30 p.m. [[Media:Lug06-bull.pdf|Bull Lustre Management Tools for Large Clusters]]
Johann Lombardi - Bull

6:00-9:00 p.m. Lustre Users Reception

===Wednesday April 19, 2006 ===

9:00-9:30 a.m. [[Media:Lug06-liquid.pdf|Converged Communications Systems Parallel File Systems and Copious IO]]
Jose Miguel Peleato - Liquid

9:30-10:00 a.m. [[Media:Lug06-hp.pdf|Key Issues Deploying Lustre at HP Customer Sites]]
Fergal Mc Carthy - HP

10:00-10:30 a.m. Indiana University's Data Capacitor
Steve Simms - Indiana University

10:30-11:00 a.m. Break

11:00-11:30 a.m. Cray Experiences Deploying Terascale Lustre Environments
Peter Rigsbee - Cray

11:30-12:00 a.m. [[Media:Lug06-ddn.pdf|Storage HW and Petascale I/O]]
Dave Fellinger - DataDirect Networks

12:00-1:00 p.m. Lunch

1:00-5:00 p.m.

Lustre Requirements Forum

-Requirements Survey

-Prioritization

-Refinement of Specific Requirements

-Break

-Implementation Scenarios

-Review

===Thursday April 20, 2006 ===

9:00-12:00 a.m.

Lustre Requirements Forum

-Requirements Survey

-Prioritization

-Refinement of Specific Requirements

-Break

-Implementation Scenarios

-Review

--------------------------------------------------------------------------------

==Registration==
Registration is open to all, however attendance is limited so please register early. While the registration deadline was Friday March 24, 2006, we will continue to accept registrations until the conference is full - however, we can no longer guarantee accommodations.

There is a conference fee of US$300 to cover the costs of hosting the meeting. Payments may be made by purchase order, credit card (all major cards accepted through Paypal), money order, or check.

-------------------------------------------------------------------------------

==Location, Travel, and Hotel==
The meeting will be held at [http://www.starwoodhotels.com/westin/property/overview/index.html?propertyID=1050 The Westin Resort, Hilton Head Island], 3 miles from the Hilton Head Island Airport (HHH) and about 45 miles from Savannah International Airport (SAV). We have negotiated a special group rate of $199/night, and reserved a block of rooms for our attendees. In addition, there is a reserved block of Government Per Diem rate rooms at $112/night.
Rooms must be booked before March 24, 2006 to take advantage of this group rate.
We are in the process of completing the reservation arrangements; please check back on February 14 for details about how to book these discounted rooms.

-------------------------------------------------------------------------------

==Contact==
To submit a presentation proposal, please send email to lustre2006-submissions@clusterfs.com. This e-mail address is being protected from spam bots, you need JavaScript enabled to view it .
For all other questions and comments related to the meeting, please send email to lustre2006@clusterfs.com. This e-mail address is being protected from spam bots, you need JavaScript enabled to view it .

* LUG Requirements Forum - [[Media:LUG-Requirements-060420-final.pdf|LUG-Requirements-060420-final.pdf]] | [http://wiki.lustre.org/images/7/78/LUG-Requirements-060420-final.xls LUG-Requirements-060420-final.xls]

Lustre User Group 2007

2011-01-20T17:44:53Z

Sbarthel:

__TOC__
The fifth annual Lustre User Group meeting was held in stylish South Beach District of Miami Beach, Florida, April 23-25, 2007, at the beautiful [http://www.thepalmshotel.com Palms Hotel].

The Lustre User Group Conference is the premier event for learning new technical information, acquiring best practices, and sharing knowledge about Lustre Technology. It's a once-a-year opportunity for users to get answers, advice, and suggestions regarding their specific Lustre implementations. Attendees will have access to experts and peers who will share their real-world experiences. You will also be able to meet with Lustre management and developers to discuss upcoming enhancements and capabilities. Register today while space is still available!

== LUG 2007 Agenda ==
===Monday April 23, 2007 ===
9:00-9:30 a.m.
[[Media:Lug-opening-talk.pdf|Welcome & Introduction]]
Dr.Peter Braam - President & CEO

9:30-10:00 a.m. Corporate Update
Kevin Canady - Vice President of Business Development

10:00-10:30 a.m. [[Media:Lug07-engineering.pdf|Engineering & QA Update]]
Peter Bojanic - Vice President of Engineering

10:30-11:00 a.m. Break

11:00-12:00 p.m. [[Media:Lug07-architecture.pdf|Lustre: Architecture & Roadmap]]
Dr.Peter Braam - President & CEO

12:00-1:00 p.m. Lunch

1:00-1:30 p.m. Lustre v1.6 Features & Benefits (Part I)
Nathan Rutman - Principal Engineer

1:30-2:00 p.m. Lustre v1.6 Features & Benefits (Part II)
Nathan Rutman - Principal Engineer

2:00-2:30 p.m. [[Media:Lug07_ornl.pdf|Lustre Center of Exellence Reports]]
Shane Canon - Oak Ridge National Lab

3:00-3:30 p.m. Break

3:30-4:00 p.m. [[Media:Lug07_sun.pdf|HPC Installations]]
Larry McIntosh - Sun

4:00-4:30 p.m. [[Media:Lustre_user_group_-_tacc_presentation.pdf|TACC Overview & Lustre Experiences]]
Karl Schulz - TACC

4:30-5:00 p.m.
[[Media:Sicortex.pdf|SiCortex and Lustre]]

6:00-7:30 p.m. Lustre Users Reception

===Tuesday April 24, 2007===

9:00-9:30 a.m. [[Media:Cealug2007.pdf|Lustre Centre of Excellence Reports: HSM]]
Stephane Thiell - French Atomic Energy Commission (CEA)

9:30-10:00 a.m. Lustre Deployments Lessons Learned
Steve Simms - Indiana University

10:00-10:30 a.m. Lustre Deployments Lessons Learned: Oil& Gas Industry
Keith Gray - BP
10:30-11:00 a.m. Break

11:00-11:30 a.m.
[[Media:Ncar_lustre.pdf|Lustre Activities]] Adam Boggs - National Center for Atmospheric Research (NCAR)

11:30-12:00 a.m. Lustre Tuning Tutorial
Nathan Rutman - Principal Engineer

12:00-1:00 p.m. Lunch

1:00-2:00 p.m. Lustre Troubleshooting Tutorial
Nathan Rutman - Principal Engineer

2:00-2:30 p.m. [[Media:Lug07-dresden.pdf|Center for Information and High Performance Computing]]
Guido Juckeland - Dresden Technical University

3:00-3:30 p.m. Break

3:30-4:00 p.m. [[Media:Ddn-lug07.pdf|Storage Architecture and Roadmap]]
Dave Fellinger - DDN

4:00-4:30 p.m. [[Media:Lug07-lustrenetworking.pdf|Lustre Networking: Features & Benefits]]
Dr. Peter Braam

4:30-5:00 p.m. [[Media:Markgary.pdf|Lustre Centre of Excellence Reports]]
Mark Gary - Lawrence Livermore National Lab

===Wednesday April 25, 2007 ===
9:00-11:30 a.m.
Lustre Requirements Forum

-Requirements Survey

-Prioritization

-Refinement of Specific Requirements

-Break

-Voting

-Review

11:30-12:00 a.m.
Wrap-up & Thanks

We look forward to seeing you at the Palms Hotel in Miami Beach! Call 1-800-550-0505 to book rooms at the Palms. Be sure to ask for the Lustre group rate.

Lustre User Group 2008

2011-01-20T17:44:27Z

Sbarthel:

__TOC__
Presentations and videos from the Lustre User Group (LUG) 2008 are available below.

[http://picasaweb.google.com/overheardinpdx/LustreUserGroup2008 LUG2008 Photo Gallery] by Carl Wimmi

==LUG 2008 AGENDA==
===APRIL 28 MONDAY===

[[Media:Bojanic.pdf|Welcome & Introduction Slides]] and [http://video.google.com/videoplay?docid=4679328954718003275&hl=en Video]
- Peter Bojanic, Director, Lustre Group, Sun

Lustre™ Business Update (slides and video not made available)
- Kevin Canady, Director of Business Development, Sun

[[Media:Barton08.pdf|Lustre Engineering Update Slides]] and [http://video.google.com/videoplay?docid=-7161557916859061133&hl=en Video]
- Eric Barton, Lustre Lead Engineer, Sun

[[Media:Lug-2008-Braam.pdf|Sun Storage Perspective & Lustre Architecture Slides]] and [http://video.google.com/videoplay?docid=-3180589066635872586&hl=en Video]
- Peter Braam, VP of Lustre, Sun

[[Media:Lustre-RoadmapUpdate-20080428.pdf|Lustre Roadmap Update Slides]] and [http://video.google.com/videoplay?docid=-3461652841731942937&hl=en Video]
- Bryon Neitzel, Lustre, Sun

[[Media:Stearman_LUG_2008.pdf|Lustre at LLNL Slides]] and [http://video.google.com/videoplay?docid=-7652222883022083898&hl=en Video]
- Marc Stearman, LLNL

[[Media:LUG-talk-canon.pdf|A Global File System with Lustre &LNET Routers Slides]] and [http://video.google.com/videoplay?docid=-826148550313733004&hl=en Video]
- Shane Canon, ORNL

[[Media:LUG08_Cray_HPCS.pdf|DARPA HPCS Project Slides]] and [http://video.google.com/videoplay?docid=-5970693206965456534&hl=en Video]
- John Carrier, Cray

[[Media:Lustre_HSM_Lug08.pdf|ILM- Lustre HSM Slides]] and [http://video.google.com/videoplay?docid=-3801023942247542718&hl=en Video]
- Aurelien Degremont, CEA

[[Media:Hedges_-LustreUsersGroup2008.pdf|Customer Support & I/O Applications from a LLNL Perspective Slides]] and [http://video.google.com/videoplay?docid=4651328086223399469&hl=en Video]
- Richard Hedges, LLNL

===APRIL 29 TUESDAY===
Keynote Speaker
- Andy Bechtolsheim (slides and video not made available)

[http://video.google.com/videoplay?docid=-1374897290924249244&hl=en FlexRex: HPC Community Portal]
- Rich Brueckner, HPC Group, Sun

[[Media:LUG08-Lustre-Application-IO.pdf|Scientific Application Performance with Lustre Slides]] and [http://video.google.com/videoplay?docid=-7545630997322244657&hl=en Video]
- Tom Wang, Lustre Group, Sun

[[Media:LUG08-Lustre-LNET-selftest.pdf|Lnet Selftest Slides]] (no Video)
- Isaac Huang, Lustre Group, Sun

[[Media:LUG08-IU-wan.pdf|BOF on Lustre over WAN - IU Slides]] (no Video)
- Eric Barton, ORNL, TACC, IU

[[Media:LUG2008-NEARSC-Cardo.pdf|Lustre Tools Session-Backup&Quota Slides]] (no Video)
- Nicholas P. Cardo, LBNL

[[Media:Shine_LUG2008.pdf|Lustre Tools Session-Shine, Administration Tool Slides]] and [http://video.google.com/videoplay?docid=569932018410904962&hl=en Video]
- Stephane Thiell, CEA

[[Media:LUG08-DDN-sm.pdf|Lustre Partner: DataDirect Networks Slides]] and [http://video.google.com/videoplay?docid=-7929228264502851730&hl=en Video]
- Dave Fellinger, DDN

'''Lustre Expertise Session (No Videos):'''

1. [[Media:LUG08-Lustre-uOSS.pdf|Userspace Server Architecture Slides]]
- Andreas Dilger

2. [[Media:Lustre_on_ZFS-Ricardo.pdf|Lustre on ZFS Slides]]
- Ricardo Correia

3. [[Media:LUG08-Lustre-NFS.pdf|NFS/pNFS export Slides]]
- Oleg Drokin

4. [[Media:LUG2008-Lustre-cmd.pdf|Clustered Metadata (CMD) Slides]]
- Andreas Dilger

5. [[Media:LUG08-head-mds-danilov.pdf|Metadata Server Stack Slides]]
- Nikita Danilov

6. [[Media:LUG08-som-vitaly-fertman.pdf|Metadata Performance with Size on MDS Slides]]
- Vitaly Fertman

===APRIL 30 WEDNESDAY===

Joint CEA/NNSA BOF on PetaScale I/O Issues (no Video)
- Mark Gary, LLNL

[[Media:LUG08-Linux_HPC_Software_Stack.pdf|Linux HPC Software Stack Slides]] and [http://video.google.com/videoplay?docid=-1289609202868980540&hl=en Video]
- Makia Minich, Lustre Group, Sun

[http://video.google.com/videoplay?docid=-2769491780923413644&hl=en Lustre Engineering Panel Q&A Video] (no Slides)
- Lustre Group

Lustre User Group 2009

2011-01-20T17:44:02Z

Sbarthel:

Thanks to all of the attendees and presenters at the 2009 Lustre User Group (LUG). We had the highest attendance in LUG's seven year history and an excellent program.

This year's LUG agenda appears below with links to the presentations and videos.

== LUG09 Agenda, Slides, and Videos ==
LUG09, a three-day event, featured a workshop and numerous presentations on select Lustre features, upcoming enhancements, site-specific experiences using Lustre, and much more.

----
<big>April 15 - Wednesday</big>

'''Lustre Workshop'''

This full-day seminar covered topics of interest to Lustre users, and featured the following presentations from our senior-level developers:

'''''LNET Mysteries''''' - Isaac Huang
:[[Media:LNet_LUG_09.pdf|''Slides'']]
:[http://slx.sun.com/node/1179274162 ''Video'']

'''''Building Lustre, Protocol Basics, and Debugging''''' - Johann Lombardi
:[[Media:Lug_johann.pdf|''Slides'']]
:[http://slx.sun.com/1179274164|''Video'']

'''''IO and Tuning''''' - Oleg Drokin
:[[Media:Oleg_Workshop_LUG_09.pdf|''Slides'']]
:[http://slx.sun.com/1179274167 ''Video'']

'''''Lustre File System and Tuning''''' - Andreas Dilger
:[[Media:Dilger_Workshop_LUG_09.pdf|''Slides'']]
:[http://slx.sun.com/1179274169 ''Video'']

----
<big>April 16 - Thursday</big>

'''Lustre Presentations and Forums'''

'''''Welcome & Introduction''''' - Peter Bojanic, Director, Lustre Group, Sun
:[[Media:PeterBojanic.pdf|''Slides'']]
:[http://slx.sun.com/1179274073 ''Video'']

'''''Lustre Scalability''''' - Eric Barton, Lead Engineer, Sun
:[[Media:Barton_LUG_2009_Scalability.pdf|''Slides'']]
:[http://slx.sun.com/1179274098 ''Video'']

'''''Lustre 1.8 Features''''' - Nathan Rutman, Lustre, Sun
:[[Media:NathanRutman.pdf|''Slides'']]
:[http://slx.sun.com/1179274124 ''Video'']

'''''Lustre Quality Initiative''''' - Robert Read, Lustre, Sun
:[[Media:RobertReadTalk1.pdf|''Slides'']]
:[http://slx.sun.com/1179274140 ''Video'']

'''''Making Movies with Lustre - Fun and Fantasy with Furry Creatures''''' - John Leedham, Daire Byrne, James Rose, Framestore
:[[Media:JamesRose.pdf|''Slides'']]
:[http://slx.sun.com/1179274187 ''Video'']

'''''Lustre at Harvard''''' - Teresa Kaltz
:[[Media:TeresaKaltz.pdf|''Slides'']]
:[http://slx.sun.com/1179274189 ''Video'']

'''''Lustre Recovery''''' - Robert Read, Lustre, Sun
:[[Media:RobertReadTalk2.pdf|''Slides'']]
:[http://slx.sun.com/1179274084 ''Video'']

'''''Lustre at Sandia''''' - Steve Monk, Sandia
:[[Media:SNL_LUG_2009.pdf|''Slides'']]
:Video not recorded for this session

'''''Lustre at CEA: TERA-100''''' - Stéphane Thiell, CEA
:[[Media:StephaneThiell.pdf|''Slides'']]
:[http://slx.sun.com/1179274102 ''Video'']

'''Sun Lustre Storage Cluster: Lessons Learned and Best Practices''' - Joey Jablonski, Sun
:[[Media:JoeyJablonski.pdf|''Slides'']]
:[http://slx.sun.com/1179274146 ''Video'']

'''Managing High Availability on a Shine-Equipped Lustre Cluster''' - Oliver Hargoaa, Bull
:[[Media:OliverHargoaa.pdf|''Slides'']]
:[http://slx.sun.com/1179274195 ''Video'']

----
<big>April 17 - Friday</big>

'''Lustre Presentations and Forums'''

'''''Lustre on Hyperion''''' - Marc Stearman, LLNL
:[[Media:LUGHyperion2009.pdf|''Slides'']]
:[http://slx.sun.com/1179274196 ''Video'']

'''''Meeting the I/O Demands of the World's Most Powerful Scientific Computing Complex''''' - Galen Shipman, ORNL
:[[Media:Gshipman_lug_2009.pdf|''Slides'']]
:[http://slx.sun.com/1179274139 ''Video'']

'''''Lustre 2.0 Overview''''' - Andreas Dilger, Lustre, Sun
:[[Media:AndreasDilger.pdf|''Slides'']]
:[http://slx.sun.com/1179274141 ''Video'']

'''''HSM: Lustre/HSM Project - Manage your Data''''' - Aurélien Degrémont, CEA
:[[Media:AurelienDegremont.pdf|''Slides'']]
:[http://slx.sun.com/1179274197 ''Video'']

'''''Lustre in a WAN Environment''''' - James Hofmann and Dardo Kleiner, Naval Research Laboratory, David McMillen, System Fabric Works
:[[Media:JamesHoffman.pdf|''Slides'']]
:[http://slx.sun.com/1179274198 ''Video'']

'''''DDN Storage and Lustre''''' - Jeff Denworth, DataDirect Networks
:[[Media:DDN_LUG_Talk_-_Public.pdf|''Slides'']]
:[http://slx.sun.com/1179274199 ''Video'']

'''''Monitoring I/O Performance on Lustre''''' - Andrew Uselton, NERSC
:[[Media:AndrewUselton.pdf|''Slides'']]
:[http://slx.sun.com/1179274200 ''Video'']

'''''Lustre as the Core of a Data Centric Best Practice HPC Workflow''''' - Bob Murphy, Sun
:[[Media:LUG_Best_Practice_HPC_Workflow.pdf|''Slides'']]
:[http://slx.sun.com/1179274158 ''Video'']

'''Lessons Learned: Lustre on the Teragrid''' - Stephen Simms, Indiana University
:[[Media:StephenSimms.pdf|''Slides'']]
:[http://slx.sun.com/1179274201 ''Video'']

'''Open Fabrics Update''' - Bill Boas, OpenFabrics Alliance
:[[Media:OpenFabrics_Alliance.pdf|''Slides'']]
:[http://slx.sun.com/1179274202 ''Video'']

'''Open Q&A with Lustre Senior Engineers''' - Lustre Group
:[http://slx.sun.com/1179274203 ''Video'']

===Historical LUGs===

* [[Lustre User Group 2008]]
* [[Lustre User Group 2007]]
* [[Lustre User Group 2006]]

===Contact Us===

If you would like to receive Lustre User Group announcements, please subscribe to the [[Lustre Mailing_Lists|lustre-announce]] mailing list.

Lustre User Group 2010

2011-01-20T17:43:16Z

Sbarthel:

Thanks to all of the attendees and presenters at the 2010 Lustre User Group (LUG). This year's event was hosted at the beautiful Seascape Resort in Monterey Bay, California, and featured a Lustre Advanced User seminar and two days of informative presentations on select Lustre features, upcoming enhancements, site-specific experiences using Lustre.

==Lustre Advanced User Seminar 2010 Agenda==

<big>'''Wednesday, April 14'''</big> - Seascape Grand Ballroom, 2nd Floor, Seascape Conference Center

{| border=1 cellpadding=0
|-
|style="background:#E0E0E0"|'''8:00 - 9:00'''
|style="background:#E0E0E0"|'''Breakfast - Riviera Room, 3rd Floor, Seascape Conference Center'''
|-
|'''9:00 - 10:30'''
|'''''[[Media:LUG-2010-tricksRev.pdf|Lustre Tips and Tricks]] '''''
:Andreas Dilger, Oracle
|-
|style="background:#E0E0E0"|'''10:30 - 11:00'''
|style="background:#E0E0E0"|'''Coffee Break - Foyer, 2nd Floor, Seascape Conference Center'''
|-
|'''11:00 - 12:00'''
|'''''[[Media:LUG_User_Seminar_Presentation_-_Hill.pdf|Administering Lustre at Scale, Lessons Learned at ORNL]]'''''
:Jason Hill, ORNL
|-
|style="background:#E0E0E0"|'''12:00 - 1:00'''
|style="background:#E0E0E0"|'''Lunch - Riviera Room, 3rd Floor, Seascape Conference Center'''
|-
|'''1:00 - 2:30'''
|'''''[[Media:Lustre_hsm_seminar_lug10.pdf|A Look Inside HSM]]'''''
:Aurélien Degrémont and Thomas Leibovici, CEA
|-
|style="background:#E0E0E0"|'''2:30 - 3:00'''
|style="background:#E0E0E0"|'''Coffee Break - Foyer, 2nd Floor, Seascape Conference Center'''
|-
|'''3:00 - 5:00'''
|'''''[[Media:A_Deep_Dive_into_Lustre_Recovery_Mechanisms.pdf|A Deep Dive into Lustre Recovery Mechanisms]]'''''
:Johann Lombardi, Oracle
|}

==LUG 2010 Agenda==

LUG 2010, a two-day event, will feature a workshop and numerous presentations on select Lustre features, upcoming enhancements, site-specific experiences using Lustre, and much more.

<big>'''Thursday, April 15 - LUG Day 1'''</big> - Seascape Grand Ballroom, 2nd Floor, Seascape Conference Center

{| border=1 cellpadding=0
|-
|style="background:#E0E0E0"|'''8:00 - 9:00'''
|style="background:#E0E0E0"|'''Breakfast - Riviera/Bayview Rooms, 3rd Floor, Seascape Conference Center'''
|-
|'''9:00 - 9:30'''
|'''''[[Media:LUG_Keynote_Presentation-Bojanic-100415.pdf|LUG Kickoff]]'''''
:Peter Bojanic, Oracle
|-
|'''9:30 - 10:00'''
|'''''[[Media:Lustre_Development_Barton_LUG_2010.pdf|Lustre Development]]'''''
:Eric Barton, Oracle
|-
|style="background:#E0E0E0"|'''10:00 - 10:30'''
|style="background:#E0E0E0"|'''Coffee Break - Foyer, 2nd Floor, Seascape Conference Center'''
|-
|'''10:30 - 11:30'''
|'''''[[Media:LUG10_Peter_Jones_1.8.x_Update.pdf|Lustre 1.8 Update]]'''''
:Peter Jones, Oracle
|-
|'''11:30 - 12:00'''
|'''''[[Media:Rread-lustre20.pdf|Lustre 2.0]]'''''
:Robert Read, Oracle
|-
|style="background:#E0E0E0"|'''12:00 - 1:00'''
|style="background:#E0E0E0"|'''Lunch - Riviera/Bayview Rooms, 3rd Floor, Seascape Conference Center'''
|-
|'''1:00 - 1:30'''
|'''''[[Media:Buisson_NUMIOA_n_multirail.pdf|Getting the Best from Lustre in a NUMIOA and Multi-rail IB Environment]]'''''
:Sebastien Buisson, Bull
|-
|'''1:30 - 2:15'''
|'''''[[Media:SNL_LUG_2010_Sandia-rev2.pdf|What RedSky and Lustre Have Accomplished]]'''''
:Steve Monk, Sandia
:Joe Mervini, Sandia
|-
|'''2:15 - 3:00'''
|'''''[[Media:Gshipman_LUG_2010.pdf|Lustre at the OLCF: Experiences and Path Forward]]'''''
:Galen Shipman, ORNL
|-
|style="background:#E0E0E0"|'''3:00 - 3:30'''
|style="background:#E0E0E0"|'''Coffee Break - Foyer, 2nd Floor, Seascape Conference Center'''
|-
|'''3:30 - 4:00'''
|'''''[[Media:Lustre_user_group_2010.pdf|Comprehensive Lustre I/O Tracking with Vampir]]'''''
:Michael Kluge, ZIH
|-
|'''4:00 - 4:30'''
|'''''[[Media:LUG2010-CLUMEQ.pdf|Lustre Deployment and Early Experiences]]'''''
:Florent Parent, Clumeq
|-
|'''4:30 - 5:00'''
|'''''[[Media:LUG2010_Simms.pdf|Indiana University's Lustre WAN - Empowering Production Workflows on the TeraGrid]]'''''
:Stephen Simms, Indiana University
|-
|'''5:00 - 5:30'''
|'''''[[Media:CEALUG2010_v1.pdf|LCE: Lustre at CEA]]'''''
:Stéphane Thiell, CEA
|-
|style="background:#E0E0E0"|'''5:30 - 6:30'''
|style="background:#E0E0E0"|'''Break'''
|-
|'''6:30'''
|Dinner Reception - Seascape Resort: To be announced at Thursday's event
|-
|}

<big>'''Friday, April 16 - LUG Day 2'''</big> - Seascape Grand Ballroom, 2nd Floor, Seascape Conference Center

{| border=1 cellpadding=0
|-
|style="background:#E0E0E0"|'''8:00 - 9:00'''
|style="background:#E0E0E0" width=574|'''Breakfast - Riviera/Bayview Rooms, 3rd Floor, Seascape Conference Center'''
|-
|'''9:00 - 9:30'''
|'''''[[Media:lustre_smp_scaling_LUG_2010.pdf|Lustre SMP Scaling Improvements]]'''''
:Liang Zhen, Oracle
|-
|'''9:30 - 10:00'''
|'''''Lustre/HSM Binding'''''
:[[Media:hsm_lug10.aurelien.pdf|Aurelien Degremont, CEA]]
:[[Media:LUG2010.hsm.hua.2.pdf|Hua Huang, Oracle]]
|-
|'''10:00 - 10:30'''
|'''''Lustre Enabled WAN in Government, NRL'''''
:Jeremy Filizetti
:[[Media:LUG2010_Filizetti_NRL.pdf|NRL]]
:[[Media:LUG2010_Filizetti_SMSi.pdf|SMSi]]
|-
|style="background:#E0E0E0"|'''10:30 - 11:00'''
|style="background:#E0E0E0"|'''Coffee Break - Foyer, 2nd Floor, Seascape Conference Center'''
|-
|'''11:00 - 11:30'''
|'''''[[Media:lug2010_BP.pdf|Hedging Our Filesystem Bet]]'''''
:Kent Blancett, BP
|-
|'''11:30 - 12:00'''
|'''''[[Media:jhammond-lug.pdf|Analysis and Recovery from Lustre Faults/Failures on Ranger]]'''''
:John Hammond, TACC
|-
|style="background:#E0E0E0"|'''12:00 - 1:00'''
|style="background:#E0E0E0"|'''Lunch - Riviera/Bayview Rooms, 3rd Floor, Seascape Conference Center'''
|-
|'''1:00 - 1:45'''
|'''''[[Media:lug2010.13-palencia.pdf|Kerberized Lustre 2.0 over the WAN]]'''''
:Josephine Palencia, PSC
|-
|'''1:45 - 2:15'''
|'''''[[Media:Cardo_LUG2010.pdf|Reaping the Benefits of MetaData]]'''''
:Nic Cardo, NERSC
|-
|style="background:#E0E0E0"|'''2:15 - 2:45'''
|style="background:#E0E0E0"|'''Coffee Break - Foyer, 2nd Floor, Seascape Conference Center'''
|-
|'''2:45 - 3:15'''
|'''''[[Media:Lustre_MacOS.pdf|Porting Lustre to Operating Systems Other than Linux]]''''':
:Ken Hornstein, NRL
|-
|'''3:15 - 3:45'''
|'''''[[Media:LUG 2010 Lustre Community.pdf|Lustre and Community Development]]'''''
:Daniel Ferber, Oracle
|-
|'''3:45'''
|LUG 2010 Concludes
|-
|}

Change Log 2.0

2011-01-20T17:42:04Z

Sbarthel:

''(Updated: Aug 2010)''

= Welcome to Lustre 2.0.0 =

This release represents a departure from the previous trains that were closely related to one another. As such, we have no previous release to show you the change from. Future 2.x releases will show the change from this or subsequent releases. All that said, you can find out more details from one of the following sources:

[http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre 2.0 Manual'']

[http://wiki.lustre.org/images/6/60/821-2077-10.pdf ''Lustre 2.0.0 Release Notes'']

'''Support for networks:'''
* socklnd - any kernel supported by Lustre,
* qswlnd - Qsnet kernel modules 5.20 and later,
* openiblnd - IbGold 1.8.2,
* o2iblnd - OFED 1.1, 1.2.0, 1.2.5, 1.3, and 1.4.1
* viblnd - Voltaire ibhost 3.4.5 and later,
* ciblnd - Topspin 3.2.0,
* iiblnd - Infiniserv 3.3 + PathBits patch,
* gmlnd - GM 2.1.22 and later,
* mxlnd - MX 1.2.10 or later,
* ptllnd - Portals 3.3 / UNICOS/lc 1.5.x, 2.0.x

'''Server support for kernels:'''
* 2.6.18-164.11.1.el5 (RHEL 5)
* 2.6.18-164.11.1.0.1.el5 (OEL 5)

'''Client support for unpatched kernels:''' see [http://wiki.lustre.org/index.php?title=Patchless_Client "Patchless Client"]
* 2.6.18-164.11.1.el5 (RHEL 5),
* 2.6.18-164.11.1.0.1.el5 (OEL 5)
* 2.6.16.60-0.42.8 (SLES 10),
* 2.6.27.19-5 (SLES11)
* 2.6.29.4-167.fc11 (FC11)

'''Recommended e2fsprogs version:'''
* 1.41.10-sun2

Managing Lustre Failover

2011-01-20T17:40:25Z

Sbarthel:

''(Updated: Feb 2010)''

For information about managing Lustre™ failover, see [http://wiki.lustre.org/manual/LustreManual20_HTML/UnderstandingFailover.html#50438253_pgfId-1304327 Chapter 3: ''Understanding Failover in Lustre''] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual''].

NFS vs. Lustre

2011-01-20T17:39:20Z

Sbarthel:

''(Updated: Oct 2009)''

''DISCLAIMER - EXTERNAL CONTRIBUTOR CONTENT''

''This content was submitted by an external contributor. We provide this information as a resource for the Lustre™ open-source community, but we make no representation as to the accuracy, completeness or reliability of this information.''

----

The following is based on a post written by Lee Ward and posted on the Lustre-discuss mailing list and a couple of corrections supplied by Daniel Kobras and Nicolas Williams have been added. Further expansion and correction is welcome.

I'll begin by motivating both NFS and Lustre. Why do they exist? What
problems do they solve.

==NFS==

Way back in the day, ethernet and the concept of a workstation got
popular. There were many tools to copy files between machines but few
ways to share a name space; Have the directory hierarchy and it's
content directly accessible to an application on a foreign machine. This
made file sharing awkward. The model was to copy the file or files to
the workstation where the work was going to be done, do the work, and
copy the results back to some, hopefully, well maintained central
machine.

There ''were'' solutions to this at the time. I recall an attractive
alternative called RFS (I believe) from the Bell Labs folks, via some
place in England if I'm remembering right, it's been a looong time after
all. It had issues though. The nastiest issue for me was that if a
client went down the service side would freeze, at least partially.
Since this could happen willy-nilly, depending on the users wishes and
how well the power button on his workstation was protected, together
with the power cord and ethernet connection, this freezing of service
for any amount of time was difficult to accept. This was so even in a
rather small collection of machines.

The problem with RFS (?) and it's cousins were that they were all
stateful. The service side depended on state that was held at the
client. If the client went down, the service side couldn't continue
without a whole lot of recovery, timeouts, etc. It was a very *annoying*
problem.

In the latter half of the 1980's (am I remembering right?) SUN proposed
an open protocol called NFS. An implementation using this protocol could
do most everything RFS(?) could but it didn't suffer the service-side
hangs. It couldn't. It was stateless. If the client went down, the
server just didn't care. If the server went down, the client had the
opportunity to either give up on the local operation, usually with an
error returned, or wait. It was always up to the user and for client
failures the annoyance was limited to the user(s) on that client.

SUN, also, wisely desired the protocol to be ubiquitous. They published
it. They wanted *everyone* to adopt it. More, they would help
competitors. SUN held interoperability bake-a-thons to help with this.
It looks like they succeeded, all around :)

Let's sum up, then. The goals for NFS were:

# Share a local file system name space across the network.
# Do it in a robust, resilient way. Pesky FS issues because some user kicked the cord out of his workstation was unacceptable.
# Make it ubiquitous. SUN was a workstation vendor. They sold servers but almost everyone had a VAX in their back pocket where they made the infrastructure investment. SUN needed the high-value machines to support this protocol.

==Lustre==

Lustre has a weird story and I'm not going to go into all of it. The
shortest, relevant, part is that while there was at least one solution
that DOE/NNSA felt acceptable, GPFS, it was not available on anything
other than an IBM platform and because DOE/NNSA had a semi-formal policy
of buying from different vendors at each of the three labs we were kind
of stuck. Other file systems, existing and imminent, at the time were
examined but they were all distributed file systems and we needed IO
'''''bandwidth''''' . We needed lots, and lots of bandwidth.

We also needed that ubiquitous thing that SUN had as one of their goals.
We didn't want to pay millions of dollars for another GPFS. We felt that
would only be painting ourselves into a corner. Whatever we did, the
result ''had'' to be open. It also had to be attractive to smaller sites
as we wanted to turn loose of the ting at some point. If it was
attractive for smaller machines we felt we would win in the long term
as, eventually, the cost to further and maintain this thing was spread
across the community.

As far as technical goals, I guess we just wanted GPFS, but open. More
though, we wanted it to survive in our platform roadmaps for at least a
decade. The actual technical requirements for the contract that DOE/NNSA
executed with HP, CFS was the sub-contractor responsible for
development, can be found here:

<http://www-cs-students.stanford.edu/~trj/SGS_PathForward_SOW.pdf>

LLNL used to host this but it's no longer there? Oh well, hopefully this
link will be good for a while, at least.

I'm just going to jump to the end and sum the goals up:

# It must do ''everything'' NFS can. We relaxed the stateless thing though, see the next item for why.
# It must support full POSIX semantics; Last writer wins, POSIX locks, etc.
# It must support all of the transports we are interested in.
# It must be scalable, in that we can cheaply attach storage and both performance (reading *and* writing) and capacity within a single mounted file system increase in direct proportion.
# We wanted it to be easy, administratively. Our goal was that it be no harder than NFS to set up and maintain. We were involving too many folks with PhDs in the operation of our machines at the time. Before you yell '''FAIL''', I'll say we did try. I'll also say we didn't make CFS responsible for this part of the task. Don't blame them overly much, OK?
# We recognized we were asking for a stateful system, we wanted to mitigate that by having some focus on resiliency. These were big machines and clients died all the time.
# While not in the SOW, we structured the contract to accomplish some future form of wide acceptance. We wanted it to be ubiquitous.

That's a lot of goals! For the technical ones, the main ones are all
pretty much structured to ask two things of what became Lustre. First,
give us everything NFS functionally does but go far beyond it in
performance. Second, give us everything NFS functionally does but make
it completely equivalent to a local file system, semantically.

There's a little more we have to consider. NFS4 is a different beast
than NFS2 or NFS3. NFS{2,3} had some serious issues that became more
prominent as time went by. First, security; It had none. Folks had
bandaged on some different things to try to cure this but they weren't
standard across platforms. Second, it couldn't do the full POSIX
required semantics. That was attacked with the NFS lock protocols but it
was such an after-thought it will always remain problematic. Third, new
authorization possibilities introduced by Microsoft and then POSIX,
called ACLs, had no way of being accomplished.

NFS4 addresses those by:

# Introducing state. (Lots of resiliency mechanisms introduced to offset the downside of this, too.) NFS4 implementations are able to handle Posix advisory locks, but unlike Lustre, they don't support full Posix filesystem semantics. For example, NFS4 still follows the traditional NFS close-to-open cache consistency model whereas with Lustre, individual write()s are atomic and become immediately visible to all clients. NFSv4 can't handle O_APPEND, and has those close-to-open semantics. Those are the two large departures from POSIX in NFSv4. NFSv4.1 also adds metadata/data separation and data distribution, much like Lustre, but with the same POSIX semantics departures mentioned above. Also, NFSv4.1's "pNFS" concept doesn't have room for "capabilities" (in the distributed filesystem sense, not in the Linux capabilities sense), which means that OSSs and MDSs have to communicate to get permissions to be enforced. There are also differences with respect to recovery, etcetera. One thing about NFS is that it's meant to be neutral w.r.t. the type of filesystem it shares. So NFSv4, for example, has features for dealing with filesystems that don't have a notion of persistent inode number. Whereas Lustre has its own on-disk format and therefore can't be used to share just any type of filesystem.
# Formalizing and offering standardized authentication headers.
# Introducing ACLs that map to equivalents in POSIX and Microsoft.

==Strengths and Weaknesses of the Two==

NFS4 does most everything Lustre can with one very important exception,
IO bandwidth.

Both seem able to deliver metadata performance at roughly the same
speeds. File create, delete, and stat rates are about the same. NetApp
seems to have a partial enhancement. They bought the Spinnaker goodies
some time back and have deployed that technology, and redirection
too(?), within their servers. The good about that is two users in
different directories *could* leverage two servers, independently, and,
so, scale metadata performance. It's not guaranteed but at least there
is the possibility. If the two users are in the same directory, it's not
much different, though, I'm thinking. Someone correct me if I'm wrong?

Both can offer full POSIX now. It's nasty in both cases but, yes, in
theory you can export mail directory hierarchies with locking.

The NFS client and server are far easier to set up and maintain. The
tools to debug issues are advanced. While the Lustre folks have done
much to improve this area, NFS is just leaps and bounds ahead. It's
easier to deal with NFS than Lustre. Just far, far easier, still.

NFS is just built in to everything. My TV has it, for heck's sake. Lustre
is, seemingly, always an add-on. It's also a moving target. We're
constantly futzing with it, upgrading, and patching. Lustre might be
compilable most everywhere we care about but building it isn't trivial.
The supplied modules are great but, still, moving targets in that we
wait for SUN to catch up to the vendor supplied changes that affect
Lustre. Given Lustre's size and interaction with other components in the
OS, that happens far more frequently than desired. NFS just plain wins
the ubiquity argument at present.

NFS IO performance does *not* scale. It's still an in-band protocol. The
data is carried in the same message as the request and is, practically,
limited in size. Reads are more scalable in writes, a popular
file-segment can be satisfied from the cache on reads but develops
issues at some point. For writes, NFS3 and NFS4 help in that they
directly support write-behind so that a client doesn't have to wait for
data to go to disk, but it's just not enough. If one streams data
to/from the store, it can be larger than the cache. A client that might
read a file already made "hot" but at a very different rate just loses.
A client, writing, is always looking for free memory to buffer content.
Again, too many of these, simultaneously, and performance descends to
the native speed of the attached back-end store and that store can only
get so big.

Lustre IO performance *does* scale. It uses a 3rd-party transfer.
Requests are made to the metadata server and IO moves directly between
the affected storage component(s) and the client. The more storage
components, the less possibility of contention between clients and the
more data can be accepted/supplied per unit time.
NFS4 has a proposed extension, called pNFS, to address this problem. It
just introduces the 3rd-party data transfers that Lustre enjoys. If and
when that is a standard, and is well supported by clients and vendors,
the really big technical difference will virtually disappear. It's been
a long time coming, though. It's still not there. Will it ever be,
really?

The answer to the NFS vs. Lustre question comes down to the workload for
a given application then, since they do have overlap in their solution
space. If I were asked to look at a platform and recommend a solution I
would worry about IO bandwidth requirements. If the platform in question
were either read-mostly and, practically, never needed sustained read or
write bandwidth, NFS would be an easy choice. I'd even think hard about
NFS if the platform created many files but all were very small; Today's
filers have very respectable IOPS rates. If it came down to IO
bandwidth, I'm still on the parallel file system bandwagon. NFS just
can't deal with that at present and I do still have the folks, in house,
to manage the administrative burden.

Done. That was useful for me. I think five years ago I might have opted
for Lustre in the "create many small files" case, where I would consider
NFS today, so re-examining the motivations, relative strengths, and
weaknesses of both was useful. As I said, I did this more as a
self-exercise than anything else but I hope you can find something
useful here, too.

Patchless Client

2011-01-20T17:38:02Z

Sbarthel: /* Versions */

''(Updated: Oct 2009)''

As of Lustre™ 1.6.0, Lustre supports running the client modules on most current "stock" kernels without the need for patches to the client kernel. Patches are still required to the Lustre ''server'' kernel, but since these nodes generally run only Lustre, this is not a major limitation. Pre-built "patchless" RPMs can be found at the [http://www.sun.com/software/products/lustre/get.jsp Lustre download site].

We strongly recommend that you use a pre-built RPM rather than building your own. However, if you need to run a kernel on the client that is not one of the supported kernels, it is possible to build from source for the kernels listed at the top the Change Log for each release (see [[Lustre Release Information]]).

The Lustre configure script will automatically detect the unpatched kernel and disable building the servers.

[lustre]$ ./configure --with-linux=/unpatched/kernel/source

=== Versions ===
See the [http://wiki.lustre.org/index.php/Lustre_Release_Information#Lustre_Test_Matrix Lustre Test Matrix] for a list of kernels that are known to work with patchless Lustre clients. Note that Oracle does not test all of these kernel versions with each Lustre release, but it is expected that kernels between the oldest and newest listed versions work with a given Lustre release.

=== Known Issues ===

Many NFS-related bugs are also addressed by the patchless client fixes.