Obsolete Lustre Wiki - User contributions [en]

Main Page

2007-08-08T19:38:48Z

Braam: /* Lustre Centres of Excellence™ */

== What is Lustre®? ==

Lustre® is a scalable, secure, robust, highly-available cluster file system. It is designed, developed and maintained by [http://www.clusterfs.com Cluster File Systems, Inc.]

The central goal is the development of a next-generation cluster file system which can serve clusters with 10,000's of nodes, provide petabytes of storage, and move 100's of GB/sec with state-of-the-art security and management infrastructure.

Lustre runs on many of the largest Linux clusters in the world, and is included by CFS's partners as a core component of their cluster offering (examples include HP StorageWorks SFS, and the Cray XT3 and XD1 supercomputers). Today's users have also demonstrated that Lustre scales down as well as it scales up, and runs in production on clusters as small as 4 and as large as 25,000 nodes.

The latest version of Lustre is always available from [http://www.clusterfs.com Cluster File Systems, Inc.] Public Open Source releases of Lustre are available under the GNU General Public License. These releases are [http://www.clusterfs.com/download.html found here], and are used in production supercomputing environments worldwide.

To be informed of Lustre releases, subscribe to the [https://mail.clusterfs.com/mailman/listinfo/lustre-announce lustre-announce mailing list].

Lustre development would not have been possible without funding and guidance from many organizations, including several U.S. National Laboratories, early adopters, and product partners.

== User Resources ==

* [http://www.clusterfs.com/download.html Lustre Downloads]
* [http://wiki.lustre.org/index.php?title=Lustre_Quick_Start Lustre Quick Start]
* [http://wiki.lustre.org/index.php?title=Mailing_Lists Mailing Lists]
* [http://manual.lustre.org Lustre Operations Manual]
* [http://wiki.lustre.org/index.php?title=Bug_Filing Filing Bugs]
* [https://bugzilla.lustre.org/showdependencytree.cgi?id=2374 Lustre Knowledge Base]

== Advanced User Resources ==
* [http://wiki.lustre.org/index.php?title=BuildLustre How to build Lustre]
* [http://wiki.lustre.org/index.php?title=Kerb_Lustre Kerberos]
* [http://wiki.lustre.org/index.php?title=LustreTuning Lustre Tuning]
* [http://wiki.lustre.org/images/7/78/LustreManual.html#Chapter_III-2._LustreProc LustreProc] - A guide on the '''proc''' tunable parameters for Lustre and their usage. It describes several of the '''proc''' tunables including those that effect the client's RPC behavior and prepare for a substantial reorganization of '''proc''' entries.
* [http://wiki.lustre.org/index.php?title=LibLustre_HowTo Liblustre HowTo]

== Lustre Centres of Excellence™ ==

* [http://ornl-lce.clusterfs.com/index.php?title=Main_Page ORNL]
* [http://www.clusterfs-mwiki.com/cea-lce CEA]
* [http://www.clusterfs-mwiki.com/llnl-lce LLNL]
* [http://www.clusterfs-mwiki.com/psc-lce/index.php?title=Main_Page PSC]
* [http://www.clusterfs-mwiki.com/tsinghua-lce Tsinghua]

== Developer Resources ==

* [http://arch.lustre.org Lustre Architecture]
* [http://wiki.lustre.org/index.php?title=Contribution_Policy Contribution Policy]
* [http://wiki.lustre.org/index.php?title=Mailing_List Developer Mailing List]
* CVS usage
** [http://wiki.lustre.org/index.php?title=Cvs_Branches CVS Branches] - How to manage branches with CVS.
** [http://wiki.lustre.org/index.php?title=Cvs_Tips CVS Tips] - Helpful things to know while using Lustre CVS.
* [http://wiki.lustre.org/index.php?title=Lustre_Debugging Debugging Lustre] - A guide to debugging Lustre.
* [http://wiki.lustre.org/index.php?title=ZFS_Resources ZFS Resources] - Learn about ZFS.
* [http://wiki.lustre.org/index.php?title=Coding_Guidelines Coding Guidelines] - Developer guidelines to avoid problems during Lustre code merges.
* [http://wiki.lustre.org/index.php?title=Portals_Ping_Client_Server Portals Ping Server Client] - Kernel modules used to test basic message passing of ports.

== CFS Development Projects ==
* [http://wiki.lustre.org/index.php?title=IOPerformanceProject I/O Performance]

== Community Development Projects==
* [http://wiki.lustre.org/index.php?title=Networking_Development Networking Development]
* [http://wiki.lustre.org/index.php?title=Diskless_Booting Diskless Booting]
* [http://wiki.lustre.org/index.php?title=Drbd_And_Lustre DRBD and Lustre]
* [http://www.bullopensource.org/lustre Bull- Open Source tools for Lustre]
* [http://www.sourceforge.net/projects/lmt LLNL- Lustre Monitoring Tool]

== Other Resources ==
* [http://wiki.lustre.org/index.php?title=Lustre_Publications Lustre Publications] - Papers and presentations about Lustre
* [http://wiki.lustre.org/index.php?title=Lustre_User_Group Lustre User Group]
** LUG Requirements Forum - [http://wiki.lustre.org/images/7/78/LUG-Requirements-060420-final.pdf LUG-Requirements-060420-final.pdf] | [http://wiki.lustre.org/images/7/78/LUG-Requirements-060420-final.xls LUG-Requirements-060420-final.xls]
** [http://www.clusterfs.com/lug2007.html Lustre User Group 2007]
** [http://www.clusterfs.com/lug2006.html Lustre User Group 2006]

Architecture - Punch and Extent Migration

2007-08-08T16:11:14Z

Braam: Architecture Punch moved to Punch and Extent Migration

== Prototypes ==

=== Punch ===

punch(inode, start, end, version, flags)

=== Migrate ===

migrate(src_inode, start, end, tgt_inode, offset, version, flags)

== references ==

http://arch.lustre.org/index.php?title=QAS_Punch

[[Category:Architecture|Punch]]

Architecture - Multiple Interfaces For LNET

2007-07-30T16:02:29Z

Braam: /* = Use Cases */

== Use Cases ==

A node may have one or more link sets - these are a set of nids that
will be used in an aggregated fashion.

# one to many, many to one, many to many and rails situations need to be supported. Precisely link sets with K elements should be able to connect to link sets with L elements.
## clients with one interface to servers with 2
## vice versa
## rail situations
# a link set requires an aggregation descriptor:
## bandwidth aggregation behavior
## link level failover/failure recovery model
### Some of these are optional or for future versions.
### These descriptors need to go into /etc/modprobe.conf
# The MGS will be reached through passing multiple remote addresses describing a failover link set
# Aggregation is desirable for links on a single or on multiple LNETs
# Utilities like lctl ping <nid> can send/packets to an individual nid of an interface and to an aggregated link set. (a link set probably needs to be named with a nid)
# Clients will connect to servers by naming the server link set. This requirement is to allow clients outside a firewall to connect to a server behind a firewall where the server has non-reachable nids (like 192.168.1.*) which might have a different meaning near the client.
# Lustre will see multiple nid's only for failover, i.e. no new connection behavior
# The nids in a link set will be made available though the LNET management node (probably the MGS) to allow dynamic server addition.
# Configuration will allow "real failover IP addresses" to be configured.
# Desirable implementation constraint: a linkset will be a nid on a lustre LNET and the routing mechanisms will be used to reach the nid and implement aggregation behavior. LNET bandwidth sharing, failure handling etc.
# It shall be possible to specify lustre configurations for simultaneous use of different linksets on the same server targets.
# If modified nids are used, they shall be big enough to contains both linkset modifiers and IP-v6 addresses
# modprobe.conf shall remain a clusterwide file

== CONFIGURATION MANAGEMENT ==

=== Lustre configuration adaptations ===

A lustre configuration specification must be able to describ linksets
for each node that shall be used during Lustre setup.

The ip2net configuration directive is extremely similar to what we
need here.

options lnet 'ip2linkgroup="eth-oss-vib-mds 192.168.0.[2-20]@tcp0:ethall;
eth-oss-vib-mds 132.6.1.[2,3]@vib0; vib-all *@vib0"'

The ethall directive is a linkset nid modifier as defined in B below.

Specifying this as a modprobe.conf parameter is very desirable,
because every node would have linkgroup descriptors which it could use
to establish routes to aggregated linksets.

The mount command can give a parameter:

mount -t lustre -o linkgroup=<name> <mgs-nids-seq>:fsname /mnt/pt

The MGS will map a linkgroup name to a linkset nid (using one of the
two alternatives below) for each server, to be used by nodes
connecting to this. These linkset nids will be in the configuration
log and can be interpreted by LNET.

This allows, for example, the MDS to connect to OSS's over IB while
clients connect to the OSS's over TCP.

mount -t lustre -o linkgroup=vib-all /dev/mds-dev /mnt/pt
mount -t lustre -o linkgroup=eth-oss-vib-mds <mgs-nids-seq>:fsname /mnt/pt

=== linkgroup indicators in NIDs ===

define linksets in /etc/modprobe.conf without a requirement to
define a unique nid, e.g.:

linkset= <linkset-name>[{<aggr params>}]( <iface list> )

linkset=eth-all{failover,noloadbalance}(eth0 eth1)

extend the syntax of the nid from nid = <address>[@<network>] to

nid = <address>[@<network>][:<linkset name>]

Now:
192.168.1.5@tcp0:eth0
192.168.1.5@tcp0:eth-all
become valid nids.

== ISSUES ==

# Suppose servers are added with a previously unspecified network. In this case the MGS needs to learn this at addition time, in particular, the MGS would have to reparse its own /etc/moprobe.conf file or get information from the new servers.
# This requirements discussion does not address the naming of nodes, which might be an additional useful requirement.

[[Category:Architecture|Multiple Network Interfaces]]
[[Category:QAS|Multiple Network Interfaces]]

ZFS Resources

2007-07-01T16:13:25Z

Braam:

This page contains useful references to ZFS:

== Training ==

* [http://www.sun.com/software/solaris/zfs_learning_center.jsp ZFS courses, fun to watch!]

== Papers & Reference Material ==

* [http://opensolaris.org/os/community/zfs/docs/ ZFS Docs at Sun]
** Presentation
** Administration guide
** Manual pages
** Disk format specification
* [http://www.sun.com/bigadmin/features/articles/zfs_part1.scalable.html Two introductory papers on ZFS]

== Presentations ==

* [http://www.opensolaris.org/os/community/zfs/docs/zfs_last.pdf ZFS: The Last Word in File Systems]

== Links on the Web ==

* [http://blogs.sun.com/bonwick/category/ZFS Jeff Bonwick's ZFS blog entries (one of ZFS' principal designers)]
* [https://developer.berlios.de/projects/zfs-fuse/ DMU and ZFS FUSE on Linux]
* [http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide# ZFS Best Practices] - A guide with tuning and configuration details.
*
* [http://blogs.sun.com/jimlaurent/entry/zfs_resources ZFS Resources]
** Best practices
** Various articles with examples to learn from
** Tuning advice
* [http://blogs.sun.com/Peerapong/entry/zfs_by_examples ZFS by Examples]