Obsolete Lustre Wiki - User contributions [en]

Lustre Center of Excellence at Oak Ridge National Laboratory

2009-11-05T16:38:35Z

Jkdawson: Uploaded and added link to the final scalability gap response.

The Sun Lustre Center of Excellence (LCE) at ORNL advances the state of Lustre for use in large scale HPC environments. As you'll see from the projects below, the ORNL LCE focuses on both systems and applications aspects of ensuring that Lustre meets the needs of DOE and the HPC community in general.

==Current projects and their status==
===Applications IO Performance===
Mike Booth, one of the LCE personnel, is investigating applications IO performance on the Lustre systems at ORNL to identify ways to achieve more consistent and higher IO performance. He is also working on enhancing the ADIOS IO layer.

Feel free to [mailto:Michael.Booth@Sun.COM contact Mike] for more information and status of his work.

[[Implementing Applications in a Lustre Environment]] contains information on achieving high performance applications IO with the Lustre file system.

==Events==

The LCE sponsors events to encourage HPC community involvement
in analyzing IO and storage requirements and identifying ways for
Lustre to address these requirements.

The next event will be an Applications Workshop this Autumn 2009.

===LCE Summit - February 2008===
*[[Media:LCE_Summit_Summary_Draft_March_14_2008.pdf|LCE Summit Meeting Summary]]
*[[Media:LCESummitSlides.pdf|LCE Summit Slides and Notes]]

===LCE Application I/O Workshop - April 16, 2008===
*[[Media:April2008ApplicationIOWorkshop.pdf|Application IO Workshop, Agenda and Notes]]
*[[Media:Lustre_workshop_WangDi.pdf|Lustre and Application IO - Wang Di's Slides]]
*[[Media:Lustre_workshop_Oleg.pdf|Lustre and Application IO - Oleg Drokin's Slides]]

===Lustre Scalability Workshop - Feb 10 & 11, 2009, ORNL===
*[[Media:Notes_on_SW1_Notes.pdf|Scalability Workshop Notes]]
*[[Media:LustreScalabilityWP_Updated.pdf|Scalability White Paper]]
*[[Media:Shipman_Feb_lustre_scalability.pdf|Galen Shipman's presentation]]
*[[Media:Eric-Barton_-_Lustre-Multi_PF_Roadmap-090130.pdf|Eric Barton's presentation]]

===Lustre Scalability Workshop - May 19 & 20, 2009, ORNL===
The May Lustre Scalability Workshop at ORNL was focused on long term (2015) HPC IO and storage requirements and on presentations on the IO objectives of the DOD HPCS program and how Lustre will achieve them.
*[[Media:Dawson_Lustre_Workshop_May_2009.pdf|Lustre Scalability Workshop, Initial Gap Response - John Dawson]]
*[[Media:Carrier_2009-05-19_ORNL_LCE_HPCS.pdf|HPCS IO - John Carrier]]
*[[Media:Newman_May_Lustre_Workshop.pdf|What is HPCS and How Does it Impact IO - Henry Newman]]
*[[Media:Shipman_May_lustre_scalability_workshop.pdf|2015 Parallel File System Requirements - Galen Shipman]]
*[[Media:Dilger_Lustre_HPCS_May_Workshop.pdf|Lustre HPCS Design Overview - Andreas Dilger]]

===Scalability Workshop Follow up===
*[[Media:Gap Response.pdf|Scalability Gap Response]] is the final version of Sun's response to the scalability gaps identified and discussed during the LCE Scalability Workshops in February and May of 2009.

==White Papers==

LCE personnel have written a variety of papers on High Performance IO and potential Lustre features. Links to these documents are below.

*[https://www.sun.com/offers/details/Peta-Scale_wp.xml Lustre Scalability - An Oak Ridge National Laboratory/Lustre Center of Excellence Paper]

*[[Media:Lce_pop_submitted.pdf|Improving I/O Performance in POP (Parallel Ocean Program)]]

* Scalability Improvements
**[[Media:Lustre_Enhancement_Report_Interval_Trees.pdf|Using Interval Tree to scale extent locks]]
**[[Media:Lustre_Enhancement_Report_UUID_Hash_Tables.pdf|Implement hash tables to scale export lookups]]
**[[Media:A_Novel_Network_Request_Scheduler_for_a_Large_Scale_Storage_System.pdf|A Novel Network Request Scheduler for a Large Scale Storage System]]

*Lustre ADIO Driver Enhancements Whitepaper
**[[Media:Lustre_ADIO_Driver_Whitepaper_0926.pdf|Lustre ADIO Driver Enhancements]]

*FMS Application IO Analysis
**[[Media:FMS_Investigation_Report_%280915%29.pdf|FMS Application IO Performance Analysis]]

==Presentations==

See the ORNL [http://wiki.lustre.org/index.php/Lustre_User_Group slides and video] from their presentation at LUG 2009.

==Press Articles==

* [http://www.nccs.gov/2009/06/30/ornl-hosts-lustre-part-ii/ ORNL Hosts Lustre Part II]

==Lustre Internals Manual==
LCE and ORNL have written a Lustre filesystem internals document the describes the internal operation of Lustre.
*[[Media:Understanding_Lustre_Filesystem_Internals.pdf|Understanding Lustre Filesystem Internals]]

== Archives of Older Material ==

This material may be out of date, and is preserved here for archive purposes.

*[[Media:Peta-Scale_wp.pdf| 2008 Paper on IO with the Lustre File System at ORNL]]

<br>
For more information about computing at Oak Ridge, click [http://www.nccs.gov/ here].

If you have questions or comments on this page or the LCE projects, contact [mailto:John.Dawson@sun.com John Dawson].

File:Gap Response.pdf

2009-11-05T04:19:34Z

Jkdawson: Final gap response to Lustre Scalability gaps identified in February Workshop.

Final gap response to Lustre Scalability gaps identified in February Workshop.

GetInvolved:Get Involved

2009-10-19T02:45:43Z

Jkdawson: Added link to NFS vs. Lustre

Find out what the Lustre Community is doing and get involved.

<div class="categoryLeft">

<big><strong>Community Events and Resources</strong></big>

* Sign up for a [[Mailing_Lists|Lustre Mailing List]].
* Access Lustre User Group (LUG) agendas, presentations, panel discussions, videos and more. LUG is the Lustre community's premier event to learn and share knowledge about Lustre technology.
** [[Lustre_User_Group|Lustre User Group 2009]]
** [[Lug_08|Lustre User Group 2008]]
** [[Lug_07|Lustre User Group 2007]]
** [[Lug_06|Lustre User Group 2006]]
* Find out how to [http://wiki.lustre.org/index.php/Contribute:Contribute contribute code or help with testing].
* Read [[Lustre_Publications|Lustre publications]] such as white papers and Lustre engineering presentations.
* Learn about some typical [[Customers|Lustre Customers]].
* Sign up for [http://www.sun.com/emrkt/hpc/news/index.html?cid=e9464 Sun HPC News] for news, articles, tips and reviews of interest to the HPC community.

</div>

<div class="categoryRight">

<big><strong>Community Development Projects</strong></big>
Interesting projects from the Lustre user community that are available for public use.

* [http://wiki.lustre.org/images/d/d9/Lustre-amanda.pdf Backup and Recovery: Amanda and Lustre] Use an Amanda client to back up a Lustre filesystem from Lustre clients.(PDF)
* [http://sourceforge.net/projects/lmt LLNL - Lustre Monitoring Tool] Tracks activity of server nodes (MDS, OSS and portals routers) for one or more Lustre filesystems.
* [http://www.bullopensource.org/lustre/ Bull - Open Source Tools for Lustre] Manage one or more Lustre filesystems from an administrative node.
* [http://sourceforge.net/projects/lustre-shine/ CEA Administration Tool for Lustre 1.6] Set up and manage a Lustre filesystem on a cluster.

</div>
<div class="categoryRight">

<big><strong>Lustre Centers of Excellence</strong></big>

Find out about active Lustre Centers of Excellence (LCEs).

* [[Sun Lustre Center of Excellence at Oak Ridge National Laboratory]]

Another Lustre Center of Excellence is located at the Naval Research Laboratory.

Additionally, several Lustre Development Centers work with Sun to make significant code and testing contributions to Lustre and the Lustre Community. These centers are located at CEA, Juelich, LLNL, and Sandia.

</div>
<div class="categoryLeft">

<big><strong>Third Party Contributions</strong></big>

Topics contributed by the Lustre user community.

* [[DRBD and Lustre]] describes the Distributed Replicated Block Device used for building high-availability clusters.
* [[Lustre FUSE]] describes how to use Lustre with the FUSE file system.
* [[Lustre DDN Tuning]] describes how to configure DDN storage arrays for use with Lustre.
* [[Debian Install]] describes how to build and install Lustre on a machine running Debian Linux.
* [[NFS vs. Lustre]] describes some of the history and rationale behind Lustre and NFS and compares and contrasts them.
</div>

NFS vs. Lustre

2009-10-19T02:37:40Z

Jkdawson:

----

DISCLAIMER - EXTERNAL CONTRIBUTOR CONTENT

This content was submitted by an external contributor. We provide this information as a resource for the Lustre open-source community, but we make no representation as to the accuracy, completeness or reliability of this information.

----

The following is based on a post written by Lee Ward and posted on the Lustre-discuss mailing list
and a couple of corrections supplied by Daniel Kobras and Nicolas Williams have been
added. Further expansion and correction is welcome.

----

I'll begin by motivating both NFS and Lustre. Why do they exist? What
problems do they solve.

==NFS==

Way back in the day, ethernet and the concept of a workstation got
popular. There were many tools to copy files between machines but few
ways to share a name space; Have the directory hierarchy and it's
content directly accessible to an application on a foreign machine. This
made file sharing awkward. The model was to copy the file or files to
the workstation where the work was going to be done, do the work, and
copy the results back to some, hopefully, well maintained central
machine.

There ''were'' solutions to this at the time. I recall an attractive
alternative called RFS (I believe) from the Bell Labs folks, via some
place in England if I'm remembering right, it's been a looong time after
all. It had issues though. The nastiest issue for me was that if a
client went down the service side would freeze, at least partially.
Since this could happen willy-nilly, depending on the users wishes and
how well the power button on his workstation was protected, together
with the power cord and ethernet connection, this freezing of service
for any amount of time was difficult to accept. This was so even in a
rather small collection of machines.

The problem with RFS (?) and it's cousins were that they were all
stateful. The service side depended on state that was held at the
client. If the client went down, the service side couldn't continue
without a whole lot of recovery, timeouts, etc. It was a very *annoying*
problem.

In the latter half of the 1980's (am I remembering right?) SUN proposed
an open protocol called NFS. An implementation using this protocol could
do most everything RFS(?) could but it didn't suffer the service-side
hangs. It couldn't. It was stateless. If the client went down, the
server just didn't care. If the server went down, the client had the
opportunity to either give up on the local operation, usually with an
error returned, or wait. It was always up to the user and for client
failures the annoyance was limited to the user(s) on that client.

SUN, also, wisely desired the protocol to be ubiquitous. They published
it. They wanted *everyone* to adopt it. More, they would help
competitors. SUN held interoperability bake-a-thons to help with this.
It looks like they succeeded, all around :)

Let's sum up, then. The goals for NFS were:

# Share a local file system name space across the network.
# Do it in a robust, resilient way. Pesky FS issues because some user kicked the cord out of his workstation was unacceptable.
# Make it ubiquitous. SUN was a workstation vendor. They sold servers but almost everyone had a VAX in their back pocket where they made the infrastructure investment. SUN needed the high-value machines to support this protocol.

==Lustre==

Lustre has a weird story and I'm not going to go into all of it. The
shortest, relevant, part is that while there was at least one solution
that DOE/NNSA felt acceptable, GPFS, it was not available on anything
other than an IBM platform and because DOE/NNSA had a semi-formal policy
of buying from different vendors at each of the three labs we were kind
of stuck. Other file systems, existing and imminent, at the time were
examined but they were all distributed file systems and we needed IO
'''''bandwidth''''' . We needed lots, and lots of bandwidth.

We also needed that ubiquitous thing that SUN had as one of their goals.
We didn't want to pay millions of dollars for another GPFS. We felt that
would only be painting ourselves into a corner. Whatever we did, the
result ''had'' to be open. It also had to be attractive to smaller sites
as we wanted to turn loose of the ting at some point. If it was
attractive for smaller machines we felt we would win in the long term
as, eventually, the cost to further and maintain this thing was spread
across the community.

As far as technical goals, I guess we just wanted GPFS, but open. More
though, we wanted it to survive in our platform roadmaps for at least a
decade. The actual technical requirements for the contract that DOE/NNSA
executed with HP, CFS was the sub-contractor responsible for
development, can be found here:

<http://www-cs-students.stanford.edu/~trj/SGS_PathForward_SOW.pdf>

LLNL used to host this but it's no longer there? Oh well, hopefully this
link will be good for a while, at least.

I'm just going to jump to the end and sum the goals up:

# It must do ''everything'' NFS can. We relaxed the stateless thing though, see the next item for why.
# It must support full POSIX semantics; Last writer wins, POSIX locks, etc.
# It must support all of the transports we are interested in.
# It must be scalable, in that we can cheaply attach storage and both performance (reading *and* writing) and capacity within a single mounted file system increase in direct proportion.
# We wanted it to be easy, administratively. Our goal was that it be no harder than NFS to set up and maintain. We were involving too many folks with PhDs in the operation of our machines at the time. Before you yell '''FAIL''', I'll say we did try. I'll also say we didn't make CFS responsible for this part of the task. Don't blame them overly much, OK?
# We recognized we were asking for a stateful system, we wanted to mitigate that by having some focus on resiliency. These were big machines and clients died all the time.
# While not in the SOW, we structured the contract to accomplish some future form of wide acceptance. We wanted it to be ubiquitous.

That's a lot of goals! For the technical ones, the main ones are all
pretty much structured to ask two things of what became Lustre. First,
give us everything NFS functionally does but go far beyond it in
performance. Second, give us everything NFS functionally does but make
it completely equivalent to a local file system, semantically.

There's a little more we have to consider. NFS4 is a different beast
than NFS2 or NFS3. NFS{2,3} had some serious issues that became more
prominent as time went by. First, security; It had none. Folks had
bandaged on some different things to try to cure this but they weren't
standard across platforms. Second, it couldn't do the full POSIX
required semantics. That was attacked with the NFS lock protocols but it
was such an after-thought it will always remain problematic. Third, new
authorization possibilities introduced by Microsoft and then POSIX,
called ACLs, had no way of being accomplished.

NFS4 addresses those by:

# Introducing state. (Lots of resiliency mechanisms introduced to offset the downside of this, too.) NFS4 implementations are able to handle Posix advisory locks, but unlike Lustre, they don't support full Posix filesystem semantics. For example, NFS4 still follows the traditional NFS close-to-open cache consistency model whereas with Lustre, individual write()s are atomic and become immediately visible to all clients. <br /> <br /> NFSv4 can't handle O_APPEND, and has those close-to-open semantics. Those are the two large departures from POSIX in NFSv4. <br /> <br /> NFSv4.1 also adds metadata/data separation and data distribution, much like Lustre, but with the same POSIX semantics departures mentioned above. Also, NFSv4.1's "pNFS" concept doesn't have room for "capabilities" (in the distributed filesystem sense, not in the Linux capabilities sense), which means that OSSs and MDSs have to communicate to get permissions to be enforced. There are also differences with respect to recovery, etcetera. <br /> <br /> One thing about NFS is that it's meant to be neutral w.r.t. the type of filesystem it shares. So NFSv4, for example, has features for dealing with filesystems that don't have a notion of persistent inode number. Whereas Lustre has its own on-disk format and therefore can't be used to share just any type of filesystem.
# Formalizing and offering standardized authentication headers.
# Introducing ACLs that map to equivalents in POSIX and Microsoft.

==Strengths and Weaknesses of the Two==

NFS4 does most everything Lustre can with one very important exception,
IO bandwidth.

Both seem able to deliver metadata performance at roughly the same
speeds. File create, delete, and stat rates are about the same. NetApp
seems to have a partial enhancement. They bought the Spinnaker goodies
some time back and have deployed that technology, and redirection
too(?), within their servers. The good about that is two users in
different directories *could* leverage two servers, independently, and,
so, scale metadata performance. It's not guaranteed but at least there
is the possibility. If the two users are in the same directory, it's not
much different, though, I'm thinking. Someone correct me if I'm wrong?

Both can offer full POSIX now. It's nasty in both cases but, yes, in
theory you can export mail directory hierarchies with locking.

The NFS client and server are far easier to set up and maintain. The
tools to debug issues are advanced. While the Lustre folks have done
much to improve this area, NFS is just leaps and bounds ahead. It's
easier to deal with NFS than Lustre. Just far, far easier, still.

NFS is just built in to everything. My TV has it, for heck's sake. Lustre
is, seemingly, always an add-on. It's also a moving target. We're
constantly futzing with it, upgrading, and patching. Lustre might be
compilable most everywhere we care about but building it isn't trivial.
The supplied modules are great but, still, moving targets in that we
wait for SUN to catch up to the vendor supplied changes that affect
Lustre. Given Lustre's size and interaction with other components in the
OS, that happens far more frequently than desired. NFS just plain wins
the ubiquity argument at present.

NFS IO performance does *not* scale. It's still an in-band protocol. The
data is carried in the same message as the request and is, practically,
limited in size. Reads are more scalable in writes, a popular
file-segment can be satisfied from the cache on reads but develops
issues at some point. For writes, NFS3 and NFS4 help in that they
directly support write-behind so that a client doesn't have to wait for
data to go to disk, but it's just not enough. If one streams data
to/from the store, it can be larger than the cache. A client that might
read a file already made "hot" but at a very different rate just loses.
A client, writing, is always looking for free memory to buffer content.
Again, too many of these, simultaneously, and performance descends to
the native speed of the attached back-end store and that store can only
get so big.

Lustre IO performance *does* scale. It uses a 3rd-party transfer.
Requests are made to the metadata server and IO moves directly between
the affected storage component(s) and the client. The more storage
components, the less possibility of contention between clients and the
more data can be accepted/supplied per unit time.
NFS4 has a proposed extension, called pNFS, to address this problem. It
just introduces the 3rd-party data transfers that Lustre enjoys. If and
when that is a standard, and is well supported by clients and vendors,
the really big technical difference will virtually disappear. It's been
a long time coming, though. It's still not there. Will it ever be,
really?

The answer to the NFS vs. Lustre question comes down to the workload for
a given application then, since they do have overlap in their solution
space. If I were asked to look at a platform and recommend a solution I
would worry about IO bandwidth requirements. If the platform in question
were either read-mostly and, practically, never needed sustained read or
write bandwidth, NFS would be an easy choice. I'd even think hard about
NFS if the platform created many files but all were very small; Today's
filers have very respectable IOPS rates. If it came down to IO
bandwidth, I'm still on the parallel file system bandwagon. NFS just
can't deal with that at present and I do still have the folks, in house,
to manage the administrative burden.

Done. That was useful for me. I think five years ago I might have opted
for Lustre in the "create many small files" case, where I would consider
NFS today, so re-examining the motivations, relative strengths, and
weaknesses of both was useful. As I said, I did this more as a
self-exercise than anything else but I hope you can find something
useful here, too.

NFS vs. Lustre

2009-10-19T02:34:50Z

Jkdawson: Initial insertion of page.

----

DISCLAIMER - EXTERNAL CONTRIBUTOR CONTENT

This content was submitted by an external contributor. We provide this information as a resource for the Lustre open-source community, but we make no representation as to the accuracy, completeness or reliability of this information.

----

The following is based on a post written by Lee Ward and posted on the Lustre-discuss mailing list
and a couple of corrections supplied by Daniel Kobras and Nicolas Williams have been
added. Further expansion and correction is welcome.

----

I'll begin by motivating both NFS and Lustre. Why do they exist? What
problems do they solve.

==NFS==

Way back in the day, ethernet and the concept of a workstation got
popular. There were many tools to copy files between machines but few
ways to share a name space; Have the directory hierarchy and it's
content directly accessible to an application on a foreign machine. This
made file sharing awkward. The model was to copy the file or files to
the workstation where the work was going to be done, do the work, and
copy the results back to some, hopefully, well maintained central
machine.

There ''were'' solutions to this at the time. I recall an attractive
alternative called RFS (I believe) from the Bell Labs folks, via some
place in England if I'm remembering right, it's been a looong time after
all. It had issues though. The nastiest issue for me was that if a
client went down the service side would freeze, at least partially.
Since this could happen willy-nilly, depending on the users wishes and
how well the power button on his workstation was protected, together
with the power cord and ethernet connection, this freezing of service
for any amount of time was difficult to accept. This was so even in a
rather small collection of machines.

The problem with RFS (?) and it's cousins were that they were all
stateful. The service side depended on state that was held at the
client. If the client went down, the service side couldn't continue
without a whole lot of recovery, timeouts, etc. It was a very *annoying*
problem.

In the latter half of the 1980's (am I remembering right?) SUN proposed
an open protocol called NFS. An implementation using this protocol could
do most everything RFS(?) could but it didn't suffer the service-side
hangs. It couldn't. It was stateless. If the client went down, the
server just didn't care. If the server went down, the client had the
opportunity to either give up on the local operation, usually with an
error returned, or wait. It was always up to the user and for client
failures the annoyance was limited to the user(s) on that client.

SUN, also, wisely desired the protocol to be ubiquitous. They published
it. They wanted *everyone* to adopt it. More, they would help
competitors. SUN held interoperability bake-a-thons to help with this.
It looks like they succeeded, all around :)

Let's sum up, then. The goals for NFS were:

# Share a local file system name space across the network.
# Do it in a robust, resilient way. Pesky FS issues because some user kicked the cord out of his workstation was unacceptable.
# Make it ubiquitous. SUN was a workstation vendor. They sold servers but almost everyone had a VAX in their back pocket where they made the infrastructure investment. SUN needed the high-value machines to support this protocol.

==Lustre==

Lustre has a weird story and I'm not going to go into all of it. The
shortest, relevant, part is that while there was at least one solution
that DOE/NNSA felt acceptable, GPFS, it was not available on anything
other than an IBM platform and because DOE/NNSA had a semi-formal policy
of buying from different vendors at each of the three labs we were kind
of stuck. Other file systems, existing and imminent, at the time were
examined but they were all distributed file systems and we needed IO
'''''bandwidth''''' . We needed lots, and lots of bandwidth.

We also needed that ubiquitous thing that SUN had as one of their goals.
We didn't want to pay millions of dollars for another GPFS. We felt that
would only be painting ourselves into a corner. Whatever we did, the
result ''had'' to be open. It also had to be attractive to smaller sites
as we wanted to turn loose of the ting at some point. If it was
attractive for smaller machines we felt we would win in the long term
as, eventually, the cost to further and maintain this thing was spread
across the community.

As far as technical goals, I guess we just wanted GPFS, but open. More
though, we wanted it to survive in our platform roadmaps for at least a
decade. The actual technical requirements for the contract that DOE/NNSA
executed with HP, CFS was the sub-contractor responsible for
development, can be found here:

<http://www-cs-students.stanford.edu/~trj/SGS_PathForward_SOW.pdf>

LLNL used to host this but it's no longer there? Oh well, hopefully this
link will be good for a while, at least.

I'm just going to jump to the end and sum the goals up:

# It must do ''everything'' NFS can. We relaxed the stateless thing though, see the next item for why.
# It must support full POSIX semantics; Last writer wins, POSIX locks, etc.
# It must support all of the transports we are interested in.
# It must be scalable, in that we can cheaply attach storage and both performance (reading *and* writing) and capacity within a single mounted file system increase in direct proportion.
# We wanted it to be easy, administratively. Our goal was that it be no harder than NFS to set up and maintain. We were involving too many folks with PhDs in the operation of our machines at the time. Before you yell '''FAIL''', I'll say we did try. I'll also say we didn't make CFS responsible for this part of the task. Don't blame them overly much, OK?
# We recognized we were asking for a stateful system, we wanted to mitigate that by having some focus on resiliency. These were big machines and clients died all the time.
# While not in the SOW, we structured the contract to accomplish some future form of wide acceptance. We wanted it to be ubiquitous.

That's a lot of goals! For the technical ones, the main ones are all
pretty much structured to ask two things of what became Lustre. First,
give us everything NFS functionally does but go far beyond it in
performance. Second, give us everything NFS functionally does but make
it completely equivalent to a local file system, semantically.

There's a little more we have to consider. NFS4 is a different beast
than NFS2 or NFS3. NFS{2,3} had some serious issues that became more
prominent as time went by. First, security; It had none. Folks had
bandaged on some different things to try to cure this but they weren't
standard across platforms. Second, it couldn't do the full POSIX
required semantics. That was attacked with the NFS lock protocols but it
was such an after-thought it will always remain problematic. Third, new
authorization possibilities introduced by Microsoft and then POSIX,
called ACLs, had no way of being accomplished.

NFS4 addresses those by:

# Introducing state. (Lots of resiliency mechanisms introduced to offset the downside of this, too.) NFS4 implementations are able to handle Posix advisory locks, but unlike Lustre, they don't support full Posix filesystem semantics. For example, NFS4 still follows the traditional NFS close-to-open cache consistency model whereas with Lustre, individual write()s are atomic and become immediately visible to all clients. <br /> <br /> NFSv4 can't handle O_APPEND, and has those close-to-open semantics. Those are the two large departures from POSIX in NFSv4. <br /> <br /> NFSv4.1 also adds metadata/data separation and data distribution, much like Lustre, but with the same POSIX semantics departures mentioned above. Also, NFSv4.1's "pNFS" concept doesn't have room for "capabilities" (in the distributed filesystem sense, not in the Linux capabilities sense), which means that OSSs and MDSs have to communicate to get permissions to be enforced. There are also differences with respect to recovery, etcetera. <br /> <br /> One thing about NFS is that it's meant to be neutral w.r.t. the type of filesystem it shares. So NFSv4, for example, has features for dealing with filesystems that don't have a notion of persistent inode number. Whereas Lustre has its own on-disk format and therefore can't be used to share just any type of filesystem.
# Formalizing and offering standardized authentication headers.
# Introducing ACLs that map to equivalents in POSIX and Microsoft.

==Strengths and Weaknesses of the Two==
-----------------------------------

NFS4 does most everything Lustre can with one very important exception,
IO bandwidth.

Both seem able to deliver metadata performance at roughly the same
speeds. File create, delete, and stat rates are about the same. NetApp
seems to have a partial enhancement. They bought the Spinnaker goodies
some time back and have deployed that technology, and redirection
too(?), within their servers. The good about that is two users in
different directories *could* leverage two servers, independently, and,
so, scale metadata performance. It's not guaranteed but at least there
is the possibility. If the two users are in the same directory, it's not
much different, though, I'm thinking. Someone correct me if I'm wrong?

Both can offer full POSIX now. It's nasty in both cases but, yes, in
theory you can export mail directory hierarchies with locking.

The NFS client and server are far easier to set up and maintain. The
tools to debug issues are advanced. While the Lustre folks have done
much to improve this area, NFS is just leaps and bounds ahead. It's
easier to deal with NFS than Lustre. Just far, far easier, still.

NFS is just built in to everything. My TV has it, for heck's sake. Lustre
is, seemingly, always an add-on. It's also a moving target. We're
constantly futzing with it, upgrading, and patching. Lustre might be
compilable most everywhere we care about but building it isn't trivial.
The supplied modules are great but, still, moving targets in that we
wait for SUN to catch up to the vendor supplied changes that affect
Lustre. Given Lustre's size and interaction with other components in the
OS, that happens far more frequently than desired. NFS just plain wins
the ubiquity argument at present.

NFS IO performance does *not* scale. It's still an in-band protocol. The
data is carried in the same message as the request and is, practically,
limited in size. Reads are more scalable in writes, a popular
file-segment can be satisfied from the cache on reads but develops
issues at some point. For writes, NFS3 and NFS4 help in that they
directly support write-behind so that a client doesn't have to wait for
data to go to disk, but it's just not enough. If one streams data
to/from the store, it can be larger than the cache. A client that might
read a file already made "hot" but at a very different rate just loses.
A client, writing, is always looking for free memory to buffer content.
Again, too many of these, simultaneously, and performance descends to
the native speed of the attached back-end store and that store can only
get so big.

Lustre IO performance *does* scale. It uses a 3rd-party transfer.
Requests are made to the metadata server and IO moves directly between
the affected storage component(s) and the client. The more storage
components, the less possibility of contention between clients and the
more data can be accepted/supplied per unit time.
NFS4 has a proposed extension, called pNFS, to address this problem. It
just introduces the 3rd-party data transfers that Lustre enjoys. If and
when that is a standard, and is well supported by clients and vendors,
the really big technical difference will virtually disappear. It's been
a long time coming, though. It's still not there. Will it ever be,
really?

The answer to the NFS vs. Lustre question comes down to the workload for
a given application then, since they do have overlap in their solution
space. If I were asked to look at a platform and recommend a solution I
would worry about IO bandwidth requirements. If the platform in question
were either read-mostly and, practically, never needed sustained read or
write bandwidth, NFS would be an easy choice. I'd even think hard about
NFS if the platform created many files but all were very small; Today's
filers have very respectable IOPS rates. If it came down to IO
bandwidth, I'm still on the parallel file system bandwagon. NFS just
can't deal with that at present and I do still have the folks, in house,
to manage the administrative burden.

Done. That was useful for me. I think five years ago I might have opted
for Lustre in the "create many small files" case, where I would consider
NFS today, so re-examining the motivations, relative strengths, and
weaknesses of both was useful. As I said, I did this more as a
self-exercise than anything else but I hope you can find something
useful here, too.

Main Page

2009-10-02T20:48:52Z

Jkdawson:

<div class="homeLeft">
<h1>High Performance and Scalability</h1>

<p>For the world's largest and most complex computing environments, the Lustre™ file system redefines high performance, scaling to tens of thousands of nodes and petabytes of storage with groundbreaking I/O and metadata throughput.
</p>

<blockquote>
'''More on Lustre performance, service, and support'''
at [http://www.sun.com/software/products/lustre/ sun.com/lustre]
</blockquote>

<br>
<strong>[http://wiki.lustre.org/index.php/HPC_Software_Workshop_and_Seminars_-_Regensburg_Germany_2009 HPC Software Workshop and Seminar - Lustre Slides]</strong>
<p>Slides from the presentations at the Lustre Advanced Seminar and Open Storage track at the Sun HPC Software Workshop held September 7-10 are posted [http://wiki.lustre.org/index.php/HPC_Software_Workshop_and_Seminars_-_Regensburg_Germany_2009 here]. Please consider joining the Sun Lustre team at SC09, November 14-20 in Portland, Oregon and also at LUG 2010, April 14-16 at Monterey Bay, California. Registration for LUG 2010 will open in November.
</p>

<br>
<strong>[[Lustre 1.8|Lustre 1.8.1]]</strong>
<p>Lustre 1.8.1 is now available for [http://www.sun.com/software/products/lustre/get.jsp download]. Lustre 1.8.1 is a maintenance release to 1.8.0 that provides a number of bug fixes and adds SLES 11 support. Read more about the 1.8 features - [[Lustre_1.8#Adaptive_Timeouts|Adaptive Timeouts]], [[Lustre_1.8#OSS_Read_Cache|OSS Read Cache]], [[Lustre_1.8#OST_Pools|OST Pools]] and [[Lustre_1.8#Version-Based_Recovery|Version-based Recovery]] and why you should upgrade.</p>

<br>
<strong>[[Testing_Path_to_Lustre_2.0_GA|Lustre 2.0 Alpha-4 Available]]</strong>
<p>We are pleased to announce that Lustre 2.0 Alpha-4 is available for download. This is the fourth step in a series of milestone based pre-releases as we move towards Lustre 2.0 GA. New milestone releases will be planned for every 4-6 weeks.
</p>

<br>
<strong> [[Lustre_User_Group|2009 Lustre User Group Slides and LUG 2010]]</strong>
<p>LUG 2009 was our most successful Lustre user event to date. For presentations and videos, see [[Lustre_User_Group|LUG 2009]]. Plan to join us for LUG 2010 April 14-16 at the [http://www.seascaperesort.com/ Seascape] resort and conference center on Monterey Bay, California. Registration for LUG 2010 will open in November.
</p>

<blockquote class="usermessage">
<h6>
<small>Send feedback on the lustre wiki to: [mailto:lustre-wiki-feedback@sun.com lustre-wiki-feedback@sun.com]</small>
</h6>
</blockquote>
</div>

<div class="homeRight">
<ul>
<li class="dnld">
<big>[[Download]]</big>
Download this open source software for free.
<ul>
<li><strong>[http://www.sun.com/software/products/lustre/get.jsp Get Lustre]</strong></li>
</ul>
</li>

<li class="lrn">
<big>[[Learn]]</big>
Find out about Lustre interoperability, features and publications.
<ul>
<li>[[Learn|Projects and Roadmap]]</li>
<li>[[Learn:Lustre_Publications|Publications]]</li>
</ul>
</li>

<li class="use">
<big>[[Use]]</big>
User resources to make Lustre perform at its best.
<ul>
<li>[[Use:Install_and_Configure|Install and Configure]]</li>
<li>[[Use:Administer|Administer]]</li>
</ul>
</li>

<li class="contribute">
<big>[[Contribute]]</big>
Developer resources and tools to contribute to Lustre.
<ul>
<li>[[Contribute:Contribute|Contribute to Lustre]]</li>
<li>[[Contribute:Contribute|Developer Resources]]</li>
</ul>
</li>

<li class="participate">
<big>[[Get Involved]]</big>
Find groups and community developer projects.
<ul>
<li>[[Get Involved|Community]]</li>
<li>[[GetInvolved:Lustre_Centers_of_Excellence|Lustre Centers of Excellence]]</li>
</ul>
</li>
</ul>
</div>

File:LustreScalabilityWP Updated.pdf

2009-09-15T21:01:57Z

Jkdawson: uploaded a new version of "File:LustreScalabilityWP Updated.pdf"

Lustre Scalability Whitepaper (Draft - March 8, 2009)

Lustre Center of Excellence at Oak Ridge National Laboratory

2009-08-17T01:13:51Z

Jkdawson: /* Applications IO Performance */

The Sun Lustre Center of Excellence (LCE) at ORNL advances the state of Lustre for use in large scale HPC environments. As you'll see from the projects below, the ORNL LCE focuses on both systems and applications aspects of ensuring that Lustre meets the needs of DOE and the HPC community in general.

==Current projects and their status==
===Applications IO Performance===
Mike Booth, one of the LCE personnel, is investigating applications IO performance on the Lustre systems at ORNL to identify ways to achieve more consistent and higher IO performance. He is also working on enhancing the ADIOS IO layer.

Feel free to [mailto:Michael.Booth@Sun.COM contact Mike] for more information and status of his work.

[[Implementing Applications in a Lustre Environment]] contains information on achieving high performance applications IO with the Lustre file system.

==Events==

The LCE sponsors events to encourage HPC community involvement
in analyzing IO and storage requirements and identifying ways for
Lustre to address these requirements.

The next event will be an Applications Workshop this Autumn 2009.

===LCE Summit - February 2008===
*[[Media:LCE_Summit_Summary_Draft_March_14_2008.pdf|LCE Summit Meeting Summary]]
*[[Media:LCESummitSlides.pdf|LCE Summit Slides and Notes]]

===LCE Application I/O Workshop - April 16, 2008===
*[[Media:April2008ApplicationIOWorkshop.pdf|Application IO Workshop, Agenda and Notes]]
*[[Media:Lustre_workshop_WangDi.pdf|Lustre and Application IO - Wang Di's Slides]]
*[[Media:Lustre_workshop_Oleg.pdf|Lustre and Application IO - Oleg Drokin's Slides]]

===Lustre Scalability Workshop - Feb 10 & 11, 2009, ORNL===
*[[Media:Notes_on_SW1_Notes.pdf|Scalability Workshop Notes]]
*[[Media:LustreScalabilityWP_Updated.pdf|Scalability White Paper]]
*[[Media:Shipman_Feb_lustre_scalability.pdf|Galen Shipman's presentation]]
*[[Media:Eric-Barton_-_Lustre-Multi_PF_Roadmap-090130.pdf|Eric Barton's presentation]]

===Lustre Scalability Workshop - May 19 & 20, 2009, ORNL===
The May Lustre Scalability Workshop at ORNL was focused on long term (2015) HPC IO and storage requirements and on presentations on the IO objectives of the DOD HPCS program and how Lustre will achieve them.
*[[Media:Dawson_Lustre_Workshop_May_2009.pdf|Lustre Scalability Workshop, Initial Gap Response - John Dawson]]
*[[Media:Carrier_2009-05-19_ORNL_LCE_HPCS.pdf|HPCS IO - John Carrier]]
*[[Media:Newman_May_Lustre_Workshop.pdf|What is HPCS and How Does it Impact IO - Henry Newman]]
*[[Media:Shipman_May_lustre_scalability_workshop.pdf|2015 Parallel File System Requirements - Galen Shipman]]
*[[Media:Dilger_Lustre_HPCS_May_Workshop.pdf|Lustre HPCS Design Overview - Andreas Dilger]]

==White Papers==

LCE personnel have written a variety of papers on High Performance IO and potential Lustre features. Links to these documents are below.

*[[Media:Peta-Scale_wp.pdf|Peta-Scale IO with the Lustre File System]]

*[[Media:Lce_pop_submitted.pdf|Improving I/O Performance in POP (Parallel Ocean Program)]]

* Scalability Improvements
**[[Media:Lustre_Enhancement_Report_Interval_Trees.pdf|Using Interval Tree to scale extent locks]]
**[[Media:Lustre_Enhancement_Report_UUID_Hash_Tables.pdf|Implement hash tables to scale export lookups]]
**[[Media:A_Novel_Network_Request_Scheduler_for_a_Large_Scale_Storage_System.pdf|A Novel Network Request Scheduler for a Large Scale Storage System]]

*Lustre ADIO Driver Enhancements Whitepaper
**[[Media:Lustre_ADIO_Driver_Whitepaper_0926.pdf|Lustre ADIO Driver Enhancements]]

*FMS Application IO Analysis
**[[Media:FMS_Investigation_Report_%280915%29.pdf|FMS Application IO Performance Analysis]]

==Presentations==

See the ORNL [http://wiki.lustre.org/index.php/Lustre_User_Group slides and video] from their presentation at LUG 2009.

==Press Articles==

* [http://www.nccs.gov/2009/06/30/ornl-hosts-lustre-part-ii/ ORNL Hosts Lustre Part II]

==Lustre Internals Manual==
LCE and ORNL have written a Lustre filesystem internals document the describes the internal operation of Lustre.
*[[Media:Understanding_Lustre_Filesystem_Internals.pdf|Understanding Lustre Filesystem Internals]]

<br>
For more information about computing at Oak Ridge, click [http://www.nccs.gov/ here].

If you have questions or comments on this page or the LCE projects, contact [mailto:John.Dawson@sun.com John Dawson].

Lustre Center of Excellence at Oak Ridge National Laboratory

2009-08-17T01:13:38Z

Jkdawson: /* Applications IO Performance */

The Sun Lustre Center of Excellence (LCE) at ORNL advances the state of Lustre for use in large scale HPC environments. As you'll see from the projects below, the ORNL LCE focuses on both systems and applications aspects of ensuring that Lustre meets the needs of DOE and the HPC community in general.

==Current projects and their status==
===Applications IO Performance===
Mike Booth, one of the LCE personnel, is investigating applications IO performance on the Lustre systems at ORNL to identify ways to achieve more consistent and higher IO performance. He is also working on enhancing the ADIOS IO layer.

Feel free to [mailto:Michael.Booth@Sun.COM contact Mike] for more information and status of his work.

[[Implementing Applications in a Lustre Environment]] contains information on achieving high performance applications IO with the Lustre file system.

==Events==

The LCE sponsors events to encourage HPC community involvement
in analyzing IO and storage requirements and identifying ways for
Lustre to address these requirements.

The next event will be an Applications Workshop this Autumn 2009.

===LCE Summit - February 2008===
*[[Media:LCE_Summit_Summary_Draft_March_14_2008.pdf|LCE Summit Meeting Summary]]
*[[Media:LCESummitSlides.pdf|LCE Summit Slides and Notes]]

===LCE Application I/O Workshop - April 16, 2008===
*[[Media:April2008ApplicationIOWorkshop.pdf|Application IO Workshop, Agenda and Notes]]
*[[Media:Lustre_workshop_WangDi.pdf|Lustre and Application IO - Wang Di's Slides]]
*[[Media:Lustre_workshop_Oleg.pdf|Lustre and Application IO - Oleg Drokin's Slides]]

===Lustre Scalability Workshop - Feb 10 & 11, 2009, ORNL===
*[[Media:Notes_on_SW1_Notes.pdf|Scalability Workshop Notes]]
*[[Media:LustreScalabilityWP_Updated.pdf|Scalability White Paper]]
*[[Media:Shipman_Feb_lustre_scalability.pdf|Galen Shipman's presentation]]
*[[Media:Eric-Barton_-_Lustre-Multi_PF_Roadmap-090130.pdf|Eric Barton's presentation]]

===Lustre Scalability Workshop - May 19 & 20, 2009, ORNL===
The May Lustre Scalability Workshop at ORNL was focused on long term (2015) HPC IO and storage requirements and on presentations on the IO objectives of the DOD HPCS program and how Lustre will achieve them.
*[[Media:Dawson_Lustre_Workshop_May_2009.pdf|Lustre Scalability Workshop, Initial Gap Response - John Dawson]]
*[[Media:Carrier_2009-05-19_ORNL_LCE_HPCS.pdf|HPCS IO - John Carrier]]
*[[Media:Newman_May_Lustre_Workshop.pdf|What is HPCS and How Does it Impact IO - Henry Newman]]
*[[Media:Shipman_May_lustre_scalability_workshop.pdf|2015 Parallel File System Requirements - Galen Shipman]]
*[[Media:Dilger_Lustre_HPCS_May_Workshop.pdf|Lustre HPCS Design Overview - Andreas Dilger]]

==White Papers==

LCE personnel have written a variety of papers on High Performance IO and potential Lustre features. Links to these documents are below.

*[[Media:Peta-Scale_wp.pdf|Peta-Scale IO with the Lustre File System]]

*[[Media:Lce_pop_submitted.pdf|Improving I/O Performance in POP (Parallel Ocean Program)]]

* Scalability Improvements
**[[Media:Lustre_Enhancement_Report_Interval_Trees.pdf|Using Interval Tree to scale extent locks]]
**[[Media:Lustre_Enhancement_Report_UUID_Hash_Tables.pdf|Implement hash tables to scale export lookups]]
**[[Media:A_Novel_Network_Request_Scheduler_for_a_Large_Scale_Storage_System.pdf|A Novel Network Request Scheduler for a Large Scale Storage System]]

*Lustre ADIO Driver Enhancements Whitepaper
**[[Media:Lustre_ADIO_Driver_Whitepaper_0926.pdf|Lustre ADIO Driver Enhancements]]

*FMS Application IO Analysis
**[[Media:FMS_Investigation_Report_%280915%29.pdf|FMS Application IO Performance Analysis]]

==Presentations==

See the ORNL [http://wiki.lustre.org/index.php/Lustre_User_Group slides and video] from their presentation at LUG 2009.

==Press Articles==

* [http://www.nccs.gov/2009/06/30/ornl-hosts-lustre-part-ii/ ORNL Hosts Lustre Part II]

==Lustre Internals Manual==
LCE and ORNL have written a Lustre filesystem internals document the describes the internal operation of Lustre.
*[[Media:Understanding_Lustre_Filesystem_Internals.pdf|Understanding Lustre Filesystem Internals]]

<br>
For more information about computing at Oak Ridge, click [http://www.nccs.gov/ here].

If you have questions or comments on this page or the LCE projects, contact [mailto:John.Dawson@sun.com John Dawson].

Lustre Center of Excellence at Oak Ridge National Laboratory

2009-08-17T01:07:19Z

Jkdawson: Added link to Lustre applications programming page

The Sun Lustre Center of Excellence (LCE) at ORNL advances the state of Lustre for use in large scale HPC environments. As you'll see from the projects below, the ORNL LCE focuses on both systems and applications aspects of ensuring that Lustre meets the needs of DOE and the HPC community in general.

==Current projects and their status==
===Applications IO Performance===
Mike Booth, one of the LCE personnel, is investigating applications IO performance on the Lustre systems at ORNL to identify ways to achieve more consistent and higher IO performance. He is also working on enhancing the ADIOS IO layer.

Feel free to [mailto:Michael.Booth@Sun.COM contact Mike] for more information and status of his work.

[[Implementing Applications in a Lustre Environment]] contains information on achieving high performance applications IO with the Lustre file system.

==Events==

The LCE sponsors events to encourage HPC community involvement
in analyzing IO and storage requirements and identifying ways for
Lustre to address these requirements.

The next event will be an Applications Workshop this Autumn 2009.

===LCE Summit - February 2008===
*[[Media:LCE_Summit_Summary_Draft_March_14_2008.pdf|LCE Summit Meeting Summary]]
*[[Media:LCESummitSlides.pdf|LCE Summit Slides and Notes]]

===LCE Application I/O Workshop - April 16, 2008===
*[[Media:April2008ApplicationIOWorkshop.pdf|Application IO Workshop, Agenda and Notes]]
*[[Media:Lustre_workshop_WangDi.pdf|Lustre and Application IO - Wang Di's Slides]]
*[[Media:Lustre_workshop_Oleg.pdf|Lustre and Application IO - Oleg Drokin's Slides]]

===Lustre Scalability Workshop - Feb 10 & 11, 2009, ORNL===
*[[Media:Notes_on_SW1_Notes.pdf|Scalability Workshop Notes]]
*[[Media:LustreScalabilityWP_Updated.pdf|Scalability White Paper]]
*[[Media:Shipman_Feb_lustre_scalability.pdf|Galen Shipman's presentation]]
*[[Media:Eric-Barton_-_Lustre-Multi_PF_Roadmap-090130.pdf|Eric Barton's presentation]]

===Lustre Scalability Workshop - May 19 & 20, 2009, ORNL===
The May Lustre Scalability Workshop at ORNL was focused on long term (2015) HPC IO and storage requirements and on presentations on the IO objectives of the DOD HPCS program and how Lustre will achieve them.
*[[Media:Dawson_Lustre_Workshop_May_2009.pdf|Lustre Scalability Workshop, Initial Gap Response - John Dawson]]
*[[Media:Carrier_2009-05-19_ORNL_LCE_HPCS.pdf|HPCS IO - John Carrier]]
*[[Media:Newman_May_Lustre_Workshop.pdf|What is HPCS and How Does it Impact IO - Henry Newman]]
*[[Media:Shipman_May_lustre_scalability_workshop.pdf|2015 Parallel File System Requirements - Galen Shipman]]
*[[Media:Dilger_Lustre_HPCS_May_Workshop.pdf|Lustre HPCS Design Overview - Andreas Dilger]]

==White Papers==

LCE personnel have written a variety of papers on High Performance IO and potential Lustre features. Links to these documents are below.

*[[Media:Peta-Scale_wp.pdf|Peta-Scale IO with the Lustre File System]]

*[[Media:Lce_pop_submitted.pdf|Improving I/O Performance in POP (Parallel Ocean Program)]]

* Scalability Improvements
**[[Media:Lustre_Enhancement_Report_Interval_Trees.pdf|Using Interval Tree to scale extent locks]]
**[[Media:Lustre_Enhancement_Report_UUID_Hash_Tables.pdf|Implement hash tables to scale export lookups]]
**[[Media:A_Novel_Network_Request_Scheduler_for_a_Large_Scale_Storage_System.pdf|A Novel Network Request Scheduler for a Large Scale Storage System]]

*Lustre ADIO Driver Enhancements Whitepaper
**[[Media:Lustre_ADIO_Driver_Whitepaper_0926.pdf|Lustre ADIO Driver Enhancements]]

*FMS Application IO Analysis
**[[Media:FMS_Investigation_Report_%280915%29.pdf|FMS Application IO Performance Analysis]]

==Presentations==

See the ORNL [http://wiki.lustre.org/index.php/Lustre_User_Group slides and video] from their presentation at LUG 2009.

==Press Articles==

* [http://www.nccs.gov/2009/06/30/ornl-hosts-lustre-part-ii/ ORNL Hosts Lustre Part II]

==Lustre Internals Manual==
LCE and ORNL have written a Lustre filesystem internals document the describes the internal operation of Lustre.
*[[Media:Understanding_Lustre_Filesystem_Internals.pdf|Understanding Lustre Filesystem Internals]]

<br>
For more information about computing at Oak Ridge, click [http://www.nccs.gov/ here].

If you have questions or comments on this page or the LCE projects, contact [mailto:John.Dawson@sun.com John Dawson].

Lustre Center of Excellence at Oak Ridge National Laboratory

2009-08-17T00:47:58Z

Jkdawson: Added link tp Applications Io page.

The Sun Lustre Center of Excellence (LCE) at ORNL advances the state of Lustre for use in large scale HPC environments. As you'll see from the projects below, the ORNL LCE focuses on both systems and applications aspects of ensuring that Lustre meets the needs of DOE and the HPC community in general.

==Current projects and their status==
===Applications IO Performance===
Mike Booth, one of the LCE personnel, is investigating applications IO performance on the Lustre systems at ORNL to identify ways to achieve more consistent and higher IO performance. He is also working on enhancing the ADIOS IO layer.
<p>
Feel free to [mailto:Michael.Booth@Sun.COM contact Mike] for more information and status of his work.

[[Implementing Applications in a Lustre Environment]] contains information on achieving high performance applications IO with the Lustre file system.

==Events==

The LCE sponsors events to encourage HPC community involvement
in analyzing IO and storage requirements and identifying ways for
Lustre to address these requirements.

The next event will be an Applications Workshop this Autumn 2009.

===LCE Summit - February 2008===
*[[Media:LCE_Summit_Summary_Draft_March_14_2008.pdf|LCE Summit Meeting Summary]]
*[[Media:LCESummitSlides.pdf|LCE Summit Slides and Notes]]

===LCE Application I/O Workshop - April 16, 2008===
*[[Media:April2008ApplicationIOWorkshop.pdf|Application IO Workshop, Agenda and Notes]]
*[[Media:Lustre_workshop_WangDi.pdf|Lustre and Application IO - Wang Di's Slides]]
*[[Media:Lustre_workshop_Oleg.pdf|Lustre and Application IO - Oleg Drokin's Slides]]

===Lustre Scalability Workshop - Feb 10 & 11, 2009, ORNL===
*[[Media:Notes_on_SW1_Notes.pdf|Scalability Workshop Notes]]
*[[Media:LustreScalabilityWP_Updated.pdf|Scalability White Paper]]
*[[Media:Shipman_Feb_lustre_scalability.pdf|Galen Shipman's presentation]]
*[[Media:Eric-Barton_-_Lustre-Multi_PF_Roadmap-090130.pdf|Eric Barton's presentation]]

===Lustre Scalability Workshop - May 19 & 20, 2009, ORNL===
The May Lustre Scalability Workshop at ORNL was focused on long term (2015) HPC IO and storage requirements and on presentations on the IO objectives of the DOD HPCS program and how Lustre will achieve them.
*[[Media:Dawson_Lustre_Workshop_May_2009.pdf|Lustre Scalability Workshop, Initial Gap Response - John Dawson]]
*[[Media:Carrier_2009-05-19_ORNL_LCE_HPCS.pdf|HPCS IO - John Carrier]]
*[[Media:Newman_May_Lustre_Workshop.pdf|What is HPCS and How Does it Impact IO - Henry Newman]]
*[[Media:Shipman_May_lustre_scalability_workshop.pdf|2015 Parallel File System Requirements - Galen Shipman]]
*[[Media:Dilger_Lustre_HPCS_May_Workshop.pdf|Lustre HPCS Design Overview - Andreas Dilger]]

==White Papers==

LCE personnel have written a variety of papers on High Performance IO and potential Lustre features. Links to these documents are below.

*[[Media:Peta-Scale_wp.pdf|Peta-Scale IO with the Lustre File System]]

*[[Media:Lce_pop_submitted.pdf|Improving I/O Performance in POP (Parallel Ocean Program)]]

* Scalability Improvements
**[[Media:Lustre_Enhancement_Report_Interval_Trees.pdf|Using Interval Tree to scale extent locks]]
**[[Media:Lustre_Enhancement_Report_UUID_Hash_Tables.pdf|Implement hash tables to scale export lookups]]
**[[Media:A_Novel_Network_Request_Scheduler_for_a_Large_Scale_Storage_System.pdf|A Novel Network Request Scheduler for a Large Scale Storage System]]

*Lustre ADIO Driver Enhancements Whitepaper
**[[Media:Lustre_ADIO_Driver_Whitepaper_0926.pdf|Lustre ADIO Driver Enhancements]]

*FMS Application IO Analysis
**[[Media:FMS_Investigation_Report_%280915%29.pdf|FMS Application IO Performance Analysis]]

==Presentations==

See the ORNL [http://wiki.lustre.org/index.php/Lustre_User_Group slides and video] from their presentation at LUG 2009.

==Press Articles==

* [http://www.nccs.gov/2009/06/30/ornl-hosts-lustre-part-ii/ ORNL Hosts Lustre Part II]

==Lustre Internals Manual==
LCE and ORNL have written a Lustre filesystem internals document the describes the internal operation of Lustre.
*[[Media:Understanding_Lustre_Filesystem_Internals.pdf|Understanding Lustre Filesystem Internals]]

<br>
For more information about computing at Oak Ridge, click [http://www.nccs.gov/ here].

If you have questions or comments on this page or the LCE projects, contact [mailto:John.Dawson@sun.com John Dawson].

Learn:Learn

2009-08-17T00:44:18Z

Jkdawson: Added link to Lustre applications programming page

The Lustre™ file system redefines I/O performance and scalability standards for the
world’s largest and most complex computing environments. Ideally suited for data-intensive applications that require the highest possible I/O performance, Lustre is an object-based cluster file system that scales to tens of thousands of nodes and petabytes of storage with groundbreaking I/O and metadata throughput.

<div class="categoryLeft">

<big><strong>Interoperability, Features and Roadmap</strong></big>

These resources detail Lustre releases and plans.

* [[Lustre_Support_Matrix|Lustre Support Matrix]] lists supported kernels and networks for current Lustre releases.

* [[Lustre_1.8|Lustre 1.8]] describes features and benefits offered by upgrading to the 1.8 branch of Lustre.

* [[Lustre_2.0|Lustre 2.0]] describes features being developed for this next-generation release of Lustre.

* [[Lustre_Roadmap|Lustre Roadmap]] provides estimated code freeze and GA dates for upcoming releases.

</div>

<div class="categoryRight">

<big><strong>Publications</strong></big>

A number of papers, presentations and publications are available for Lustre. Use these resources to learn about the benefits offered by Lustre and plans for future development.

* [[Lustre_Publications|Lustre white papers, case studies, engineering presentations]]
** [[Lustre_Documentation|Lustre Documentation]]
** [[Lustre_Launch|Lustre All-Hands Meeting 3/08]]
** [[ Lustre All-Hands Meeting 12/08]]
* [http://manual.lustre.org/index.php?title=Main_Page#Lustre_Operations_Manual Lustre Operations Manual]
* [http://arch.lustre.org/index.php?title=Main_Page Lustre Architecture]

</div>

<div class="categoryLeft">

<big><strong>Lustre Projects and Future Features</strong></big>

To learn about active Lustre projects that span releases, or are features that are not in Lustre 1.8 or 2.0 releases, go to this [[Projects|Lustre Projects]] page.

</div>

<div class="categoryRight">

<big><strong>Training and Internals</strong></big>

Lustre training is available from Sun Microsystems.

* Lustre training, [http://www.sun.com/training/catalog/courses/CL-400.xml Administering Lustre Based Clusters (CL-400)], is available from Sun Microsystems.

* [[Lustre_Internals|Lustre Internals]] (formerly part of Lustre advanced training) are also available, covering complex, code-level transactions.

[[Implementing Applications in a Lustre Environment]] contains information on achieving high performance applications IO with the Lustre file system.

</div>

Learn:Learn

2009-08-17T00:41:07Z

Jkdawson:

The Lustre™ file system redefines I/O performance and scalability standards for the
world’s largest and most complex computing environments. Ideally suited for data-intensive applications that require the highest possible I/O performance, Lustre is an object-based cluster file system that scales to tens of thousands of nodes and petabytes of storage with groundbreaking I/O and metadata throughput.

<div class="categoryLeft">

<big><strong>Interoperability, Features and Roadmap</strong></big>

These resources detail Lustre releases and plans.

* [[Lustre_Support_Matrix|Lustre Support Matrix]] lists supported kernels and networks for current Lustre releases.

* [[Lustre_1.8|Lustre 1.8]] describes features and benefits offered by upgrading to the 1.8 branch of Lustre.

* [[Lustre_2.0|Lustre 2.0]] describes features being developed for this next-generation release of Lustre.

* [[Lustre_Roadmap|Lustre Roadmap]] provides estimated code freeze and GA dates for upcoming releases.

</div>

<div class="categoryRight">

<big><strong>Publications</strong></big>

A number of papers, presentations and publications are available for Lustre. Use these resources to learn about the benefits offered by Lustre and plans for future development.

* [[Lustre_Publications|Lustre white papers, case studies, engineering presentations]]
** [[Lustre_Documentation|Lustre Documentation]]
** [[Lustre_Launch|Lustre All-Hands Meeting 3/08]]
** [[ Lustre All-Hands Meeting 12/08]]
* [http://manual.lustre.org/index.php?title=Main_Page#Lustre_Operations_Manual Lustre Operations Manual]
* [http://arch.lustre.org/index.php?title=Main_Page Lustre Architecture]

</div>

<div class="categoryLeft">

<big><strong>Lustre Projects and Future Features</strong></big>

To learn about active Lustre projects that span releases, or are features that are not in Lustre 1.8 or 2.0 releases, go to this [[Projects|Lustre Projects]] page.

</div>

<div class="categoryRight">

<big><strong>Training and Internals</strong></big>

Lustre training is available from Sun Microsystems.

* Lustre training, [http://www.sun.com/training/catalog/courses/CL-400.xml Administering Lustre Based Clusters (CL-400)], is available from Sun Microsystems.

* [[Lustre_Internals|Lustre Internals]] (formerly part of Lustre advanced training) are also available, covering complex, code-level transactions.

Information on achieving high performance applications IO with the Lustre file system can be found at [[Implementing Applications in a Lustre Environment]].

</div>

Clustered Metadata

2009-08-16T20:41:20Z

Jkdawson: Added link to HPCS CMD Design Doc PDF

= Clustered Metadata Design (in progress) =

This document describes the design of the clustered metadata handling
for Lustre. This material depends on other Lustre design, such as:
:
* General recovery
* Orphan Recovery
* Metadata Write Back caching

The draft Clustered Metadata Design is available as a PDF:

* [[Media:HPCS_CMD_06_15_09.pdf|Clustered Metadata Design]]

== Introduction ==

Overall the clustered metadata handling is structured as follows.

* A cluster of metadata servers manage a collection of inode groups. Each inode group is a Lustre device exporting the usual metadata api, augmented with a few operations specifically crafted for metadata clustering. We call these collections of inodes inode groups.
* Directory formats for file systems used on the MDS devices are changed to introduce a allow directory entries to contain an inode group and identifier of the inode.
* A logical clustered metadata driver is introduced below the client Lustre file system write back cache driver that maintains connections with the MDS servers.
* There is a single metadata protocol that is used by the client file system to make updates on the MDS's and by the MDS's to make updates involving other MDS's.
* There is a single recovery protocol that is used by the clients - MDS and MDS-MDS service.
* Directories can be split across multiple MDS nodes. In that case a primary MDS directory inode contains an extended attribute that points at other MDS inodes which we call directory objects.

== Configuration management and Startup ==

The configuration will name an MDS server, and optionally a failover
node, which hold the root inode for a fileset. Clients will contact
that MDS for the root inode during mount, as they do already.

They will also fetch from it a clustering descriptor. The clustering
descriptor contains a header and an array lists what inode groups are
served by what server.

Through normal mechanisms clients will wait and probe for available
metadata servers, during startup and cluster transitions. When new
servers are found or configurations have changed they can update their
clustering descriptor as they update the LOV striping descriptor for
OST's.

== Data Structures ==

The fid will contain a new 32 bit integer to name the inode group.

Directory entries will contain a new 32 bit integer to name the inode
group.

Directory inodes on the MDS, when large, contain a new EA which is a
descriptor of how the directory is split over directory objects,
residing on other MDS's. This EA is subject to ordinary concurrency
control by the MDS holding the inode. The EA is virtually identical
to the LOV EA.

== The clustered metadata client (CMC) ==

Client systems will have the write back client (WBD) or client file
system directly communicate with the CMC driver: it offers the
metadata api to the file system and uses the metadata api offered by a
collection of MDC drivers. Each MDC driver managed the metadata
traffic to one.

The function of the CMC is very simple: it figures out from the
command issued what MDC to use. This is based on:
* the inode groups in the request
* a hash value of names used in the request, combined with the EA of a primary inode involved in the request.
* for readdir the directory offset combined with the EA of the primary inode
* the clustering descriptor

In any case every command is dispatched to a single metadata server,
the clients will not engage more than one metadata server for a single
request.

The api changes here are minimal and the client part of the
implementation is very trivial.

== MDS implementation ==

For the most part, operations are extremely similar or identical to
what they were before. In some cases multiple mds servers are
involved in updates.

Getattr, open, readdir, setattr and lookup methods are unaffected.

Methods adding entries to directories are modified in some cases:

* '''mkdir''' always create the new directory on another MDS
* '''unlink, rmdir, rename''': may involve more than one MDS
* '''large directories''' all operations making updates to directories can cause a directory split. The directory split is discussed below.
* '''other operations''' If no splits large directories are encountered all other operations proceed as they are executed on one MDS.

=== Directory Split ===

A directory that is growing larger will be split. There is a fairly heavy penalty associated with splitting the directory and also with renames in within split directories. Moreover, at the point of splitting, inodes become remote and will incur a penalty upon unlink.

Probably it is best to delay the split until the directory is fairly large, and then to split over several nodes, to avoid further splits being necessary soon afterwards.

== Recovery ==

=== Transaction Replay ===

The MDS - MDS interaction is managed as follows. The node approached
with a request change is made the coordinator of the transaction.

==== Mechanisms ====

The coordinator will first establish that the transaction can commit on all nodes, by acquiring locks on directories and checking for available space existing entries with the same name etc. It may also first perform a directory split if the size is becoming too large, and more MDS nodes are still available.

All nodes involved in the transaction need to have a transaction sequence number to place the transaction into their sequence and allow correctly replay.

At this point the coordinator will:
* start a transaction locally.
* It will then report the transaction sequence number to all other nodes involved in the transaction.
* These nodes will commit (in memory as usual), write a journal record for replay and reply to the coordinator.
* The coordinator will then commit its own transaction.
* The replay log records are subject to normal log commit cancelation messages, but on the coordinator commit messages must be received from all other nodes before the record will be canceled.

In this way if the results of the transaction survive on any of the nodes, they can be replayed on all.

==== Cluster crashes and the transaction sequence ====

If the cluster crashes abruptly, there is the opportunity for transactions to be in progress affecting multiple nodes. Dependencies between the transactions must be managed to ensure serializibility of the protocol.

Example: In transaction one, a node X creates directories a. Then in transaction 2 a cross MDS node rename moves a file with a directory entry on node Y into this directory. It is now possible for this file to lose its directory entry on Y and for the transaction on X not to commit. More complex examples exist.

We see 4 solutions:

* Disk commit delay locks: dependent transactions many not commit before the parent transaction commits.
* Commit acks: transactions may not proceed until previous pre-requisite transactions have committed.
* Synchronous NVRAM journal on all MDS nodes
* Shared journal among all MDS nodes

The first and last method offer the most opportunity to proceed without synchronous disk writes. The last method involves contention on a shared resource.

Although exhaustive analysis remains, it is clear that rename and splitting the directories are the primary culprits. Hence, we wonder about the following policy related issues:
* Only split really large directories, say after 1M entries.
* Do not needlessly create subdirectories on other nodes. A much better policy is likely to keep directories with one owner, or possibly one client system generating them together.

==== Replay ====

To order transaction sequences Lustre uses reply ack's: the acks
server only one purpose to release a lock that enforces ordering of
the transaction sequence. In the case where MDS operations involve
more than server, the reply "ack" from the primary to secondary
servers should only be sent after the client has sent the ack to the
first server. This MDS-MDS reply ack is now not really an ack anymore
but a simple lock cancelation review.

Clients will replay lost transactions to the mds which they originally
engaged for the request.

Orphaned children will be cleaned up only
after replay completes to allow orphaned objects to be re-used during
replay.

=== Failover ===

The configuration data can designate a standby MDS that will take over
from a failed MDS. By organizing the servers in one or more rings,
the nearest working left neighbor MDS can be the failover node. This
leads to a simple scheme with multiple failover nodes, avoiding quorum
and other complications beyond what is needed for two node clusters.

== Locking ==

We believe locking can be done in fid order as it is currently done on the MDS.

Lustre Projects

2009-08-16T19:32:15Z

Jkdawson: Added Link to HPCS page

<big><strong>Current Lustre Projects and Future Features</strong></big>

This page lists Lustre projects that will may span multiple releases, or are features-to-be that have not yet been assigned a release.

For features not here and already assigned to a release, look at the [[Learn | Lustre Releases]] page.

Projects
<ul>
<li>[[MetaData_Project | MetaData Project]]
<li>[[Integrating_Hadoop_with_Lustre | Integrating Lustre and Hadoop]]
<li>[[Lustre HPCS Activities]]
</ul>

Features
<ul>
<li>[[Metadata_Clustering | Clustered MetaData ]]
<li>[[Windows_Native_Client | Windows Native Client]]
</ul>

If you do not see a project here and do not see it listed in any of the releases as a feature, contact [mailto:Daniel.Ferber@sun.com Dan Ferber].

Lustre Publications

2009-08-12T20:05:14Z

Jkdawson: Corrected two typos

__TOC__
Informative sources regarding Lustre technology and its application.

You'll find user documentation along with videos and podcasts describing Lustre at a high-level, engineering presentations from Lustre engineers, and detailed descriptions of applications in blue prints and white papers produced by research organizations.

===Lustre Documentation===

{| border=1 cellspacing=0
|+Lustre Documentation
|-
!Title
!Description/Source
!Date
|-
|[http://manual.lustre.org/index.php?title=Main_Page#Lustre_Operations_Manual Lustre Operations Manual]||The Lustre Operations Manual provides users with installation, configuration, tuning, monitoring and troubleshooting information.||2009
|-
|}

===Videos & Podcasts===

{| border=1 cellspacing=0
|+Videos & Podcasts from Sun
|-
!Title
!Description/Source
!Date
|-
|[http://www.rce-cast.com/index.php/Podcast/rce-14-lustre-cluster-filesystem.html '''RCE 14: Lustre Cluster File System''']|| Research Computing and Engineering interview with Andreas Dilger, one of the principal file system architects for the Lustre file system.||2009
|-
|[http://link.brightcove.com/services/player/bcpid1640183659?bctid=8899392001 '''Linux HPC Software Stack''']||Sun Systems Engineer Larry McIntosh provides an overview of Sun HPC software reference stack for Lustre.||2008
|-
|[http://channelsun.sun.com/video/lustre/1653611906 '''Lustre Overview by Peter Bojanic''']||Learn about the Lustre parallel file system, the newest addition to the Sun HPC portfolio, which is designed to meet the demands of the world's largest high performance clusters.||December 7, 2007
|-
|[http://channelsun.sun.com/video/storage+cluster/8901697001 '''Sun Storage Cluster''']||Find out how Sun simplifies the deployment of Lustre-based storage.|| January 9, 2009
|-
||[http://channelsun.sun.com/video/radio+hpc+-+episode+11/10049527001 '''Radio HPC - Episode 11''']||Tony Warner chats with Voltaire's Brian Forbes about their companies' partnership in the InfiniBand space, and Peter Bojanic clues us in on what's new with Sun's Lustre file system. ||February 3, 2009
|-
|}

===White Papers===

{| border=1 cellspacing=0
|+White Papers
|-
!Title
!Description/Source
!Date
|-
||[http://www.sun.com/software/products/lustre/docs/Lustre-networking.pdf '''Lustre File System Networking: High-Performance Features and Flexible Support for a Wide Array of Networks''']||Information about Lustre networking that can be used to plan cluster file system deployments for optimal performance and scalability. Covered topics include Lustre message passing, Lustre Network Drivers, and routing in Lustre networks, and the paper describes how these features can be used to improve cluster storage management.|| November 2008
|-
||[http://www.sun.com/software/products/lustre/docs/lustrefilesystem_wp.pdf '''Lustre File System: High-Performance Storage Architecture and Scalable Cluster File System''']||Basic information about the Lustre file system. Covered topics include general characteristics and markets in which Lustre has a strong presence, a typical Lustre file system configuration, an overview of Lustre networking (LNET), an introduction of Lustre capabilities that support high availability and rolling upgrades, discussion of file storage in a Lustre file system, additional features, and information about a how a Lustre file system compares to other shared file systems.||October 2008
|-
||[http://www.sun.com/offers/docs/open_petascale_computing.pdf '''Pathways to Open Petascale Computing''']||Derived from Sun’s innovative design approach and experience with very large , supercomputing deployments, the Sun Constellation System provides the world's ﬁrst , open petascale computing environment — one built entirely with open and standard , hardware and software technologies. Cluster architects can use the Sun Constellation , System to design and rapidly deploy tightly-integrated, efﬁcient, and cost-effective , supercomputing grids and clusters that scale predictably from a few teraﬂops to over a , petaﬂop. With a totally modular approach, processors, memory, interconnect fabric, and storage can all be scaled independently depending on individual needs. ||June 2008
|-
|[http://www.datadirectnet.com/resource-downloads/best-practices-for-architecting-a-lustre-based-storage-environment-registration '''Best Practices for Architecting a Lustre-Based Storage Environment''']|| DDN ||March 2008
|-
||[http://www.sun.com/software/products/lustre/docs/Peta-Scale_wp.pdf '''Peta-Scale I/O with the Lustre File System''']||Describes low-level infrastructure in the Lustre file system that addresses scalability in very large clusters. Covered topics include scalable I/O, locking policies and algorithms to cope with scale, implications for recovery, and other scalability issues. / ORNL ||February 2008
|-
||[http://wiki.lustre.org/images/4/49/WP_BestPractices_Lustre_DDN_032108.pdf '''Best Practices for Architecting a Lustre-based Storage Environment''']|| A series of best practices to consider when deploying a highly-reliable, high-performance Lustre environment. Covered topics include storage infrastructure failover, maximizing computational capability by minimizing I/O overhead, ensuring predictable striped file performance, and protecting large, persistent data stores. / DataDirect Networks||2008
|-
||[http://wiki.lustre.org/images/2/20/Lustre_wan_tg07.pdf '''Wide Area Filesystem Performance using Lustre on the TeraGrid''']|| TeraGrid 2007 conference / Indiana University|| June 2007
|-
||[http://wiki.lustre.org/images/d/db/Yu_lustre.pdf '''Exploiting Lustre File Joining for Effective Collective IO''']||Proceedings of the CCGrid'07 / ORNL || May 2007
|-
||[http://wiki.lustre.org/images/7/79/Thumper-BP-6.pdf '''Tokyo Tech Tsubame Grid Storage Implementation''']||By Syuuichi Ihara [http://www.sun.com/blueprints/0507/820-2187.html Also a Sun BluePrints Publication]|| May 2007
|-
||[http://wiki.lustre.org/index.php?title=Image:Hpc_cats_wp.pdf '''Optimizing Storage and I/O For Distributed Processing On Enterprise & High Performance Compute(HPC)Systems For Mask Data Preparation Software (CATS)''']||Glenn Newell, Sr.IT Solutions Mgr, Naji Bekhazi, Director of R&D,Mask Data Prep (CATS), Ray Morgan,Sr.Product Marketing Manager,Mask Data Prep(CATS)/ Synopsys||2007
|-
||[http://wiki.lustre.org/index.php?title=Image:Lustre_wan_tg07.pdf '''Wide Area Filesystem Performance Using Lustre on the TeraGrid''']||Teragrid 2007 Conference, Madison,WI / TeraGrid||2007
|-
|[http://wiki.lustre.org/images/3/3f/Larkin_paper.pdf '''Guidelines for Efficient Parallel I/O on the Cray XT3/XT4''' (PDF)]||Jeff Larkin, Mark Fahey, proceedings of CUG 2007 / Cray User Group|| 2007
|-
|[http://wiki.lustre.org/images/b/b9/Canon_slides.pdf '''XT7? Integrating and Operating a Conjoined XT3+XT4 System''']||Presented by ORNL on CUG 2007 [http://wiki.lustre.org/images/f/fa/Canon_paper.pdf (Presentation also available.)] / Cray User Group ||2007
|-
|[http://wiki.lustre.org/images/7/77/A_Center-Wide_FS_using_Lustre.pdf '''A Center-Wide File System using Lustre''']||Shane Canon, Sarp Oral, proceedings of CUG 2006 \ Cray User Group|| 2006
|-
||[http://wiki.lustre.org/images/d/d8/Cac06_lustre.pdf '''Benefits of High Speed Interconnects to Cluster File Systems: A Case Study with Lustre''']||Parallel and Distributed Processing Symposium, 2006. IPDPS 2006. Lustre performance comparison when using InfiniBand and Quadrics interconnects. You can also [http://nowlab.cse.ohio-state.edu/publications/conf-papers/2006/yu-cac06.pdf download the paper at the OSU site]. / Ohio State University || 2006
|-
||[http://wiki.lustre.org/images/f/fc/MSST-2006-paper.pdf '''Coordinating Parallel Hierarchical Storage Management in Object-base Cluster File Systems''']||MSST2006, Conference on Mass Storage Systems and Technologies (May 2006) / University of Minnesota ||2006
|-
||[http://wiki.lustre.org/images/0/0b/Karlsruhe0604.pdf '''Experiences with HP SFS/Lustre at SSCK''']||SGPFS 5 in Stuttgart / Karlsruhe Lustre Talks || 4.4.2006
|-
||[http://wiki.lustre.org/images/7/7c/Karlsruhe0503.pdf '''Filesystems on SSCK's HP XC6000''']|| Einführungsveranstaltung im Rechenzentrum / Karlsruhe Lustre Talks || 2005
|-
||[http://wiki.lustre.org/images/9/95/Karlsruhe0510.pdf '''Experiences & Performance of SFS/Lustre Cluster File System in Production'''] || HP-CAST 4 in Krakau / Karlsruhe Lustre Talks ||10.5.2005
|-
||[http://wiki.lustre.org/images/5/5f/Karlsruhe0506.pdf '''ISC 2005 in Heidelberg''']|| Karlsruhe Lustre Talks ||24.6.2005
|-
||[http://wiki.lustre.org/images/1/17/Karlsruhe0511.pdf '''Experiences with 10 Months HP SFS/Lustre in HPC Production''']||HP-CAST 5 in Seattle / Karlsruhe Lustre Talks || 11.11.2005
|-
||[http://wiki.lustre.org/images/a/aa/Karlsruhe0512.pdf '''Performance Monitoring in a HP SFS Environment''']||HP-CCN in Seattle / Karlsruhe Lustre Talks || 12.11.2005
|-
|[http://wiki.lustre.org/images/9/95/Our_Collateral_selecting-a-cfs.pdf '''Selecting a cluster file system''']||CFS||Nov 2005
|-
||[http://wiki.lustre.org/images/8/81/LciPaper.pdf '''Shared Parallel Filesystem in Heterogeneous Linux Multi-Cluster Environment''']||Proceedings of the 6th LCI International Conference on Linux Clusters: The HPC Revolution (2005). The management issues mentioned in the last part of this paper have been addressed. The same paper is also [http://linuxclustersinstitute.org/Linux-HPC-Revolution/Archive/PDF05/17-Oberg_M.pdf available at the CU site.] / University of Colorado, Boulder ||2005
|-
|[http://wiki.lustre.org/images/d/d2/Ols2003.pdf '''Lustre: Building a cluster file system for 1,000 node clusters''']||A technical presentation about our successes and mistakes during 2002-2003 / CFS||Summer 2003
|-
|-
|}

===BluePrints===

{| border=1 cellspacing=0
|+BluePrints
|-
!Title
!Description/Source
!Date
|-class="even"
|[http://wikis.sun.com/display/BluePrints/Lustre+File+System+-+Demo+Quick+Start+Guide '''Lustre File System - Demo Quick Start Guide''']||A simple cookbook for non-Linux experts on how to set up a Linux-based Lustre file system using small servers, workstations, PCs, or other available hardware for demonstration purposes.|| 2009
|-
|[http://wikis.sun.com/display/BluePrints/Implementing+the+Lustre+File+System+with+Sun+Storage ''' Implementing the Lustre File System with Sun Storage''']||Describes an implementation of the Sun Lustre file system as a scalable storage cluster using Sun Fire servers, and highspeed/lowlatency InfiniBand interconnects.||2009
|-class="even"
||[http://wikis.sun.com/display/BluePrints/Tokyo+Tech+Tsubame+Grid+Storage+Implementation '''Tokyo Tech Tsubame Grid Storage Implementation''']||This Sun BluePrints™ article describes the storage architecture of the Tokyo Tech , TSUBAME grid, as well as the steps for installing and configuring the Lustre file system , within the storage architecture.||2009
|-
||[http://wikis.sun.com/download/attachments/31395541/820-5304.pdf?version=1 '''Sun Storage and Archive Solution for HPC: Sun BluePrints Reference Architecture''']||To help customers address an almost bewildering set of architectural challenges, Sun has developed the Sun Storage and Archive Solution for HPC, a reference architecture that , can be easily customized to meet specific application goals and business requirements.||May 2008
|-
|}

===Presentations===

{| border=1 cellspacing=0
|+Presentations
|-
!Title
!Description/Source
!Date
|-
||[http://indico.cern.ch/materialDisplay.py?contribId=23&sessionId=10&materialId=slides&confId=27391 '''Lustre cluster in production at GSI''']||Presented by SCHöN, Walter / HEPiX Talks || May 2008
|-
||[http://indico.cern.ch/materialDisplay.py?contribId=28&sessionId=10&materialId=slides&confId=27391 '''Final Report from File Systems Working Group''']||Presented by MASLENNIKOV, Andrei / HEPiX Talks || May 2008
|-
||[http://indico.cern.ch/contributionDisplay.py?contribId=17&sessionId=10&confId=27391 '''Setting up a simple Lustre Filesystem'''] ||Presented by Stephan Wiesand. Slides on HPPix site: [https://indico.desy.de/getFile.py/access?contribId=26&sessionId=40&resId=1&materialId=slides&confId=257 Storage Evaluations at BNL] / HEPiX Talks || May 2008
|-
||[http://wiki.lustre.org/images/d/da/Storage_Evaluations%40BNL.pdf '''Storage Evaluations at BNL'''] ||Presented by Robert Petkus - BNL. Slides on HEPix site: [https://indico.desy.de/getFile.py/access?contribId=44&sessionId=39&resId=0&materialId=slides&confId=257 Lustre Experience at CEA/DIF] / HEPiX Talks || May 2008
|-
||[http://wiki.lustre.org/images/5/58/DIF.pdf '''Lustre Experience at CEA/DIF''']||Presented by J-Ch Lafoucriere / HEPiX Talks || April 23-27, 2008
|-
||[http://wiki.lustre.org/images/f/fa/Canon_paper.pdf '''XT7? Integrating and Operating a Conjoined XT3+XT4 System''']|| This paper describes the processes and tools used to move production work from the pre-existing XT3 to the new system incorporating that same XT3, including novel application of Lustre routing capabilities. / Cray User Group [http://wiki.lustre.org/images/b/b9/Canon_slides.pdf (Paper also available)]|| 2007
|-
|[http://wiki.lustre.org/images/e/ef/Using_IOR_to_Analyze_IO_Performance.pdf '''Using IOR to Analyze the I/O Performance''']|| Presented by Hongzhang Shan,John Shalf (NERSC) on CUG 2007 / Cray User Group ||2007
|-
|[http://wiki.lustre.org/images/a/a3/Gelato-2004-05.pdf '''Lustre state and production installations''']||Presentation on gelato.org meeting / CFS||May 2004
|-
|[http://wiki.lustre.org/images/e/ea/Lustre-usg-2003.pdf '''Lustre File System ''']||Presentation on the state of Lustre in mid-2003 and the path towards Lustre1.0 / CFS||Summer 2003
|-
|[http://wiki.lustre.org/images/d/d2/Ols2003.pdf '''Lustre: Building a cluster file system for 1,000 node clusters''']||A technical presentation about our successes and mistakes during 2002-2003 / CFS||Summer 2003
|-
|}

====Presentations from Lustre Engineers, March 2008====

Once a year, the Lustre Engineering team gathers to discuss new features under development and testing efforts. This week-long event is known as the Lustre all-hands meeting. The Development and QE presentations made at these all-hands meetings are available here.

Also available at [[Learn:Lustre_All-Hands_Meeting_3/08|Lustre All-Hands Meeting_3/08]] and [[Learn:Lustre_All-Hands_Meeting_3/08|Lustre All-Hands Meeting_12/08]].

{| border=1 cellspacing=0
|+December 2008: Lustre Launch in Beijing
|-
!Title
!Description
!Date
|-
|[http://wiki.lustre.org/images/c/c1/WindowsPort.pdf '''Lustre Windows Porting''']||Matt Wu|| December 2008
|-
||[http://wiki.lustre.org/images/f/f8/VBR-TOI.pdf '''VBR-TOI''']||Mikhail Pershin|| December 2008
|-
||[http://wiki.lustre.org/images/0/0b/Simplified_InteropRecovery.pdf '''Simplified Interoperability Recovery''']||Huang Hua|| December 2008
|-
||[http://wiki.lustre.org/images/c/cd/RecoveryTalk_2009.pdf '''Recovery Overview''']|| Robert Read|| December 2008
|-
||[http://wiki.lustre.org/images/e/e7/Quotas-TOI.pdf '''Quotas-TOI''']||Yong Fan|| December 2008
|-
||[http://wiki.lustre.org/images/e/ee/QualityInitiativeTalk.pdf '''Quality Initiative Talk''']||Robert Read|| December 2008
|-
||[http://wiki.lustre.org/images/8/8c/PNFS_SpeakerNotes.pdf pNFS '''Speaker Notes''']||Oleg Drokin|| December 2008
|-
||[http://wiki.lustre.org/images/0/07/PNFS.pdf '''pNFS''']||Oleg Drokin|| December 2008
|-
||[http://wiki.lustre.org/images/9/97/OST_Pools.pdf '''OST Pools''']||Nathan Rutman|| December 2008
|-
||[http://wiki.lustre.org/images/f/ff/OST_Migration_RAID1_SNS.pdf '''OST Migration RAID1 SNS''']||Andreas Dilger|| December 2008
|-
||[http://wiki.lustre.org/images/d/d5/ORNL_Experience_SpeakerNotes.pdf '''ORNL Experience Speaker Notes''']||Oleg Drokin|| December 2008
|-
||[http://wiki.lustre.org/images/d/d9/ORNL_Experience.pdf '''ORNL Experience''']||Oleg Drokin|| December 2008
|-
||[http://wiki.lustre.org/images/2/2f/NRS.pdf '''Lustre NRS Simulation''']||Yingjin Qian, Wang Di|| December 2008
|-
||[http://wiki.lustre.org/images/1/1d/LustreInterop_1_8.pdf '''Lustre Interoperability 1.8''']||Huang Hua|| December 2008
|-
||[http://wiki.lustre.org/images/e/ed/LustreAllHands_12_2009_Keynote.pdf '''Lustre All-Hands Meeting 12/2008 - Keynote''']||Peter Bojanic|| December 2008
|-
||[http://wiki.lustre.org/images/b/b7/Lustre_on_DMU.pdf '''Lustre on DMU''']||Alex Zhuravlev|| December 2008
|-
||[http://wiki.lustre.org/images/5/57/LLOG_Overview.pdf '''LLOG Overview''']||Alex Zhuravlev|| December 2008
|-
||[http://wiki.lustre.org/images/d/dd/HDFS.pdf '''HDFS''']||Wang Di|| December 2008
|-
||[http://wiki.lustre.org/images/5/5b/GIT_Overview.pdf '''GIT Overview''']||Robert Read|| December 2008
|-
||[http://wiki.lustre.org/images/2/2d/COS-TOI.pdf COS-TOI '''COS-TOI''']||Alexander Zarochentsev|| December 2008
|-
||[http://wiki.lustre.org/images/6/66/CLIO-TOI.pdf '''CLIO-TOI''']||Nikita Danilov|| December 2008
|-
||[http://wiki.lustre.org/images/1/1d/CLIO.pdf '''CLIO''']||Nikita Danilov|| December 2008
|-
||[http://wiki.lustre.org/images/2/28/ROC-TOI.pdf '''Read-Only Cache - TOI''']||Alex Zhuravlev|| December 2008
|-
|}

{| border=1 cellspacing=0
|+March 2008: Lustre Launch in Beijing
|-
!Title
!Description
!Date
|-class="even"
|Development||||
|-
|[http://wiki.lustre.org/images/a/a3/RMG-process-0308.pdf '''RMG Processes''']||Andreas Dilger|| March 17, 2008
|-
||[http://wiki.lustre.org/images/8/8d/Eeb-launch-0308.pdf '''Lustre Development Strategy''']||Eric Barton|| March 17, 2008
|-
||[http://wiki.lustre.org/images/6/6c/Cmd-0308.pdf '''CMD'''] || Yury Umanets|| March 17, 2008
|-
||[http://wiki.lustre.org/images/f/f4/Mdt.pdf '''HEAD MDS'''] ||Nikita Danilov|| March 17, 2008
|-
||[http://wiki.lustre.org/images/d/d9/Uss-0308.pdf '''User Space Servers''']||Alex Zhuravlev|| March 17, 2008
|-
||[http://wiki.lustre.org/images/2/29/DMU-0308.pdf '''DMU''']||Ricardo Correia|| March 17, 2008
|-
||[http://wiki.lustre.org/images/2/22/Recovery_overview-0308.pdf '''Recovery''']||Mike Pershin|| March 17, 2008
|-
||[http://wiki.lustre.org/images/c/c6/LDLM-0308.pdf '''DLM''']||Oleg Drokin|| March 17, 2008
|-
||[http://wiki.lustre.org/images/0/02/Lnet-0308.pdf '''LNET''']||Issac Huang|| March 17, 2008
|-
|-class="even"
|Quality Engineering||||
|-
||[http://wiki.lustre.org/images/9/92/Jd.day1-0308.pdf '''Day 1'''][http://wiki.lustre.org/images/2/25/Jd.day2-0308.pdf '''Day 2''']||JD|| March 17, 2008
|-
||[http://wiki.lustre.org/images/7/72/Lustre-release%26weekly-testing-030.pdf '''Lustre Release & weekly testing''']||Jian Yu|| March 17, 2008
|-
||[http://wiki.lustre.org/images/6/62/Buildsys-0308.pdf '''Build System''']||Yibin Wang|| March 17, 2008
|-
||[http://wiki.lustre.org/images/a/af/Head-testing-0308.pdf '''HEAD testing''']||Zheng Chen|| March 17, 2008
|-
||[http://wiki.lustre.org/images/c/cf/Latest.b1_6-0308.pdf '''b1.6 testing''']||Peng Ye|| March 17, 2008
|-
||[http://wiki.lustre.org/images/0/0b/Perf.testing-0308.pdf '''Performance testing''']||Jack Chen|| March 17, 2008
|-
||[http://wiki.lustre.org/images/6/60/Automation-0308.pdf '''Automation''']||Minh Diep|| March 17, 2008
|-
||[http://wiki.lustre.org/images/9/9f/Cham_accep-sm.pdf '''Acc-small''']||Elena Gryaznova|| March 17, 2008
|-
|}

===Archive (Prior to 2003)===

<big><strong>From CFS</strong></big>

* [http://wiki.lustre.org/images/6/6f/T10-062002.pdf '''Lustre: Scalable Clustered Object Storage''']
** A technical presentation on Lustre.
** June 2002

* [http://wiki.lustre.org/images/b/b5/001_lustretechnical-fall2002.pdf '''Lustre - the inter-galactic cluster file system?''']
** A technical overview of Lustre from 2002.
** June 2002

* [http://wiki.lustre.org/images/7/79/Intragalactic-2001.pdf '''Lustre Light: a simpler fully functional cluster file system''']
** September 2001

* [http://wiki.lustre.org/images/c/c9/LustreSystemAnatomy.pdf '''Lustre System Anatomy''']
** Lustre component overview.

* [http://wiki.lustre.org/images/a/af/Intergalactic-062001.pdf '''Lustre: the intergalactic file system for the international labs?''']
** Presentation for Linux World and elsewhere on Lustre and Next-Generation Data Centers
** June 2001

* [http://wiki.lustre.org/images/4/44/Obdcluster.pdf '''The object-based storage cluster file systems and parallel I/O''']
** Sandia presentation on Lustre and Linux clustering

* [http://wiki.lustre.org/images/a/a2/Sdi-clusters.pdf '''Linux clustering and storage management''']
** Powerpoint slides of an overview of cluster and OBD technology

* [http://wiki.lustre.org/images/8/81/Lustre-sow-dist.pdf '''Lustre Technical Project Summary''']
** A Lustre roadmap presented to address the [http://wiki.lustre.org/images/7/70/SGSRFP.pdf Tri-Labs/DOD SGS File System RFP].
** July 2001

* [http://wiki.lustre.org/images/b/bd/Dfsprotocols.pdf '''File Systems for Clusters from a Protocol Perspective''']
** A comparative description of several distributed file systems.
** Proc. Second Extreme Linux Topics Workshop, Monterey CA, June 1999.

* [http://www.pdl.cs.cmu.edu/NASD '''CMU NASD project''']

* [http://wiki.lustre.org/images/2/24/Osd-r03.pdf '''Working draft T10 OSD''']
** A standards effort exists in the T10 OSD working group proposal.
** October 2000

Lustre HPCS Activities

2009-08-08T03:57:02Z

Jkdawson: Second draft of HPCS page

The DARPA High Productivity Computing Systems (HPCS) project is a program created to ensure that US Government agencies continue to have access to the advanced high-performance computing technologies needed to fulfill their missions. The program objectives include achieving extremely high IO performance and file system scale and reliability. [http://www.cray.com Cray Inc.] has partnered with Sun to use Lustre to achieve these goals.

General information on the HPCS program is available at:

*http://www.highproductivity.org/

*http://www.darpa.mil/ipto/programs/hpcs/hpcs.asp

John Carrier of Cray, Inc. presented an overview of the HPCS IO project at the Lustre User Group meeting in April of 2008. The slides and video of John Carrier's presentation can be found here:

*[http://wiki.lustre.org/images/1/16/LUG08_Cray_HPCS.pdf DARPA HPCS Project Slides] and *[http://video.google.com/videoplay?docid=-5970693206965456534&hl=en Video] - John Carrier, Cray Inc.

There were three presentations on the HPCS IO project at the Scalability Workshop held by the [[Sun Lustre Center of Excellence at Oak Ridge National Laboratory]] in May, 2009.

*[[Media:Carrier_2009-05-19_ORNL_LCE_HPCS.pdf|HPCS IO - John Carrier, Cray Inc.]]
*[[Media:Newman_May_Lustre_Workshop.pdf|What is HPCS and How Does it Impact IO - Henry Newman, Instrumental Inc.]]
*[[Media:Dilger_Lustre_HPCS_May_Workshop.pdf|Lustre HPCS Design Overview - Andreas Dilger, Sun Microsystems]]

The Lustre team has prepared several design documents as part of the HPCS program. These designs are:

*[[Media:JKD_Wiki_V1_2009_08_07_Lustre_HPCS_Overview.pdf|Lustre HPCS Design Overview]]
*[[Media:FSCK_Design-2009-06-15-09.pdf|Filesystem Integrity Check Design]]
*[[Media:End-to-End-Integrity-2009-06-15.pdf|End to End Data Integrity Design]]
*[[Media:Channel Bonding_06_15_09.pdf|LNet Channel Bonding Design]]
*[[Media:Rebuild_performance-2009-06-15.pdf|Rebuild Performance Design]]
*[[Media:HPCS_CMD_06_15_09.pdf|Clustered Metadata Design]]

File:HPCS CMD 06 15 09.pdf

2009-08-08T03:31:11Z

Jkdawson: Clustered Metadat design doc for HPCS.

Clustered Metadat design doc for HPCS.

File:Rebuild performance-2009-06-15.pdf

2009-08-08T03:30:30Z

Jkdawson: Rebuild performance design for HPCS.

Rebuild performance design for HPCS.

File:Channel Bonding 06 15 09.pdf

2009-08-08T03:29:53Z

Jkdawson: Channel bonding design doc for HPCS.

Channel bonding design doc for HPCS.

File:End-to-End-Integrity-2009-06-15.pdf

2009-08-08T03:29:04Z

Jkdawson: E2E Integrity Design for HPCS.

E2E Integrity Design for HPCS.

File:FSCK Design-2009-06-15-09.pdf

2009-08-08T03:28:00Z

Jkdawson: FSCK Design for HPCS.

FSCK Design for HPCS.

File:JKD Wiki V1 2009 08 07 Lustre HPCS Overview.pdf

2009-08-08T03:27:09Z

Jkdawson: Current version of HPCS File System Design Document. Has most of Carrier's comments incorporated but still need a couple minor questions answered by Andreas.

Current version of HPCS File System Design Document. Has most of Carrier's comments incorporated but still need a couple minor questions answered by Andreas.

Lustre HPCS Activities

2009-08-08T03:17:37Z

Jkdawson:

The DARPA High Productivity Computing Systems (HPCS) project is a program created to ensure that US Government agencies continue to have access to the advanced high-performance computing technologies needed to fulfill their missions. The program objectives include achieving extremely high IO performance. [http://www.cray.com Cray Inc.] has partnered with Sun to use Lustre to achieve these goals.

General information on the HPCS program is available at:

*http://www.highproductivity.org/

*http://www.darpa.mil/ipto/programs/hpcs/hpcs.asp

John Carrier of Cray, Inc. presented an overview of the HPCS IO project at the Lustre User Group meeting in April of 2008. The slides and video of John Carrier's presentation can be found here:

*[http://wiki.lustre.org/images/1/16/LUG08_Cray_HPCS.pdf DARPA HPCS Project Slides] and *[http://video.google.com/videoplay?docid=-5970693206965456534&hl=en Video] - John Carrier, Cray Inc.

There were three presentations on the HPCS IO project at the Scalability Workshop held by the [[Sun Lustre Center of Excellence at Oak Ridge National Laboratory]] in May, 2009.

*[[Media:Carrier_2009-05-19_ORNL_LCE_HPCS.pdf|HPCS IO - John Carrier, Cray Inc.]]
*[[Media:Newman_May_Lustre_Workshop.pdf|What is HPCS and How Does it Impact IO - Henry Newman, Instrumental Inc.]]
*[[Media:Dilger_Lustre_HPCS_May_Workshop.pdf|Lustre HPCS Design Overview - Andreas Dilger, Sun Microsystems]]

The Lustre team has prepared several design documents as part of the HPCS program.

Lustre HPCS Activities

2009-08-07T21:27:16Z

Jkdawson: First draft os Lustre HPCS page

The DARPA High Productivity Computing Systems project isa program created to ensure that US Government agencies continue to have access to the advanced high-performance computing technologies needed to fulfill their missions

Includes a number of “Mission Partners”(DOE, DoD, NSF, NSA, NNSA, etc) who work with DARPA to ensure that the systems developed under the program will meet their current and future needs

Commercial viability of the resulting system design is a key goal

Online resources

http://www.highproductivity.org/

http://www.darpa.mil/ipto/programs/hpcs/hpcs.asp

Lustre Center of Excellence at Oak Ridge National Laboratory

2009-07-09T16:58:47Z

Jkdawson: Added titles to workshop presentation links

The Sun Lustre Center of Excellence (LCE) at ORNL advances the state of Lustre for use in large scale HPC environments. As you'll see from the projects below, the ORNL LCE focuses on both systems and applications aspects of ensuring that Lustre meets the needs of DOE and the HPC community in general.

==Current projects and their status==
===Applications IO Performance===
Mike Booth, one of the LCE personnel, is investigating applications IO performance on the Lustre systems at ORNL to identify ways to achieve more consistent and higher IO performance. He is also working on enhancing the ADIOS IO layer.
<p>
Feel free to [mailto:Michael.Booth@Sun.COM contact Mike] for more information and status of his work.

==Events==

The LCE sponsors events to encourage HPC community involvement
in analyzing IO and storage requirements and identifying ways for
Lustre to address these requirements.

The next event will be an Applications Workshop this Autumn 2009.

===LCE Summit - February 2008===
*[[Media:LCE_Summit_Summary_Draft_March_14_2008.pdf|LCE Summit Meeting Summary]]
*[[Media:LCESummitSlides.pdf|LCE Summit Slides and Notes]]

===LCE Application I/O Workshop - April 16, 2008===
*[[Media:April2008ApplicationIOWorkshop.pdf|Application IO Workshop, Agenda and Notes]]
*[[Media:Lustre_workshop_WangDi.pdf|Lustre and Application IO - Wang Di's Slides]]
*[[Media:Lustre_workshop_Oleg.pdf|Lustre and Application IO - Oleg Drokin's Slides]]

===Lustre Scalability Workshop - Feb 10 & 11, 2009, ORNL===
*[[Media:Notes_on_SW1_Notes.pdf|Scalability Workshop Notes]]
*[[Media:LustreScalabilityWP_Updated.pdf|Scalability White Paper]]
*[[Media:Shipman_Feb_lustre_scalability.pdf|Galen Shipman's presentation]]
*[[Media:Eric-Barton_-_Lustre-Multi_PF_Roadmap-090130.pdf|Eric Barton's presentation]]

===Lustre Scalability Workshop - May 19 & 20, 2009, ORNL===
The May Lustre Scalability Workshop at ORNL was focused on long term (2015) HPC IO and storage requirements and on presentations on the IO objectives of the DOD HPCS program and how Lustre will achieve them.
*[[Media:Dawson_Lustre_Workshop_May_2009.pdf|Lustre Scalability Workshop, Initial Gap Response - John Dawson]]
*[[Media:Carrier_2009-05-19_ORNL_LCE_HPCS.pdf|HPCS IO - John Carrier]]
*[[Media:Newman_May_Lustre_Workshop.pdf|What is HPCS and How Does it Impact IO - Henry Newman]]
*[[Media:Shipman_May_lustre_scalability_workshop.pdf|2015 Parallel File System Requirements - Galen Shipman]]
*[[Media:Dilger_Lustre_HPCS_May_Workshop.pdf|Lustre HPCS Design Overview - Andreas Dilger]]

==White Papers==

LCE personnel have written a variety of papers on High Performance IO and potential Lustre features. Links to these documents are below.

*[[Media:Peta-Scale_wp.pdf|Peta-Scale IO with the Lustre File System]]

*[[Media:Lce_pop_submitted.pdf|Improving I/O Performance in POP (Parallel Ocean Program)]]

* Scalability Improvements
**[[Media:Lustre_Enhancement_Report_Interval_Trees.pdf|Using Interval Tree to scale extent locks]]
**[[Media:Lustre_Enhancement_Report_UUID_Hash_Tables.pdf|Implement hash tables to scale export lookups]]
**[[Media:A_Novel_Network_Request_Scheduler_for_a_Large_Scale_Storage_System.pdf|A Novel Network Request Scheduler for a Large Scale Storage System]]

*Lustre ADIO Driver Enhancements Whitepaper
**[[Media:Lustre_ADIO_Driver_Whitepaper_0926.pdf|Lustre ADIO Driver Enhancements]]

*FMS Application IO Analysis
**[[Media:FMS_Investigation_Report_%280915%29.pdf|FMS Application IO Performance Analysis]]

==Presentations==

See the ORNL [http://wiki.lustre.org/index.php/Lustre_User_Group slides and video] from their presentation at LUG 2009.

==Press Articles==

* [http://www.nccs.gov/2009/06/30/ornl-hosts-lustre-part-ii/ ORNL Hosts Lustre Part II]

==Lustre Internals Manual==
LCE and ORNL have written a Lustre filesystem internals document the describes the internal operation of Lustre.
*[[Media:Understanding_Lustre_Filesystem_Internals.pdf|Understanding Lustre Filesystem Internals]]

<br>
For more information about computing at Oak Ridge, click [http://www.nccs.gov/ here].

If you have questions or comments on this page or the LCE projects, contact [mailto:John.Dawson@sun.com John Dawson].

GetInvolved:Get Involved

2009-07-08T18:02:21Z

Jkdawson: FIxed link to LCE at ORNL

Find out about what the Lustre Community is doing, and get involved.

<div class="categoryLeft">

<big><strong>Community</strong></big>

Lustre user community events and resources.

* Sign up for one or more [[Mailing_Lists|Lustre Mailing Lists]]
* [[Lustre_User_Group|Lustre User Group 2009]] Lustre community's premier event to learn and share knowledge about Lustre technology.
** [[Lug_08|Lustre User Group 2008]] Agenda, presentations and panel discussions from the 2008 program (slide decks and videos available).
** [[Lug_07|Lustre User Group 2007]] Agenda and presentations from the 2007 program (slide decks available).
** [[Lug_06|Lustre User Group 2006]] Agenda and presentations from the 2006 program (slide decks available).
* Developers, find out how to [[Contribution_Policy|contribute code or test]].
* Read case studies and other [[Lustre_Publications|Lustre publications]], including Lustre engineering presentations.

</div>
<div class="categoryRight">

<big><strong>Community Development Projects</strong></big>
Interesting projects from the Lustre user community that are available for public use.

* [http://wiki.lustre.org/images/d/d9/Lustre-amanda.pdf Backup and Recovery: Amanda and Lustre] Use an Amanda client to back up a Lustre filesystem from Lustre clients.(PDF)
* [http://sourceforge.net/projects/lmt LLNL - Lustre Monitoring Tool] Tracks activity of server nodes (MDS, OSS and portals routers) for one or more Lustre filesystems.
* [http://www.bullopensource.org/lustre/ Bull - Open Source Tools for Lustre] Manage one or more Lustre filesystems from an administrative node.
* [http://shine-wiki.async.eu/wiki/Home CEA Administration Tool for Lustre 1.6] Set up and manage a Lustre filesystem on a cluster.

</div>
<div class="categoryRight">

<big><strong>Lustre Centers of Excellence</strong></big>

Find out about active Lustre Centers of Excellence (LCEs).

* [[Sun Lustre Center of Excellence at Oak Ridge National Laboratory]]
<p>
Other Lustre Centers of Excellence are located at: Naval Research Laboratory.
<br>
<br>
Additionally, there are several Lustre Development Centers that work with Sun to make significant code and testing contributions to Lustre and the Lustre Community. These centers are located at CEA, Juelich, LLNL, and Sandia.

Lustre Center of Excellence at Oak Ridge National Laboratory

2009-07-08T17:20:14Z

Jkdawson:

The Sun Lustre Center of Excellence (LCE) at ORNL advances the state of Lustre for use in large scale HPC environments. As you'll see from the projects below, the ORNL LCE focuses on both systems and applications aspects of ensuring that Lustre meets the needs of DOE and the HPC community in general.

==Current projects and their status==
===Applications IO Performance===
Mike Booth, one of the LCE personnel, is investigating applications IO performance on the Lustre systems at ORNL to identify ways to achieve more consistent and higher IO performance. He is also working on enhancing the ADIOS IO layer.
<p>
Feel free to [mailto:Michael.Booth@Sun.COM contact Mike] for more information and status of his work.

==Events==

The LCE sponsors events to encourage HPC community involvement
in analyzing IO and storage requirements and identifying ways for
Lustre to address these requirements.

The next event will be an Applications Workshop this Autumn 2009.

===LCE Summit - February 2008===
*[[Media:LCE_Summit_Summary_Draft_March_14_2008.pdf|LCE Summit Meeting Summary]]
*[[Media:LCESummitSlides.pdf|LCE Summit Slides and Notes]]

===LCE Application I/O Workshop - April 16, 2008===
*[[Media:April2008ApplicationIOWorkshop.pdf|Application IO Workshop, Agenda and Notes]]
*[[Media:Lustre_workshop_WangDi.pdf|Lustre and Application IO - Wang Di's Slides]]
*[[Media:Lustre_workshop_Oleg.pdf|Lustre and Application IO - Oleg Drokin's Slides]]

===Lustre Scalability Workshop - Feb 10 & 11, 2009, ORNL===
*[[Media:Notes_on_SW1_Notes.pdf|Scalability Workshop Notes]]
*[[Media:LustreScalabilityWP_Updated.pdf|Scalability White Paper]]
*[[Media:Shipman_Feb_lustre_scalability.pdf|Galen Shipman's presentation]]
*[[Media:Eric-Barton_-_Lustre-Multi_PF_Roadmap-090130.pdf|Eric Barton's presentation]]

===Lustre Scalability Workshop - May 19 & 20, 2009, ORNL===
The May Lustre Scalability Workshop at ORNL was focused on long term (2015) HPC IO and storage requirements and on presentations on the IO objectives of the DOD HPCS program and how Lustre will achieve them.
*[[Media:Dawson_Lustre_Workshop_May_2009.pdf|Dawson's presentation]]
*[[Media:Carrier_2009-05-19_ORNL_LCE_HPCS.pdf|Carrier's presentation]]
*[[Media:Newman_May_Lustre_Workshop.pdf|Newman's presentation]]
*[[Media:Shipman_May_lustre_scalability_workshop.pdf|Shipman's presentation]]
*[[Media:Dilger_Lustre_HPCS_May_Workshop.pdf|Dilger's presentation]]

==White Papers==

LCE personnel have written a variety of papers on High Performance IO and potential Lustre features. Links to these documents are below.

*[[Media:Peta-Scale_wp.pdf|Peta-Scale IO with the Lustre File System]]

*[[Media:Lce_pop_submitted.pdf|Improving I/O Performance in POP (Parallel Ocean Program)]]

* Scalability Improvements
**[[Media:Lustre_Enhancement_Report_Interval_Trees.pdf|Using Interval Tree to scale extent locks]]
**[[Media:Lustre_Enhancement_Report_UUID_Hash_Tables.pdf|Implement hash tables to scale export lookups]]
**[[Media:A_Novel_Network_Request_Scheduler_for_a_Large_Scale_Storage_System.pdf|A Novel Network Request Scheduler for a Large Scale Storage System]]

*Lustre ADIO Driver Enhancements Whitepaper
**[[Media:Lustre_ADIO_Driver_Whitepaper_0926.pdf|Lustre ADIO Driver Enhancements]]

*FMS Application IO Analysis
**[[Media:FMS_Investigation_Report_%280915%29.pdf|FMS Application IO Performance Analysis]]

==Presentations==

See the ORNL [http://wiki.lustre.org/index.php/Lustre_User_Group slides and video] from their presentation at LUG 2009.

==Press Articles==

* [http://www.nccs.gov/2009/06/30/ornl-hosts-lustre-part-ii/ ORNL Hosts Lustre Part II]

==Lustre Internals Manual==
LCE and ORNL have written a Lustre filesystem internals document the describes the internal operation of Lustre.
*[[Media:Understanding_Lustre_Filesystem_Internals.pdf|Understanding Lustre Filesystem Internals]]

<br>
For more information about computing at Oak Ridge, click [http://www.nccs.gov/ here].

If you have questions or comments on this page or the LCE projects, contact [mailto:John.Dawson@sun.com John Dawson].

Lustre Center of Excellence at Oak Ridge National Laboratory

2009-07-08T17:05:36Z

Jkdawson: Created page with 'The Sun Lustre Center of Excellence (LCE) at ORNL advances the state of Lustre for use in large scale HPC environments. As you'll see from the projects below, the ORNL LCE focuse...'

The Sun Lustre Center of Excellence (LCE) at ORNL advances the state of Lustre for use in large scale HPC environments. As you'll see from the projects below, the ORNL LCE focuses on both systems and applications aspects of ensuring that Lustre meets the needs of DOE and the HPC community in general.

==Current projects and their status==
===Applications IO Performance===
Mike Booth, one of the LCE personnel, is investigating applications IO performance on the Lustre systems at ORNL to identify way to achieve more consistent and higher IO performance. He is also working on enhancing the ADIOS IO layer.
<p>
Feel free to [mailto:Michael.Booth@Sun.COM contact Mike] for more information and status of his work.

==Events==

The LCE sponsors events to encourage HPC community involvement
in analyzing IO and storage requirements and identifying ways for
Lustre to address these requirements.

The next event will be an Applications Workshop this Autumn 2009.

===LCE Summit - February 2008===
*[[Media:LCE_Summit_Summary_Draft_March_14_2008.pdf|LCE Summit February 2008 Meeting Summary]]
*[[Media:LCESummitSlides.pdf|LCE Summit February 2008 Slides and Notes]]

===LCE Application I/O Workshop - April 16, 2008===
*[[Media:April2008ApplicationIOWorkshop.pdf|April 2008 Application IO Workshop, Agenda and Notes]]
*[[Media:Lustre_workshop_WangDi.pdf|Lustre and Application IO - Wang Di's Slides]]
*[[Media:Lustre_workshop_Oleg.pdf|Lustre and Apllication IO - Oleg Drokin's Slides]]

===Lustre Scalability Workshop - Feb 10 & 11, 2009, ORNL===
*[[Media:Notes_on_SW1_Notes.pdf|February Scalability Workshop Notes]]
*[[Media:LustreScalabilityWP_Updated.pdf|Scalability White Paper]]
*[[Media:Shipman_Feb_lustre_scalability.pdf|Galen Shipman's presentation]]
*[[Media:Eric-Barton_-_Lustre-Multi_PF_Roadmap-090130.pdf|Eric Barton's presentation]]

===Lustre Scalability Workshop - May 19 & 20, 2009, ORNL===
The May Lustre Scalability Workshop at ORNL was focused on long term (2015) HPC IO and storage requirements and on presentations on the IO objectives of the DOD HPCS program and how Lustre will achieve them.
*[[Media:Dawson_Lustre_Workshop_May_2009.pdf|Dawson's presentation]]
*[[Media:Carrier_2009-05-19_ORNL_LCE_HPCS.pdf|Carrier's presentation]]
*[[Media:Newman_May_Lustre_Workshop.pdf|Newman's presentation]]
*[[Media:Shipman_May_lustre_scalability_workshop.pdf|Shipman's presentation]]
*[[Media:Dilger_Lustre_HPCS_May_Workshop.pdf|Dilger's presentation]]

==White Papers==

LCE personnel have written a variety of papers on High Performance IO and potential Lustre features. Links to these documents are below.

*[[Media:Peta-Scale_wp.pdf|Peta-Scale IO with the Lustre File System]]

*[[Media:Lce_pop_submitted.pdf|Improving I/O Performance in POP (Parallel Ocean Program)]]

* Scalability Improvements
**[[Media:Lustre_Enhancement_Report_Interval_Trees.pdf|Using Interval Tree to scale extent locks]]
**[[Media:Lustre_Enhancement_Report_UUID_Hash_Tables.pdf|Implement hash tables to scale export lookups]]
**[[Media:A_Novel_Network_Request_Scheduler_for_a_Large_Scale_Storage_System.pdf|A Novel Network Request Scheduler for a Large Scale Storage System]]

*Lustre ADIO Driver Enhancements Whitepaper
**[[Media:Lustre_ADIO_Driver_Whitepaper_0926.pdf|Lustre ADIO Driver Enhancements]]

*FMS Application IO Analysis
**[[Media:FMS_Investigation_Report_%280915%29.pdf|FMS Application IO Performance Analysis]]

==Presentations==

See the ORNL [http://wiki.lustre.org/index.php/Lustre_User_Group slides and video] from their presentation at LUG 2009.

==Press Articles==

* [http://www.nccs.gov/2009/06/30/ornl-hosts-lustre-part-ii/ ORNL Hosts Lustre Part II]

==Lustre internals manual==
LCE and ORNL have written a Lustre filesystem internals document the describes the internal operation of Lustre.
*[[Media:Understanding_Lustre_Filesystem_Internals.pdf|Understanding Lustre Filesystem Internals]]

<br>
For more information about computing at Oak Ridge, click [http://www.nccs.gov/ here].

If you have questions or comments on this page or the LCE projects, contact [mailto:John.Dawson@sun.com John Dawson].

GetInvolved:Get Involved

2009-07-08T01:00:15Z

Jkdawson:

Find out about what the Lustre Community is doing, and get involved.

<div class="categoryLeft">

<big><strong>Community</strong></big>

Lustre user community events and resources.

* Sign up for one or more [[Mailing_Lists|Lustre Mailing Lists]]
* [[Lustre_User_Group|Lustre User Group 2009]] Lustre community's premier event to learn and share knowledge about Lustre technology.
** [[Lug_08|Lustre User Group 2008]] Agenda, presentations and panel discussions from the 2008 program (slide decks and videos available).
** [[Lug_07|Lustre User Group 2007]] Agenda and presentations from the 2007 program (slide decks available).
** [[Lug_06|Lustre User Group 2006]] Agenda and presentations from the 2006 program (slide decks available).
* Developers, find out how to [[Contribution_Policy|contribute code or test]].
* Read case studies and other [[Lustre_Publications|Lustre publications]], including Lustre engineering presentations.

</div>
<div class="categoryRight">

<big><strong>Community Development Projects</strong></big>
Interesting projects from the Lustre user community that are available for public use.

* [http://wiki.lustre.org/images/d/d9/Lustre-amanda.pdf Backup and Recovery: Amanda and Lustre] Use an Amanda client to back up a Lustre filesystem from Lustre clients.(PDF)
* [http://sourceforge.net/projects/lmt LLNL - Lustre Monitoring Tool] Tracks activity of server nodes (MDS, OSS and portals routers) for one or more Lustre filesystems.
* [http://www.bullopensource.org/lustre/ Bull - Open Source Tools for Lustre] Manage one or more Lustre filesystems from an administrative node.
* [http://shine-wiki.async.eu/wiki/Home CEA Administration Tool for Lustre 1.6] Set up and manage a Lustre filesystem on a cluster.

</div>
<div class="categoryRight">

<big><strong>Lustre Centers of Excellence</strong></big>

Find out about active Lustre Centers of Excellence (LCEs).

* [http://wiki.lustre.org/index.php/ORNL_LCE Lustre Centre of Excellence at Oak Ridge National Laboratory]
<p>
Other Lustre Centers of Excellence are located at: Naval Research Laboratory.
<br>
<br>
Additionally, there are several Lustre Development Centers that work with Sun to make significant code and testing contributions to Lustre and the Lustre Community. These centers are located at CEA, Juelich, LLNL, and Sandia.

GetInvolved:Get Involved

2009-07-03T04:57:57Z

Jkdawson: Changed link to new ORNL LCE page

Find out about what the Lustre Community is doing, and get involved.

<div class="categoryLeft">

<big><strong>Community</strong></big>

Lustre user community events and resources.

* Sign up for one or more [[Mailing_Lists|Lustre Mailing Lists]]
* [[Lustre_User_Group|Lustre User Group 2009]] Lustre community's premier event to learn and share knowledge about Lustre technology.
** [[Lug_08|Lustre User Group 2008]] Agenda, presentations and panel discussions from the 2008 program (slide decks and videos available).
** [[Lug_07|Lustre User Group 2007]] Agenda and presentations from the 2007 program (slide decks available).
** [[Lug_06|Lustre User Group 2006]] Agenda and presentations from the 2006 program (slide decks available).
* Developers, find out how to [[Contribution_Policy|contribute code or test]].
* Read case studies and other [[Lustre_Publications|Lustre publications]], including Lustre engineering presentations.

</div>
<div class="categoryRight">

<big><strong>Community Development Projects</strong></big>
Interesting projects from the Lustre user community that are available for public use.

* [http://wiki.lustre.org/images/d/d9/Lustre-amanda.pdf Backup and Recovery: Amanda and Lustre] Use an Amanda client to back up a Lustre filesystem from Lustre clients.(PDF)
* [http://sourceforge.net/projects/lmt LLNL - Lustre Monitoring Tool] Tracks activity of server nodes (MDS, OSS and portals routers) for one or more Lustre filesystems.
* [http://www.bullopensource.org/lustre/ Bull - Open Source Tools for Lustre] Manage one or more Lustre filesystems from an administrative node.
* [http://shine-wiki.async.eu/wiki/Home CEA Administration Tool for Lustre 1.6] Set up and manage a Lustre filesystem on a cluster.

</div>
<div class="categoryRight">

<big><strong>Lustre Centres of Excellence</strong></big>

Find out about active Lustre Centres of Excellence (LCEs).

* [http://wiki.lustre.org/index.php/ORNL_LCE Lustre Centre of Excellence at Oak Ridge National Laboratory]
<p>
Other Lustre Centers of Excellence are located at: Naval Research Laboratory.
<br>
<br>
Additionally, there are several Lustre Development Centers that work with Sun to make significant code and testing contributions to Lustre and the Lustre Community. These centers are located at CEA, Juelich, LLNL, and Sandia.

File:Dilger Lustre HPCS May Workshop.pdf

2009-06-28T19:05:39Z

Jkdawson: Andreas Dilger's presentation at May Scalability workshop.

Andreas Dilger's presentation at May Scalability workshop.

File:Shipman May lustre scalability workshop.pdf

2009-06-28T19:04:00Z

Jkdawson: Galen Shipman's presentation at May Scalability workshop.

Galen Shipman's presentation at May Scalability workshop.

File:Newman May Lustre Workshop.pdf

2009-06-28T19:01:14Z

Jkdawson: Henry Newman's presentation at May Scalability workshop.

Henry Newman's presentation at May Scalability workshop.

File:Carrier 2009-05-19 ORNL LCE HPCS.pdf

2009-06-28T18:58:37Z

Jkdawson: John Carrier's presentation at May Scalability workshop.

John Carrier's presentation at May Scalability workshop.

File:Dawson Lustre Workshop May 2009.pdf

2009-06-28T18:57:01Z

Jkdawson: John Dawson's presentation at May Scalability workshop.

John Dawson's presentation at May Scalability workshop.

File:Eric-Barton - Lustre-Multi PF Roadmap-090130.pdf

2009-06-28T18:14:59Z

Jkdawson: Eric Barton's presentation, February Scalability Workshop

Eric Barton's presentation, February Scalability Workshop

File:Shipman Feb lustre scalability.pdf

2009-06-28T18:11:28Z

Jkdawson: Galen Shipman's presentation, February Scalability Workshop

Galen Shipman's presentation, February Scalability Workshop

File:Understanding Lustre Filesystem Internals.pdf

2009-06-28T02:30:11Z

Jkdawson: Lustre internals document prepared by ORNL and the LCE. Based on Lustre 1.6.

Lustre internals document prepared by ORNL and the LCE. Based on Lustre 1.6.

File:Apps IO working.pdf

2009-06-28T01:51:07Z

Jkdawson: Working document that summarizes the status of Mike Booth's Applications IO work.

Working document that summarizes the status of Mike Booth's Applications IO work.

File:Notes on SW1 Notes.pdf

2009-06-27T18:49:08Z

Jkdawson: Scalability Workshop Notes from February 10-11, 2009

Scalability Workshop Notes from February 10-11, 2009

File:LustreScalabilityWP Updated.pdf

2009-06-27T18:37:35Z

Jkdawson: Lustre Scalability Whitepaper (Draft - March 8, 2009)

Lustre Scalability Whitepaper (Draft - March 8, 2009)

File:FMS Investigation Report (0915).pdf

2009-06-26T18:10:41Z

Jkdawson: FMS Performance Investigation Report

FMS Performance Investigation Report

File:Lustre ADIO Driver Whitepaper 0926.pdf

2009-06-26T18:06:06Z

Jkdawson: Lustre ADIO Collective Write Driver - Whitepaper

Lustre ADIO Collective Write Driver - Whitepaper

File:Lustre workshop Oleg.pdf

2009-06-26T17:55:56Z

Jkdawson: Lustre & Application IO - Oleg Drokin's Slides

Lustre & Application IO - Oleg Drokin's Slides

File:Lustre workshop WangDi.pdf

2009-06-26T17:51:50Z

Jkdawson: Lustre & Application IO Slides

Lustre & Application IO Slides

File:April2008ApplicationIOWorkshop.pdf

2009-06-26T17:24:12Z

Jkdawson: April 2008 Application IO Workshop Slides and Notes

April 2008 Application IO Workshop Slides and Notes

File:LCESummitSlides.pdf

2009-06-26T17:19:05Z

Jkdawson: LCE Summit February 2008 Slides and Notes

LCE Summit February 2008 Slides and Notes

File:LCE Summit Summary Draft March 14 2008.pdf

2009-06-26T03:35:06Z

Jkdawson:

File:A Novel Network Request Scheduler for a Large Scale Storage System.pdf

2009-06-26T03:11:33Z

Jkdawson: A Novel Network Request Scheduler for a Large Scale Storage System

A Novel Network Request Scheduler for a Large Scale Storage System