WARNING: This is the _old_ Lustre wiki, and it is in the process of being retired. The information found here is all likely to be out of date. Please search the new wiki for more up to date information.

Difference between revisions of "Lustre Project List"

From Obsolete Lustre Wiki
Jump to navigationJump to search
Line 10: Line 10:
 
!Tracking Bug
 
!Tracking Bug
 
!Brief Description
 
!Brief Description
 +
|-
 +
|ioctl() number cleanups
 +
|1
 +
|kernel
 +
|[https://bugzilla.lustre.org/show_bug.cgi?id=20731 20731]
 +
|<small>Clean up Linux IOC numbering to properly use "size" field so that mixed 32- and 64-bit kernel/userspace ioctls work correctly.  Attention needs to be paid to maintaining userspace compatibility for a number of releases, so the old ioctl() numbers cannot simply be removed.</small>
 
|-
 
|-
 
|Improve testing Efficiency
 
|Improve testing Efficiency
|1
+
|3
 
|shell, test
 
|shell, test
 
|
 
|
|<small>Improve the performance, efficiency, and coverage of acc-smImproved test scheduling.  Dynamic cluster configuration.  Virtual machines for functional tests.</small>
+
|<small>Improve the performance, efficiency, and coverage of the acceptance-small.sh test scriptsMore advanced work includes improved test scheduling.  Dynamic cluster configuration.  Virtual machines for functional tests.</small>
 
|-
 
|-
 
|Config save/edit/restore
 
|Config save/edit/restore
Line 28: Line 34:
 
|[https://bugzilla.lustre.org/show_bug.cgi?id=21658 21658]
 
|[https://bugzilla.lustre.org/show_bug.cgi?id=21658 21658]
 
|<small>Add a low-level metadata unit test to allow measuring performance of the metadata stack without having connected clients, similar and/or integrated to the obdfilter survey (echo client, echo server)</small>
 
|<small>Add a low-level metadata unit test to allow measuring performance of the metadata stack without having connected clients, similar and/or integrated to the obdfilter survey (echo client, echo server)</small>
 +
|-
 +
|Readdir with large requests
 +
|4
 +
|MDS, RPC
 +
|[https://bugzilla.lustre.org/show_bug.cgi?id=17833 17833]
 +
|<small>Read directory pages in large chunks instead of the current page-at-a-time reads from the client.  This will improve readdir performance somewhat, and reduce load on the MDS.  It is expected to be significant over WAN high-latency links.</small>
 
|-
 
|-
 
|32TB ldiskfs filesystems
 
|32TB ldiskfs filesystems
Line 35: Line 47:
 
|<small>Single OST sizes larger than 16TB.  This is largely supported in newer ext4 filesystems (e.g. RHEL5.4, RHEL6), but thorough testing and some bug fixing work may be needed in obdfilter (1.8, 2.0) or OFD (2.x), and other work may be needed in client (all versions).
 
|<small>Single OST sizes larger than 16TB.  This is largely supported in newer ext4 filesystems (e.g. RHEL5.4, RHEL6), but thorough testing and some bug fixing work may be needed in obdfilter (1.8, 2.0) or OFD (2.x), and other work may be needed in client (all versions).
 
</small>
 
</small>
 +
|-
 +
|All RPCs pass a lock handle
 +
|5
 +
|DLM, RPC
 +
|[https://bugzilla.lustre.org/show_bug.cgi?id=22849 22849]
 +
|<small>For protocol correctness, and improved performance, it would be desirable for all RPCs that are done with a client lock held to send the lock handle along with the request.  For OST requests this means all read, write, truncate operations (unless "lockless") should include a lock handle.  This allows the OST to validate the request is being done by a client that holds the correct locks, and allows lockh->lock->object lookups to avoid OI lookups in most cases.</small>
 +
|-
 +
|OST Space Management (Basic)
 +
|6
 +
|HSM, layout
 +
|[https://bugzilla.lustre.org/show_bug.cgi?id=13107 13107]
 +
|<small>Simple migration capability - transparently migrate objects/files between OSTs (blocking application writes, or aborting migration during contention); evacuate OSTs and move file data to other OSTs; add new OST and balance data on it.  The OST doesn't really need to understand this, only the MDS (for LOV EA rewrite) and client (LOV EA rewrite); HSM project implements layout lock support and policy engine for automatic space management.</small>
 
|-
 
|-
 
|Imperative recovery
 
|Imperative recovery

Revision as of 15:42, 5 September 2010

List of Lustre Features and Projects

Below is a list of Lustre features and projects that are just waiting for someone to start working on them. They are listed roughly in order of increasing complexity, but this is highly dependent upon the coding skills of the developer and their familiarity with the Lustre code base.

Feature Complexity Required skills Tracking Bug Brief Description
ioctl() number cleanups 1 kernel 20731 Clean up Linux IOC numbering to properly use "size" field so that mixed 32- and 64-bit kernel/userspace ioctls work correctly. Attention needs to be paid to maintaining userspace compatibility for a number of releases, so the old ioctl() numbers cannot simply be removed.
Improve testing Efficiency 3 shell, test Improve the performance, efficiency, and coverage of the acceptance-small.sh test scripts. More advanced work includes improved test scheduling. Dynamic cluster configuration. Virtual machines for functional tests.
Config save/edit/restore 3 MGS, llog, config 17094 Need to be able to backup/edit/restore the client/MDS/OSS config llog files after a writeconf. One reason is for config recovery if the config llog becomes corrupted. Another reason is that all of the filesystem tunable parameters (including all of the OST pool definitions) are stored in the config llog and are lost if a writeconf is done.
mdd-survey tools for performance analysis 3 obdfilter-survey, mdd, benchmarking 21658 Add a low-level metadata unit test to allow measuring performance of the metadata stack without having connected clients, similar and/or integrated to the obdfilter survey (echo client, echo server)
Readdir with large requests 4 MDS, RPC 17833 Read directory pages in large chunks instead of the current page-at-a-time reads from the client. This will improve readdir performance somewhat, and reduce load on the MDS. It is expected to be significant over WAN high-latency links.
32TB ldiskfs filesystems 5 ldiskfs, obdfilter 20063 Single OST sizes larger than 16TB. This is largely supported in newer ext4 filesystems (e.g. RHEL5.4, RHEL6), but thorough testing and some bug fixing work may be needed in obdfilter (1.8, 2.0) or OFD (2.x), and other work may be needed in client (all versions).

All RPCs pass a lock handle 5 DLM, RPC 22849 For protocol correctness, and improved performance, it would be desirable for all RPCs that are done with a client lock held to send the lock handle along with the request. For OST requests this means all read, write, truncate operations (unless "lockless") should include a lock handle. This allows the OST to validate the request is being done by a client that holds the correct locks, and allows lockh->lock->object lookups to avoid OI lookups in most cases.
OST Space Management (Basic) 6 HSM, layout 13107 Simple migration capability - transparently migrate objects/files between OSTs (blocking application writes, or aborting migration during contention); evacuate OSTs and move file data to other OSTs; add new OST and balance data on it. The OST doesn't really need to understand this, only the MDS (for LOV EA rewrite) and client (LOV EA rewrite); HSM project implements layout lock support and policy engine for automatic space management.
Imperative recovery 6 recovery, RPC 18767 Reduce recovery time by having the server notify clients after recovery has completed instead of waiting for the client to timeout the RPC before it begins recovery.