Architecture - MPI IO and NetCDF

Note: The content on this page reflects the state of design of a Lustre feature at a particular point in time and may contain outdated information.

Definitions
ADIO - Abstract device interface for parallel I/O. Here, it specially means ADIO driver inside MPI.

HDF5 - Hierarchical Data Format (HDF5) is a set of software libraries and machine-independent standard for storing scientific data(metadata and array data) in files. Here, it specially means the HDF5 library.

NetCDF - NetCDF (network Common Data Form) is a set of software libraries and machine-independent data formats that support the creation, access, and sharing of array-oriented scientific data. Similar as HDF5, but do not support parallel I/O.

Background
Good Parallel I/O performance on lustre not only depends on the proper I/O pattern of the software, but also depends on the good behavior of special filesystem and its ADIO driver. This paper will discuss this topic in three fields.


 * 1) Lustre ADIO driver improvements.
 * 2) Lustre filesystem internal improvments for parallel I/o.
 * 3) HDF5 and netcdf with lustre