LibLustre How-To Guide

Liblustre A How to Guide

Introduction to liblustre

For some Lustre versions, a library version of the Lustre client file system (liblustre) is now available. Liblustre gives a user application (linked with the library) access to Lustre file systems, without needing to mount Lustre (VFS) on the client. The key goals for the library are to provide a portable mechanism to access Lustre from different POSIX-compliant operating systems, and to provide access from microkernel-based systems.

Currently, liblustre is still under development and only works on linux (i386 & x86_64, not tested on ia64).

In this document, we will discuss how to use liblustre.

Note that liblustre is NOT a required element of Lustre. liblustre is not required or even recommended for running Lustre on Linux. Instead, you should use the Lustre (VFS) client filesystem to mount Lustre directly. liblustre does NOT support multi-threaded applications.

Generally speaking, liblustre implements the Lustre client filesystem in user space. The liblustre component links LNET and libsysio together to form a shared library which can be used by applications to perform file I/O.

Building Clients and Servers for liblustre

When using liblustre, servers are first built/configured in the usual way as described in the How to Guide to Lustre. By default, liblustre is built unless "./configure --disable-liblustre" is specified.

The following liblustre files are located in lustre/liblustre:

• liblustre/liblustre.so
• liblustre/tests/

How to Use liblustre

First, you must run the Lustre servers (MDS/OSTs) without ACLs and accept connections on insecure ports. For example, add options lnet networks=tcp(eth0) accept=all to /etc/modprobe.conf, then run the script sh llmount.sh (if you do not already have a mounted Lustre filesystem to test against). This sets up a MDS, OST and client on the local machine. You should umount the Lustre client at /mnt/lustre to avoid confusion between the liblustre client and the normal VFS client.

Mount Target

liblustre needs to know the mount target before connecting to a Lustre server. The format is similar to the following:

   mgs_nid:/profile_name

• mgs_nid is the actual hostname of the MGS (or IP address if you do not have proper name resolution set up).

• profile_name is the profile name of the client mount point, also called the filesystem name in many configurations.

For additional information on mount target, refer to Lustre documentation. This can be passed to most liblustre programs via the environment variable LIBLUSTRE_MOUNT_TARGET.

Sanity Test

There is a statically built-in liblustre test program at lustre/liblustre/tests/sanity. You can use this test to verify if liblustre is working properly:

   sanity --target mgs_nid:/profile_name

How It Works

If we run a program, e.g. iozone, then we will use LD_PRELOAD to load liblustre.so first. The start function of liblustre.so will mount a lustre partition on certain directories, e.g. /mnt/lustre. Furthermore, following loaded iozone's functions calls such as open/read/write... will dynamically linked with implementations in liblustre.so, instead of libc in the usual case. Thus, we can intercept filesystem-related system calls and translate them into Lustre commands.

Necessary Environmental Variables

There is a simple script lustre/utils/lrun which set some ENVs:

• mount point

   LIBLUSTRE_MOUNT_POINT=${LIBLUSTRE_MOUNT_POINT:-"/mnt/lustre"}

This is where liblustre mounts the remote Lustre filesystem. The default location is /mnt/lustre. Make sure it exists on your system.

• mount target

   LIBLUSTRE_MOUNT_TARGET=${LIBLUSTRE_MOUNT_TARGET:-"your_mount_target"}

Before using liblustre, you need to fill in the actual mount target (e.g. mdshost:/mdsname/client).

• shared library

   LD_PRELOAD=${LD_PRELOAD:-"/usr/lib/liblustre.so"}

Make sure you have installed lustre/liblustre/liblustre.so in this location.

• server port

   LNET_ACCEPT_PORT=${LNET_ACCEPT_PORT:-"40"}

This is optional, necessary only when the Lustre server doesn't listen on the default port (e.g. server uses the lnet option "accept_port" to change its port). It's only meaningful for the socklnd.

• other lnet options

You'd need to set LNET_ROUTES for the liblustre client to use routing to access the remote filesystem.

Running Programs over liblustre

Until now, only a small number of applications have been tested with liblustre:

• iozone
• IOR
• simul

And also several standard UNIX commands:

• ls
• touch
• rm
• mkdir
• rmdir
• mv
• cp
• find
• grep

You need to prepend 'lrun' before the programs that you intend to run:

  lrun iozone -f /mnt/lustre/ioz_testfile -r 256k -s 1g
   
  lrun mkdir /mnt/lustre/testdir
  lrun touch /mnt/lustre/testdir/testfile
  lrun cp /etc/fstab /mnt/lustre/testdir
  lrun ls /mnt/lustre/testdir
  lrun find /mnt/lustre/
  lrun .....

Important Note

liblustre is somewhat sensitive to the glibc version. We are using glibc-2.2.5 and glic-2.3.2.

Liblustre (from old "Building Lustre" topic)

The Lustre library client, liblustre, relies on libsysio, which is a library that provides POSIX-like file and name space support for remote file systems from the application program address space. To obtain libsysio, go to: http://sourceforge.net/projects/libsysio/

NOTE: Liblustre is not for general use. It was created to work with specific hardware (Cray) and should never be used with other hardware.

Development of libsysio has continued ever since it was first targeted for use with Lustre. First, check out the b_lustre branch from the libsysio CVS repository. This provides the libsysio version that is compatible with Lustre. Once checked out, use the following steps to build libsysio.

* $ sh autogen.sh 
* $ ./configure --with-sockets 
* $ make 

Once libsysio is built, use the following commands to build liblustre.

* $ ./configure --with-lib –with-sysio=/path/to/libsysio/source
* $ make