LibLustre How-To Guide

Liblustre A How to Guide

Liblustre Introduction

A library version of the Lustre client file system is now available for some versions of Lustre, it gives an user application linked with the library access to Lustre file systems without the need to mount Lustre (VFS) on the client. The key goals for the library are to provide a portable mechanism to access Lustre from different POSIX compliant operating systems, to provide access from microkernel based systems and from the Windows operating system.

Currently, liblustre is still under development and only works on linux (i386 & x86_64, not tested on ia64).

In this document, we will discuss how to use liblustre.

Note that liblustre is NOT a required element of Lustre. liblustre is not required or even recommended for running Lustre on Linux, you should instead use the Lustre (VFS) client filesystem to mount Lustre directly.

Generally speaking, liblustre implements the Lustre client filesystem in user space. The liblustre component links LNET and libsysio together to form a shared library which could be used by applications to perform File I/O.

Building clients and servers for the Lustre library version

When using liblustre, servers are first built/configured in the usual way as described in Guide to Lustre. liblustre is built by default unless "./configure --disable-liblustre" is specified.

The liblustre files are located in lustre/liblustre:

• liblustre/liblustre.so
• liblustre/tests/

How to use liblustre

First of all you must be running Lustre servers (MDS/OSTs) without ACLs and accepting connections on insecure ports. For example, add options lnet networks=tcp(eth0) accept=all to /etc/modprobe.conf, then run the script MDS_MOUNT_OPTS="noacl" sh llmount.sh. This will set up a MDS, OST, and client on the local machine. You should umount the Lustre client at /mnt/lustre.

Mount Target

liblustre needs to know mount target before connect to lustre server. the format is like following:

   mds_nid:/mds_name/profile_name

• mds_nid is actually the hostname of MDS (or IP address if you don't have proper name resolution set up).
• mds_name is the name of the MDS. In the .xml file, it appears in following line:

      <mds uuid='mds_uuid' name='mds_name'>
      

• profile is the profile name of the client mountpoint, often client in many configs.

Please refer to lustre documents for more mount target explaination. This can be passed to most liblustre programs via the environment variable LIBLUSTRE_MOUNT_TARGET.

sanity test

there's a statically built-in liblustre test program at lustre/liblustre/tests/sanity. you could use it to verify whether liblustre work properly:

   sanity --target mds_nid:/mdsname/profile_name

How It Works

If we run a program, e.g. iozone, we'll use LD_PRELOAD to load liblustre.so at first. The start function of liblustre.so will mount a lustre partion on certain dir, e.g. /mnt/lustre. Furthermore, following loaded iozone's functions calls such as open/read/write... will dynamically linked with the implementations in liblustre.so instead of libc in the usually case. Thus we could intercept the filesystem related system calls and translate into lustre commands.

Necessary Environment Variables

There's a simple script lustre/utils/lrun which set some ENVs: a. mount point

   LIBLUSTRE_MOUNT_POINT=${LIBLUSTRE_MOUNT_POINT:-"/mnt/lustre"}
  

this is where liblustre will mount remote lustre filesystem. default is /mnt/lustre, please make sure it exist on your system:

b. mount target

   LIBLUSTRE_MOUNT_TARGET=${LIBLUSTRE_MOUNT_TARGET:-"your_mount_target"}
   

you need fill in the actual target before using liblustre (e.g. mdshost:/mdsname/client).

c. shared library

   LD_PRELOAD=${LD_PRELOAD:-"/usr/lib/liblustre.so"}
   

make sure you'v install lustre/liblustre/liblustre.so into the this place.

running programs over liblustre

Until now, only a very small number of applications have been tested with liblustre:

 * iozone
 * IOR
 * simul

and also several standard unix command:

 * ls
 * touch
 * rm
 * mkdir
 * rmdir
 * mv
 * cp
 * find
 * grep

You need prepend 'lrun' before the programs you intend to run:

  {{{
  lrun iozone -f /mnt/lustre/ioz_testfile -r 256k -s 1g

  lrun mkdir /mnt/lustre/testdir
  lrun touch /mnt/lustre/testdir/testfile
  lrun cp /etc/fstab /mnt/lustre/testdir
  lrun ls /mnt/lustre/testdir
  lrun find /mnt/lustre/
  lrun .....
  }}}

Important Note liblustre is somewhat sensitive to glibc version, what we are using is glibc-2.2.5 and glic-2.3.2.

---

*FrontPage