WARNING: This is the _old_ Lustre wiki, and it is in the process of being retired. The information found here is all likely to be out of date. Please search the new wiki for more up to date information.
LibLustre How-To Guide: Difference between revisions
Line 24: | Line 24: | ||
== How to use liblustre == | == How to use liblustre == | ||
First | First, you must run the Lustre servers (MDS/OSTs) without ACLs and accept connections on insecure ports. For example, add '''options lnet networks=tcp(eth0) accept=all''' to /etc/modprobe.conf, then run the script '''MDS_MOUNT_OPTS="noacl" sh llmount.sh'''. This sets up a MDS, OST and client on the local machine. You should umount the Lustre client at /mnt/lustre. | ||
=== Mount Target === | === Mount Target === | ||
liblustre needs to know ''mount target'' before | liblustre needs to know the ''mount target'' before connecting to a Lustre server. The format is similar to the following: | ||
mds_nid:/mds_name/profile_name | mds_nid:/mds_name/profile_name | ||
* '''mds_nid''' is | * '''mds_nid''' is the actual hostname of the MDS (or IP address if you do not have proper name resolution set up). | ||
* '''mds_name''' is the name of the MDS. In the .xml file, it appears in following line: | * '''mds_name''' is the name of the MDS. In the .xml file, it appears in following line: | ||
<mds uuid='mds_uuid' name='mds_name'> | <mds uuid='mds_uuid' name='mds_name'> | ||
* '''profile''' is the profile name of the client | * '''profile''' is the profile name of the client mount point, often called '''client''' in many configurations. | ||
For additional information on mount target, refer to Lustre documentation. This can be passed to most liblustre programs via the environment variable LIBLUSTRE_MOUNT_TARGET. | |||
=== | === Sanity Test === | ||
There is a statically built-in liblustre test program at ''lustre/liblustre/tests/sanity''. You can use this test to verify if liblustre is working properly: | |||
sanity --target mds_nid:/mdsname/profile_name | sanity --target mds_nid:/mdsname/profile_name | ||
=== How It Works === | === How It Works === | ||
If we run a program, e.g. iozone, we | |||
If we run a program, e.g. iozone, then we will use LD_PRELOAD to load ''liblustre.so'' first. The start function of ''liblustre.so'' will mount a lustre partition on certain directories, e.g. /mnt/lustre. Furthermore, following loaded iozone's functions calls such as open/read/write... will dynamically linked with implementations in ''liblustre.so'', instead of ''libc'' in the usual case. Thus, we can intercept filesystem-related system calls and translate them into Lustre commands. | |||
=== Necessary Environment Variables === | === Necessary Environment Variables === | ||
There | |||
There is a simple script ''lustre/utils/lrun'' which set some ENVs: | |||
* mount point | |||
LIBLUSTRE_MOUNT_POINT=${LIBLUSTRE_MOUNT_POINT:-"/mnt/lustre"} | LIBLUSTRE_MOUNT_POINT=${LIBLUSTRE_MOUNT_POINT:-"/mnt/lustre"} | ||
This is where liblustre mounts the remote Lustre filesystem. The default location is ''/mnt/lustre''. Make sure it exists on your system. | |||
* mount target | |||
LIBLUSTRE_MOUNT_TARGET=${LIBLUSTRE_MOUNT_TARGET:-"your_mount_target"} | LIBLUSTRE_MOUNT_TARGET=${LIBLUSTRE_MOUNT_TARGET:-"your_mount_target"} | ||
you need fill in the actual target | Before using liblustre, you need to fill in the actual mount target (e.g. mdshost:/mdsname/client). | ||
* shared library | |||
LD_PRELOAD=${LD_PRELOAD:-"/usr/lib/liblustre.so"} | LD_PRELOAD=${LD_PRELOAD:-"/usr/lib/liblustre.so"} | ||
Make sure you have installed lustre/liblustre/liblustre.so in this location. | |||
=== Running Programs over liblustre === | |||
Until now, only a small number of applications have been tested with liblustre: | |||
* iozone | * iozone | ||
* IOR | * IOR | ||
* simul | * simul | ||
And also several standard UNIX commands: | |||
* ls | * ls | ||
* touch | * touch | ||
Line 87: | Line 93: | ||
* grep | * grep | ||
You need prepend 'lrun' before the programs you intend to run: | You need to prepend 'lrun' before the programs that you intend to run: | ||
lrun iozone -f /mnt/lustre/ioz_testfile -r 256k -s 1g | lrun iozone -f /mnt/lustre/ioz_testfile -r 256k -s 1g | ||
Line 99: | Line 105: | ||
'''Important Note''' | '''Important Note''' | ||
liblustre is somewhat sensitive to glibc version | |||
liblustre is somewhat sensitive to the glibc version. We are using glibc-2.2.5 and glic-2.3.2. | |||
---- | ---- | ||
*'''[http://wiki.lustre.org/index.php?title=Main_Page FrontPage]''' | *'''[http://wiki.lustre.org/index.php?title=Main_Page FrontPage]''' |
Revision as of 13:05, 19 May 2008
Liblustre A How to Guide
Introduction to liblustre
For some Lustre versions, a library version of the Lustre client file system (liblustre) is now available. Liblustre gives a user application (linked with the library) access to Lustre file systems, without needing to mount Lustre (VFS) on the client. The key goals for the library are to provide a portable mechanism to access Lustre from different POSIX-compliant operating systems, and to provide access from microkernel-based systems and from the Windows operating system.
Currently, liblustre is still under development and only works on linux (i386 & x86_64, not tested on ia64).
In this document, we will discuss how to use liblustre.
Note that liblustre is NOT a required element of Lustre. liblustre is not required or even recommended for running Lustre on Linux. Instead, you should use the Lustre (VFS) client filesystem to mount Lustre directly.
Generally speaking, liblustre implements the Lustre client filesystem in user space. The liblustre component links LNET and libsysio together to form a shared library which can be used by applications to perform file I/O.
Building clients and servers for liblustre
When using liblustre, servers are first built/configured in the usual way as described in the How to Guide to Lustre. By default, liblustre is built unless "./configure --disable-liblustre" is specified.
The following liblustre files are located in lustre/liblustre:
- liblustre/liblustre.so
- liblustre/tests/
How to use liblustre
First, you must run the Lustre servers (MDS/OSTs) without ACLs and accept connections on insecure ports. For example, add options lnet networks=tcp(eth0) accept=all to /etc/modprobe.conf, then run the script MDS_MOUNT_OPTS="noacl" sh llmount.sh. This sets up a MDS, OST and client on the local machine. You should umount the Lustre client at /mnt/lustre.
Mount Target
liblustre needs to know the mount target before connecting to a Lustre server. The format is similar to the following:
mds_nid:/mds_name/profile_name
- mds_nid is the actual hostname of the MDS (or IP address if you do not have proper name resolution set up).
- mds_name is the name of the MDS. In the .xml file, it appears in following line:
<mds uuid='mds_uuid' name='mds_name'>
- profile is the profile name of the client mount point, often called client in many configurations.
For additional information on mount target, refer to Lustre documentation. This can be passed to most liblustre programs via the environment variable LIBLUSTRE_MOUNT_TARGET.
Sanity Test
There is a statically built-in liblustre test program at lustre/liblustre/tests/sanity. You can use this test to verify if liblustre is working properly:
sanity --target mds_nid:/mdsname/profile_name
How It Works
If we run a program, e.g. iozone, then we will use LD_PRELOAD to load liblustre.so first. The start function of liblustre.so will mount a lustre partition on certain directories, e.g. /mnt/lustre. Furthermore, following loaded iozone's functions calls such as open/read/write... will dynamically linked with implementations in liblustre.so, instead of libc in the usual case. Thus, we can intercept filesystem-related system calls and translate them into Lustre commands.
Necessary Environment Variables
There is a simple script lustre/utils/lrun which set some ENVs:
- mount point
LIBLUSTRE_MOUNT_POINT=${LIBLUSTRE_MOUNT_POINT:-"/mnt/lustre"}
This is where liblustre mounts the remote Lustre filesystem. The default location is /mnt/lustre. Make sure it exists on your system.
- mount target
LIBLUSTRE_MOUNT_TARGET=${LIBLUSTRE_MOUNT_TARGET:-"your_mount_target"}
Before using liblustre, you need to fill in the actual mount target (e.g. mdshost:/mdsname/client).
- shared library
LD_PRELOAD=${LD_PRELOAD:-"/usr/lib/liblustre.so"}
Make sure you have installed lustre/liblustre/liblustre.so in this location.
Running Programs over liblustre
Until now, only a small number of applications have been tested with liblustre:
- iozone
- IOR
- simul
And also several standard UNIX commands:
- ls
- touch
- rm
- mkdir
- rmdir
- mv
- cp
- find
- grep
You need to prepend 'lrun' before the programs that you intend to run:
lrun iozone -f /mnt/lustre/ioz_testfile -r 256k -s 1g lrun mkdir /mnt/lustre/testdir lrun touch /mnt/lustre/testdir/testfile lrun cp /etc/fstab /mnt/lustre/testdir lrun ls /mnt/lustre/testdir lrun find /mnt/lustre/ lrun .....
Important Note
liblustre is somewhat sensitive to the glibc version. We are using glibc-2.2.5 and glic-2.3.2.