WARNING: This is the _old_ Lustre wiki, and it is in the process of being retired. The information found here is all likely to be out of date. Please search the new wiki for more up to date information.

Difference between revisions of "LibLustre How-To Guide"

From Obsolete Lustre Wiki
Jump to navigationJump to search
 
(34 intermediate revisions by 5 users not shown)
Line 1: Line 1:
= Liblustre A How to Guide =
+
<small>''(Updated: Feb 2010)''</small>
== Introduction to liblustre ==
+
__TOC__
 +
For most Lustre™ versions, a library version of the Lustre client file system (''liblustre'') is available. ''liblustre'' gives a user application (linked with the library) access to Lustre file systems, without needing to mount Lustre (VFS) on the client. The key goals for the library are to provide a portable mechanism to access Lustre from different POSIX-compliant operating systems, and to provide access from microkernel-based systems.
  
For some Lustre versions, a library version of the Lustre client file system (liblustre) is now available. Liblustre gives a user application (linked with the library) access to Lustre file systems, without needing to mount Lustre (VFS) on the client. The key goals for the library are to provide a portable mechanism to access Lustre from different POSIX-compliant operating systems, and to provide access from microkernel-based systems and from the Windows operating system.
+
Currently, ''liblustre'' is still under development and only works on linux (i386 & x86_64, not tested on ia64).
  
Currently, liblustre is still under development and only works on linux (i386 & x86_64, not tested on ia64).
+
In this document, we discuss how to use ''liblustre''.
  
In this document, we will discuss how to use liblustre.   
+
'''''Note:''''' ''liblustre'' is not required or even recommended for running Lustre on Linux. Most users will not use ''liblustre''. Instead, you should use the Lustre (VFS) client file system to mount Lustre directly''liblustre'' does ''NOT'' support multi-threaded applications.
  
'''Note that liblustre is NOT a required element of Lustre.'''
+
'''''Note:''''' ''liblustre'' is not widely tested as part of Lustre release testing, and is currently maintained only as a courtesy to the Lustre community.
liblustre is not required or even recommended for running Lustre on Linux. Instead, you should use the Lustre (VFS) client filesystem to mount Lustre directly.
 
  
Generally speaking, liblustre implements the Lustre client filesystem in user space. The liblustre component links LNET and libsysio together to form a shared library which could be used by applications to perform file I/O.
+
Generally speaking, ''liblustre'' implements the Lustre client file system in user space. The ''liblustre'' component links LNET and ''libsysio'' together to form a shared library which can be used by applications to perform file I/O.
  
== Building clients and servers for the Lustre library version ==
+
== Building Clients and Servers for liblustre ==
  
When using liblustre, servers are first built/configured in the usual way as described in [https://projects.clusterfs.com/lustre/LustreHowto Guide to Lustre]. liblustre is built by default unless "./configure --disable-liblustre" is specified.
+
When using liblustre, servers are first built/configured in the usual way as described in [[Building and Installing Lustre from Source Code]]. By default, liblustre is built unless "./configure --disable-liblustre" is specified.
 +
 
 +
The following liblustre files are located in lustre/liblustre:
  
The liblustre files are located in lustre/liblustre:
 
 
* liblustre/liblustre.so
 
* liblustre/liblustre.so
 
* liblustre/tests/
 
* liblustre/tests/
  
== How to use liblustre ==
+
== How to Use liblustre ==
  
First of all you must be running Lustre servers (MDS/OSTs) without ACLs and accepting connections on insecure ports. For example, add '''options lnet networks=tcp(eth0) accept=all''' to /etc/modprobe.conf, then run the script '''MDS_MOUNT_OPTS="noacl" sh llmount.sh'''.  This will set up a MDS, OST, and client on the local machine. You should umount the Lustre client at /mnt/lustre.
+
First, you must configure the networking on Lustre servers (MDS/OSTs) to accept connections on insecure ports. For example, add ''options lnet networks=tcp(eth0) accept=all'' to /etc/modprobe.conf before mounting the server(s) as usual.  If you do not have any Lustre filesystem, the ''llmount.sh'' script will format temporary MDT and OST filesystems and start up MDS, OSS, and clients on that machine. You should umount the Lustre client at /mnt/lustre to avoid confusion between the liblustre client and the normal VFS client if you are also using that node for the liblustre client.
  
 
=== Mount Target ===
 
=== Mount Target ===
  
liblustre needs to know ''mount target'' before connect to lustre server. the format is like following:
+
liblustre needs to know the ''mount target'' before connecting to a Lustre server. The format is similar to the following:
  
     mds_nid:/mds_name/profile_name
+
     mgs_nid:/profile_name
  
* '''mds_nid''' is actually the hostname of MDS (or IP address if you don't have proper name resolution set up).
+
* ''mgs_nid'' is the actual hostname of the MGS (or IP address if you do not have proper name resolution set up).
* '''mds_name''' is the name of the MDS. In the .xml file, it appears in following line:
+
     
 +
* ''profile_name'' is the profile name of the client mount point, also called the ''filesystem name'' in many configurations.
  
      <mds uuid='mds_uuid' name='mds_name'>
+
For additional information on mount target, refer to Lustre documentation. This can be passed to most liblustre programs via the environment variable LIBLUSTRE_MOUNT_TARGET.
     
+
 
* '''profile''' is the profile name of the client mountpoint, often '''client''' in many configs.
+
=== Sanity Test ===
 +
A statically built-in liblustre test program ''lustre/liblustre/tests/sanity'' is also included in the lustre-tests RPM package. You can use this test to verify if liblustre is working properly:
  
Please refer to lustre documents for more mount target explaination.  This can be passed to most liblustre programs via the environment variable LIBLUSTRE_MOUNT_TARGET.
+
    sanity --target mgs_nid:/profile_name
  
=== sanity test ===
+
=== How It Works ===
  
there's a statically built-in liblustre test program at ''lustre/liblustre/tests/sanity''. you could use it to verify whether liblustre work properly:
+
If we run an existing program, e.g. iozone, then we will use LD_PRELOAD to load ''liblustre.so'' first. The start function of ''liblustre.so'' will mount a lustre partition on certain directories, e.g. /mnt/lustre. Furthermore, following loaded iozone's functions calls such as open, read, write, etc. will dynamically linked with implementations in ''liblustre.so'', instead of ''libc'' in the usual case.  Thus, we can intercept filesystem-related system calls and translate them into Lustre commands.
  
    sanity --target mds_nid:/mdsname/profile_name
+
=== Necessary Environmental Variables ===
  
=== How It Works ===
+
A simple script ''lustre/utils/lrun'' sets some environment variables:
If we run a program, e.g. iozone, we'll use LD_PRELOAD to load ''liblustre.so'' at first. The start function of ''liblustre.so'' will mount a lustre partion on certain dir, e.g. /mnt/lustre. Furthermore, following loaded iozone's functions calls such as open/read/write... will dynamically linked with the implementations in ''liblustre.so'' instead of ''libc'' in the usually case. Thus we could intercept the filesystem related system calls and translate into lustre commands.
 
  
=== Necessary Environment Variables ===
+
* mount point
There's a simple script ''lustre/utils/lrun'' which set some ENVs:
 
a. mount point
 
 
    
 
    
 
     LIBLUSTRE_MOUNT_POINT=${LIBLUSTRE_MOUNT_POINT:-"/mnt/lustre"}
 
     LIBLUSTRE_MOUNT_POINT=${LIBLUSTRE_MOUNT_POINT:-"/mnt/lustre"}
 
+
:This is where liblustre mounts the remote Lustre filesystem. The default location is ''/mnt/lustre''. Make sure this directory exists on the client system.
this is where liblustre will mount remote lustre filesystem. default is ''/mnt/lustre'', please make sure it exist on your system:
 
  
b. mount target
+
* mount target
 
      
 
      
 
     LIBLUSTRE_MOUNT_TARGET=${LIBLUSTRE_MOUNT_TARGET:-"your_mount_target"}
 
     LIBLUSTRE_MOUNT_TARGET=${LIBLUSTRE_MOUNT_TARGET:-"your_mount_target"}
   
+
:Before using liblustre, you need to fill in the mount target (e.g. mdshost:/profile_name).
you need fill in the actual target before using liblustre (e.g. mdshost:/mdsname/client).
 
  
c. shared library
+
* shared library
 
      
 
      
 
     LD_PRELOAD=${LD_PRELOAD:-"/usr/lib/liblustre.so"}
 
     LD_PRELOAD=${LD_PRELOAD:-"/usr/lib/liblustre.so"}
   
+
:Make sure you have installed lustre/liblustre/liblustre.so in this location.
make sure you'v install lustre/liblustre/liblustre.so into the this place.
+
 
 +
* server port
 +
 
 +
    LNET_ACCEPT_PORT=${LNET_ACCEPT_PORT:-"988"}
 +
:This is optional, necessary only when the Lustre server doesn't listen on the default port (e.g. server uses the lnet option "accept_port" to change its port). It's only meaningful for the socklnd (TCP network).
 +
 
 +
* other LNET options
 +
 
 +
:You'd need to set LNET_ROUTES for the liblustre client to use routing to access the remote filesystem.
 +
 
 +
=== Running Programs over liblustre ===
 +
 
 +
Until now, only a small number of applications have been tested with liblustre:
  
=== running programs over liblustre ===
 
Until now, only a very small number of applications have been tested with liblustre:
 
 
* iozone
 
* iozone
 
* IOR
 
* IOR
 
* simul
 
* simul
  
and also several standard unix command:
+
And also several standard UNIX commands:
 +
 
 
* ls
 
* ls
 
* touch
 
* touch
Line 86: Line 95:
 
* grep
 
* grep
  
You need prepend 'lrun' before the programs you intend to run:
+
You need to prepend ''lrun'' before the programs that you intend to run:
 
   
 
   
 
   lrun iozone -f /mnt/lustre/ioz_testfile -r 256k -s 1g
 
   lrun iozone -f /mnt/lustre/ioz_testfile -r 256k -s 1g
Line 96: Line 105:
 
   lrun find /mnt/lustre/
 
   lrun find /mnt/lustre/
 
   lrun .....
 
   lrun .....
 
'''Important Note'''
 
liblustre is somewhat sensitive to glibc version, what we are using is glibc-2.2.5 and glic-2.3.2.
 
----
 
*'''[http://wiki.lustre.org/index.php?title=Main_Page FrontPage]'''
 

Latest revision as of 10:19, 22 February 2010

(Updated: Feb 2010)

For most Lustre™ versions, a library version of the Lustre client file system (liblustre) is available. liblustre gives a user application (linked with the library) access to Lustre file systems, without needing to mount Lustre (VFS) on the client. The key goals for the library are to provide a portable mechanism to access Lustre from different POSIX-compliant operating systems, and to provide access from microkernel-based systems.

Currently, liblustre is still under development and only works on linux (i386 & x86_64, not tested on ia64).

In this document, we discuss how to use liblustre.

Note: liblustre is not required or even recommended for running Lustre on Linux. Most users will not use liblustre. Instead, you should use the Lustre (VFS) client file system to mount Lustre directly. liblustre does NOT support multi-threaded applications.

Note: liblustre is not widely tested as part of Lustre release testing, and is currently maintained only as a courtesy to the Lustre community.

Generally speaking, liblustre implements the Lustre client file system in user space. The liblustre component links LNET and libsysio together to form a shared library which can be used by applications to perform file I/O.

Building Clients and Servers for liblustre

When using liblustre, servers are first built/configured in the usual way as described in Building and Installing Lustre from Source Code. By default, liblustre is built unless "./configure --disable-liblustre" is specified.

The following liblustre files are located in lustre/liblustre:

  • liblustre/liblustre.so
  • liblustre/tests/

How to Use liblustre

First, you must configure the networking on Lustre servers (MDS/OSTs) to accept connections on insecure ports. For example, add options lnet networks=tcp(eth0) accept=all to /etc/modprobe.conf before mounting the server(s) as usual. If you do not have any Lustre filesystem, the llmount.sh script will format temporary MDT and OST filesystems and start up MDS, OSS, and clients on that machine. You should umount the Lustre client at /mnt/lustre to avoid confusion between the liblustre client and the normal VFS client if you are also using that node for the liblustre client.

Mount Target

liblustre needs to know the mount target before connecting to a Lustre server. The format is similar to the following:

   mgs_nid:/profile_name
  • mgs_nid is the actual hostname of the MGS (or IP address if you do not have proper name resolution set up).
  • profile_name is the profile name of the client mount point, also called the filesystem name in many configurations.

For additional information on mount target, refer to Lustre documentation. This can be passed to most liblustre programs via the environment variable LIBLUSTRE_MOUNT_TARGET.

Sanity Test

A statically built-in liblustre test program lustre/liblustre/tests/sanity is also included in the lustre-tests RPM package. You can use this test to verify if liblustre is working properly:

   sanity --target mgs_nid:/profile_name

How It Works

If we run an existing program, e.g. iozone, then we will use LD_PRELOAD to load liblustre.so first. The start function of liblustre.so will mount a lustre partition on certain directories, e.g. /mnt/lustre. Furthermore, following loaded iozone's functions calls such as open, read, write, etc. will dynamically linked with implementations in liblustre.so, instead of libc in the usual case. Thus, we can intercept filesystem-related system calls and translate them into Lustre commands.

Necessary Environmental Variables

A simple script lustre/utils/lrun sets some environment variables:

  • mount point
   LIBLUSTRE_MOUNT_POINT=${LIBLUSTRE_MOUNT_POINT:-"/mnt/lustre"}
This is where liblustre mounts the remote Lustre filesystem. The default location is /mnt/lustre. Make sure this directory exists on the client system.
  • mount target
   LIBLUSTRE_MOUNT_TARGET=${LIBLUSTRE_MOUNT_TARGET:-"your_mount_target"}
Before using liblustre, you need to fill in the mount target (e.g. mdshost:/profile_name).
  • shared library
   LD_PRELOAD=${LD_PRELOAD:-"/usr/lib/liblustre.so"}
Make sure you have installed lustre/liblustre/liblustre.so in this location.
  • server port
   LNET_ACCEPT_PORT=${LNET_ACCEPT_PORT:-"988"}
This is optional, necessary only when the Lustre server doesn't listen on the default port (e.g. server uses the lnet option "accept_port" to change its port). It's only meaningful for the socklnd (TCP network).
  • other LNET options
You'd need to set LNET_ROUTES for the liblustre client to use routing to access the remote filesystem.

Running Programs over liblustre

Until now, only a small number of applications have been tested with liblustre:

  • iozone
  • IOR
  • simul

And also several standard UNIX commands:

  • ls
  • touch
  • rm
  • mkdir
  • rmdir
  • mv
  • cp
  • find
  • grep

You need to prepend lrun before the programs that you intend to run:

  lrun iozone -f /mnt/lustre/ioz_testfile -r 256k -s 1g
   
  lrun mkdir /mnt/lustre/testdir
  lrun touch /mnt/lustre/testdir/testfile
  lrun cp /etc/fstab /mnt/lustre/testdir
  lrun ls /mnt/lustre/testdir
  lrun find /mnt/lustre/
  lrun .....