Lustre FUSE

Note - while this page is out of date, it does contain important information and code for using Lustre with FUSE, and provides a starting point for anyone interested to work on this.

I (Stuart Midgley) have undertaken a number of projects with FUSE recently and been impressed with how easy it is to develop user space file systems. I then started using our Cray XT3 and saw that you could dynamically load a module to wrap your system calls to easily port a code to the XT3. This lead me to the idea that I should be able to build the example FUSE code (which creates a mirror file system) against liblustre and then have full access to the lustre file system from workstations without having to use a patched kernel or even the lustre kernel modules.

To that end, here is how to do it :)

I have only tested this build process against Centos4.4 linux  dhcp232:~ > uname -a Linux dhcp232 2.6.9-42.0.8.EL #1 Tue Jan 30 12:10:22 EST 2007 i686 i686 i386 GNU/Linux a single node lustre cluster  dhcp236:~ > uname -a Linux dhcp236 2.6.9-42.0.3.EL_lustre.1.5.97smp #1 SMP Thu Jan 11 17:55:05 MST 2007 i686 i686 i386 GNU/Linux dhcp236:~ > df -h Filesystem           Size  Used Avail Use% Mounted on /dev/hda3              50G  3.1G   45G   7% / /dev/hda1            487M   19M  443M   4% /boot none                  93M     0   93M   0% /dev/shm /dev/hda6            1.8G   86M  1.6G   6% /lustre/mdt /dev/hda5             20G  738M   18G   4% /lustre/ost and Lustre 1.5.97.

IMPORTANT you need to have your lustre file system accepting all connections  options lnet networks=tcp(eth0) accept=all in /etc/modprobe.conf

---

Lets get into it:

Download relevent files
* Download FUSE 2.6.3 (this is what all the patches apply to) * Download Lustre 1.6b7 (1.5.97) (this is the version patches apply to) * Download (originally from ) to allow FUSE to build against Centos4.4 kernel * Download (originally from ) which rectifies a debug problem, a statfs problem, the `SYSIO_INTERFACE_NAME` inconsistency with `mount`, and a linking problem * Download (originally from  ) which is the Lustre FUSE application file system

Building FUSE
* un-tar-compress FUSE  tar zxf fuse-2.6.3.tar.gz * apply the patch to allow to build on Centos4.4  patch -d fuse-2.6.3 -p1 < fuse-2.6.3-rhel-build.patch * build and install FUSE  cd fuse-2.6.3 ./configure make sudo -s make install modprobe fuse * you should have fuse built, installed and the kernel module up and running * Now make sure users can mount file systems  chmod guo+wr /dev/fuse

Building liblustre
* un-tar-compress Lustre  tar zxf lustre-1.5.97.tar.gz * apply the patches  patch -d lustre-1.5.97 -p1 < liblustre.diff * build liblustre  setenv CFLAGS '-DSYSIO_LABEL_NAMES="fuse_" -D_FILE_OFFSET_BITS=64 -D_HAVE_STATVFS' cd lustre-1.5.97 ./configure --enable-liblustre make sudo -s make install remember you should be using `tcsh`. Maybe you don't need the `make install`, but hey everything goes in `/usr` right? You wanted all of lustre installed right? * ok, you should now have liblustre in `/usr/lib`  ls -l /usr/lib/liblustre*

Building lustrefuse
Now this is the bit we have all been waiting for :) * Build lustrefuse  cd lustrefuse-0.1 ./configure make sudo -s make install * and now mount lustre  mkdir ~/lustre setenv LIBLUSTRE_MOUNT_TARGET your.lustre.server:/your.lustre.fs /usr/local/bin/lustrefuse ~/lustre cd ~/lustre ls df -h * oh oh oh... there it is :) * bonnie++ and iozone happily run... albeit a bit slowly :)

Things to be done or cleaned up
* performance, performance, performance * lustrefuse should probably cache data so that larger chunks are presented to liblustre * threading? * Sun to work on liblustre for performance? * get patches listed into FUSE and lustre * add support for the entire FUSE api * um... did I mention performance?

And the real reason for this? The MacOSX port.
I notice while hacking around inside Lustre that their looks to be Darwin ports of everything... but the code isn't provided and liblustre doesn't build. It would be really good if Sun got liblustre to build on MacOSX which would allow mac's to mount lustre file systems.