Lustre FUSE

DISCLAIMER - EXTERNAL CONTRIBUTOR CONTENT

This content was submitted by an external contributor. We provide this information as a resource for the Lustre™ open-source community, but we make no representation as to the accuracy, completeness or reliability of this information.

---

Note: This page is out of date. However, it contains important information and code for using Lustre with FUSE and provides a starting point for anyone interested to work on this.

I (Stuart Midgley) have undertaken a number of projects with FUSE recently and been impressed with how easy it is to develop user space file systems. I then started using our Cray XT3 and saw that you could dynamically load a module to wrap your system calls to easily port a code to the XT3. This lead me to the idea that I should be able to build the example FUSE code (which creates a mirror file system) against liblustre and then have full access to the lustre file system from workstations without having to use a patched kernel or even the lustre kernel modules.

To that end, here is how to do it :)

I have only tested this build process against Centos4.4 linux

dhcp232:~ > uname -a
Linux dhcp232 2.6.9-42.0.8.EL #1 Tue Jan 30 12:10:22 EST 2007 i686 i686 i386 GNU/Linux

a single node lustre cluster

dhcp236:~ > uname -a
Linux dhcp236 2.6.9-42.0.3.EL_lustre.1.5.97smp #1 SMP Thu Jan 11 17:55:05 MST 2007 i686 i686 i386 GNU/Linux
dhcp236:~ > df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/hda3              50G  3.1G   45G   7% /
/dev/hda1             487M   19M  443M   4% /boot
none                   93M     0   93M   0% /dev/shm
/dev/hda6             1.8G   86M  1.6G   6% /lustre/mdt
/dev/hda5              20G  738M   18G   4% /lustre/ost

and Lustre 1.5.97.

IMPORTANT you need to have your lustre file system accepting all connections

options lnet networks=tcp(eth0) accept=all

in /etc/modprobe.conf

---

Lets get into it:

Download relevent files

• Download FUSE 2.6.3 (this is what all the patches apply to)
• Download Lustre 1.6b7 (1.5.97) (this is the version patches apply to)
• Download File:FUSE-2.6.3-rhel-build.patch (originally from [1]) to allow FUSE to build against Centos4.4 kernel
• Download File:Liblustre-1.5.97.diff (originally from [2]) which rectifies a debug problem, a statfs problem, the `SYSIO_INTERFACE_NAME` inconsistency with `mount()`, and a linking problem
• Download File:LustreFUSE-0.1.tgz (originally from [3] ) which is the Lustre FUSE application file system

Building FUSE

* un-tar-compress FUSE 

tar zxf fuse-2.6.3.tar.gz

* apply the patch to allow to build on Centos4.4

patch -d fuse-2.6.3 -p1 < fuse-2.6.3-rhel-build.patch

* build and install FUSE

cd fuse-2.6.3
./configure
make
sudo -s
make install
modprobe fuse

* you should have fuse built, installed and the kernel module up and running
* Now make sure users can mount file systems 

chmod guo+wr /dev/fuse

Building liblustre

* un-tar-compress Lustre 

tar zxf lustre-1.5.97.tar.gz

* apply the patches

patch -d lustre-1.5.97 -p1 < liblustre.diff

* build liblustre

setenv CFLAGS '-DSYSIO_LABEL_NAMES="fuse_" -D_FILE_OFFSET_BITS=64 -D_HAVE_STATVFS'
cd lustre-1.5.97
./configure --enable-liblustre
make
sudo -s
make install

remember you should be using `tcsh`. Maybe you don't need the `make install`, but hey everything goes in `/usr` right? You wanted all of lustre installed right?

* ok, you should now have liblustre in `/usr/lib` 

ls -l /usr/lib/liblustre*

Building lustrefuse

Now this is the bit we have all been waiting for :)

* Build lustrefuse 

cd lustrefuse-0.1
./configure
make
sudo -s
make install

* and now mount lustre

mkdir ~/lustre
setenv LIBLUSTRE_MOUNT_TARGET your.lustre.server:/your.lustre.fs
/usr/local/bin/lustrefuse ~/lustre
cd ~/lustre
ls
df -h

* oh oh oh... there it is :)
* bonnie++ and iozone happily run... albeit a bit slowly :)

Things to be done or cleaned up

• performance, performance, performance
• lustrefuse should probably cache data so that larger chunks are presented to liblustre
• threading?
• Sun to work on liblustre for performance?
• get patches listed into FUSE and lustre
• add support for the entire FUSE api
• um... did I mention performance?

And the real reason for this? The MacOSX port.

I notice while hacking around inside Lustre that their looks to be Darwin ports of everything... but the code isn't provided and liblustre doesn't build. It would be really good if Sun got liblustre to build on MacOSX which would allow mac's to mount lustre file systems.