WARNING: This is the _old_ Lustre wiki, and it is in the process of being retired. The information found here is all likely to be out of date. Please search the new wiki for more up to date information.

Lustre FUSE

From Obsolete Lustre Wiki
Revision as of 21:07, 2 September 2009 by Adilger (talk | contribs) (attach files locally so they will not be lost, minor clarification)
Jump to navigationJump to search

Note - while this page is out of date, it does contain important information and code for using Lustre with FUSE, and provides a starting point for anyone interested to work on this.

I (Stuart Midgley) have undertaken a number of projects with FUSE recently and been impressed with how easy it is to develop user space file systems. I then started using our Cray XT3 and saw that you could dynamically load a module to wrap your system calls to easily port a code to the XT3. This lead me to the idea that I should be able to build the example FUSE code (which creates a mirror file system) against liblustre and then have full access to the lustre file system from workstations without having to use a patched kernel or even the lustre kernel modules.

To that end, here is how to do it :)

I have only tested this build process against Centos4.4 linux

dhcp232:~ > uname -a
Linux dhcp232 2.6.9-42.0.8.EL #1 Tue Jan 30 12:10:22 EST 2007 i686 i686 i386 GNU/Linux

a single node lustre cluster

dhcp236:~ > uname -a
Linux dhcp236 2.6.9-42.0.3.EL_lustre.1.5.97smp #1 SMP Thu Jan 11 17:55:05 MST 2007 i686 i686 i386 GNU/Linux
dhcp236:~ > df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/hda3              50G  3.1G   45G   7% /
/dev/hda1             487M   19M  443M   4% /boot
none                   93M     0   93M   0% /dev/shm
/dev/hda6             1.8G   86M  1.6G   6% /lustre/mdt
/dev/hda5              20G  738M   18G   4% /lustre/ost

and Lustre 1.5.97.

IMPORTANT you need to have your lustre file system accepting all connections

options lnet networks=tcp(eth0) accept=all

in /etc/modprobe.conf

Lets get into it:

Download relevent files

* Download FUSE 2.6.3 (this is what all the patches apply to)
* Download Lustre 1.6b7 (1.5.97) (this is the version patches apply to)
* Download File:FUSE-2.6.3-rhel-build-patch (originally from [1]) to allow FUSE to build against Centos4.4 kernel
* Download File:Liblustre-1.5.97.diff (originally from [2]) which rectifies a debug problem, a statfs problem, the `SYSIO_INTERFACE_NAME` inconsistency with `mount()`, and a linking problem
* Download File:LustreFUSE-0.1.tgz (originally from [3] ) which is the Lustre FUSE application file system

Building FUSE

* un-tar-compress FUSE 
tar zxf fuse-2.6.3.tar.gz
* apply the patch to allow to build on Centos4.4
patch -d fuse-2.6.3 -p1 < fuse-2.6.3-rhel-build.patch
* build and install FUSE
cd fuse-2.6.3
sudo -s
make install
modprobe fuse
* you should have fuse built, installed and the kernel module up and running
* Now make sure users can mount file systems 
chmod guo+wr /dev/fuse

Building liblustre

* un-tar-compress Lustre 
tar zxf lustre-1.5.97.tar.gz
* apply the patches
patch -d lustre-1.5.97 -p1 < liblustre.diff
* build liblustre
cd lustre-1.5.97
./configure --enable-liblustre
sudo -s
make install

remember you should be using `tcsh`. Maybe you don't need the `make install`, but hey everything goes in `/usr` right? You wanted all of lustre installed right?

* ok, you should now have liblustre in `/usr/lib` 
ls -l /usr/lib/liblustre*

Building lustrefuse

Now this is the bit we have all been waiting for :)

* Build lustrefuse 
cd lustrefuse-0.1
sudo -s
make install
* and now mount lustre
mkdir ~/lustre
setenv LIBLUSTRE_MOUNT_TARGET your.lustre.server:/your.lustre.fs
/usr/local/bin/lustrefuse ~/lustre
cd ~/lustre
df -h
* oh oh oh... there it is :)
* bonnie++ and iozone happily run... albeit a bit slowly :)

Things to be done or cleaned up

* performance, performance, performance
* lustrefuse should probably cache data so that larger chunks are presented to liblustre
* threading?
* Sun to work on liblustre for performance?
* get patches listed into FUSE and lustre
* add support for the entire FUSE api
* um... did I mention performance?

And the real reason for this? The MacOSX port.

I notice while hacking around inside Lustre that their looks to be Darwin ports of everything... but the code isn't provided and liblustre doesn't build. It would be really good if Sun got liblustre to build on MacOSX which would allow mac's to mount lustre file systems.