I (Stuart Midgley) have undertaken a number of projects with FUSE recently and been impressed with how easy it is to develop user space file systems. I then started using our Cray XT3 and saw that you could dynamically load a module to wrap your system calls to easily port a code to the XT3. This lead me to the idea that I should be able to build the example FUSE code (which creates a mirror file system) against liblustre and then have full access to the lustre file system from workstations without having to use a patched kernel or even the lustre kernel modules.
To that end, here is how to do it :)
I have only tested this build process against Centos4.4 linux
dhcp232:~ > uname -a Linux dhcp232 2.6.9-42.0.8.EL #1 Tue Jan 30 12:10:22 EST 2007 i686 i686 i386 GNU/Linuxa single node lustre cluster
dhcp236:~ > uname -a Linux dhcp236 2.6.9-42.0.3.EL_lustre.1.5.97smp #1 SMP Thu Jan 11 17:55:05 MST 2007 i686 i686 i386 GNU/Linux dhcp236:~ > df -h Filesystem Size Used Avail Use% Mounted on /dev/hda3 50G 3.1G 45G 7% / /dev/hda1 487M 19M 443M 4% /boot none 93M 0 93M 0% /dev/shm /dev/hda6 1.8G 86M 1.6G 6% /lustre/mdt /dev/hda5 20G 738M 18G 4% /lustre/ostand Lustre 1.5.97.
IMPORTANT you need to have your lustre file system accepting all connections
options lnet networks=tcp(eth0) accept=allin /etc/modprobe.conf
Lets get into it:
Download relevent files
* Download FUSE 2.6.3 (this is what all the patches apply to) * Download Lustre 1.6b7 (1.5.97) (this is the version patches apply to) * Download FUSE patch to allow fuse to build against Centos4.4 kernel * Download liblustre patch which rectifies a debug problem, a statfs problem, the `SYSIO_INTERFACE_NAME` inconsistency with `mount()`, and a linking problem * Download sysio statvfs patch which adds `statvfs()` to liblustre to allow `df`'s to work * Download lustrefuse filesystem application which is the FUSE application file system
* un-tar-compress FUSE
tar zxf fuse-2.6.3.tar.gz
* apply the patch to allow to build on Centos4.4
patch -d fuse-2.6.3 -p1 < fuse-2.6.3-rhel-build.patch
* build and install FUSE
cd fuse-2.6.3 ./configure make sudo -s make install modprobe fuse
* you should have fuse built, installed and the kernel module up and running * Now make sure users can mount file systems
chmod guo+wr /dev/fuse
* un-tar-compress Lustre
tar zxf lustre-1.5.97.tar.gz
* apply the patches
patch -d lustre-1.5.97 -p1 < liblustre.diff patch -d lustre-1.5.97 -p1 < sysio-statvfs.diff
* build liblustre
setenv CFLAGS '-DSYSIO_LABEL_NAMES="fuse_" -D_FILE_OFFSET_BITS=64 -D_HAVE_STATVFS' cd lustre-1.5.97 ./configure --enable-liblustre make sudo -s make install
remember you should be using `tcsh`. Maybe you don't need the `make install`, but hey everything goes in `/usr` right? You wanted all of lustre installed right?
* ok, you should now have liblustre in `/usr/lib`
ls -l /usr/lib/liblustre*
Now this is the bit we have all been waiting for :)
* Build lustrefuse
cd lustrefuse-0.1 ./configure make sudo -s make install
* and now mount lustre
mkdir ~/lustre setenv LIBLUSTRE_MOUNT_TARGET your.lustre.server:/your.lustre.fs /usr/local/bin/lustrefuse ~/lustre cd ~/lustre ls df -h
* oh oh oh... there it is :) * bonnie++ and iozone happily run... be it slowely :)
Things to be done or cleaned up
* performance, performance, performance * lustrefuse should probably cache data so that larger chunks are presented to liblustre * threading? * ClusterFS to work on liblustre for performance? * get patches listed into fuse and lustre * add support for the entire fuse api * um... did I mention performance?
And the real reason for this? The MacOSX port.
I notice while hacking around inside Lustre that their looks to be Darwin ports of everything... but the code isn't provided and liblustre doesn't build. It would be really good if CFS got liblustre to build on MacOSX which would allow mac's to mount lustre file systems.