Architecture - Libcfs

Note: The content on this page reflects the state of design of a Lustre feature at a particular point in time and may contain outdated information.

Summary

Libcfs provides an API comprising fundamental primitives and subsystems - e.g. process management and debugging support - used throughout Lnet, Lustre and associated utilities. This API defines a portable runtime environment implemented consistently on all supported build targets.

Requirements

Runtime Environment

Libcfs provides both process and utility environments. The process environment provides runtime support for instances of Lnet and Lustre (client and server) running both in kernel and user space. The utility environment provides runtime support for userspace application programs used to configure, control and test Lnet and Lustre.

The specific build target is exposed to libcfs users via macro definitions which allow conditional compilation and build-time assertions. There are separate macro definitions at each level - e.g. utility v. process at the top level, kernel v. userspace within process etc. The main targets are listed below...

Process Environment

Kernel

Linux is the only fully supported target at this time. Code exists for Darwin and Windows which may be resurrected as required. A Solaris target will be required for native Solaris lustre clients.

Both hard and soft interrupt contexts are supported within LNDs (NB these may only call back into Lnet in thread context). Sift interrupt context is supported for timer callbacks. All other code must run in thread context.

Userspace

Userspace targets are broadly posix runtimes. Single threaded runtimes (e.g. catamount) elide all locking primitives. Multi-threaded runtimes use posix threads. Signal safety is not defined.

Utility Environment

APIs available in the utility environment are a subset of those available in the process environment for the multi-threaded userspace target.

C dialect

Libcfs (and Lnet and Lustre) requires C99 syntax for structure member initialization (e.g. {.member = val}).

APIs

Libcfs defines abstract types and APIs to operate on them. These can be grouped broadly as follows...

• Scalar types with explicit precision (e.g. __u64)
  • Byte swapping
  • 64 bit arithmetic
• Time
  • Wallclock time
  • Ticks
  • Timers
• Debugging
  • CDEBUG etc
  • stack dumping etc
• Lists
• NID and process ID abstractions
• Process Management
  • Process creation and termination
  • Process attributes
  • Locking
  • Wait queues and/or condition variables
• Memory management
  • Allocate/Free
  • Leak detection
  • Mapping
• etc (list incomplete)...

Header Files

All libcfs declarations for all build targets and runtimes must be included via a single public header file e.g...

 #include <libcfs.h>

Target-specific header files as determined by preprocessor directives are then included from this single public header.

The graph of header file dependencies within Libcfs must be a simple tree.

Compile-time assertions must be provided to ensure no libcfs header files apart from the single public header are included directly.

The build system must define the macro "__LIBCFS_UTIL__" when building utilities so that the correct environment is provided.

Conversion Process

It is the intention that all parts of Lnet and Lustre which are sufficiently generic should use Libcfs APIs, however this has to be a gradual process so that conversion does not block ongoing Lustre development - i.e. it must be possible for individual subsystems to be converted independently.

The biggest obstacle is Liblustre. It currently duplicates much functionality that should be provided by Libcfs however gradual conversion within Liblustre is impossible - name clashes break the userspace build once libcfs headers are included.

Common Lnet code will benefit from conversion to the Libcfs process management and synchronisation APIs since that will provide support for the multi-threaded userspace process environment. This in turn will provide better concurrency and therefore better CPU utilisation on SMPs.

The extent to which LNDs exploit Libcfs APIs depends utterly on the LND. The most generic LNDs (e.g. userspace socklnd) should use Libcfs APIs almost exclusively. The most specific LNDs (e.g. viblnd) may hardly use Libcfs at all (e.g. only debug and allocation).