Architecture - User Level Access

Note: The content on this page reflects the state of design of a Lustre feature at a particular point in time and may contain outdated information.

Summary

The LNET userspace API driver exports the LNET API to userspace so that userspace lustre servers and clients can use all existing kernel LNET infrastructure.

Requirements

LNET API

Userspace processes can use the LNET API transparently to communicate with other processes of all flavours.

Multi-user

LNET users are distinguished by LNET PID. This PID may be shared by several O/S processes which may in turn be multi-threaded. Processes sharing the same LNET PID may share ME, MD and EQ handles. Kernel threads all share the same PID.

PID assignment

PID is requested in LNetNIInit(). If this is LNET_PID_ANY, a new unique PID is created and given the real userid of the caller. If the requested PID exists already, the caller may be permitted to join it. If it does not exist, the caller may be permitted to create it.

Privilege

Privileged users may create or join any PID. Unprivileged users must set the "userflag" when requesting a PID and my only join an existing PID if the caller's real userid matches the PID's userid.

Isolation

Replies, acks and events for a given PID are isolated from other PIDs, including future incarnations of the same PID.

Cleanup

A PID is destroyed when all its processes have finalized or terminated. At this time, all outstanding communications are completed with failure in finite time and all associated state is freed.

Parameter checking

NO userspace LNET API parameters are trusted. Checking may occur at any time - e.g. invalid buffer addresses may be discovered when a communication completes rather than when it was initiated.

Memory

User pages bound to an MD are locked down until the MD has been destroyed.

EQ Callbacks

EQ callbacks in userspace are not supported. Userspace processes may dedicate one or more threads to process events eagerly.

Thread safety

All exported LNET API calls are thread-safe.

Zero-copy

Communications to/from userspace buffers are zero-copy where the kernel LND supports it.

Resource Limits

LNET kernel module parameters allow resource limits (e.g. extant PIDs, MEs, MDs, concurrent GETs and PUTs) to be set for two classes of user: privileged and unprivileged. These limits include global maxima to limit the total resource that either class of process may use in aggregate. Either class may be set to infinity or zero. The normal "getrlimit" resource limits will apply to pages associated with MDs.