Architecture - Space Manager

Note: The content on this page reflects the state of design of a Lustre feature at a particular point in time and may contain outdated information.

Purpose
This page describes uses cases for HSM space manager

Definitions

 * space manager: Lustre component which manages the filesystem free space: punching filesystem objects and moving them to the archiving system using migration policy. It is responsible for checking the OST, OST pool and file system usage.
 * purge: Action on a file to free allocated data without changing file attributes (size, mtime, ...). This action keeps first and last file blocks on disks.
 * trigger: Process or event in the file system which causes some purges.
 * migration policy: Rules describing the filters used to select migration candidates. They could be based on access time or object size by example.
 * scanner: Entity responsible for fastly building a candidate file list using policies that will be used for migration and purge.
 * waterwarks: High watermark and low watermark are 2 space usage values. When the high watermark is reached, a purge event is raised to free enough objects until the space usage does not exceed the low watermark. Watermarks could be defined for OST, OST pool or filesystem.

pre-migration

 * 1) Periodically, the space manager asks the scanner for candidate objects matching the policy.
 * 2) For each candidate, it requests the coordinator to copy out this object (See corresponding scenarios).
 * 3) When the migration is finished or aborted, the space manager is warned.
 * 4) If the migration successed, the file is tagged being up-to-date in the archiving system.

purge

 * 1) The space manager receives a purge trigger.
 * 2) It uses the scanner to build a candidate list to purge.
 * 3) It purges as much as needed file system objects, depending on policy, to fall short of the limit. Depending on the object type, the purge requests are sent to MDT or OST.
 * 4) The purge is finished when the requested threshold is reached

Implementation Constraints

 * A filesystem object can be purged only if an up-to-date copy is available on another storage (another OST pools or an external storage).
 * As space over high-watermark is wasted, the HW must be very high, so purge action must be very fast to avoid reaching file system getting full.