Architecture - Fileset

Summary
A user application (or Lustre internal features) may want to perform an action on a very large set of files. Such actions might include migration to slower storage, purging of old files, or replication to a proxy server. A fileset is an efficient representation of these file identifiers (fids).

The definition of any particular fileset is left to an external agent; no search features will be included in Lustre itself (excluding Least Recently Used files, which is probably only efficiently tracked within Lustre). Typically searches for files with particular metadata characteristics will be done a database that mirrors the Lustre file tree via a ChangeLog. The files matching these criteria will be added to a fileset via a Lustre fileset API.

Filesets will generally come in two flavors: arbitrary collections of files, or a full file tree. See Enumeration below.

Definitions

 * Fileset : an arbitrary subset of files from within a single filesystem's namespace.
 * Consumer : an entity acting on the contents of a fileset
 * Internal consumer : a Lustre internal feature using a fileset (e.g. fileset client mount, maybe replicator, migrator)
 * External consumer : an entity external to Lustre using a fileset. This may be limited to a user of a fileset client mount, and no access to any other representation of a fileset is needed. see Client Access below.
 * type : fileset type, see Enumeration below

Use Cases

 * compliance


 * workset


 * backup

Dynamic
Search results may be returned slowly, or new files that meet the search criteria may be added to the filesystem. In those cases, it should be possible to add (or remove) items to an existing fileset. The fileset should in turn notify consumers of the fileset. Alternately, some filesets may be defined to be static.

Persistence
The workset case implies a fileset must be persistent across server / client reboots.

Specification
It may be desirable for a remote site to specify a fileset that should be locally proxied (i.e. pull instead of push). A fileset name is probably useful for this. (e.g. a client requests mirror fileset 'bunnyrabbit' on local proxy servers)

Coherence
Files referenced in the fileset must be coherent with the original file. E.g. if a file referenced by a fileset is moved, the fileset should reflect the new file location. If a file in a fileset is deleted, the file should disappear from the fileset. Maybe this can be achieved by having the fileset take appropriate locks on the original files.

Coherence requirements: - unlink - rename - move to a new directory - file metadata (access time, perms, owner, etc.)

Note that if changing the above would cause a file to no longer meet the original search criteria that generated that fileset, it is up to the search generator to (eventually) remove it from the fileset. There are two exception to this rule, where the file should be removed from the fileset automatically:
 * 1. unlink
 * 2. move of a file included by virtue of its location in a file tree to a location outside of that tree (see Enumeration below)

Fileset as Object
Depending on the intended use, some filesets may be represented more efficiently than others, or may require different descriptors or methods. Implementing filesets as objects with variable attributes and methods may provide broad but efficient coverage of the range of uses. For example, one common type of fileset may be "a user's home directory", which could be efficiently represented as a single directory fid.

Hashing
When performing an action on large filesets or large numbers of filesets, we must be able to distribute load across multiple servers to insure performant operation. This is true for internal consumers, but perhaps this function should be offloaded to a distributed application for external consumers.

For example, 10,000 filesets are to be replicated independently. A changelog per fileset may not scale well, and instead we may need a scalable algorithm to find the results for each fileset from a global changelog.

ChangeLog
It may be useful to have a per-fileset changelog maintained for audit or replication purposes. A fileset-specific changelog could be used to provide migration/replication-related events specific to the fileset to migration agents. The agents would then use this information e.g. to abort / commence copying a file.

However, maintaining a per-fileset changelog may not scale. At some point, it make make more sense to process a common global changelog.

Multiple Membership
A file may be part of multiple filesets. A type 2 fileset may implicitly include other type 2 filesets. Operations on a file should affect all filesets it belongs to, and vice-versa.

Fileset API
The user API for filesets should include the following functionality:
 * Start a new fileset
 * Add items to a fileset
 * Remove items from a fileset
 * Delete a fileset
 * Initiate activity of an internal consumer (e.g. migrate fileset bunny from poolA to poolB)
 * Provide client access to a fileset (see Client Access below)

Enumeration
Fileset enumeration should be handled in two ways:
 * Type 1. An explicit enumeration of files or directories. Files within directories are not included in the fileset unless explicitly listed as well.
 * Type 2. Inclusive file trees. All files / subdirectories below enumerated directories are included in the fileset.

We should have provision for using both types of filesets. In fact, with some per-entry flags, we can define "mixed" filesets including both of the above (each entry in a fileset may be type 1 (flat=single file) or type 2 (tree). Perhaps a 3rd type; "not_included" would be a useful definition as well, to specifically exclude a particular subdirectory from a type 2 fileset.

Storage
Permanent fileset definitions would probably be stored on the MDT (as opposed to the MGS) for scalability and namespace-related locking.

Maintenance
The UI for maintaining filesets might reasonably be run through lfs similar to pools:
 * 1) lfs fileset_new  Define a new fileset
 * 2) lfs fileset_add    ...  Add the named files to the fileset; define type 1 or type 2
 * 3) lfs fileset_remove   Remove the named file from the fileset
 * 4) lfs fileset_destroy Remove the definition of the fileset

Client Access
For arbitrary user access to the files in a fileset, a mechanism like mount(8) seems like it would provide a clear, simple way to retrieve a fileset. (Command format might be "mount -t lustre mgs://fsname/fileset mntpt")

For type 1 filesets, a hierarchical namespace defined by the files and directories in the fileset would be constructed locally. Directories would all be read/execute-only; a client cannot add new entries into the fileset by creating files in the fileset hierarchy. Regular files would keep their normal access permissions.

For type 2 filesets, the mount point would act exactly like a subtree of the full lustre fs.