Architecture - Fileset - Revision history

Docadmin at 21:10, 22 January 2010

2010-01-22T21:10:11Z

← Older revision		Revision as of 14:10, 22 January 2010
Line 1:		Line 1:
	'''''Note:''''' The content on this page reflects the state of design of a Lustre feature at a particular point in time and may contain outdated information.		'''''Note:''''' ''The content on this page reflects the state of design of a Lustre feature at a particular point in time and may contain outdated information.''

	== Summary ==		== Summary ==

Docadmin: Protected "Architecture - Fileset" ([edit=sysop] (indefinite) [move=sysop] (indefinite))

2010-01-21T19:42:59Z

Protected "Architecture - Fileset" ([edit=sysop] (indefinite) [move=sysop] (indefinite))

← Older revision	Revision as of 12:42, 21 January 2010
(No difference)

Docadmin: /* Enumeration */

2010-01-19T18:21:21Z

Enumeration

← Older revision		Revision as of 11:21, 19 January 2010
Line 187:		Line 187:
	=== Enumeration ===		=== Enumeration ===
	Fileset enumeration should be handled in two ways:		Fileset enumeration should be handled in two ways:
	;Type 1. An explicit enumeration of files or directories. Files within directories are ''not'' included in the fileset unless explicitly listed as well.		* Type 1. An explicit enumeration of files or directories. Files within directories are ''not'' included in the fileset unless explicitly listed as well.
	;Type 2. Inclusive file trees. ''All'' files / subdirectories below enumerated directories are included in the fileset.		* Type 2. Inclusive file trees. ''All'' files / subdirectories below enumerated directories are included in the fileset.

	We should have provision for using both types of filesets. In fact, with some per-entry flags, we can define "mixed" filesets including both of the above (each entry in a fileset may be type 1 (flat=single file) or type 2 (tree). Perhaps a 3rd type; "not_included" would be a useful definition as well, to specifically exclude a particular subdirectory from a type 2 fileset.		We should have provision for using both types of filesets. In fact, with some per-entry flags, we can define "mixed" filesets including both of the above (each entry in a fileset may be type 1 (flat=single file) or type 2 (tree). Perhaps a 3rd type; "not_included" would be a useful definition as well, to specifically exclude a particular subdirectory from a type 2 fileset.



	=== Storage ===		=== Storage ===

Docadmin: /* Summary */

2010-01-18T23:57:41Z

Summary

← Older revision		Revision as of 16:57, 18 January 2010
Line 1:		Line 1:
			'''''Note:''''' The content on this page reflects the state of design of a Lustre feature at a particular point in time and may contain outdated information.

	== Summary ==		== Summary ==

Docadmin: /* Maintenance */

2010-01-14T22:24:12Z

Maintenance

← Older revision		Revision as of 15:24, 14 January 2010
Line 198:		Line 198:
	==== Maintenance ====		==== Maintenance ====
	The UI for maintaining filesets might reasonably be run through lfs similar to		The UI for maintaining filesets might reasonably be run through lfs similar to
	[~~http://arch.lustre.org/index.php?title=Pools_of_targets~~ pools]:		[[Architecture - Pools of targets\|pools]]:
	# lfs fileset_new <fileset name> Define a new fileset		# lfs fileset_new <fileset name> Define a new fileset
	# lfs fileset_add <fileset name> <options> <filename1> <filename2> ... Add the named files to the fileset; define type 1 or type 2		# lfs fileset_add <fileset name> <options> <filename1> <filename2> ... Add the named files to the fileset; define type 1 or type 2

Docadmin: /* References */

2010-01-14T22:23:21Z

References

← Older revision		Revision as of 15:23, 14 January 2010
Line 213:		Line 213:
	== References ==		== References ==
	; [https://bugzilla.lustre.org/show_bug.cgi?id=14168 bug 14168]		; [https://bugzilla.lustre.org/show_bug.cgi?id=14168 bug 14168]
	; ~~[http://arch.lustre.org/index.php?title=Server_Changelogs server changelogs]~~		; [[Architecture - Changelogs\|server changelogs]]
	[[~~Category:~~Architecture\|~~Fileset]]~~
	~~[[Category:Team_Rabbit~~]]

Docadmin: /* Fileset API */

2010-01-14T22:22:08Z

Fileset API

← Older revision		Revision as of 15:22, 14 January 2010
Line 174:		Line 174:
	=== Fileset API ===		=== Fileset API ===
	The user API for filesets should include the following functionality:		The user API for filesets should include the following functionality:
	- Start a new fileset		* Start a new fileset
	- Add items to a fileset		* Add items to a fileset
	- Remove items from a fileset		* Remove items from a fileset
	- Delete a fileset		* Delete a fileset
	- Initiate activity of an internal consumer (e.g. migrate fileset bunny from poolA to poolB)		* Initiate activity of an internal consumer (e.g. migrate fileset bunny from poolA to poolB)
	- Provide client access to a fileset (see Client Access below)		* Provide client access to a fileset (see Client Access below)

	== Implementation Notes ==		== Implementation Notes ==

Docadmin: /* ChangeLog */

2010-01-14T22:21:37Z

ChangeLog

← Older revision		Revision as of 15:21, 14 January 2010
Line 165:		Line 165:

	=== ChangeLog ===		=== ChangeLog ===
	It may be useful to have a per-fileset [~~http://arch.lustre.org/index.php?title=Server_Changelogs~~ changelog] maintained for audit or replication purposes. A fileset-specific changelog could be used to provide migration/replication-related events specific to the fileset to migration agents. The agents would then use this information e.g. to abort / commence copying a file.		It may be useful to have a per-fileset [[Architecture - Changelogs\|changelog]] maintained for audit or replication purposes. A fileset-specific changelog could be used to provide migration/replication-related events specific to the fileset to migration agents. The agents would then use this information e.g. to abort / commence copying a file.

	However, maintaining a per-fileset changelog may not scale. At some point, it make make more sense to process a common global changelog.		However, maintaining a per-fileset changelog may not scale. At some point, it make make more sense to process a common global changelog.

Docadmin: /* Use Cases */

2010-01-14T22:20:53Z

Use Cases

← Older revision		Revision as of 15:20, 14 January 2010
Line 62:		Line 62:
	\|'''Stimulus source:'''\|\| Compliance policy dictates removal of old files		\|'''Stimulus source:'''\|\| Compliance policy dictates removal of old files
	\|-align="left"		\|-align="left"
	\|'''Environment:'''\|\| Database has recent FS information (from watching a [~~http://arch.lustre.org/index.php?title=Server_Changelogs~~ ChangeLog])		\|'''Environment:'''\|\| Database has recent FS information (from watching a [[Architecture - Changelogs\|ChangeLog]])
	\|-align="left"		\|-align="left"
	\|'''Artifact:'''\|\| Fileset, type 1		\|'''Artifact:'''\|\| Fileset, type 1

Docadmin: /* Summary */

2010-01-14T22:20:02Z

Summary

← Older revision		Revision as of 15:20, 14 January 2010
Line 3:		Line 3:
	A user application (or Lustre internal features) may want to perform an action on a very large set of files. Such actions might include migration to slower storage, purging of old files, or replication to a proxy server. A '''fileset''' is an efficient representation of these file identifiers (fids).		A user application (or Lustre internal features) may want to perform an action on a very large set of files. Such actions might include migration to slower storage, purging of old files, or replication to a proxy server. A '''fileset''' is an efficient representation of these file identifiers (fids).

	The definition of any particular fileset is left to an external agent; no search features will be included in Lustre itself (excluding Least Recently Used files, which is probably only efficiently tracked within Lustre). Typically searches for files with particular metadata characteristics will be done a database that mirrors the Lustre file tree via a [~~http://arch.lustre.org/index.php?title=Server_Changelogs~~ ChangeLog]. The files matching these criteria will be added to a fileset via a Lustre fileset API.		The definition of any particular fileset is left to an external agent; no search features will be included in Lustre itself (excluding Least Recently Used files, which is probably only efficiently tracked within Lustre). Typically searches for files with particular metadata characteristics will be done a database that mirrors the Lustre file tree via a [[Architecture - Changelogs\|ChangeLog]]. The files matching these criteria will be added to a fileset via a Lustre fileset API.

	Filesets will generally come in two flavors: arbitrary collections of files, or a full file tree. See Enumeration below.		Filesets will generally come in two flavors: arbitrary collections of files, or a full file tree. See Enumeration below.