Architecture - Space Manager - Revision history

Docadmin at 21:24, 22 January 2010

2010-01-22T21:24:27Z

← Older revision		Revision as of 14:24, 22 January 2010
Line 1:		Line 1:
	'''''Note:''''' The content on this page reflects the state of design of a Lustre feature at a particular point in time and may contain outdated information.		'''''Note:''''' ''The content on this page reflects the state of design of a Lustre feature at a particular point in time and may contain outdated information.''

	== Purpose ==		== Purpose ==

Docadmin: Protected "Architecture - Space Manager" ([edit=sysop] (indefinite) [move=sysop] (indefinite))

2010-01-22T19:18:54Z

Protected "Architecture - Space Manager" ([edit=sysop] (indefinite) [move=sysop] (indefinite))

← Older revision	Revision as of 12:18, 22 January 2010
(No difference)

Docadmin: /* Uses Cases */

2010-01-22T19:18:26Z

Uses Cases

← Older revision		Revision as of 12:18, 22 January 2010
Line 18:		Line 18:
	!id !! quality attribute !! summary		!id !! quality attribute !! summary
	\|-		\|-
	\|pre-migration \|\| performance \|\| The space manager is responsible for copying-out candidate files to the HSM when possible (see [[~~HSM_Migration~~]]).		\|pre-migration \|\| performance \|\| The space manager is responsible for copying-out candidate files to the HSM when possible (see [[Architecture - HSM Migration\|HSM Migration]]).
	\|-		\|-
	\|purge \|\| availability \|\| The space manager is requested to purge candidate files when a policy limit is reached.		\|purge \|\| availability \|\| The space manager is requested to purge candidate files when a policy limit is reached.

Docadmin: /* Purpose */

2010-01-19T00:21:13Z

Purpose

← Older revision		Revision as of 17:21, 18 January 2010
Line 1:		Line 1:
			'''''Note:''''' The content on this page reflects the state of design of a Lustre feature at a particular point in time and may contain outdated information.

	== Purpose ==		== Purpose ==
	This page describes uses cases for HSM space manager		This page describes uses cases for HSM space manager

Docadmin: /* Space management interactions */

2010-01-15T00:04:24Z

Space management interactions

← Older revision		Revision as of 17:04, 14 January 2010
Line 104:		Line 104:

	[[Image:space_manager.png\|Space management schema]]		[[Image:space_manager.png\|Space management schema]]

	~~[[Category:Architecture\|Space Manager]]~~

Docadmin: moved Space Manager to Architecture - Space Manager

2010-01-14T21:49:55Z

moved Space Manager to Architecture - Space Manager

← Older revision	Revision as of 14:49, 14 January 2010
(No difference)

Docadmin: 1 revision

2010-01-14T21:01:09Z

1 revision

New page

== Purpose ==
This page describes uses cases for HSM space manager

== Definitions ==
;space manager: Lustre component which manages the filesystem free space: punching filesystem objects and moving them to the archiving system using migration policy. It is responsible for checking the OST, OST pool and file system usage.
;purge: Action on a file to free allocated data without changing file attributes (size, mtime, ...). This action keeps first and last file blocks on disks.
;trigger: Process or event in the file system which causes some purges.
;migration policy: Rules describing the filters used to select migration candidates. They could be based on access time or object size by example.
;scanner: Entity responsible for fastly building a candidate file list using policies that will be used for migration and purge.
;waterwarks: High watermark and low watermark are 2 space usage values. When the high watermark is reached, a purge event is raised to free enough objects until the space usage does not exceed the low watermark. Watermarks could be defined for OST, OST pool or filesystem.

== Uses Cases ==

{| border=1 cellspacing=0 cellpadding=3
|-
!id !! quality attribute !! summary
|-
|pre-migration || performance || The space manager is responsible for copying-out candidate files to the HSM when possible (see [[HSM_Migration]]).
|-
|purge || availability || The space manager is requested to purge candidate files when a policy limit is reached.
|-
|}

== Feature ==

=== pre-migration ===

{|border=1 cellspacing="0" cellpading="4"
|-align="left"
|colspan=2|'''Scenario:''' || The space manager copies out candidate objects to the archiving system to be able to quickly purge them when needed.
|-align="left"
|colspan=2|'''Business Goals:''' || File system disponibility.
|-align="left"
|colspan=2|'''Relevant QA's:'''|| Performance.
|-align="left"
|rowspan="6" writing-mode="vertical"|'''details'''
| '''Stimulus:'''|| Periodic event.
|-align="left"
|'''Stimulus source:'''|| N/A.
|-align="left"
|'''Environment:'''|| Lustre filesystem objects.
|-align="left"
|'''Artifact:'''|| Candidate objects to be copied out.
|-align="left"
|'''Response:'''|| Periodically, the space manager asks for a candidate list and requests the coordinator for migration of those objects. At the end, those objects are tagged being up-to-date in the HSM and ready to be punched if needed.
|-align="left"
|'''Response measure:'''|| Volume of candidate objects ready to be purged increased.
|-align="left"
|colspan=2|'''Questions:'''|| None.
|-align="left"
|colspan=2|'''Issues:'''|| None.
|-
|}

=== purge ===

{|border=1 cellspacing="0" cellpading="4"
|-align="left"
|colspan=2|'''Scenario:''' || A policy limit is reached and the space manager purges enough objects provided by the scanner to satisfy the policy rules.
|-align="left"
|colspan=2|'''Business Goals:''' || Virtually extends the file system capacity using an archiving system.
|-align="left"
|colspan=2|'''Relevant QA's:'''|| Availability.
|-align="left"
|rowspan="6" writing-mode="vertical"|'''details'''
| '''Stimulus:'''|| A trigger is raised.
|-align="left"
|'''Stimulus source:'''|| Space manager or administrator request.
|-align="left"
|'''Environment:'''|| Lustre file system objects.
|-align="left"
|'''Artifact:'''|| Purge candidate objects.
|-align="left"
|'''Response:'''|| When a policy limit is reached inside the filesystem, the space manager is used to punch as many objects as needed to fall short of the limit.
|-align="left"
|'''Response measure:'''|| Objects have been purged.
|-align="left"
|colspan=2|'''Questions:'''|| None.
|-align="left"
|colspan=2|'''Issues:'''|| None.
|-
|}

==Scenarios==

===pre-migration===

#Periodically, the ''space manager'' asks the ''scanner'' for candidate objects matching the policy.
#For each candidate, it requests the ''coordinator'' to copy out this object (See corresponding scenarios).
#When the migration is finished or aborted, the space manager is warned.
#If the migration successed, the file is tagged being up-to-date in the archiving system.

===purge===
#The ''space manager'' receives a purge trigger.
#It uses the ''scanner'' to build a candidate list to purge.
#It purges as much as needed file system objects, depending on policy, to fall short of the limit. Depending on the object type, the purge requests are sent to MDT or OST.
#The purge is finished when the requested threshold is reached

==Implementation Constraints==
*A filesystem object can be purged only if an up-to-date copy is available on another storage (another OST pools or an external storage).
*As space over high-watermark is wasted, the HW must be very high, so purge action must be very fast to avoid reaching file system getting full.

==Space management interactions==

[[Image:space_manager.png|Space management schema]]

[[Category:Architecture|Space Manager]]