Architecture - Recovery Failures

Note: The content on this page reflects the state of design of a Lustre feature at a particular point in time and may contain outdated information.

Recovery Terminology

Replay	A client resends a transaction that had previously been executed and for which a reply was sent, but was lost in a server crash, because it was not committed.
Collecting clients	A server waiting for a short period to allow more clients to connect during recovery.
Version controlled replay	clients connecting at any time and are allowed to replay updates to anything that did not change since the update was made before the server failure.
Version based cache revalidation	Cached client data may be retained if transactions it depended on were replayed.
Commit on sharing	before new or updated metadata is shared with a node that is not the updater or creator, the metadata is committed.
Commit on sharing sequences	Attributes or data of objects that belong to one and the same inode sequence are committed before nodes other than the updator or creator can access them.
Eviction	A client flushing its cache because the server indicates it does not have the cached data anymore and possibly cannot re-create it.

Recovery Architectures

A. All clients reconnect:	After a server failure client collecting. If all clients reconnect they replay all missing transactions. All data lost in the server failure is restored on the server and in client caches. (Currently implemented by Lustre on Linux)
B. Version recovery:	After a server failure first method A is tried. If it fails all connecting clients are subject to are subject to collecting clients, version controlled replay and version based cache validation. Nodes that see mismatching versions for cached items are evicted.
C. Version checking (aka relaxed recovery requirement)	This is version based cache revalidation, without replay. Not further discussed.

This can be combined with other mechanisms:

• Commit on sharing or
• Commit on sharing sequences - just mentioned for historical relevance, not further documented or discussed.

Use cases

Summary

id	quality attribute	summary
create_in_one_dir	performance, availability	many clients create files in one directory.
race_crash_and_share	availability	a client is making a lot of updates. it starts computing, the server crashes and another client wants to access these updates.

Sharing among clients

Scenario:		Clients sharing recently modified/created metadata
Business Goals:		Maximize availability
Relevant QA's:		Performance & Availability
details	Stimulus:	One client creates or modifies metadata, another one accesses it
	Stimulus source:	two or more client systems, one making updates.
	Environment:	server failures
	Artifact:	recoverable state in client caches
	Response:	see below
	Response measure:	How many clients recover?
Questions:		What does customer want?
Issues:		None.

responses

A	unless both clients reconnect both will be evicted.
B	if the client making the updates/creates reconnects first both clients will recover, even if there is a delayed reconnect for the second client.

Individual client performance

Scenario:		A single client make a burst of file system updates or many clients making bursts file system updates without sharing metadata. Other clients (accessing clients) access some of the updates.
Business Goals:		Maximize performance & availability
Relevant QA's:		Performance & Availability
details	Stimulus:	One client creates or modifies metadata, another one accesses it
	Stimulus source:	one or more client systems making updates.
	Environment:	server failures
	Artifact:	recoverable state in the cluster
	Response:	throughput of cluster during the updates and count of recoverable clients
	Response measure:	Performance
Questions:		What does customer want?
Issues:		None.

Responses

A	Fast performance. All clients must be present to avoid evicting all clients.
B	Fast performance. If the updates are independent of each other all updating clients can reconnect at any time and perform a replay. An accessing client will recover if and only if it reconnects after all updating clients preceding its access have recovered. On catamount this leads to a high likelihood of recovery failures.
B with commit on sharing	Every accessing client will cause a cache flush, possibly degrading performance heavily. Any accessing client reconnectingwill recover.