Architecture - Request Redirection

Note: The content on this page reflects the state of design of a Lustre feature at a particular point in time and may contain outdated information.

Definitions

Target OST

an OST which file data were initially written to, it receives data access requests from clients, manages locks, maintains information about data cached by clients and dedicated servers, maintains persistent and in-code redirect information.

Client

an application which accesses data from target OST

Migration

moving file data from one set of OSTs to another

Collaborative cache

read only cache distributed over clients or dedicated cache servers. Target OSTs maintain in-core information about clients and cache servers and data they are caching and redirect read accesses to appropriate cache server.

Flash cache

write only cache. There are dedicated flash cache servers. Target OSTs maintain (persistent?) information about data cached by flash cache servers and redirect write accesses to them.

Summary

Request Redirection is a mechanism which allows target OST to redirect client requests to other servers. Target OST may decide to redirect a request in hope to improve system throughput or may have to redirect in case when it does not store requested data anymore.

Requirements

Universality

clients get redirected to instances of data created via different ways with the same request redirection mechanism

Flexibility

the request redirection mechanism can allow clients to specify preferences (for example, "do not redirect me", or "let me choose myself")

Extensibility

adding new ways to create instances of data should require no or minimal changes to request redirection mechanism

Availability

requested data are accessible either at servers to which a client is redirected or at target OST

Centralization

target OST keeps all the information about possible redirections, all lock requests get sent to it, it does locking and optionally can redirect a client to another server where data access will happen without further locking

Modes

redirection information can be stored persistently when it has to survive reboots (for example in case of migration) or it can be maintained in memory only (for example in case of collaborative cache)

API

request redirection mechanism provides means for clients to send/receive redirection information to/from a target OST, means for target OST to store that information

Multiplexing

there may be several instances of the same data. Request redirection mechanism should be able to deal with that. For example, in case of collaborative cache target OST has to be able to find clients which are caching requested data and to choose where to redirect.

Use Cases

id	quality attribute	summary
collaborative cache populating	performance,scalability	client holds a read lock on data extent, sends read request to target OST
collaborative cache redirection	performance,scalability	client sends read lock request to target OST
flash cache redirection	performance	client sends write lock request, there is flash server in the filesystem
client reads or writes migrated data	availability	client sends lock request to target OST for data extent, data are not on target OST due to migration
data server crash	availability	data server crashes and gets up
update persistent redirect information	availability	migration is in progress: data extent

collaborative cache populating

Scenario:		a client sends read request to target OST, there are no other caches for the data, the client is running OST service
Business Goals:		populate collaborative cache
Relevant QA's:		performance, scalability
details	Stimulus:	client read request
	Stimulus source:	client
	Environment:	the requested data are not cached by anybody
	Artifact:	requested data
	Response:	target OST reads requested data and sends them to client, if the client node has OST service then the target OST makes a record (in-core) that the requested data extent is cached by this client node
	Response measure:	target OST knows that certain data are cached by certain client node
Questions:
Issues:		Client node may not desire to participate in collaborative cache, this can be controlled with preferences

collaborative cache redirection

Scenario:		a client sends read lock request to target OST, the requested data extent is cached by another client
Business Goals:		offload target OST by redirecting read request to hopefully less loaded node
Relevant QA's:		performance, scalability
details	Stimulus:	client read lock request
	Stimulus source:	client
	Environment:	the requested data extent is cached by another client
	Artifact:	requested data
	Response:	target OST grants lock, checks its records and sees that the data are available via collaborative cache. The request is serviced in accordance with its preferences: list of nodes caching the data range can be returned, or client node in local network can be choosen, etc
	Response measure:	lock is granted, the client knows where the data can be fetched
Questions:
Issues:

flash cache redirection

Scenario:		a client sends write lock request to target OST, there is a flash cache in the filesystem
Business Goals:
Relevant QA's:		performance
details	Stimulus:	client write lock request
	Stimulus source:	client
	Environment:	flash cache is capable to perform this write
	Artifact:	write lock
	Response:	target OST grants lock (all other locks are revoked, caches are released), checks its records to see if the data were already written to flash cache server, selects flash cache server which is able to do this write, redirect the client to selected flash cache server
	Response measure:	the client has write lock and knows where to sent write request to
Questions:		the client has to send a notification to target OST when flash server completes the write, so that target OST could make appropriate redirect record. Only after that the client releases the write lock.
Issues:		How is it guaranteed that the flash server will complete the write?

client accesses a file which migrates

Scenario:		client accesses a file which is migrating to other OST and accessed extent of data is not available on target OST
Business Goals:		allow migration and client access to work simultaneously
Relevant QA's:		Availability
details	Stimulus:	Clients needs to access data to do its job
	Stimulus source:	Client
	Environment:	Data requested by the client were migrated to other data server or to several servers
	Artifact:	lock request
	Response:	target OST grants the lock, checks its redirect records and sees that the data are already on another server, redirects the client to that server
	Response measure:	lock is granted to the client, client knows where it can fetch data from, client does not have to wait until migration completes, migration continues
Questions:		What does target OST do if requested data extent migrated to several servers? It can return either array of redirections or redirection for first part of extent only
Issues:

data server crash

Scenario:		a data server crashes while a migration agent copies a file hosted by the data server
Business Goals:		incorrect redirection is not allowed
Relevant QA's:		availabiliy
details	Stimulus:	OST crash
	Stimulus source:	power failure
	Environment:	at time of crash a migration agent worked with data hosted on the crashed data server
	Artifact:	data server
	Response:	when the data server is up again, none of RID update requests from the agent are lost
	Response measure:
Questions:		is there anything to do about recovering? I guess no, if agents follow simple rules interacting with a data server.
Issues:		Migration agent is responsible for resending RID update requests which the data server did not complete before crash

agent sends a RID update request to data server

Scenario:		agent sends to data server a request to update the RID
Business Goals:		keep data server aware of real data location
Relevant QA's:		availabiliy
details	Stimulus:	Agent copied data somewhere
	Stimulus source:	Agent
	Environment:	Agent copied data to target data seerver, so source data server's RID has to be updated
	Artifact:	RID of data server
	Response:	data server adds new record to its RID and sends to agent completion notification
	Response measure:
Questions:
Issues:		Migration agent has to send RID update request after copied data are written to disk on target data server

Questions

1. Is there need for request redirection mechanism to be involved into filesystem replication? Hopefully not, because of significant overhead of RID maintainence.

2. in case of replication it may happen that data server can either serve a request locally or redirect the request somewhere else. Who is to make a choice?

References

bug 14174

Simple Space Balance Migration