Docadmin at 21:16, 22 January 2010

2010-01-22T21:16:37Z

← Older revision		Revision as of 14:16, 22 January 2010
Line 1:		Line 1:
	'''''Note:''''' The content on this page reflects the state of design of a Lustre feature at a particular point in time and may contain outdated information.		'''''Note:''''' ''The content on this page reflects the state of design of a Lustre feature at a particular point in time and may contain outdated information.''

	== Summary ==		== Summary ==

Docadmin: Protected "Architecture - Metadata API" ([edit=sysop] (indefinite) [move=sysop] (indefinite))

2010-01-22T18:05:44Z

Protected "Architecture - Metadata API" ([edit=sysop] (indefinite) [move=sysop] (indefinite))

← Older revision	Revision as of 11:05, 22 January 2010
(No difference)

Docadmin: /* final lookup */

2010-01-19T18:48:43Z

final lookup

← Older revision		Revision as of 11:48, 19 January 2010
Line 377:		Line 377:
	=== final lookup ===		=== final lookup ===

	[[Architecture - New_Metadata_API\|~~New_Metadata_API~~]]		[[Architecture - New_Metadata_API\|New Metadata API]]

Docadmin at 00:01, 19 January 2010

2010-01-19T00:01:53Z

← Older revision		Revision as of 17:01, 18 January 2010
Line 1:		Line 1:
			'''''Note:''''' The content on this page reflects the state of design of a Lustre feature at a particular point in time and may contain outdated information.

	== Summary ==		== Summary ==

Docadmin: /* final lookup */

2010-01-14T23:24:38Z

final lookup

← Older revision		Revision as of 16:24, 14 January 2010
Line 375:		Line 375:
	=== final lookup ===		=== final lookup ===

			[[Architecture - New_Metadata_API\|New_Metadata_API]]
	[[~~Category:~~Architecture\|~~Metadata API]]~~
	[[New_Metadata_API]]

Docadmin: moved Metadata API to Architecture - Metadata API

2010-01-14T21:54:46Z

moved Metadata API to Architecture - Metadata API

← Older revision	Revision as of 14:54, 14 January 2010
(No difference)

Docadmin: 1 revision

2010-01-14T21:01:10Z

1 revision

New page

== Summary ==

Metadata API is a set of methods llite layer uses to access and manipulate metadata. This page starts from describing existing API. As time goes MdAPI should evolve to become clear and sufficiently portable.

== Definitions ==

; '''Lock''': LDLM lock
; '''resolving''': a triple of parent/name pointing to child

== Requirements ==

# clear and sufficient (no tricks needed to use it)
# portable

== Cache Model ==

Current implementation supposes four types of cache:
# resolving (dentry in linux)
# attributes (inode in linux)
# directory (pagecache to store directory entries in linux)
# open files

All of them expect open files are maintained by the kernel with help from filesystem driver (llite module of lustre).

All these caches are protected by DLM locks. A special type of lock - inode bitlock - was introduced. Each lock may have 3 bits:
# LOOKUP - protects all resolvings to given object (all because of hardlinks) + permissions
# UPDATE - protects all metadata attributes except permissions
# OPEN - allows client to cache open state of a file

Depending on actual operation client combines bits to access data. For example,
to do intermediate lookup where we only need to make sure the component still exists and permissions haven't changed, LOOKUP bit is required. Once resolving of intermediate component is cached, client doesn't need to lookup it on server again. And given operations within directory don't need LOOKUP bit, client's resolving is still in cache even if another client create/unlink file in that directory.

== Intents ==

Intents is a mechanism to save RPCs which is the most expensive part of metadata handling due to network latency.

Let's imagine open("file", O_CREAT). Linux VFS breaks this request into series of calls to filesystem driver:
# '''lookup''' to find '''file''' if it exists
# if '''file''' doesn't exist, then '''create''' is called
# then '''open''' is called

Thus, 2 or 3 RPCs could be needed, if we follow usual model. Instead, we send lookup RPC with a special structure called '''intent''' which describes real intent of the operation, MDS does all required work, then '''create''' and '''open''' use results returned to the first operation.

In reality any operation starts with locking, so usually it's ENQUEUE RPC which carries intent.

There are few intents defined currently:

{| cellspacing="0" border="1"
! intent !! purpose !! details
|-
| IT_OPEN
| open file/directory
| may return LOOKUP + OPEN lock
|-
| IT_CREAT
| create file
| used with IT_OPEN, may return LOOKUP + OPEN lock
|-
| IT_READDIR
| readdir
| used to access/cache directories, may return LOOKUP + UPDATE lock
|-
| IT_GETATTR
| lookup/getattr
| used to lookup and get attr (final lookup), may return LOOKUP + UPDATE lock
|-
| IT_LOOKUP
| lookup
| used for intermediate lookups, may return LOOKUP lock
|-
| IT_UNLINK
| unused
| --
|-
| IT_TRUNC
| unused
| --
|-
| IT_GETXATTR
| unused
| --
|}

== Recovery ==

== Use Cases ==

{| cellspacing="0" border="1"
! ID !! Quality Attribute !! Summary
|-
| intermediate lookup
| usability
| lookup of intermediate component of filename
|-
| final lookup
| usability
| lookup of final component of filename
|-
| cached lookup
| performance
| how can user check resolving is still valid
|-
| resolving invalidate
| usability
| how to invalidate cached resolving
|-
| cached negative resolving
| performance
| how can user check name still does not exist
|-
| negative resolving invalidate
| usability
| how can user check name is created now
|-
| mkdir
| usability
| creating of new directory
|-
| rmdir
| usability
| directory removal
|-
| link
| usability
|
|-
| unlink
| usability
|
|-
| unlink open file
| usability, availability
| file should be protected from removal until it's closed
|-
| rename
| usability
|
|-
| stat
| usability
|
|-
| cached stat
| performance
|
|-
| stat invalidate
| usability
|
|-
| readdir
| usability
|
|-
| cached readdir
| performance
|
|-
| readdir invalidate
| usability
|
|-
| open
| usability
|
|-
| cached open
| performance
|
|-
| open invalidate
| usability
|
|-
| close
| usability
|
|-
| open recovery
| availability
| during MDS failover, clients recover opens to provide posix semantics of unlink
|-
| open by fid
| availability
| used by nfsd
|-
| lockless
| usability
| in some cases results can return with no lock
|}

== Existing MdAPI ==

=== Data Structures ===

struct lustre_intent_data {
int it_disposition;
int it_status;
__u64 it_lock_handle;
void *it_data;
int it_lock_mode;
};

struct lookup_intent {
int it_op;
int it_flags;
int it_create_mode;
union {
struct lustre_intent_data lustre;
} d;
};

struct mdc_op_data {
struct ll_fid fid1;
struct ll_fid fid2;
struct ll_fid fid3;
struct ll_fid fid4;
__u64 mod_time;
const char *name;
int namelen;
__u32 create_mode;
__u32 suppgids[2];
void *data;
};

struct lustre_md {
struct mds_body *body;
struct lov_stripe_md *lsm;
#ifdef CONFIG_FS_POSIX_ACL
struct posix_acl *posix_acl;
#endif
};

=== mdc_intent_lock ===

int mdc_intent_lock(struct obd_export *exp, struct mdc_op_data *op_data,
void *lmm, int lmmsize, struct lookup_intent *it,
int lookup_flags, struct ptlrpc_request **reqp,
ldlm_blocking_callback cb_blocking, int extra_lock_flags)

; '''exp''': export for the connection to MDS
; '''op_data''': fids/name/etc - operation args
; '''lmm''': buffer to store striping info
; '''lmmsize''': available space in lmm buffer
; '''it''': intent
; '''lookup_flags''': unused currently
; '''reqp''': where to store address of request (to be used to access server's reply)
; '''cb_blocking''': callback routine, called when correspondent lock is being cacelled - to invalidate cache
; '''extra_flags''': extra flags to be passed LDLM

mdc_intent_lock() is the major entry point for many metadata operations. as it was described above, some metadata operations are implemented with intents. if we didn't have intents, then operation could look like the following:

lock(dir);
lookup(dir, name);
unlock(dir);

with intents (and currently) it looks this way:

lock(dir, intent, reply);
find lookup result in reply;

mdc_intent_lock() is that lock() in pseudo-code.

mdc_intent_lock() is used to lookup name, retrieve attributes, create regular files and open files/directories. it usually returns DLM lock for result object. actual bits of the lock may depend on intent and server's policy. if mdc_intent_lock() returns with a lock, then data in reply is subject to cache.

=== mdc_set_lock_data ===

void mdc_set_lock_data(__u64 *l, void *data)

; '''l''': lock handle
; '''data''': data to be stored in lock's private field

the function is used to associate lock with some cache entity so that at time lock is being cancelled we can find that entity easily

=== mdc_readpage ===

int mdc_readpage(struct obd_export *exp, struct ll_fid *fid, __u64 offset,
struct page *page, struct ptlrpc_request **request)

; '''exp''': export for connection to MDS
; '''fid''': fid of the directory to be read
; '''offset''': where to start reading from
; '''page''': where to store result ('''LINUX-SPECIFIC YET''')
; '''request''': request where newer attributes of the directory can be fetched from

=== mdc_setattr ===

int mdc_setattr(struct obd_export *exp, struct mdc_op_data *op_data,
struct iattr *iattr, void *ea, int ealen, void *ea2, int ea2len,
struct ptlrpc_request **request)

; '''exp''': connection to MDS
; '''op_data''': operation arguments
; '''iattr''': new attributes (platform specific yet)
; '''ea''': striping info buffer
; '''ealen''': striping info buffer length
; '''ea2''': currently unused
; '''ea2len''': currently unused
; '''request''': where address of request will be stored

mdc_setattr() sends sync RPC to MDS to change attributes. it does not return any lock, so attributes in reply isn't subject to cache.

=== mdc_getattr ===

int mdc_getattr(struct obd_export *exp, struct ll_fid *fid,
obd_valid valid, unsigned int ea_size,
struct ptlrpc_request **request)

; '''exp''': connection to MDS
; '''fid''': object of interest
; '''valid''': attributes of interest
; '''ea_size''': how many bytes of striping info client can store
; '''reuqest''': where address of request will be stored (to access reply)

mdc_getattr() sends synchronous RPC to MDS. it never returns lock.

=== mdc_close ===

int mdc_close(struct obd_export *exp, struct obdo *oa,
struct obd_client_handle *och, struct ptlrpc_request **request)

; '''exp''': export for connection to MDS
; '''oa''':
; '''och''': openhandle returned by mdc_intent_lock() at open time
; '''request''': where address of request will be stored (to access reply)

=== mdc_link ===

int mdc_link(struct obd_export *exp, struct mdc_op_data *op_data,
struct ptlrpc_request **request)

=== mdc_unlink ===

int mdc_unlink(struct obd_export *exp, struct mdc_op_data *op_data,
struct ptlrpc_request **request)

=== mdc_create ===

int mdc_create(struct obd_export *exp, struct mdc_op_data *op_data,
const void *data, int datalen, int mode, __u32 uid, __u32 gid,
__u32 cap_effective, __u64 rdev, struct ptlrpc_request **request)

=== mdc_rename ===

int mdc_rename(struct obd_export *exp, struct mdc_op_data *op_data,
const char *old, int oldlen, const char *new, int newlen,
struct ptlrpc_request **request)

=== mdc_req2lustre_md ===

int mdc_req2lustre_md(struct ptlrpc_request *req, int offset,
struct obd_export *exp,
struct lustre_md *md)

== Examples ==

=== intermediate lookup ===

VFS parses filename, finds intermediate (not last) component and starts lookup:
# local cache (dcache) is checked
# if dentry is found in dcache, then revalidation is called (llite's revalidation check whether lock with LOOKUP bit is still granted to the client)
# if dentry isn't found in dcache, then lookup method is called
# llite's lookup prepares intent and sends ENQUEUE RPC with the intent
# MDS receives RPC, finds intent and execute it: name is looked up to fid, correspondent attributes are fetched, DLM lock is acquired for this resolving (LOOKUP). all this is put in the reply
# client gets fid and attributes, finds or creates new inode for given fid, adds one more entry into dcache

ll_lookup_it()
mdc_intent_lock(IT_LOOKUP)
mdc_enqueue()
iget()
ldlm_lock_decref()

=== final lookup ===

[[Category:Architecture|Metadata API]]
[[New_Metadata_API]]

Architecture - Metadata API - Revision history

Docadmin at 21:16, 22 January 2010

Docadmin: Protected "Architecture - Metadata API" ([edit=sysop] (indefinite) [move=sysop] (indefinite))

Docadmin: /* final lookup */

Docadmin at 00:01, 19 January 2010

Docadmin: /* final lookup */

Docadmin: moved Metadata API to Architecture - Metadata API

Docadmin: 1 revision