From Obsolete Lustre Wiki
Jump to navigationJump to search
Revision as of 14:01, 14 January 2010
Definitions
- CROW (CReate On Write)
- the technique to optimize the create performance by deferring actual OSS objects creation until the first modify event (write or setattr) occur
CROW Architecture
MDS_LOV |
Allocate object FIDs and store them into LOV EA without object creation RPC to OST
|
obdfilter |
Implement object creation during first write/setattr request
|
Use Cases
Summary
id |
quality attribute |
summary
|
object_creation |
performance, usability |
Only save objects FIDs in EA during create operation on MDS. Create objects on OSTs at first write/setattr. Don't use precreation mechanism.
|
wrong_creation |
usability |
The re-creation of just destroyed objects are properly recognized and objects are not re-created
|
objects |
usability |
The MD object can exist with stripe info but without OSS object itself
|
fid |
usability |
Uniform FIDs are used to identify objects. MD FID and OSS FID are cross-referred using EA
|
cache |
availability, usability |
Client allocates both MD and OSS FIDs before doing create request to MDS.
|
recovery |
availability, usability |
Recoverable through re-creation of lost objects and/or deletion of orphaned objects in case of single-point failures.
|
cmd |
usability |
CMD is supported.
|
tests |
testability |
sanity and recovery tests.
|
quota |
security, usability |
compatible with quota.
|
scalability |
availability, usability |
more scalable than pre-creation mechanism
|
inodes_reservation |
usability |
OST should have enough inodes for all allocated FIDs
|
wrong_creation
Scenario: |
The OST objects being unlinked are re-created again by setattr.
|
Business Goals: |
Avoid re-creation of objects on OST
|
Relevant QA's: |
Usability & availability
|
details
|
Stimulus: |
OST objects are unlinked then late setattr or write come and create objects again.
|
Stimulus source: |
Object destroy and setattr come to OST from different nodes, therefore setattr can come later in case of network or server failure and consequent recovery.
|
Environment: |
MDS, OST
|
Artifact: |
There is no way to know about was the object already destroyed or just not created yet.
|
Response: |
Determine the state of non-existent OST objects - not yet created or destroyed already or cluster-wide serialization setattr vs. unlink
|
Response measure: |
Objects are not re-created.
|
Questions: |
No.
|
Issues: |
There is no clear understanding yet how to achieve the goal. The key can be the MDS FID of object. If it is exists then OST objects are not yet created, otherwise they was destroyed already.
serialize setattr vs unlink on MDS? bzzz
|
cache
Scenario: |
The FIDs for OST objects allocation on MDS can appear later than client needs them.
|
Business Goals: |
Caching MDS and disconnected operations should work
|
Relevant QA's: |
Usability
|
details
|
Stimulus: |
Though the creation of OST objects is postponed the FIDs for them should be allocated during create and saved at MDS in LOV EA. Doing that on MDS can invoke problems.
|
Stimulus source: |
Caching MDS, disconnected operations
|
Environment: |
Client
|
Artifact: |
Caching MD or disconnected operations can send requests to MDS with delay, doing MDS job. Meanwhile the client should have FIDs for OST objects without delays to work with OST.
|
Response: |
Allocate OST FIDs at client during create and pass them to MDS along with create request in LOV EA data.
|
Response measure: |
Client has valid LOV EA with OST object FIDs right after create operation.
|
Questions: |
If the clients are doing OST object creation before notifying the MDS then there is no way for the MDS/OST to clean up orphan objects if the client crashes before sending LOV EA to the MDS. Possibly the MDS would need to track the last-used objid for each sequence, and clients need to flush files+LOV_EAs to the MDS in objid order?
|
Issues: |
No.
|
inodes_reservation
Scenario: |
FIDs are allocated during create() but inodes on OST are created later, so there can be no free inodes for already allocated FIDs
|
Business Goals: |
All object FIDs should have inode
|
Relevant QA's: |
Usability
|
details
|
Stimulus: |
The object FIDs are allocated and stored in LOV EA on MDS but there are no free inodes at the moment of write/setattr on OST.
|
Stimulus source: |
Applications
|
Environment: |
OST
|
Artifact: |
The FID allocation is done earlier than getting inode on OST.
|
Response: |
Reserve OST inodes for future FID allocations.
|
Response measure: |
Any allocated FID for object shall get inode on OST or allocation should fail.
|
Questions: |
The DMU does not have a (practical) inode count limit like ldiskfs does, but will return ENOSPC when there is no free space left to create a new inode. This is equivalent to ENOSPC due to no free space for data, so maybe no reservation is needed in this case.
|
Issues: |
No.
|
Implementation Constraints
1. Use existing API and protocols
References