Windows Native Client

Google Doc Questions and Comments

Lustre Windows Client Questions

This page is to be populated by Lustre Group engineers with information, questions, highlights and risks associated with native Windows client porting efforts.

Branch details (Nikita)

Information:

branch for OSR: b_winnt_port

it's based on b_client_io_layering, which, in turn, is based on HEAD

b_winnt_port uses HEAD lnet

this branch is in public CVS repository

Questions:

is separate lnet branch necessary?

repository access control and visibility?

Compiling on Windows (Matt Wu)

Information: gcc & vc incompatible issues: mingw for intermediate modules and wdk for libcfs/tdilnd/llite(final fs driver) minimum modifications are needed. building process will do extra steps to process codes

share codes to be made generic (llite/libcfs/tdilnd) 1) structure dot initialization to be converted to code (around 500 LOC in all client modules, evaluated on HEAD) 2) typeof,MACRO to be avoided, the same to other issues which can't easily be done by a program 3) "long" and "long long" to be avoided 4) automatic conversion of CDEBUG, constant suffix ...

automatic builds for different targets 1) on Windows: automatically generate windows style makefiles (DIRS/MAKEFILE/SOURCES) and mingw makefile 2) on Linux: provide a mingw cross-compile environment for Linux people to assure not breaking Windows codes. Only adopted to common codes obdclass: windows-specific codes to be moved to windows llite

Questions:

Can we make b_winnt_port a complete windows branch with all above issues resolved ? It eases windows development, but increases complexity on syncing and landing

Windows doesn't permit low-level page/task manipulations, while current Linux codes deponds much on them, especially for CLIO routines. like current->journal_info, Windows has "Thread Local Storage" array in every thread for similar purpose, but we can not guarantee other programs' access to it. File system driver has another structure (context control block) to maintain the session related information.

CLIO API (Nikita)

Information:

most up to date version of CLIO is in b_client_io_layering branch

detailed interface documentation is automatically generated from annotated source code by doxygen. high level design: HLD/client-io-layering.lyx

detailed level design: DLD/client-io-layering-dld.lyx

a list of Quality Attribute Scenarios (architectural use cases): http://arch.lustre.org/index.php?title=QAS_Client_Cleanup

a to-do/known-issues list: http://spreadsheets.google.com/a/lustre.org/ccc?key=pIIxuIuTZWiZ2P5wvELYoRQ&hl=en /* Hmm... access permissions? */

top-level Bugzilla ticket: https://bugzilla.lustre.org/show_bug.cgi?id=14166

Questions:

scope of file handles. It looks like in Windows kernel handles for kernel objects (such as files and volumes) are attached to a particular process and can be used only in the context of this process. Is this correct? If so, care has to be taken to work around this issue in ptlrpc (asynchronous request handling), and dlm (lock ASTs). [matt: Windows kernel uses FILE/DEVICE_OBJECT directly rather than Handle. Handle could also be created in kenrel space and will stay valid under arbitrary thread contexts.]

OwPurgeCache() and OwFlushCache() works on the whole files only. We need an extent or page/block based interface to support shared-IO mode efficiently.

can we get an access to the source code of sample FSDK based file system drivers?

locking/sharing model: mandatory/advisory, requested by user-space/kernel?

FS_READ, FS_WRITE: synchronous? Should they return when IO operation completes?

What are the consistency expectations of FSDK? [matt: most of FSDK-style interfaces are synchronous. but FSDK lib can be asynchous and allow concurrent I/O. I'm not sure of FSDK internals. Need to confirm with OSR.]

FS_READ: read region is partially cached. What component handles local (intra-node) locking?

MD API (Alex)

Semantic:

FSDK doesn't allow to unlink open files?

does access to file's attribute imply open?

does lookup through directory imply preceding open?

Caching: what exactly does FSDK cache, how and when: parent/name->object, object attributes and directory entries

how can we control this? (except through OwPurgeCache() and OwFlushCache())

Intents: how can we learn purpose of FS_LOOKUP/FS_LOOKUP_PATH: intermediate component, final component, file to be open, file to be created

is there a way to store context over single syscall in FSD?

should we use FS_LOOKUP_PATH to implement intents?

Next Steps for OSR

QUESTION (PSB):

What development activities will OSR undertake while support Sun efforts are underway? Please identify a set of deliverable tasks and estimate timeframe.

Matt: Could OSR send us FSDK ? The FSDK examples can help us to well understand FSDK interfaces. FSDK core library is distributed in binary. Would it be fine for us ?

Page purging stays a risk. Cache can't be purged for mmapped files. We either play with stale data or cancel lock revocation. Hope OSR can propose a solution.

FSDK doesn't provide cache manipulation support on file data/directory page cache to it's client fs. It's not a good idea to maintain memory pools in the client file system while FSDK/Cache manager maintain another copy.

a) No way to read data from cache in FSDK internal interface. The client file system can only talk to the underlying disk or network layer. Though we can get the FileObject from it's orignal Irp, it might conflict with FSDK (OSR doesn't recommend).

b) Cannot maintain directory page cache. Windows file system driver normally creates a stream file object for directory inode, the it can perform page / cache operations upon the stream file object. It needs FSDK to work together to handle these virtual stream file objects (to avoid possible conflicts).

Milestones

- Working Linux Lustre cluster (vmware or otherwise) - Michael MacDonald to assist asap? - Mounting on windows – requires Matt’s branch to work - File Browsing - File I/O - Integrate with CFS testing - Performance

Methodology

OSR has a reasonable idea how to get skeleton code together for 1-3, but needs a review of lock revocation and metadata caches before finalizing 3, from someone like Oleg. I expect some assistance from Matt suggesting skeletons etc can help tremendously. Nikita should help architect a skeleton for the file I/O architecture.

Contributors:

OSR Dan Root – project manager Tony Mason – design lead, working with Mark

CFS Bryon Neitzel - Project Manager Matt – owns libraries, compiler issues, LNET Nikita – file I/O guidance Oleg ? – Metadata guidance Eeb – control landing from the branch Braam / Eeb – architecture, abstractions

Risks

- all risks are fairly likely and highly disruptive

Loss of CFS attention – CFS project manager Poorly defined details – CFS documentation team Out of date libraries – Matt Wu Compiler issues – Matt Wu Code base continues to move – CFS development team Getting it right vs working – Nikita / Oleg Landing into 2.0 – Barton

Implementation strategy

OSR will work on skeletons / prototypes and request review of these when ready.

Project management

Frequent status reports with risk update. OSR agreed to weekly status reports and bi-weekly conference calls.

Budget

Peter Braam and Tony were not yet concerned about the budget situation, but the original estimates were too optimistic, in particular because CFS made no progress between October 2007 and April 2008.

Meeting History

Windows Client Meeting, Monday, 2008-04-21.

Eric Barton Peter Bojanic Nikita Danilov Alex Tomas Matt Wu Bryon Neitzel

A Google Document was created for questions from the development staff for Braam's meeting with OSR this week. http://docs.google.com/a/lustre.org/Doc?id=d8jgmdp_20fpxwdgb. Alex and Nikita to document questions by Wednesday.

A branch has been created for use by OSR. b_winnt_port branch of lustre-core module (sharing HEAD lnet so far)

Comments from the Meeting:

- eeb: need a clean interface that porting will use. - eeb: need metadata interface that doesn't expose internal stuff to another platform

- Matt to continue documentation, compile issues, and porting all client components that are common to other platforms (LOV, LDLM, OSC, MDC, LVM, PTLRPC, etc) Eric would like to be involved before too much design work is done. - Matt has drafted a doc titled gcc_vc.pdf that describes the differences between the gcc and vc compilers, and some proposed solutions. This was sent to Dan Root, et al, on Jan 20. Eric to provide feedback.

- Alex wants to learn more about Windows. understand simple use cases.

- would OSR implement the file handle stuff? (do we need to export an MD FID lookup?)

- Get de-brief from Braam on meeting, have him give direction to Alex on md api

           - Alex to send a list of questions to braam

- Put a question in doc about what OSR can be doing?

- Need an OSR work plan as a result of Braam meeting, so the Lustre team knows all the dependencies and dates on items we're responsible for.

- Do we need an LNET branch? Give them a snapshot of all of the sources. Otherwise we don't know what they're implementing to. Nikita says current client IO can't be used for implementation, as it's not far enough along yet.

- Provide branch names: b_winnt _port, HEAD version of LNET. OSR will need their own private branch when they start to commit code. Need to sort out branch situation when OSR is ready to check in code. Their code cannot live in a pubic branch.