Architecture - Interoperability fids zfs - Revision history

Adilger: /* NEW.0 */ Fix LLOG and ECHO group numbers error to match FID_SEQ_LLOG/FID_SEQ_ECHO

2012-10-04T07:19:58Z

NEW.0: Fix LLOG and ECHO group numbers error to match FID_SEQ_LLOG/FID_SEQ_ECHO

← Older revision		Revision as of 00:19, 4 October 2012
Line 610:		Line 610:
	; '''FID''' : File IDentifier generated by client from range allocated by the seq service. First 0x400 sequences [2<sup>33</sup>, 2<sup>33</sup> + 0x400] are reserved for system use. Note that on ldiskfs MDTs that IGIF FIDs can use inode numbers starting at 12, but this is in the IGIF SEQ rangeand does not conflict with assigned FIDs.		; '''FID''' : File IDentifier generated by client from range allocated by the seq service. First 0x400 sequences [2<sup>33</sup>, 2<sup>33</sup> + 0x400] are reserved for system use. Note that on ldiskfs MDTs that IGIF FIDs can use inode numbers starting at 12, but this is in the IGIF SEQ rangeand does not conflict with assigned FIDs.

	; '''IGIF''' : Inode and Generation In FID, a surrogate FID used to globally identify an existing object on OLD formatted MDT file system. This would only be used on MDT0 in a ~~CMD~~ filesystem, because there are not expected to be any OLD formatted ~~CMD~~ filesystems. Belongs to a sequence in [12, 2<sup>32</sup> - 1] range, where sequence number is inode number, and inode generation is used as OID. '''NOTE''': This assumes no more than 2<sup>32</sup>-1 inodes exist in the MDT filesystem, which is the maximum possible for an ldiskfs backend. '''NOTE''': This assumes that the reserved ext3/ext4/ldiskfs inode numbers [0-11] are never visible to clients, which has always been true.		; '''IGIF''' : Inode and Generation In FID, a surrogate FID used to globally identify an existing object on OLD formatted MDT file system. This would only be used on MDT0 in a DNE filesystem, because there are not expected to be any OLD formatted DNE filesystems. Belongs to a sequence in [12, 2<sup>32</sup> - 1] range, where sequence number is inode number, and inode generation is used as OID. '''NOTE''': This assumes no more than 2<sup>32</sup>-1 inodes exist in the MDT filesystem, which is the maximum possible for an ldiskfs backend. '''NOTE''': This assumes that the reserved ext3/ext4/ldiskfs inode numbers [0-11] are never visible to clients, which has always been true.

	; '''IDIF''' : object ID in FID, a surrogate FID used to globally identify an existing object on OLD formatted OST file system. Belongs to a sequence in [2<sup>32</sup>, 2<sup>33</sup> - 1]. Sequence number is calculated as:		; '''IDIF''' : object ID in FID, a surrogate FID used to globally identify an existing object on OLD formatted OST file system. Belongs to a sequence in [2<sup>32</sup>, 2<sup>33</sup> - 1]. Sequence number is calculated as:
Line 624:		Line 624:
	; '''OST_MDT0''' : Surrogate FID used to identify an existing object on OLD formatted OST filesystem. Belongs to the reserved sequence 0, and is used internally prior to the introduction of FID-on-OST, at which point IDIF will be used to identify objects as residing on a specific OST.		; '''OST_MDT0''' : Surrogate FID used to identify an existing object on OLD formatted OST filesystem. Belongs to the reserved sequence 0, and is used internally prior to the introduction of FID-on-OST, at which point IDIF will be used to identify objects as residing on a specific OST.

	; '''~~ECHO~~''' : for ~~testing OST IO performance~~ the object sequence 1 is used. This is compatible with both OLD and NEW.1 namespaces, as this SEQ number is in the ext3/ldiskfs reserved inode range and does not conflict with IGIF sequence numbers.		; '''LLOG''' : for Lustre Log objects the object sequence 1 is used. This is compatible with both OLD and NEW.1 namespaces, as this SEQ number is in the ext3/ldiskfs reserved inode range and does not conflict with IGIF sequence numbers.

	; '''~~LLOG~~''' : for ~~Lustre Log objects~~ the object sequence 2 is used. This is compatible with both OLD and NEW.1 namespaces, as this SEQ number is in the ext3/ldiskfs reserved inode range and does not conflict with IGIF sequence numbers.		; '''ECHO''' : for testing OST IO performance the object sequence 2 is used. This is compatible with both OLD and NEW.1 namespaces, as this SEQ number is in the ext3/ldiskfs reserved inode range and does not conflict with IGIF sequence numbers.

	; '''OST_MDT1''' .. '''OST_MAX''' : for testing with multiple MDTs the object sequence 3 through 9 is used, allowing direct mapping of MDTs 1 through 7 respectively, for a total of 8 MDTs including '''OST_MDT0'''. This matches the legacy CMD project "group" mappings. However, this SEQ range is only for testing prior to any production ~~CMD~~ release, as the objects in this range conflict across all OSTs, as the OST index is not part of the FID.		; '''OST_MDT1''' .. '''OST_MAX''' : for testing with multiple MDTs the object sequence 3 through 9 is used, allowing direct mapping of MDTs 1 through 7 respectively, for a total of 8 MDTs including '''OST_MDT0'''. This matches the legacy CMD project "group" mappings. However, this SEQ range is only for testing prior to any production DNE release, as the objects in this range conflict across all OSTs, as the OST index is not part of the FID.

Adilger: minor updates to FID_SEQ table

2010-07-12T23:48:51Z

minor updates to FID_SEQ table

← Older revision		Revision as of 16:48, 12 July 2010
Line 1:		Line 1:
	'''''Note:''''' ''The content on this page reflects the state of design of a Lustre feature at a particular point in time and may contain outdated information.''		'''''Note:''''' ''The content on this page reflects the state of design of a Lustre feature at a particular point in time and may contain both outdated information and unimplemented functionality.''

	== Summary ==		== Summary ==
Line 567:		Line 567:
	\|fields \|\|SEQ \|\|OID \|\|VER		\|fields \|\|SEQ \|\|OID \|\|VER
	\|-		\|-
	\|FID_SEQ_OST_MDT0 \|\|=0 \|\| \|\|		\|FID_SEQ_OST_MDT0 \|\|= 0 \|\| \|\|
	\|-		\|-
	\|FID_SEQ_LLOG \|\|=1 \|\| \|\|		\|FID_SEQ_LLOG \|\|= 1 \|\| \|\|
	\|-		\|-
	\|FID_SEQ_ECHO \|\|=2 \|\| \|\|		\|FID_SEQ_ECHO \|\|= 2 \|\| \|\|
	\|-		\|-
	\|FID_SEQ_OST_MDT1 \|\|=3 \|\| \|\|		\|FID_SEQ_OST_MDT1 \|\|= 3 \|\| \|\|
	\|-		\|-
	\|FID_SEQ_OST_MAX \|\|=9 (=FID_SEQ_OST_MDT7) \|\| \|\|		\|FID_SEQ_OST_MAX \|\|= 9 (=FID_SEQ_OST_MDT7) \|\| \|\|
	\|-		\|-
	\|FID_SEQ_IGIF \|\|=12 \|\| \|\|		\|FID_SEQ_IGIF \|\|= 12 \|\| \|\|
			\|-
			\|FID_SEQ_IGIF_MAX \|\|= 0xffffffff \|\| \|\|
	\|-		\|-
	\|FID_SEQ_IDIF \|\|=0x100000000 \|\| \|\|		\|FID_SEQ_IDIF \|\|=0x100000000 \|\| \|\|
Line 590:		Line 592:
	\|-		\|-
	\|-		\|-
	\|obdo/lmm/oinfo(OLD)\|\|o_seq:64 (FID_SEQ_OST_MDT0) \|\|o_id_lo:48\|\|o_id_hi:16		\|obdo/lmm/oinfo(OLD)\|\|o_seq:64 [FID_SEQ_OST_MDT0] \|\|o_id_lo:48\|\|o_id_hi:16
	\|-		\|-
	\|obdo/lmm/oinfo(NEW.1)\|\|o_seq:64 (FID_SEQ_{IDIF,NORMAL})\|\|o_id_lo:32\|\|o_id_hi:32		\|obdo/lmm/oinfo(NEW.1)\|\|o_seq:64 [FID_SEQ_{IDIF,NORMAL}]\|\|o_id_lo:32\|\|o_id_hi:32
	\|-		\|-
	\|lu_fid \|\|f_seq:64 \|\|f_oid:32 \|\|f_ver:32		\|lu_fid \|\|f_seq:64 \|\|f_oid:32 \|\|f_ver:32
Line 598:		Line 600:
	\|IGIF \|\|0:32, ino:32 [12,FID_SEQ_IGIF_MAX] \|\|gen:32 \|\|0:32		\|IGIF \|\|0:32, ino:32 [12,FID_SEQ_IGIF_MAX] \|\|gen:32 \|\|0:32
	\|-		\|-
	\|IDIF \|\|0:31, 1:1, ost_idx:16,o_id_hi:16 \|\|o_id_lo:32\|\|~~o_id_hi~~:16		\|IDIF \|\|0:31, 1:1, ost_idx:16,o_id_hi:16 \|\|o_id_lo:32\|\|o_id_hi_hi:16
	\|-		\|-
	\|reserved \|\|[FID_SEQ_START,FID_SEQ_START+0x3ff]\|\|f_oid:32 \|\|f_ver:32		\|reserved \|\|[FID_SEQ_START,FID_SEQ_START+0x3ff]\|\|f_oid:32 \|\|f_ver:32
Line 647:		Line 649:
	There are 2 important recovery scenarios related to interoperability:		There are 2 important recovery scenarios related to interoperability:

	* OLD.x client reconnects to MDT after a fail-over and learns that it has to switch back to the OLD protocol, because server was downgraded. Client has to replay requests, but before that they have to be converted into OLD protocol format. This requires changing message format and going from ~~fids~~ to inode numbers (storage cookies). If a ~~fid~~ in is ~~igif~~ format it can be converted to inode number according to the reverse of ~~igif~~ generation algorithm. If a ~~fid~~ is client-generated, then '''KABOOM'''! Client has to evict itself, because it doesn't know old-format inode number. '''Q? Is there a better solution?'''. What to do with RPCs that old server cannot handle at all: SEQ_QUERY? Again, eviction seems to be the only option.		* OLD.x client reconnects to MDT after a fail-over and learns that it has to switch back to the OLD protocol, because server was downgraded. Client has to replay requests, but before that they have to be converted into OLD protocol format. This requires changing message format and going from client-assigned FIDs to inode/generation numbers (storage cookies). If a FID in is IGIF format it can be converted to inode number according to the reverse of IGIF generation algorithm. If a FID is client-generated, then '''KABOOM'''! Client has to evict itself, because it doesn't know old-format inode number. '''Q? Is there a better solution?'''. What to do with RPCs that old server cannot handle at all: SEQ_QUERY? Again, eviction seems to be the only option.

	* OLD.x client reconnects to MDT and determines that it has to switch to the new protocol, because MDT was upgraded to NEW.0. To replay RPCs, client has to convert them to the NEW format. This includes message format conversion and going from inode numbers to ~~fids~~. For RPCs that already include inode number as an argument, ~~igif fid is~~ used. For CREATE RPC that requires fid in NEW protocol there are two options:		* OLD.x client reconnects to MDT and determines that it has to switch to the new protocol, because MDT was upgraded to NEW.0. To replay RPCs, client has to convert them to the NEW format. This includes message format conversion and going from inode/generation numbers to FIDs. For RPCs that already include inode number as an argument, IGIF FID can be used. For CREATE RPC that requires fid in NEW protocol there are two options:
	** client supplies fill-in-~~fid~~. NEW.0 server recognizes this as a request to generate ~~fid~~ on the server, and uses special sequence range reserved for this purpose to allocate a ~~fid~~ from. Note that this sequence cannot be exhausted, as there is single MDT in the cluster at that point, which means it has full control over complete ~~fid~~ space.		** client supplies fill-in-FID. NEW.0 server recognizes this as a request to generate FID on the server, and uses special sequence range reserved for this purpose to allocate a FID from. Note that this sequence cannot be exhausted, as there is single MDT in the cluster at that point, which means it has full control over complete FID space.
	** client supplies inode number as in usual OLD protocol replay. Server detects this and creates inode with given inode number. This has certain drawbacks:		** client supplies inode number as in usual OLD protocol replay. Server detects this and creates inode with given inode number. This has certain drawbacks:
	*** a dependency on ext3-wantedi patch is re-introduced, and		*** a dependency on ext3-wantedi patch is re-introduced, and
	*** backward-compatibility code is introduced in NEW.0 release, which we are trying to avoid.		*** backward-compatibility code is introduced in NEW.0 release, which we are trying to avoid.

	~~--15 May 2008 (PDT)~~

Docadmin: /* NEW.0 */

2010-07-12T00:46:47Z

NEW.0

Show changes

Docadmin: /* Recovery */

2010-02-03T23:43:42Z

Recovery

← Older revision		Revision as of 16:43, 3 February 2010
Line 625:		Line 625:
	*** backward-compatibility code is introduced in NEW.0 release, which we are trying to avoid.		*** backward-compatibility code is introduced in NEW.0 release, which we are trying to avoid.

	--~~Nikita 12:16,~~ 15 May 2008 (PDT)		--15 May 2008 (PDT)

Docadmin: /* Summary */

2010-02-03T23:43:02Z

Summary

← Older revision		Revision as of 16:43, 3 February 2010
Line 4:		Line 4:

	This document describes an architecture for client, server, network, and storage interoperability during migration from 1.6-based, fidless Lustre clusters, using ldiskfs as a back-end file system, to clusters based on fids and zfs file system.		This document describes an architecture for client, server, network, and storage interoperability during migration from 1.6-based, fidless Lustre clusters, using ldiskfs as a back-end file system, to clusters based on fids and zfs file system.

	~~This document originates from an internal [https://wikis.lustre.org/intra/index.php/CMD3_interoperability_architecture wiki page].~~

	== Definitions ==		== Definitions ==

Docadmin at 21:13, 22 January 2010

2010-01-22T21:13:27Z

← Older revision		Revision as of 14:13, 22 January 2010
Line 1:		Line 1:
	'''''Note:''''' The content on this page reflects the state of design of a Lustre feature at a particular point in time and may contain outdated information.		'''''Note:''''' ''The content on this page reflects the state of design of a Lustre feature at a particular point in time and may contain outdated information.''

	== Summary ==		== Summary ==

Docadmin: Protected "Architecture - Interoperability fids zfs" ([edit=sysop] (indefinite) [move=sysop] (indefinite))

2010-01-21T19:59:22Z

Protected "Architecture - Interoperability fids zfs" ([edit=sysop] (indefinite) [move=sysop] (indefinite))

← Older revision	Revision as of 12:59, 21 January 2010
(No difference)

Docadmin: /* Summary */

2010-01-18T23:59:18Z

Summary

← Older revision		Revision as of 16:59, 18 January 2010
Line 1:		Line 1:
			'''''Note:''''' The content on this page reflects the state of design of a Lustre feature at a particular point in time and may contain outdated information.

	== Summary ==		== Summary ==

Docadmin: /* NEW.0 */

2010-01-18T23:41:19Z

NEW.0

← Older revision		Revision as of 16:41, 18 January 2010
Line 600:		Line 600:
	resource[] = {seq, oid, ver, hash};		resource[] = {seq, oid, ver, hash};

	Note that while the o_id may be larger than the 2<sup>33 reserved sequence numbers for IDIF, in all production releases the OLD o_gr field is always 0, so although it is possible for more than 8B objects to have been created on a single OST the non-zero ''oid'' field of the FID-based resource will distinguish the lock resources from each other.		Note that while the o_id may be larger than the 2<sup>33</sup> reserved sequence numbers for IDIF, in all production releases the OLD o_gr field is always 0, so although it is possible for more than 8B objects to have been created on a single OST the non-zero ''oid'' field of the FID-based resource will distinguish the lock resources from each other.

Docadmin: /* Recovery */

2010-01-14T22:34:50Z

Recovery

← Older revision		Revision as of 15:34, 14 January 2010
Line 625:		Line 625:
	*** backward-compatibility code is introduced in NEW.0 release, which we are trying to avoid.		*** backward-compatibility code is introduced in NEW.0 release, which we are trying to avoid.

	--~~[[User:~~Nikita~~\|Nikita]]~~ 12:16, 15 May 2008 (PDT)		--Nikita 12:16, 15 May 2008 (PDT)