GSS / Kerberos - Revision history

Adilger: Undo revision 12284 by Delphia Mulcahey (talk)

2013-04-12T16:26:46Z

Undo revision 12284 by Delphia Mulcahey (talk)

← Older revision		Revision as of 09:26, 12 April 2013
Line 43:		Line 43:
	kadmin> addprinc -randkey lustre_mds/mds_host.domain@REALM		kadmin> addprinc -randkey lustre_mds/mds_host.domain@REALM
	kadmin> ktadd -e aes128-cts:normal lustre_mds/mds_host.domain@REALM		kadmin> ktadd -e aes128-cts:normal lustre_mds/mds_host.domain@REALM
	*Install the keytab on the MDS node. ~~[http://www.slow-computer-solutions.org/ fix my slow computer]~~		*Install the keytab on the MDS node.

	3. Configure OSS node:		3. Configure OSS node:

Delphia Mulcahey: /* Configuration */

2011-04-06T13:33:02Z

Configuration

← Older revision		Revision as of 06:33, 6 April 2011
Line 43:		Line 43:
	kadmin> addprinc -randkey lustre_mds/mds_host.domain@REALM		kadmin> addprinc -randkey lustre_mds/mds_host.domain@REALM
	kadmin> ktadd -e aes128-cts:normal lustre_mds/mds_host.domain@REALM		kadmin> ktadd -e aes128-cts:normal lustre_mds/mds_host.domain@REALM
	*Install the keytab on the MDS node.		*Install the keytab on the MDS node. [http://www.slow-computer-solutions.org/ fix my slow computer]

	3. Configure OSS node:		3. Configure OSS node:

Sbarthel: /* Secure MGC - MGS connection */

2011-01-20T18:41:13Z

Secure MGC - MGS connection

← Older revision		Revision as of 11:41, 20 January 2011
Line 187:		Line 187:
	By default, MGS accept RPCs with any flavor. But sysad can configure MGS to only accept certain flavor from certain network. The syntax is similar but with target as a special "_mgs":		By default, MGS accept RPCs with any flavor. But sysad can configure MGS to only accept certain flavor from certain network. The syntax is similar but with target as a special "_mgs":
	mgs> lctl conf_param _mgs.srpc.flavor.<network>=flavor		mgs> lctl conf_param _mgs.srpc.flavor.<network>=flavor
	'''NOTE: apply inappropriate flavor may lead to a node never be able to communicate with MGS until restart. So use it carefully.~~'''~~		'''NOTE:''' apply inappropriate flavor may lead to a node never be able to communicate with MGS until restart. So use it carefully.

	== Cross-Realms Authentication ==		== Cross-Realms Authentication ==
	Due to idmap functionality is missing, we don't support cross-realm authentication currently.		Due to idmap functionality is missing, we don't support cross-realm authentication currently.

Ericm: /* Setting Security Flavors */

2010-09-07T21:39:32Z

Setting Security Flavors

← Older revision		Revision as of 14:39, 7 September 2010
Line 133:		Line 133:
	If there a existing rule of <spec> part, it will overwritten.		If there a existing rule of <spec> part, it will overwritten.

	To delete a rule~~, just leave <flavor> part be empty~~:		To delete a rule:
	mgs> lctl conf_param <spec>=		mgs> lctl conf_param -d <spec>

	Current rule set could be obtained by:		Current rule set could be obtained by:

Sbarthel: /* required packages */

2010-06-09T18:13:54Z

required packages

← Older revision		Revision as of 11:13, 9 June 2010
Line 68:		Line 68:
	*For MIT Kerberos 1.3.x, only ''des-cbc-md5'' works because a known issue between libgssapi and Kerberos library.		*For MIT Kerberos 1.3.x, only ''des-cbc-md5'' works because a known issue between libgssapi and Kerberos library.

	== ~~required~~ packages ==		== Required packages ==
	Every node should have follow packages installed:		Every node should have follow packages installed:
	* '''''libgssapi''''' version 0.10 or higher. Some newer Linux distributions already come with it. If not, build & install from source: http://www.citi.umich.edu/projects/nfsv4/linux/libgssapi/libgssapi-0.11.tar.gz		* '''''libgssapi''''' version 0.10 or higher. Some newer Linux distributions already come with it. If not, build & install from source: http://www.citi.umich.edu/projects/nfsv4/linux/libgssapi/libgssapi-0.11.tar.gz

Sbarthel at 18:11, 9 June 2010

2010-06-09T18:11:09Z

← Older revision		Revision as of 11:11, 9 June 2010
Line 1:		Line 1:
	'''Note: ~~only~~ HEAD branch ~~support~~ GSS/Kerberos functionality~~, and it's~~ subject to changes at any time, backward compatibility is NOT guaranteed.~~'''~~		'''Note:''' Only the HEAD branch supports GSS/Kerberos functionality. It is subject to changes at any time, and backward compatibility is NOT guaranteed.

	= Kerberos Lustre Setup =		= Kerberos Lustre Setup =

Sbarthel: Created page with ''''Note: only HEAD branch support GSS/Kerberos functionality, and it's subject to changes at any time, backward compatibility is NOT guaranteed.''' = Kerberos Lustre Setup = ==...'

2010-06-09T18:10:26Z

Created page with ''''Note: only HEAD branch support GSS/Kerberos functionality, and it's subject to changes at any time, backward compatibility is NOT guaranteed.''' = Kerberos Lustre Setup = ==...'

New page

'''Note: only HEAD branch support GSS/Kerberos functionality, and it's subject to changes at any time, backward compatibility is NOT guaranteed.'''

= Kerberos Lustre Setup =

== Security Flavor ==
A security flavor is a string to describe what kind authentication and data transformation be performed upon a PTLRPC connection. It covers both RPC message and BULK data.

The support flavors are described in following table:

{|border=1 cellspacing=0
|bgcolor=#E6E6E6| Base Flavor||bgcolor= #E6E6E6|Authentication||bgcolor=#E6E6E6|RPC Message Protection||bgcolor=#E6E6E6|Bulk Data Protection||bgcolor=#E6E6E6|Notes
|-
|'''''null'''''||N/A ||N/A ||N/A '''[*]''' ||Almost no performance overhead. The on-wire rpc format is compatible with old versions (1.4.x, 1.6.x, 1.8.x).
|-
|'''''plain'''''||N/A ||N/A ||checksum||(obsolete)
|-
|'''''krb5n'''''||GSS/Kerberos5 ||null||checksum (adler32)||No protection of rpc message, adler32 checksum protection of bulk data, light performance overhead.
|-
|'''''krb5a'''''||GSS/Kerberos5 ||partly integrity (krb5)||checksum (adler32)||Only header of rpc message is integrity protected, adler32 checksum protection of bulk data, more performance overhead compare to krb5n.
|-
|'''''krb5i'''''||GSS/Kerberos5 ||integrity (krb5)||integrity (krb5)||transformation algorithm is determined by actual Kerberos algorithms in use; Heavy performance penalty.
|-
|'''''krb5p'''''||GSS/Kerberos5 ||privacy (krb5)||privacy (krb5)||transformation privacy protection algorithm is determined by actual Kerberos algorithms in use; The heaviest performance penalty.
|}

'''[*]''' In Lustre 1.4 and 1.6 it is possible to enable bulk data checksumming to provide integrity checking using CRC32. In 1.6.5 this is expected to be the default behaviour, using the Adler32 mechanism by default (lower CPU overhead than CRC32).

In the future, we may want to support customize flavor to some extend. For example, allow set different flavors for RPC message and bulk data.

== Kerberos Setup ==
=== Distribution ===
* We only support MIT Kerberos 5, version from 1.3.x to latest 1.6.x.

=== Configuration ===
1. Configure client nodes:
*For each client node, create a lustre_root principal and generate keytab.
kadmin> addprinc -randkey lustre_root/client_host.domain@REALM
kadmin> ktadd -e aes128-cts:normal lustre_root/client_host.domain@REALM
*Install the keytab on the client node.

2. Configure MDS node:
*For each MDS node, create a lustre_mds principal and generate keytab.
kadmin> addprinc -randkey lustre_mds/mds_host.domain@REALM
kadmin> ktadd -e aes128-cts:normal lustre_mds/mds_host.domain@REALM
*Install the keytab on the MDS node.

3. Configure OSS node:
*For each OSS node, create a lustre_oss principal and generate keytab.
kadmin> addprinc -randkey lustre_oss/oss_host.domain@REALM
kadmin> ktadd -e aes128-cts:normal lustre_oss/oss_host.domain@REALM
*Install the keytab on the OSS node.

NOTES:
*The ''host.domain'' should be the FQDN in your network, otherwise server might not recognize any GSS request.

*As an alternative of the client keytab, if you want to save the trouble of assigning unique keytab for each client node, you can create a general lustre_root principal and its keytab, and install the same keytab on as many client nodes as you want. '''But be aware that in this way one compromised client means all clients are insecure'''.
kadmin> addprinc -randkey lustre_root@REALM
kadmin> ktadd -e aes128-cts:normal lustre_root@REALM

*To merge keytab files, you need the tool '''''ktutil''''', for more details please refers to manual of ktutil.

*Lustre support following ''enctypes'' for MIT Kerberos 5 version 1.4 or higher:
**''des-cbc-md5''
**''des3-hmac-sha1''
**''aes128-cts''
**''aes256-cts''

*For MIT Kerberos 1.3.x, only ''des-cbc-md5'' works because a known issue between libgssapi and Kerberos library.

== required packages ==
Every node should have follow packages installed:
* '''''libgssapi''''' version 0.10 or higher. Some newer Linux distributions already come with it. If not, build & install from source: http://www.citi.umich.edu/projects/nfsv4/linux/libgssapi/libgssapi-0.11.tar.gz
* '''''keyutils'''''

== Kernel & Environment ==
* System wide configuration:
On Each node (MDT, OST, Client) following line should be added into /etc/fstab to be automatically mounted
nfsd /proc/fs/nfsd nfsd defaults 0 0
Each MDT and Client node add following line into /etc/request-key.conf:
create lgssc * * /usr/sbin/lgss_keyring %o %k %t %d %c %u %g %T %P %S
Note you might need to replace '''/usr/sbin/lgss_keyring''' in above line to the actual path to lgss_keyring binary in your setting.

* Networking:
If you are using network which is '''NOT''' TCP or Infiniband (e.g. Quadrics Elan, Myrinet, etc), you need configure a '''''/etc/lustre/nid2hostname''''' on '''each''' server node (MDT & OST), which is a simple script to translate NID into hostname. Following is sample on a Elan cluster:

#!/bin/bash
set -x
exec 2>/tmp/$(basename $0).debug

# convert a NID for a LND to a hostname, for GSS for example

# called with thre arguments: lnd netid nid
# $lnd will be string "QSWLND", "GMLND", etc.
# $netid will be number in hex string format, like "0x16", etc.
# $nid has the same format as $netid
# output the corresponding hostname, or error message leaded by a '@' for error logging.

lnd=$1
netid=$2
nid=$3

# uppercase the hex
nid=$(echo $nid | tr '[abcdef]' '[ABCDEF]')
# and convert to decimal
nid=$(echo -e "ibase=16\n${nid/#0x}" | bc)
case $lnd in
QSWLND) # simply stick "mtn" on the front
echo "mtn$nid"
;;
*) echo "@unknown LND: $lnd"
;;
esac

== Build Lustre ==
Enable GSS during configuration:

./configure --enable-gss --other-options

== Running ==
=== GSS Daemons ===
Make sure start the daemon process '''lsvcgssd''' on each OST and MDT node before starting Lustre. The command syntax is:
lsvcgssd [-f] [-v]
* ''-f'': running at foreground instead of as daemon, thus output error/warning messages to front console instead of system log.
* ''-v'': increase verbosity by 1. The default is 0, maximum is 4.

=== Setting Security Flavors ===
Note: If nothing specified, by default all RPC connections will use '''''null'''''.

On MGS there's a persistent sptlrpc rule database, by specifying set of rules you can change security flavors between nodes. A rule is in form of:
<spec>=<flavor>
Rules can be manipulated on MGS node. To add a rule:
mgs> lctl conf_param <spec>=<flavor>
If there a existing rule of <spec> part, it will overwritten.

To delete a rule, just leave <flavor> part be empty:
mgs> lctl conf_param <spec>=

Current rule set could be obtained by:
msg> cat /proc/fs/lustre/mgs/<mgs-name>/live/<fs-name> | grep "srpc.flavor"

'''Note''':
* Rules have persistent storage on MGS, so it applied across re-mount.
* It doesn't matter in which order you add a set of rules, lustre keep rules in certain order or priority.
* After you changed a rule, usually it will take the system within 1 minutes to apply the new rules to all nodes, depend on system load.
* Before you change a rule, make sure affected nodes are ready for the new security flavor. E.g. you changed flavor from '''''null''''' to '''''krb5p''''' but GSS/Kerberos env is not properly configured on affected nodes, those nodes might be evicted because they can't communicate with others.
* You can also specify rules via device on-disk parameters, by mke2fs.lustre or tune2fs.lustre. The syntax is the same, and the rule only applied to connections to this specific target (MDT/OST).

=== Rules Syntax & Examples ===
The general syntax is:
<target>.srpc.flavor.<network>[.<direction>]=flavor

* <target>: could be filesystem name, or specific MDT/OST device name. For example, ''lustre'', ''lustre-MDT0000'', ''lustre-OST0001'', etc.
* <network>: LNET network name of the RPC initiator. For example, ''tcp0'', ''elan1'', ''o2ib0''.
* <direction>: could be one of ''cli2mdt'', ''cli2ost'', ''mdt2mdt'', ''mdt2ost''. In most cases you don't need to specify <direction> part.

Examples:
* Apply ''krb5i'' on '''ALL''' connections:
mgs> lctl conf_param lustre.srpc.flavor.default=krb5i

* Nodes in network ''tcp0'' use ''krb5p''; All other nodes use ''null''
mgs> lctl conf_param lustre.srpc.flavor.tcp0=krb5p
mgs> lctl conf_param lustre.srpc.flavor.default=null

* Nodes in network ''tcp0'' use ''krb5p''; Nodes in ''elan1'' use ''plain''; Amount other nodes, clients use ''krb5i'' to MDT/OST, MDT use ''null'' to other MDTs, MDT use ''plain'' to OSTs.
mgs> lctl conf_param lustre.srpc.flavor.tcp0=krb5p
mgs> lctl conf_param lustre.srpc.flavor.elan1=plain
mgs> lctl conf_param lustre.srpc.flavor.default.cli2mdt=krb5i
mgs> lctl conf_param lustre.srpc.flavor.default.cli2ost=krb5i
mgs> lctl conf_param lustre.srpc.flavor.default.mdt2mdt=null
mgs> lctl conf_param lustre.srpc.flavor.default.mdt2ost=plain

=== Authenticate Normal Users ===
On client nodes, a non-root user need '''''kinit''''' before accessing Lustre, just like other Kerberized applications.
* Required by kerberos, the user's principal (''username@REALM'') should be added into KDC.
* Client and MDT nodes should have the same user database, i.e. the user name and uid/gid translation.
A use could destroy the established security contexts before logout, by "lfs flushctx":

lfs flushctx [-k]

Here "-k" means also destroy the on-disk kerberos credential cache, equals to "kdestroy", otherwise it only destroy established contexts in Lustre kernel memory.

== Secure MGC - MGS connection ==
Each node can specify what flavor to use to connect to MGS, by option '''''mgssec=flavor''''' upon mounting a target device or client. By default ''null'' is chosen. Once a flavor is chosen, it can't be changed until umount.

Because each node has only one connection to MGS, so if there's more than one target device or client on a single node, all the "mgssec=" specification must be the same. Or simply missing option "mgssec=" means "using currently chosen flavor.

By default, MGS accept RPCs with any flavor. But sysad can configure MGS to only accept certain flavor from certain network. The syntax is similar but with target as a special "_mgs":
mgs> lctl conf_param _mgs.srpc.flavor.<network>=flavor
'''NOTE: apply inappropriate flavor may lead to a node never be able to communicate with MGS until restart. So use it carefully.'''

== Cross-Realms Authentication ==
Due to idmap functionality is missing, we don't support cross-realm authentication currently.