http://wiki.old.lustre.org/api.php?action=feedcontributions&user=Delphia+Mulcahey&feedformat=atomObsolete Lustre Wiki - User contributions [en]2024-03-29T09:06:03ZUser contributionsMediaWiki 1.35.5http://wiki.old.lustre.org/index.php?title=GSS_/_Kerberos&diff=12284GSS / Kerberos2011-04-06T13:33:02Z<p>Delphia Mulcahey: /* Configuration */</p>
<hr />
<div>'''Note:''' Only the HEAD branch supports GSS/Kerberos functionality. It is subject to changes at any time, and backward compatibility is NOT guaranteed.<br />
<br />
= Kerberos Lustre Setup =<br />
<br />
== Security Flavor ==<br />
A security flavor is a string to describe what kind authentication and data transformation be performed upon a PTLRPC connection. It covers both RPC message and BULK data.<br />
<br />
The support flavors are described in following table:<br />
<br />
{|border=1 cellspacing=0<br />
|bgcolor=#E6E6E6| Base Flavor||bgcolor= #E6E6E6|Authentication||bgcolor=#E6E6E6|RPC Message Protection||bgcolor=#E6E6E6|Bulk Data Protection||bgcolor=#E6E6E6|Notes<br />
|-<br />
|'''''null'''''||N/A ||N/A ||N/A '''[*]''' ||Almost no performance overhead. The on-wire rpc format is compatible with old versions (1.4.x, 1.6.x, 1.8.x).<br />
|-<br />
|'''''plain'''''||N/A ||N/A ||checksum||(obsolete)<br />
|-<br />
|'''''krb5n'''''||GSS/Kerberos5 ||null||checksum (adler32)||No protection of rpc message, adler32 checksum protection of bulk data, light performance overhead.<br />
|-<br />
|'''''krb5a'''''||GSS/Kerberos5 ||partly integrity (krb5)||checksum (adler32)||Only header of rpc message is integrity protected, adler32 checksum protection of bulk data, more performance overhead compare to krb5n. <br />
|-<br />
|'''''krb5i'''''||GSS/Kerberos5 ||integrity (krb5)||integrity (krb5)||transformation algorithm is determined by actual Kerberos algorithms in use; Heavy performance penalty. <br />
|-<br />
|'''''krb5p'''''||GSS/Kerberos5 ||privacy (krb5)||privacy (krb5)||transformation privacy protection algorithm is determined by actual Kerberos algorithms in use; The heaviest performance penalty.<br />
|}<br />
<br />
'''[*]''' In Lustre 1.4 and 1.6 it is possible to enable bulk data checksumming to provide integrity checking using CRC32. In 1.6.5 this is expected to be the default behaviour, using the Adler32 mechanism by default (lower CPU overhead than CRC32).<br />
<br />
In the future, we may want to support customize flavor to some extend. For example, allow set different flavors for RPC message and bulk data.<br />
<br />
== Kerberos Setup ==<br />
=== Distribution ===<br />
* We only support MIT Kerberos 5, version from 1.3.x to latest 1.6.x.<br />
<br />
=== Configuration ===<br />
1. Configure client nodes:<br />
*For each client node, create a lustre_root principal and generate keytab.<br />
kadmin> addprinc -randkey lustre_root/client_host.domain@REALM<br />
kadmin> ktadd -e aes128-cts:normal lustre_root/client_host.domain@REALM <br />
*Install the keytab on the client node.<br />
<br />
2. Configure MDS node:<br />
*For each MDS node, create a lustre_mds principal and generate keytab.<br />
kadmin> addprinc -randkey lustre_mds/mds_host.domain@REALM<br />
kadmin> ktadd -e aes128-cts:normal lustre_mds/mds_host.domain@REALM<br />
*Install the keytab on the MDS node. [http://www.slow-computer-solutions.org/ fix my slow computer]<br />
<br />
3. Configure OSS node:<br />
*For each OSS node, create a lustre_oss principal and generate keytab.<br />
kadmin> addprinc -randkey lustre_oss/oss_host.domain@REALM<br />
kadmin> ktadd -e aes128-cts:normal lustre_oss/oss_host.domain@REALM<br />
*Install the keytab on the OSS node.<br />
<br />
NOTES:<br />
*The ''host.domain'' should be the FQDN in your network, otherwise server might not recognize any GSS request.<br />
<br />
*As an alternative of the client keytab, if you want to save the trouble of assigning unique keytab for each client node, you can create a general lustre_root principal and its keytab, and install the same keytab on as many client nodes as you want. '''But be aware that in this way one compromised client means all clients are insecure'''.<br />
kadmin> addprinc -randkey lustre_root@REALM<br />
kadmin> ktadd -e aes128-cts:normal lustre_root@REALM<br />
<br />
*To merge keytab files, you need the tool '''''ktutil''''', for more details please refers to manual of ktutil.<br />
<br />
*Lustre support following ''enctypes'' for MIT Kerberos 5 version 1.4 or higher:<br />
**<u>''des-cbc-md5''</u><br />
**<u>''des3-hmac-sha1''</u><br />
**<u>''aes128-cts''</u><br />
**<u>''aes256-cts''</u><br />
<br />
*For MIT Kerberos 1.3.x, only ''des-cbc-md5'' works because a known issue between libgssapi and Kerberos library.<br />
<br />
== Required packages ==<br />
Every node should have follow packages installed:<br />
* '''''libgssapi''''' version 0.10 or higher. Some newer Linux distributions already come with it. If not, build & install from source: http://www.citi.umich.edu/projects/nfsv4/linux/libgssapi/libgssapi-0.11.tar.gz<br />
* '''''keyutils'''''<br />
<br />
== Kernel & Environment ==<br />
* System wide configuration:<br />
On Each node (MDT, OST, Client) following line should be added into /etc/fstab to be automatically mounted<br />
nfsd /proc/fs/nfsd nfsd defaults 0 0 <br />
Each MDT and Client node add following line into /etc/request-key.conf:<br />
create lgssc * * /usr/sbin/lgss_keyring %o %k %t %d %c %u %g %T %P %S<br />
Note you might need to replace '''/usr/sbin/lgss_keyring''' in above line to the actual path to lgss_keyring binary in your setting.<br />
<br />
* Networking:<br />
If you are using network which is '''NOT''' TCP or Infiniband (e.g. Quadrics Elan, Myrinet, etc), you need configure a '''''/etc/lustre/nid2hostname''''' on '''each''' server node (MDT & OST), which is a simple script to translate NID into hostname. Following is sample on a Elan cluster:<br />
<br />
#!/bin/bash<br />
set -x<br />
exec 2>/tmp/$(basename $0).debug<br />
<br />
# convert a NID for a LND to a hostname, for GSS for example<br />
<br />
# called with thre arguments: lnd netid nid<br />
# $lnd will be string "QSWLND", "GMLND", etc.<br />
# $netid will be number in hex string format, like "0x16", etc.<br />
# $nid has the same format as $netid<br />
# output the corresponding hostname, or error message leaded by a '@' for error logging.<br />
<br />
lnd=$1<br />
netid=$2<br />
nid=$3<br />
<br />
# uppercase the hex<br />
nid=$(echo $nid | tr '[abcdef]' '[ABCDEF]')<br />
# and convert to decimal<br />
nid=$(echo -e "ibase=16\n${nid/#0x}" | bc)<br />
case $lnd in<br />
QSWLND) # simply stick "mtn" on the front<br />
echo "mtn$nid"<br />
;;<br />
*) echo "@unknown LND: $lnd"<br />
;;<br />
esac<br />
<br />
== Build Lustre ==<br />
Enable GSS during configuration:<br />
<br />
./configure --enable-gss --other-options<br />
<br />
== Running ==<br />
=== GSS Daemons ===<br />
Make sure start the daemon process '''lsvcgssd''' on each OST and MDT node before starting Lustre. The command syntax is:<br />
lsvcgssd [-f] [-v]<br />
* ''-f'': running at foreground instead of as daemon, thus output error/warning messages to front console instead of system log.<br />
* ''-v'': increase verbosity by 1. The default is 0, maximum is 4.<br />
<br />
=== Setting Security Flavors ===<br />
Note: If nothing specified, by default all RPC connections will use '''''null'''''.<br />
<br />
On MGS there's a persistent sptlrpc rule database, by specifying set of rules you can change security flavors between nodes. A rule is in form of:<br />
<spec>=<flavor><br />
Rules can be manipulated on MGS node. To add a rule:<br />
mgs> lctl conf_param <spec>=<flavor><br />
If there a existing rule of <spec> part, it will overwritten.<br />
<br />
To delete a rule:<br />
mgs> lctl conf_param -d <spec><br />
<br />
Current rule set could be obtained by:<br />
msg> cat /proc/fs/lustre/mgs/<mgs-name>/live/<fs-name> | grep "srpc.flavor"<br />
<br />
'''Note''':<br />
* Rules have persistent storage on MGS, so it applied across re-mount.<br />
* It doesn't matter in which order you add a set of rules, lustre keep rules in certain order or priority.<br />
* After you changed a rule, usually it will take the system within 1 minutes to apply the new rules to all nodes, depend on system load.<br />
* Before you change a rule, make sure affected nodes are ready for the new security flavor. E.g. you changed flavor from '''''null''''' to '''''krb5p''''' but GSS/Kerberos env is not properly configured on affected nodes, those nodes might be evicted because they can't communicate with others.<br />
* You can also specify rules via device on-disk parameters, by mke2fs.lustre or tune2fs.lustre. The syntax is the same, and the rule only applied to connections to this specific target (MDT/OST).<br />
<br />
=== Rules Syntax & Examples ===<br />
The general syntax is:<br />
<target>.srpc.flavor.<network>[.<direction>]=flavor<br />
<br />
* <target>: could be filesystem name, or specific MDT/OST device name. For example, ''lustre'', ''lustre-MDT0000'', ''lustre-OST0001'', etc.<br />
* <network>: LNET network name of the RPC initiator. For example, ''tcp0'', ''elan1'', ''o2ib0''.<br />
* <direction>: could be one of ''cli2mdt'', ''cli2ost'', ''mdt2mdt'', ''mdt2ost''. In most cases you don't need to specify <direction> part.<br />
<br />
Examples:<br />
* Apply ''krb5i'' on '''ALL''' connections:<br />
mgs> lctl conf_param lustre.srpc.flavor.default=krb5i<br />
<br />
* Nodes in network ''tcp0'' use ''krb5p''; All other nodes use ''null''<br />
mgs> lctl conf_param lustre.srpc.flavor.tcp0=krb5p<br />
mgs> lctl conf_param lustre.srpc.flavor.default=null<br />
<br />
* Nodes in network ''tcp0'' use ''krb5p''; Nodes in ''elan1'' use ''plain''; Amount other nodes, clients use ''krb5i'' to MDT/OST, MDT use ''null'' to other MDTs, MDT use ''plain'' to OSTs.<br />
mgs> lctl conf_param lustre.srpc.flavor.tcp0=krb5p<br />
mgs> lctl conf_param lustre.srpc.flavor.elan1=plain<br />
mgs> lctl conf_param lustre.srpc.flavor.default.cli2mdt=krb5i<br />
mgs> lctl conf_param lustre.srpc.flavor.default.cli2ost=krb5i<br />
mgs> lctl conf_param lustre.srpc.flavor.default.mdt2mdt=null<br />
mgs> lctl conf_param lustre.srpc.flavor.default.mdt2ost=plain<br />
<br />
=== Authenticate Normal Users ===<br />
On client nodes, a non-root user need '''''kinit''''' before accessing Lustre, just like other Kerberized applications.<br />
* Required by kerberos, the user's principal (''username@REALM'') should be added into KDC.<br />
* Client and MDT nodes should have the same user database, i.e. the user name and uid/gid translation.<br />
A use could destroy the established security contexts before logout, by "lfs flushctx":<br />
<br />
lfs flushctx [-k]<br />
<br />
Here "-k" means also destroy the on-disk kerberos credential cache, equals to "kdestroy", otherwise it only destroy established contexts in Lustre kernel memory.<br />
<br />
== Secure MGC - MGS connection ==<br />
Each node can specify what flavor to use to connect to MGS, by option '''''mgssec=flavor''''' upon mounting a target device or client. By default ''null'' is chosen. Once a flavor is chosen, it can't be changed until umount.<br />
<br />
Because each node has only one connection to MGS, so if there's more than one target device or client on a single node, all the "mgssec=" specification must be the same. Or simply missing option "mgssec=" means "using currently chosen flavor.<br />
<br />
By default, MGS accept RPCs with any flavor. But sysad can configure MGS to only accept certain flavor from certain network. The syntax is similar but with target as a special "_mgs":<br />
mgs> lctl conf_param _mgs.srpc.flavor.<network>=flavor<br />
'''NOTE:''' apply inappropriate flavor may lead to a node never be able to communicate with MGS until restart. So use it carefully.<br />
<br />
== Cross-Realms Authentication ==<br />
Due to idmap functionality is missing, we don't support cross-realm authentication currently.</div>Delphia Mulcahey