Coding Guidelines

(Updated: Jan 2010)

Beautiful Code

A note from Eric Barton, a Lustre pioneer:

More important than the physical layout of code (which is covered in detail below) is the idea that the code should be beautiful to read.

What makes code beautiful to me? Fundamentally, it's readability and obviousness. The code must not have secrets but should flow easily, pleasurably and accurately off the page and into the mind of the reader.

How do I think beautiful code is written? Like this...

• The author must be confident and knowledgeable and proud of her work. She must understand what the code should do, the environment it must work in, all the combinations of inputs, all the valid outputs, all the possible races and all the reachable states. She must grok it.

• Names must be well chosen. The meaning a human reader attaches to a name can be orthogonal to what the compiler does with it, so it's just as easy to mislead as it is to inform. "Does exactly what it says on the tin" is a popular UK English expression describing something that does exactly what it tells you it's going to do, no more and no less. For example, if I open a tin labeled "soap", I expect the contents to help me wash and maybe even smell nice. If it's no good at removing dirt, I'll be disappointed. If it removes the dirt but burns off a layer of skin with it, I'll be positively upset. The name of a procedure, a variable or a structure member should tell you something informative about the entity without misleading - just "what it says on the tin".

• Names must be well chosen. Local, temporary variables can almost always remain relatively short and anonymous, while names in global scope must be unique. In general, the wider the context you expect to use the name in, the more unique and informative the name should be. Don't be scared of long names if they help to make_the_code_clearer, but do_not_let_things_get_out_of_hand either - we don't write COBOL. Related names should be obvious, unambiguous and avoid naming conflicts with other unrelated names, e.g. by using a consistent prefix. This applies to all API procedures (if not all procedures period) within a given subsystem. Similarly, unique member names for global structures, using a prefix to identify the parent structure type, helps readability.

• Names must be well chosen. Don't choose names that are easily confused - especially not if the compiler can't even tell the difference when you make a spelling mistake. i and j aren't the worst example - rq_reqmsg and rq_repmsg are much worse (and taken from our own code!!!).

• Names must be well chosen. I can't emphasize this issue enough - I hope you get the point.

• Assertions must be used intelligently. They combine the roles of active comment and software fuse. As an active comment they tell you something about the program that you can trust more than a comment. And as a software fuse, they provide fault isolation between subsystems by letting you know when and where invariant assumptions are violated. Overuse must be avoided - it hurts performance without helping readability - and any other use is just plain wrong. For example, assertions must never be used to validate data read from disk or the network. Network and disk hardware does fail and Lustre has to handle that - it can't just crash. The same goes for user input. Checking data copied in from userspace with assertions just opens the door for a denial of service attack.

• Formatting and indentation rules should be followed intelligently. The visual layout of the code on the page should lend itself to being read easily and accurately - it just looks clean and good.
  • Separate "ideas" should be separated clearly in the code layout using blank lines that group related statements and separate unrelated statements.
  • Procedures should not ramble on. You must be able to take in the meaning of a procedure without scrolling past page after page of code or parsing deeply nested conditionals and loops. The 80-column rule is there for a reason.
  • Declarations are easier to refer to while scanning the code if placed in a block locally to, but visually separate from, the code that uses them. Readability is further enhanced by limiting declarations to one per line and aligning types and names vertically.
  • Parameters in multi-line procedure calls should be aligned so that they are visually contained by their brackets.
  • Brackets should be used in complex expressions to make operator precedence clear.
  • Conditional boolean (if (expr)), scalar (if (val != 0)) and pointer (if (ptr != NULL)) expressions should be written consistently.
  • Formatting and indentation rules should not be followed slavishly. If you're faced with either breaking the 80-chars-per-line rule or the parameter indentation rule or creating an obscure helper function, then the 80-chars-per-line rule might have to suffer. The overriding consideration is how the code reads.

I could go on, but I hope you get the idea. Just think about the poor reader when you're writing, and whether your code will convey its meaning naturally, quickly and accurately, without room for misinterpretation.

I didn't mention clever as a feature of beautiful code because it's only one step from clever to tricky - consider...

  t = a; a = b; b = t; /* dumb swap */

  a ^= b; b ^= a; a ^= b;  /* clever swap */

You could feel quite pleased that the clever swap avoids the need for a local temporary variable - but is that such a big deal compared with how quickly, easily and accurately the reader will read it? This is a very minor example which can almost be excused because the "cleverness" is confined to a tiny part of the code. But when clever code gets spread out, it becomes much harder to modify without adding defects. You can only work on code without screwing up if you understand the code and the environment it works in completely. Or to put it more succinctly...

Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it. - Brian W. Kernighan

IMHO, beautiful code helps code quality because it improves communication between the code author and the code reader. Since everyone maintaining and developing the code is a code reader as well as a code author, the quality of this communication can lead either to a virtuous circle of improving quality, or a vicious circle of degrading quality. You, dear reader, will determine which.

---

Style and Formatting Guidlelines

All of our rules for formatting, wrapping, parenthesis, brace placement, etc., are originally derived from the Linux kernel rules, which are basically K&R style.

Whitespace

Whitespace gets its own section because unnecessary whitespace changes can cause spurious merge conflicts when code is landed and updated in a distributed development environment. Please ensure that you comply with the guidelines in this section to avoid these issues. We've included default formatting rules for emacs and vim to help make it easier.

• Tabs should be used in all 2.3 and later lustre/, lnet/ and libcfs/ files. This matches the upstream Linux kernel coding style, and is the default method of code indentation.

• NOTE NOTE NOTE The use of tabs for indentation is a reversal from previous Lustre coding guidelines, since May 2012 and Lustre 2.3. This is being done in order to facilitate code integration with the Linux kernel. All new patches should be submitted using tabs for ALL modified lines in the patch. If there are 6 or fewer lines using spaces for indentation between two lines changed by a patch, then all of the intervening lines should also have the indentation changed to use tabs. Similarly, if there are only a handful of lines at the start or end a modified function or test that are still using spaces for indentation, convert all of the lines in that function or test to use tabs. In this manner, we can migrate consistent chunks of code over to tabs without having a 250kLOC patch breaking the commit history of every line of code, and also avoid breaking code that is in existing branches/patches that still need to merge. In a year or so, lines that still use spaces (i.e. those that are not under active development) will be converted to using tabs for indentation as well.

• All lines should wrap at 80 characters. If it's getting too hard to wrap at 80 characters, you probably need to rearrange conditional order or break it up into more functions.

right:

void func_helper(...)
{
        do_sth2_1;

        if (cond3)
	        do_sth_which_needs_a_very_long_line_to_read_clearly;

        do_sth2_2;
}

void func (...)
{
        if (!cond1)
                return;

        do_sth1_1;

        if (cond 2)
                func_helper(...)

        do_sth1_2;
}

wrong:

void func(...)
{
        if (cond1) {
                do_sth1_1;
                if (cond2) {
                        do_sth2_1;
                        if (cond3) {
                                do_sth_which_needs_a_very_long_line_to_read_clearly;
                        }
                        do_sth2_2;
                }
                do_sth1_2;
        }
}

• Do not include spaces or tabs on blank lines or at the end of lines. Please ensure you remove all instances of these in any patches you submit to Bugzilla. You can find them with grep or in vim using the following regexps:

 /[ \t]$/

Alternatively, if you use vim, you can put this line in your vimrc file, which will highlight whitespace at the end of lines and spaces followed by tabs in indentation (only works for C/C++ files):

let c_space_errors=1

Or you can use this command, which will make tabs and whitespace at the end of lines visible for all files (but a bit more discretely):

set list listchars=tab:>\ ,trail:$

In emacs, you can use (whitespace-mode) or (whitespace-visual-mode) depending on the version. You could also consider using (flyspell-prog-mode).

C Language Features

• Don't use inline unless you're doing something so performance critical that the function call overhead will make a difference -- in other words: almost never. It makes debugging harder and overuse can actually hurt performance by causing instruction cache or stack overflow.

• Use typedef carefully...
  • Do not create a new integer typedef without a good reason.
  • Always postfix typedef names with _t so that they can be identified clearly in the code.
  • Never typedef pointers. The * makes C pointer declarations obvious. Hiding it inside a typedef just obfuscates the code.

• Do not embed assignments inside boolean expressions. Although this can make the code more concise, it doesn't necessarily make it more elegant and you increase the risk of confusing "=" with "==" or getting operator precedence wrong if you skimp on brackets. It's even easier to make mistakes when reading the code, so it's much safer simply to avoid it altogether.

 right:
        ptr = malloc(size);
        if (ptr != NULL) {
                ...

 wrong:
        if ((ptr = malloc(size)) != NULL) {
                ...

• Conditional expressions read more clearly if only boolean expressions are implicit (i.e., non-boolean and pointer expressions compare explicitly with 0 and NULL respectively.)

 right:
        if (!writing &&          /* not writing? */
            inode != NULL &&     /* valid inode? */
            ref_count == 0)      /* no more references? */
                do_this();

 wrong:
        if (writing == 0 &&      /* not writing? */
            inode &&             /* valid inode? */
            !ref_count)          /* no more references? */
                do_this();

• Use parentheses to help readability and reduce the chance of operator precedence errors, but not so heavily that it is difficult to determine which parentheses are a matched pair.

 right:
        if (a->a_field == 3 ||
            ((b->b_field & BITMASK1) && (c->c_field & BITMASK2)))
                do this();

 wrong:
        if (a->a_field == 3 || b->b_field & BITMASK1 && c->c_field & BITMASK2)
                 do this()

 wrong:
        if (((a->a_field == 3) || ((b->b_field & (BITMASK1)) &&
            (c->c_field & (BITMASK2)))))
                 do this()

Lustre Guidelines

• The types and printf()/printk() formats used by Lustre code are:

        __u64                LPU64/LPX64/LPD64 (unsigned, hex, signed)
        __u32/int            %u/%x/%d (unsigned, hex, signed)
        (unsigned)long long  %llu/%llx/%lld
        loff_t               %lld after a cast to long long (unfortunately)

• Functions and files should be documented using the Doxygen markup style:

/**
 * Implements cl_page_operations::cpo_make_ready() method for Linux.
 *
 * This is called to yank a page referred to by \a slice from the transfer
 * cache and to send it out as a part of transfer. This function try-locks
 * the page. If try-lock failed, page is owned by some concurrent IO, and
 * should be skipped (this is bad, but hopefully rare situation, as it usually
 * results in transfer being shorter than possible).
 *
 * \param  env     lu environment for large temporary stack variables
 * \param  slice   per-layer page structure being prepared
 * \retval 0       success, page can be placed into transfer
 * \retval -EAGAIN page is either used by concurrent IO has been
 *                 truncated. Skip it.
 */
static int vvp_page_make_ready(const struct lu_env *env,
                               const struct cl_page_slice *slice)
{

• Use list_for_each_entry() instead of list_for_each followed by list_entry

• When using sizeof() it should be used on the variable itself, rather than specifying the type of the variable, so that if the variable changes type/size then sizeof() will be correct:

 right:
         int *array;

         OBD_ALLOC(array, 10 * sizeof(*array));

 wrong:
         OBD_ALLOC(array, 10 * sizeof(int));     /* breaks if array becomes __u64 */

 wrong:
         OBD_ALLOC(array, 10 * sizeof(array));   /* This is the pointer size */
    

• When allocating/freeing a single struct, use OBD_ALLOC_PTR() for clarity:

 right:
         OBD_ALLOC_PTR(mds_body);
         OBD_FREE_PTR(mds_body);

 wrong:
         OBD_ALLOC(mds_body, sizeof(*mds_body));
         OBD_FREE(mds_body, sizeof(*mds_body));

• Do not embed operations inside assertions. If assertions are disabled for performance reasons this code will not be executed.

 right:
        len = strcat(foo, bar);
        LASSERT(len > 0);

 wrong:
        LASSERT(strcat(foo, bar) > 0);

• Messages on the console (CERROR, CWARN, LCONSOLE_*) should print the OBD device name or filesystem name where the error is hit, since there are usually multiple targets running on a single server. The error messages should also include enough information to make some useful diagnosis of the problem (e.g. FID and/or filename, client NID, etc). Otherwise, there is little value in having printed the error, but then having to try and reproduce the problem to diagnose it:

right:
        LCONSOLE_INFO("MDS %s: %s now active, resetting orphans\n",
                      obd->obd_name, obd_uuid2str(uuid));
        LCONSOLE_WARN("%s: new disk, initializing\n", obd->obd_name);
        CERROR("%s: unsupported incompat filesystem feature(s) %x\n", obd->obd_name,

wrong:
        CERROR("Cannot get thandle\n");
        CERROR("NULL bitmap!\n");
        CERROR("invalid event\n");

• Error messages that print a numeric error value should print it at the end of the line in a consistent format:

right:
        CERROR("%s: error invoking upcall %s %s %s: rc = %d",
        CERROR("%s: cannot open/create O: rc = %d\n", obd->obd_name,rc);

wrong:
        CERROR("err %d on param '%s'\n", rc, ptr);
        CERROR("Can't get index (%d)\n", rc);

Layout

• Code can be much more readable if the simpler actions are taken first in a set of tests. Re-ordering conditions like this also eliminates excessive nesting.

right:
        list_for_each_entry(...) {

                if (!condition1) {
                        do_sth1;
                        continue;
                }

                do_sth2_1;
                do_sth2_2;
                ...
                do_sth2_N;

                if (!condition2)
                        break;

                do_sth3_1;
                do_sth3_2;
                ...
                do_sth3_N;
        }
wrong:
        list_for_each_entry(...) {
                if (condition1) {
                        do_sth2_1;
                        do_sth2_2;
                        ...
                        do_sth2_N;
                        if (condition2) {
                                do_sth3_1;
                                do_sth3_2;
                                ...
                                do_sth3_N;
                                continue;
                        }               
                        break;
                } else {
                        do_sth1;
                }
        }

• Variable should be declared one per line, type and name, even if there are multiple variables of the same type. For maximum readability, the names should be aligned on the same column, preferably with longer declarations at the top.

 right:
        int           len;
        int           count;
        struct inode *inode;

 wrong:
        int len, count;
        struct inode *inode;

• Variable declarations should be kept to an internal scope, if practical and reasonable, to simplify understanding of where these variables are used:

 right:
        int len;
        
        if (len > 0) {
                int           count;
                struct inode *inode = iget(foo);

                count = inode->i_size;
                :
        }

• Even for short conditionals, the operation should be on a separate line:

 right:
        if (foo)
                bar();
 wrong:
        if (foo) bar();

• When you wrap a line containing parenthesis, start the next line after the parenthesis so that the expression or argument is visually bracketed.

 right:
        variable = do_something_complicated(long_argument, longer_argument,
                                            longest_argument(sub_argument,
                                                             foo_argument),
                                            last_argument);

        if (some_long_condition(arg1, arg2, arg3) < some_long_value &&
            another_long_condition(very_long_argument_name,
                                   another_long_argument_name) >
            second_long_value) {
                ...

 wrong:

        variable = do_something_complicated(long_argument, longer_argument,
                longest_argument(sub_argument, foo_argument),
                last_argument);

        if (some_long_condition(arg1, arg2, arg3) < some_long_value &&
                another_long_condition(very_long_argument_name,
                        another_long_argument_name) >
                second_long_value) {
                ...

• If you're wrapping an expression, put the operator at the end of the line. If there are no parentheses to which to align the start of the next line, just indent 8 more spaces.

        off = le32_to_cpu(fsd->fsd_client_start) +
                cl_idx * le16_to_cpu(fsd->fsd_client_size);

• Binary and ternary (but not unary) operators should be separated from their arguments by one space.

 right:
        a++;
        b |= c;
        d = (f > g) ? 0 : 1;

• Function calls should be nestled against the parentheses, the parentheses should crowd the arguments, and one space should appear after commas:

 right: 
        do_foo(bar, baz);

 wrong:
        do_foo ( bar,baz );

• Put a space between if, for, while etc. and the following parenthesis. Put a space after each semicolon in a for statement.

 right:
        for (a = 0; a < b; a++)
        if (a < b || a == c)
        while (1)
 wrong:
        for( a=0; a<b; a++ )
        if( a<b || a==c )
        while( 1 )

• Opening braces should be on the same line as the line that introduces the block, except for function calls. Bare closing braces (i.e. not else or while in do/while) get their own line.

int foo(void)
{
        if (bar) {
                this();
                that();
        } else if (baz) {
                stuff();
        } else {
                other_stuff();
        }

        do {
                cow();
        } while (condition);
}

• If one part of a compound if block has braces, all should.

 right:
        if (foo) {
                bar();
                baz();
        } else {
                salmon();
        }

 wrong:
        if (foo) {
                bar();
                baz();
        } else
                moose();

• When you define a macro, protect callers by placing parentheses round every parameter reference in the body. Line up the backslashes of multi-line macros to help readability. Use a do/while (0) block with no trailing semicolon to ensure multi-statement macros are syntactically equivalent to procedure calls.

/* right */
#define DO_STUFF(a)                              \
do {                                             \
        int b = (a) + MAGIC;                     \
        do_other_stuff(b);                       \
} while (0)

/* wrong */
#define DO_STUFF(a) \
do { \
        int b = a + MAGIC; \
        do_other_stuff(b); \
} while (0);

• If you write conditionally compiled code in a procedure body, make sure you do not create unbalanced braces, quotes, etc. This really confuses editors that navigate expressions or use fonts to highlight language features. It can often be much cleaner to put the conditionally compiled code in its own helper function which, by good choice of name, documents itself too.

/* right */
static inline int invalid_dentry(struct dentry *d)
{
#ifdef DCACHE_LUSTRE_INVALID
        return d->d_flags & DCACHE_LUSTRE_INVALID;
#else
        return d_unhashed(d);
#endif
}

int do_stuff(struct dentry *parent)
{
        if (invalid_dentry(parent)) {
                ...
        
/* wrong */
int do_stuff(struct dentry *parent)
{
#ifdef DCACHE_LUSTRE_INVALID
        if (parent->d_flags & DCACHE_LUSTRE_INVALID) {
#else
        if (d_unhashed(parent)) {
#endif
                ...

• If you nest preprocessor commands, use spaces to visually delineate:

#ifdef __KERNEL__
# include <goose>
# define MOOSE steak
#else
# include <mutton>
# define MOOSE prancing
#endif

• For very long #ifdefs, include the conditional with each #endif to make it readable:

#ifdef __KERNEL__
# if LINUX_VERSION_CODE >= KERNEL_VERSION(2,5,0)
/* lots
   of
   stuff */
# endif /* KERNEL_VERSION(2,5,0) */
#else /* !__KERNEL__ */
# if HAVE_FEATURE
/* more
 * stuff */
# endif
#endif /* __KERNEL__ */

• Comments should have the leading '/*' on the same line as the comment and the trailing '*/' at the end of the last comment line. Intermediate lines should start with a '*' aligned with the '*' on the first line:

        /* This is a short comment */

        /* This is a multi-line comment.  I wish the line would wrap already,
         * as I don't have much to write about. */

• Function declarations absolutely should NOT go into .c files, unless they are forward declarations for static functions that can't otherwise be moved before the caller. Instead, the declaration should go into the most "local" header available (preferably *_internal.h for a given piece of code).

• Structure and constant declarations should not be declared in multiple places. Put the struct into the most "local" header possible. If it is something that is passed over the wire, it needs to go into lustre_idl.h and needs to be correctly swabbed when the RPC message is unpacked.

• The types and printf/printk formats used by Lustre code are:

        __u64                 LPU64/LPX64/LPD64 (unsigned, hex, signed)
        size_t                LPSZ (or cast to int and use %u / %d)
        __u32/int             %u/%x/%d (unsigned, hex, signed)
        (unsigned) long long  %llu/%llx/%lld
        loff_t                %lld after a cast to long long (unfortunately)

• For Autoconf macros, follow the style suggested in the autoconf manual.

        AC_CACHE_CHECK([for EMX OS/2 environment], [ac_cv_emxos2],
        [AC_COMPILE_IFELSE([AC_LANG_PROGRAM([], [return __EMX__;])],
                           [ac_cv_emxos2=yes],
                           [ac_cv_emxos2=no])])

or_even

       AC_CACHE_CHECK([for EMX OS/2 environment],
                      [ac_cv_emxos2],
                      [AC_COMPILE_IFELSE([AC_LANG_PROGRAM([],
                                                          [return __EMX__;])],
                                         [ac_cv_emxos2=yes],
                                         [ac_cv_emxos2=no])])

---