Testing Lustre Code

We recommend a "test early, test often" approach to testing. If you are developing a new feature for Lustre, designing tests to exercise the new feature early in the development process will allow you to test your code as you develop it.

If you are fixing a bug in Lustre, creating a regression test up front will ensure that you can reproduce the reported problem and then verify that it has been fixed. And it will save you the effort of testing the fix manually and then creating a separate regression test later to submit with your bug fix.

We provide several tools to help with testing Lustre code:
 * Acceptance test suite. A Lustre testing framework from which a suite of acceptance tests called "acceptance-small" can be run is described in the next section.
 * POSIX compliance test suite.Instructions for using a POSIX Compliance Suite for texting a Lustre file system is described in POSIX Testing. How is this used?

Using the Lustre Testing Framework
Before you submit code, it must pass the Lustre acceptance test suite, called "acceptance-small". We recommend you run the test suite often so that you can find out as soon as possible if your code changes result in a regression. The acceptance-small test suite is run using the script acceptance-small.sh, which is located in the lustre/tests directory of a compiled Lustre tree. For more details, see Acceptance Small (acc-sm) Testing on Lustre

* This runs a number of test scripts that are also run by the ltest (Buffalo) test harness on Lustre test clusters.

Note: Please set the "acc-sm passed" flag on the attachment for each individual branch that was tested in Bugzilla to indicate this. refer to submitting code topic

Test Scripts
Ensure while working on bugs that there are existing tests that exercise the failing code, or that you have added a new test that exercises the functionality called from one of the existing tests:

In most cases, if a defect is found in Lustre, a valid test does not exist for this functionality. It is easiest to start fixing a bug by developing a test case (a scripted subtest of the above!) that causes the bug to be hit, then fix the bug, and finally verify the bug has been fixed by passing the new test.

Bypassing Failures

 * If you are regularly hitting failures in any of these tests that exist in the parent branch, you should check for a bug on the failure or file a new bug if one has not yet been opened.
 * If the bug is preventing you from completing the tests, then set the environment variables to skip these specific tests until you or someone else fixes them.
 * For example, to skip sanity.sh subtest 36g and 65, replay-single.sh subtest 42, and all of insanity.sh set in your environment:
 * export SANITY_EXCEPT="36g 65"
 * export REPLAY_SINGLE_EXCEPT=42
 * export INSANITY=no
 * You can also skip tests on the command line. For example, when running acceptance-small:
 * SANITY_EXCEPT="36g 65" REPLAY_SINGLE_EXCEPT=42 INSANITY=no ./acceptance-small.sh
 * The test framework is very flexible, and it is a very easy "hands-off" way of running testing while you are doing other things, like coding.
 * Questions/problems with the test framework should be emailed to the lustre-discuss mailing list, so all Lustre users can benefit from improving and documenting it.
 * If you do not run the whole test suite regularly, you have no idea whether a bug is added from your code or not, and you will waste a lot of time looking.
 * You can also skip tests on the command line. For example, when running acceptance-small:
 * SANITY_EXCEPT="36g 65" REPLAY_SINGLE_EXCEPT=42 INSANITY=no ./acceptance-small.sh
 * The test framework is very flexible, and it is a very easy "hands-off" way of running testing while you are doing other things, like coding.
 * Questions/problems with the test framework should be emailed to the lustre-discuss mailing list, so all Lustre users can benefit from improving and documenting it.
 * If you do not run the whole test suite regularly, you have no idea whether a bug is added from your code or not, and you will waste a lot of time looking.
 * The test framework is very flexible, and it is a very easy "hands-off" way of running testing while you are doing other things, like coding.
 * Questions/problems with the test framework should be emailed to the lustre-discuss mailing list, so all Lustre users can benefit from improving and documenting it.
 * If you do not run the whole test suite regularly, you have no idea whether a bug is added from your code or not, and you will waste a lot of time looking.
 * Questions/problems with the test framework should be emailed to the lustre-discuss mailing list, so all Lustre users can benefit from improving and documenting it.
 * If you do not run the whole test suite regularly, you have no idea whether a bug is added from your code or not, and you will waste a lot of time looking.

Test Framework Options
The examples below show how to run a full test or sub-tests from the acceptance-small suite.
 * Run all tests including "standard" tests (sanity*, liblustre) with lov.sh setup; recovery*.sh; and replay*.sh.

$ cd lustre/tests $ sh acceptance-small.sh
 * Run only the recovery-small.sh, replay-single.sh, and conf-sanity.sh tests.

$ ACC_SM_ONLY="recovery-small replay-single conf-sanity" sh acceptance-small.sh


 * Run acceptance-small with a different configuration (expects myth.sh to generate myth.xml).

$ CONFIGS="myth" sh acceptance-small.sh
 * Run only tests 1, 3, 4, 6, 9 in sanity.sh with the lov.sh configuration.

$ ONLY="1 3 4 6 9" NAME=lov sh sanity.sh


 * Skip tests 1 ... 30 and run remaining tests in sanity.sh.

$ EXCEPT="`seq 1 30`" sh sanity.sh


 * Clean up after a lov.sh test failure (normally the system is left mounted for debugging after a failure).

$ NAME=lov sh llmountcleanup.sh


 * Clean up replay-single.sh after a test failure (normally the system is left mounted for debugging after a failure).

$ ONLY=cleanup sh replay-single.sh

Adding New Tests

 * Adding a test to one of the above scripts is easy, and failures can be injected directly into the Lustre kernel code via OBD_FAIL_CHECK, OBD_FAIL_RACE, OBD_FAIL_TIMEOUT, and sysctl -w lustre.fail_loc

Note: You can use the OBD_FAIL_CHECK or OBD_FAIL_TIMEOUT hooks in Lustre to monitor failures.