WARNING: This is the _old_ Lustre wiki, and it is in the process of being retired. The information found here is all likely to be out of date. Please search the new wiki for more up to date information.

Difference between revisions of "POSIX Compliance Testing"

From Obsolete Lustre Wiki
Jump to navigationJump to search
 
(45 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 +
<small>''(Updated: Sep 2009)''</small>
 +
__TOC__
 
This page provides instructions for installing and running the POSIX compliance suite of file system tests.
 
This page provides instructions for installing and running the POSIX compliance suite of file system tests.
 +
 +
'''''Note:''''' The Lustre™ code is not completely POSIX compliant, so test results may show some errors. If you have questions about the test results, submit a request for assistance through [[Reporting Bugs|Bugzilla]].
  
 
== Quick Start ==
 
== Quick Start ==
  
Several versions of the POSIX compliance suite are provided on our FTP site (ftp://ftp.lustre.org:/pub/benchmarks/posix/). Each version is gcc and architecture specific. You will need to determine which version of gcc you are running locally ({{{gcc -v}}})[[Is this correct?]] and then download the appropriate tarball.
+
Several versions of the POSIX compliance suite available to  [http://downloads.lustre.org/public/tools/benchmarks/posix/ download]. Each version is gcc and architecture specific. You will need to determine which version of gcc you are running locally (''{{{gcc -v}}}'') and then download the appropriate tarball.
  
If a package isn't available for your particular combination of gcc+archtecture, follow the [[#Detailed Instructions|Detailed Instructions]] provided below.
+
If a package isn't available for your particular combination of gcc+architecture, see [[POSIX Compliance Testing#Building and Running a POSIX Compliance Test Suite on Lustre|Building and Running a POSIX Compliance Test Suite on Lustre]].
  
 
The following quick start versions are provided:
 
The following quick start versions are provided:
* [ftp://ftp.lustre.org:/pub/benchmarks/posix/one-step-gcc2.96-i686.tgz one-step-gcc2.96-i686.tgz]
+
* [http://downloads.lustre.org/public/tools/benchmarks/posix/one-step-gcc2.96-i686.tgz one-step-gcc2.96-i686.tgz]
* [ftp://ftp.lustre.org:/pub/benchmarks/posix/one-step-gcc2.96-ia64.tgz one-step-gcc2.96-ia64.tgz]
+
* [http://downloads.lustre.org/public/tools/benchmarks/posix//one-step-gcc2.96-ia64.tgz one-step-gcc2.96-ia64.tgz]
* [ftp://ftp.lustre.org:/pub/benchmarks/posix/one-step-gcc3.04-i686.tgz one-step-gcc3.04-i686.tgz]
+
* [http://downloads.lustre.org/public/tools/benchmarks/posix/one-step-gcc3.04-i686.tgz one-step-gcc3.04-i686.tgz]
* [ftp://ftp.lustre.org:/pub/benchmarks/posix/one-step-gcc3.2-i686.tgz one-step-gcc3.2-i686.tgz]
+
* [http://downloads.lustre.org/public/tools/benchmarks/posix/one-step-gcc3.2-i686.tgz one-step-gcc3.2-i686.tgz]
  
 
=== Installing Using the Quick Start Script ===
 
=== Installing Using the Quick Start Script ===
1. Download the test suite and quick start script into ''/usr/src/posix'':
+
1. Download the following scripts into ''/usr/src/posix'':
* one-step-gcc<gcc version>-<arch>.tgz
+
* Test script: ''one-step-gcc<gcc version>-<arch>.tgz''
* one-step-setup.sh
+
* Quick start script: ''one-step-setup.sh''
Both are available from ftp://ftp.lustre.org:/pub/benchmarks/posix/ [[This link doesn't work]]
+
Both are available at [http://downloads.lustre.org/public/tools/benchmarks/posix/ downloads.lustre.org/public/tools/benchmarks/posix/].
  
 
2. Run the setup script by entering:
 
2. Run the setup script by entering:
Line 33: Line 37:
 
</pre>  
 
</pre>  
  
'''''Note:''''' The quick start procedure only works with the paths ''/home/tet'' and ''/mnt/lustre''. If you want to change the paths, follow the [[#Bulding from Scratch|Building from Scratch]] instructions below and create a new tarball.
+
'''''Note:''''' The quick start procedure only works with the paths ''/home/tet'' and ''/mnt/lustre''. If you want to change the paths, follow the [[POSIX Compliance Testing#Building from Scratch|Building from Scratch]] instructions below and create a new tarball.
  
 
=== Running the Test Suite===
 
=== Running the Test Suite===
Line 43: Line 47:
 
</pre>
 
</pre>
  
== Detailed Instructions ==
+
== Building and Running a POSIX Compliance Test Suite on Lustre ==
 +
This section describes how to build and run a POSIX compliance test suite for a compiler and architecture for which we don't provide a quick start package.
  
 
=== Building from Scratch ===
 
=== Building from Scratch ===
This section describes how to install POSIX for Lustre testing:
+
This section describes how to build a POSIX compliance suite for testing a Lustre file system:
  
1. Download all the POSIX files in ftp://ftp.lustre.org:/pub/benchmarks/posix/
+
1. Download all the POSIX files in [http://downloads.lustre.org/public/tools/benchmarks/posix/ downloads.lustre.org/public/tools/benchmarks/posix/]
  
*''tet_vsxgen_2.0.tgz''
+
*''tet_vsxgen_3.02.tgz''
*''lts_vsx-pcts-1.0.1.2.tgz''
+
*''lts_vsx-pcts2.0beta2.tgz''
 
*''install.sh''
 
*''install.sh''
 
*''myscen.bld''
 
*''myscen.bld''
 
*''myscen.exec''
 
*''myscen.exec''
 +
 +
'''''Note:''''' We now use the latest release of the [http://www.opengroup.org/testing/linux-test/lsb-vsx.html LSB-VSX POSIX test suite] (''lts_vsx-pcts2.0beta2.tgz'') and the generic TET/VSXgen framework (''tet_vsxgen_3.02.tgz''). In this release, the issue of "''getgroups() did not return NGROUPS_MAX''" has been fixed.
  
 
2. ''DO NOT'' configure or mount a Lustre filesystem yet.
 
2. ''DO NOT'' configure or mount a Lustre filesystem yet.
  
3. Run the {{{install.sh}}} script and select ''/home/tet'' for the root directory for the test suite installation.  Say 'y' to installing the users and groups. Accept the defaults for the packages to install.
+
3. Run the ''{{{install.sh}}}'' script and select ''/home/tet'' for the root directory for the test suite installation.  Say 'y' to installing the users and groups. Accept the defaults for the packages to install.
  
 
4. Create a temporary directory for holding the POSIX tests while they are being built:
 
4. Create a temporary directory for holding the POSIX tests while they are being built:
Line 76: Line 83:
 
Most of the default answers are correct, except the root directory from which to run the testsets. For this you should specify ''/mnt/lustre/TESTROOT''. For "Install pseudolanguages?", answer 'n'.
 
Most of the default answers are correct, except the root directory from which to run the testsets. For this you should specify ''/mnt/lustre/TESTROOT''. For "Install pseudolanguages?", answer 'n'.
  
7. When it asks you "Install scripts into TESTROOT/BIN..?", do not answer right away. Using another terminal -- because stopping the script doesn't work -- replace the files ''/home/tet/test_sets/scen.exec'' and ''/home/tet/test_sets/scen.bld'' with ''myscen.exec'' and ''myscen.bld'' which you downloaded, i.e:
+
7. When the script asks you "Install scripts into TESTROOT/BIN..?", do not stop the script from running (this doesn't work). Instead, use another terminal to replace existing files with files you downloaded by entering:
*''cp .../myscen.bld  /home/tet/test_sets/scen.bld''
+
<pre>
*''cp .../myscen.exec /home/tet/test_sets/scen.exec''
+
cp .../myscen.bld  /home/tet/test_sets/scen.bld
This will limit the tests that are run to those relevant for filesystems, and avoid '''hours''' of other tests on sockets, math, stdio, libc, shell, etc.
+
cp .../myscen.exec /home/tet/test_sets/scen.exec
 +
</pre>
 +
This will confine the tests that are run to those relevant for file systems, avoiding ''hours'' of running other tests on sockets, math, stdio, libc, shell, etc.
 +
 
 +
8. Continue with the installation at this point. Answer 'y' to the "Build testsets" question.  
  
8. Continue with the installation at this point, and answer 'y' to the "Build testsets" question. It will proceed to build and install all of the filesystem tests, and will then run them all as well. Even though it is running them on a local filesystem, this is a valuable baseline for comparison with the behaviour of Lustre. It should put the results into ''/home/tet/test_sets/results/0002e/journal'', and I suggest renaming or symlinking this directory to ''/home/tet/test_sets/results/ext3/journal'' (or whatever the name of the local filesystem is that the test was run on).
+
The script proceeds to build and install all the file system tests and then runs them all. Even though the script is running them on a local file system, this is a valuable baseline for comparison with the behavior of Lustre.  
  
9. Running the full test should only take 5 minutes or so. Answer 'n' to re-running just the failed tests, and you are done.  The results (in a very lengthy table) are in ''/home/tet/test_sets/results/report''.
+
The results are put into ''/home/tet/test_sets/results/0002e/journal''. It is suggested that you rename or symlink this directory to ''/home/tet/test_sets/results/ext3/journal'' (or the name of the local file system that the test was run on).
  
10. At this point you need to '''save''' the test suite for running further tests on a Lustre filesystem.  Tar up the tests, so we don't have to rebuild each time:
+
Running the full test should only take 5 minutes or so.  
*''tar cvzf TESTROOT.tgz -C /mnt/lustre TESTROOT''
 
  
You will probably also want to remove the installed tests at this time, to save a bit space, and more importantly to avoid confusion if you forget to mount your Lustre filesystem before running the test.
+
9. Answer 'n' to re-running just the failed tests, and you are done.  The results (in a very lengthy table) are in ''/home/tet/test_sets/results/report''.
  
=== Running the Suite against Lustre, and Checking your Results ===
+
10. At this point you need to ''save'' the test suite to use for running further tests on a Lustre file system. Tar up the tests to avoid rebuilding each time by entering:
1. As root, set up your Lustre filesystem, mounted on ''/mnt/lustre'' (e.g. ''sh llmount.sh'' and untar the POSIX tests back to their home:
+
<pre>
*''tar --same-owner -xzpvf /path/to/tarball/TESTROOT.tgz -C /mnt/lustre''
+
tar cvzf TESTROOT.tgz -C /mnt/lustre TESTROOT
 +
</pre>
  
2. As the vsx0 user, you can re-run the tests as many times as you want. If you are newly su'd or logged in as the vsx0 user, you will need to source the environment with '. profile' so that your path and other environment is set up correctly. You run the tests with:
+
You will may also want to remove the installed tests to save space, and more importantly to avoid confusion if you forget to mount your Lustre file system before running the test.
*''. /home/tet/profile''
 
*''tcc -e -s scen.exec -a /mnt/lustre/TESTROOT -p''
 
  
3. Each new result is put in a new directory under ''/home/tet/test_sets/results'' and is given a directory name similar to ''0004e'' (an increasing number and ending with '''e''' for test '''e'''xecution, or '''b''' for '''b'''uilding the tests).  
+
=== Running the POSIX Test Suite against Lustre and checking your results ===
 +
1. As root, set up your Lustre filesystem mounted on ''/mnt/lustre'' (e.g. ''sh llmount.sh'') and untar the POSIX tests back to their home by entering:
 +
<pre>
 +
tar --same-owner -xzpvf /path/to/tarball/TESTROOT.tgz -C /mnt/lustre
 +
</pre>
 +
 
 +
2. As the vsx0 user, you can re-run the tests as many times as you want. If you are newly su'd or logged in as the vsx0 user, you will need to source the environment with '. profile' so that your path and other environment is set up correctly. Run the tests by entering:
 +
<pre>
 +
. /home/tet/profile
 +
tcc -e -s scen.exec -a /mnt/lustre/TESTROOT -p
 +
</pre>
 +
 
 +
3. Each new result is put in a new directory under ''/home/tet/test_sets/results'' and is given a directory name similar to ''0004e'' (an increasing number and ending with ''e'' for test ''e''xecution, or ''b'' for ''b''uilding the tests).  
 
    
 
    
To look at a formatted report:
+
To look at a formatted report, enter:
*''vrpt results/0004e/journal | less''
+
<pre>
 +
vrpt results/0004e/journal | less
 +
</pre>
 
    
 
    
 
Some tests are "Unsupported", "Untested", or "Not In Use", which does not necessarily indicate a problem.  
 
Some tests are "Unsupported", "Untested", or "Not In Use", which does not necessarily indicate a problem.  
  
 
To compare two test results, use the command:
 
To compare two test results, use the command:
*''vrptm results/ext3/journal results/0004e/journal | less''
+
<pre>
 +
vrptm results/ext3/journal results/0004e/journal | less
 +
</pre>
 
    
 
    
This is more interesting than looking at the result of a single test since it helps to find test failures that are specific to the filesystem instead of the Linux VFS or kernel. Up to 6 test results can be compared at once. It is often useful to rename the results directory to have more interesting names so that they have meaning in the future (e.g. ''before_unlink_fix'', or similar).
+
This is more interesting than looking at the result of a single test since it helps to find test failures that are specific to the file system instead of the Linux VFS or kernel. Up to 6 test results can be compared at once.  
 +
 
 +
It is often useful to rename the results directory to have more meaningful names (such as ''before_unlink_fix'').
  
 
=== Isolating and Debugging Failures ===
 
=== Isolating and Debugging Failures ===
  
1. When failures happen, you'll need to gather some more information about what is happening at runtime. For example, some tests may cause kernel panics depending on your configuration. The POSIX suite doesn't have debugging enabled by default, so it's useful to turn on the debugging options of VSX. There are two debug options of note in the config file ''tetexec.cfg'', under your TESTROOT directory:
+
When failures occur, you'll need to gather more information about what is happening at runtime. For example, some tests may cause kernel panics depending on your configuration.
:i. ''VSX_DBUG_FILE=output_file'' - If you're running the test under UML with hostfs support, you should use a file on the hostfs as the debug output file. In the case of a crash, the debug output will then be safely written to the debug file. '''NOTE:''' the default value for this option puts the debug log under your test directory in ''/mnt/lustre/TESTROOT'', which is not horribly useful if you do experience a kernel panic and lustre (or your machine) crashes.
+
* The POSIX compliance suite doesn't have debugging enabled by default, so it is useful to turn on the debugging options of VSX. Two debug options of note reside in the config file ''tetexec.cfg'', under your TESTROOT directory:
:ii. ''VSX_DBUG_FLAGS=xxxxx'' - For detailed info about debug flags, please refer to the documentation included with the POSIX suite. The following example will make VSX output ''all'' debug messages:
 
  
    VSX_DBUG_FLAGS=t:d:n:f:F:L:l,2:p:P
+
:*''VSX_DBUG_FILE=output_file'' - If you're running the test under UML with ''hostfs'' support, you should use a file on the ''hostfs'' as the debug output file. In the case of a crash, the debug output will then be safely written to the debug file.
 +
::'''''Note:''''' the default value for this option puts the debug log under your test directory in ''/mnt/lustre/TESTROOT'', which may not be useful if you experience a kernel panic and lustre (or your machine) crashes.
  
2. VSX is based on the TET framework which provides common libraries for VSX. You can also have TET print out verbose debug messages by inserting the option -T when running the tests, e.g.
+
:*''VSX_DBUG_FLAGS=xxxxx'' - For detailed information about debug flags, refer to the documentation included with the POSIX suite. The following example will make VSX output ''all'' debug messages:
+
:<pre>VSX_DBUG_FLAGS=t:d:n:f:F:L:l,2:p:P
    tcc -Tall5 -e -s scen.exec -a /mnt/lustre/TESTROOT -p 2>&1 | tee /tmp/POSIX-command-line-output.log
+
</pre>
   
+
* VSX is based on the TET framework which provides common libraries for VSX. You can have TET print out verbose debug messages by inserting the option -T when running the tests:
3. VSX will print out detailed messages in the report for failed tests. This will include the test strategy, what kind of operations done by the test suite, and what's going wrong. Each subtest (e.g. 'access', 'create') usually contains many single tests; the report will show exactly which single testing fails. In this case, you can find more infomation directly from the VSX source code. For example: the 5th single test of subtest ''chmod'' failed, you could look at the source:
+
<pre>
*''/home/tet/test_sets/tset/POSIX.os/files/chmod/chmod.c''
+
tcc -Tall5 -e -s scen.exec -a /mnt/lustre/TESTROOT -p 2>&1 | tee /tmp/POSIX-command-line-output.log
 
+
  </pre>
which contains a single test array:
+
* VSX will print out detailed messages in the report for failed tests. This will include the test strategy, what kind of operations are done by the test suite, and what's going wrong.  
  
 +
Each subtest (e.g. 'access', 'create') usually contains many single tests. The report will show exactly which single testing fails. In this case, you can find more information directly from the VSX source code. For example, if the 5th single test of subtest ''chmod'' failed, you could look at the source:
 +
<pre>
 +
/home/tet/test_sets/tset/POSIX.os/files/chmod/chmod.c
 +
</pre>
 +
...which contains a single test array:
 +
<pre>
 
     public  struct tet_testlist tet_testlist[] = {
 
     public  struct tet_testlist tet_testlist[] = {
 
         test1, 1,
 
         test1, 1,
Line 153: Line 186:
 
         NULL, 0
 
         NULL, 0
 
     };
 
     };
 +
</pre>
  
 +
If this single test is causing problems, as in the case of a kernel panic, or if you are trying to isolate a single failure, it may be useful to edit the ''tet_testlist'' array down to the single test in question and then recompile the test suite. You can then create a new tarball of the resulting TESTROOT directory, named appropriately (e.g. ''TESTROOT-chmod-5-only.tgz'') and re-run the POSIX suite using the steps above.
  
If this single test is causing real problems, as in the case of a kernel panic, or if you are trying to isolate a single failure, it may be useful to edit the ''tet_testlist'' array down to the single test in question and then recompile the test suite. You can then create a new tarball of the resulting TESTROOT directory, named appropriately (e.g. ''TESTROOT-chmod-5-only.tgz'') and re-run the POSIX suite using the steps above. It may also be helpful to edit the ''scen.exec'' file to only run the test set in question:
+
It may also be helpful to edit the ''scen.exec'' file to only run the test set in question:
 
+
<pre>
 
   all
 
   all
 
         "total tests in POSIX.os 1"
 
         "total tests in POSIX.os 1"
 
         /tset/POSIX.os/files/chmod/T.chmod
 
         /tset/POSIX.os/files/chmod/T.chmod
 +
</pre>
  
 
+
'''''Note:''''' Rebuilding individual POSIX tests is not straightforward due to the reliance on tcc. One option is to substitute edited source files into the source tree while following the manual installation procedure outlined above and let the existing POSIX install scripts do the work. The installation scripts (specifically ''/home/tet/test_sets/run_testsets.sh'') contain relevant commands for building the test suite -- something akin to ''tcc -p -b -s $HOME/scen.bld $*'' -- but these commands may not work outside the scripts. Please let us know if you manage to get better mileage rebuilding these tests.
'''NOTE:''' rebuilding the individual POSIX tests is not exactly straightforward due to the reliance on tcc. I usually end up substituting my edited source files into the source tree while following the manual installation procedure outlined above and letting the existing POSIX install scripts do the work for me. The installation scripts (specifically ''/home/tet/test_sets/run_testsets.sh'') do contain relevant commands for building the test suite --- something akin to ''tcc -p -b -s $HOME/scen.bld $*'' -- but I've never had it work outside of the script I found it in. Please update this documentation if you manage to get better mileage rebuilding these tests.
 

Latest revision as of 14:35, 23 March 2010

(Updated: Sep 2009)

This page provides instructions for installing and running the POSIX compliance suite of file system tests.

Note: The Lustre™ code is not completely POSIX compliant, so test results may show some errors. If you have questions about the test results, submit a request for assistance through Bugzilla.

Quick Start

Several versions of the POSIX compliance suite available to download. Each version is gcc and architecture specific. You will need to determine which version of gcc you are running locally ({{{gcc -v}}}) and then download the appropriate tarball.

If a package isn't available for your particular combination of gcc+architecture, see Building and Running a POSIX Compliance Test Suite on Lustre.

The following quick start versions are provided:

Installing Using the Quick Start Script

1. Download the following scripts into /usr/src/posix:

  • Test script: one-step-gcc<gcc version>-<arch>.tgz
  • Quick start script: one-step-setup.sh

Both are available at downloads.lustre.org/public/tools/benchmarks/posix/.

2. Run the setup script by entering:

cd /usr/src/posix
sh one-step-setup.sh

4. Edit the configuration file /mnt/lustre/TESTROOT/tetexec.cfg to enter appropriate values for your system.

5. Save the TESTROOT for running lustre tests by entering:

cd /mnt/lustre
tar zcvf /usr/src/posix/TESTROOT.tgz TESTROOT

Note: The quick start procedure only works with the paths /home/tet and /mnt/lustre. If you want to change the paths, follow the Building from Scratch instructions below and create a new tarball.

Running the Test Suite

To run the test suite, enter:

su - vxs0
. ../profile
tcc -e -a /mnt/lustre/TESTROOT -s scen.exec -p

Building and Running a POSIX Compliance Test Suite on Lustre

This section describes how to build and run a POSIX compliance test suite for a compiler and architecture for which we don't provide a quick start package.

Building from Scratch

This section describes how to build a POSIX compliance suite for testing a Lustre file system:

1. Download all the POSIX files in downloads.lustre.org/public/tools/benchmarks/posix/

  • tet_vsxgen_3.02.tgz
  • lts_vsx-pcts2.0beta2.tgz
  • install.sh
  • myscen.bld
  • myscen.exec

Note: We now use the latest release of the LSB-VSX POSIX test suite (lts_vsx-pcts2.0beta2.tgz) and the generic TET/VSXgen framework (tet_vsxgen_3.02.tgz). In this release, the issue of "getgroups() did not return NGROUPS_MAX" has been fixed.

2. DO NOT configure or mount a Lustre filesystem yet.

3. Run the {{{install.sh}}} script and select /home/tet for the root directory for the test suite installation. Say 'y' to installing the users and groups. Accept the defaults for the packages to install.

4. Create a temporary directory for holding the POSIX tests while they are being built:

mkdir -p /mnt/lustre/TESTROOT;chown vsx0.vsxg0 !$

5. Log in as the test user:

su - vsx0

6. To build the test suite, run

../setup.sh

Most of the default answers are correct, except the root directory from which to run the testsets. For this you should specify /mnt/lustre/TESTROOT. For "Install pseudolanguages?", answer 'n'.

7. When the script asks you "Install scripts into TESTROOT/BIN..?", do not stop the script from running (this doesn't work). Instead, use another terminal to replace existing files with files you downloaded by entering:

cp .../myscen.bld  /home/tet/test_sets/scen.bld
cp .../myscen.exec /home/tet/test_sets/scen.exec

This will confine the tests that are run to those relevant for file systems, avoiding hours of running other tests on sockets, math, stdio, libc, shell, etc.

8. Continue with the installation at this point. Answer 'y' to the "Build testsets" question.

The script proceeds to build and install all the file system tests and then runs them all. Even though the script is running them on a local file system, this is a valuable baseline for comparison with the behavior of Lustre.

The results are put into /home/tet/test_sets/results/0002e/journal. It is suggested that you rename or symlink this directory to /home/tet/test_sets/results/ext3/journal (or the name of the local file system that the test was run on).

Running the full test should only take 5 minutes or so.

9. Answer 'n' to re-running just the failed tests, and you are done. The results (in a very lengthy table) are in /home/tet/test_sets/results/report.

10. At this point you need to save the test suite to use for running further tests on a Lustre file system. Tar up the tests to avoid rebuilding each time by entering:

tar cvzf TESTROOT.tgz -C /mnt/lustre TESTROOT

You will may also want to remove the installed tests to save space, and more importantly to avoid confusion if you forget to mount your Lustre file system before running the test.

Running the POSIX Test Suite against Lustre and checking your results

1. As root, set up your Lustre filesystem mounted on /mnt/lustre (e.g. sh llmount.sh) and untar the POSIX tests back to their home by entering:

tar --same-owner -xzpvf /path/to/tarball/TESTROOT.tgz -C /mnt/lustre

2. As the vsx0 user, you can re-run the tests as many times as you want. If you are newly su'd or logged in as the vsx0 user, you will need to source the environment with '. profile' so that your path and other environment is set up correctly. Run the tests by entering:

. /home/tet/profile
tcc -e -s scen.exec -a /mnt/lustre/TESTROOT -p

3. Each new result is put in a new directory under /home/tet/test_sets/results and is given a directory name similar to 0004e (an increasing number and ending with e for test execution, or b for building the tests).

To look at a formatted report, enter:

vrpt results/0004e/journal | less

Some tests are "Unsupported", "Untested", or "Not In Use", which does not necessarily indicate a problem.

To compare two test results, use the command:

vrptm results/ext3/journal results/0004e/journal | less

This is more interesting than looking at the result of a single test since it helps to find test failures that are specific to the file system instead of the Linux VFS or kernel. Up to 6 test results can be compared at once.

It is often useful to rename the results directory to have more meaningful names (such as before_unlink_fix).

Isolating and Debugging Failures

When failures occur, you'll need to gather more information about what is happening at runtime. For example, some tests may cause kernel panics depending on your configuration.

  • The POSIX compliance suite doesn't have debugging enabled by default, so it is useful to turn on the debugging options of VSX. Two debug options of note reside in the config file tetexec.cfg, under your TESTROOT directory:
  • VSX_DBUG_FILE=output_file - If you're running the test under UML with hostfs support, you should use a file on the hostfs as the debug output file. In the case of a crash, the debug output will then be safely written to the debug file.
Note: the default value for this option puts the debug log under your test directory in /mnt/lustre/TESTROOT, which may not be useful if you experience a kernel panic and lustre (or your machine) crashes.
  • VSX_DBUG_FLAGS=xxxxx - For detailed information about debug flags, refer to the documentation included with the POSIX suite. The following example will make VSX output all debug messages:
VSX_DBUG_FLAGS=t:d:n:f:F:L:l,2:p:P
  • VSX is based on the TET framework which provides common libraries for VSX. You can have TET print out verbose debug messages by inserting the option -T when running the tests:
tcc -Tall5 -e -s scen.exec -a /mnt/lustre/TESTROOT -p 2>&1 | tee /tmp/POSIX-command-line-output.log
 
  • VSX will print out detailed messages in the report for failed tests. This will include the test strategy, what kind of operations are done by the test suite, and what's going wrong.

Each subtest (e.g. 'access', 'create') usually contains many single tests. The report will show exactly which single testing fails. In this case, you can find more information directly from the VSX source code. For example, if the 5th single test of subtest chmod failed, you could look at the source:

/home/tet/test_sets/tset/POSIX.os/files/chmod/chmod.c

...which contains a single test array:

     public  struct tet_testlist tet_testlist[] = {
         test1, 1,
         test2, 2,
         test3, 3,
         test4, 4,
         test5, 5,
         test6, 6,
         test7, 7,
         test8, 8,
         test9, 9,
         test10, 10,
         test11, 11,
         test12, 12,
         test13, 13,
         test14, 14,
         test15, 15,
         test16, 16,
         test17, 17,
         test18, 18,
         test19, 19,
         test20, 20,
         test21, 21,
         test22, 22,
         test23, 23,
        NULL, 0
     };

If this single test is causing problems, as in the case of a kernel panic, or if you are trying to isolate a single failure, it may be useful to edit the tet_testlist array down to the single test in question and then recompile the test suite. You can then create a new tarball of the resulting TESTROOT directory, named appropriately (e.g. TESTROOT-chmod-5-only.tgz) and re-run the POSIX suite using the steps above.

It may also be helpful to edit the scen.exec file to only run the test set in question:

   all
        "total tests in POSIX.os 1"
        /tset/POSIX.os/files/chmod/T.chmod

Note: Rebuilding individual POSIX tests is not straightforward due to the reliance on tcc. One option is to substitute edited source files into the source tree while following the manual installation procedure outlined above and let the existing POSIX install scripts do the work. The installation scripts (specifically /home/tet/test_sets/run_testsets.sh) contain relevant commands for building the test suite -- something akin to tcc -p -b -s $HOME/scen.bld $* -- but these commands may not work outside the scripts. Please let us know if you manage to get better mileage rebuilding these tests.