Differences between revisions 19 and 21 (spanning 2 versions)
Revision 19 as of 2009-06-05 00:34:59
Size: 8971
Editor: WillMaier
Comment: Note ls -R compatibility issues.
Revision 21 as of 2009-11-14 07:07:38
Size: 9723
Editor: StuartMarks
Comment:
Deletions are marked like this. Additions are marked like this.
Line 8: Line 8:
Line 17: Line 16:

This finds all scripts in the tests/ directory named test-* and executes
them. The scripts can be either shell scripts or Python. Each test is
run in a temporary directory that is removed when the test is complete.
This finds all scripts in the tests/ directory named test-* and executes them. The scripts can be either shell scripts or Python. Each test is run in a temporary directory that is removed when the test is complete.
Line 30: Line 26:

A test-<x> succeeds if the script returns success and its output
matches test-<x>.out. If the new output doesn't match, it is stored in
test-<x>.err.
A test-<x> succeeds if the script returns success and its output matches test-<x>.out. If the new output doesn't match, it is stored in test-<x>.err.
Line 54: Line 47:


Running tests under Windows is a bit harder, see WindowsTestingPlan for details.
One option that comes in handy when running tests repeatedly is `--local`. By default, `run-tests.py` installs Mercurial into its temporary directory for each run of the test suite. You can save several seconds per run with `--local`, which tells `run-tests.py` simply to use the local `hg` script and library. The catch: if you edit the code during a long test suite run, different tests will run with different code. It's best to use `--local` when you are running the same test script many times, as often happens during development.

Running tests under Windows is a bit harder; see WindowsTestingPlan for details.
Line 61: Line 54:

Creating a regression test is easy. Simply create a shell script that executes the necessary commands to exercise Mercurial. 
Creating a regression test is easy. Simply create a shell script that executes the necessary commands to exercise Mercurial.
Line 66: Line 58:
{{{
#!/bin/sh

hg init
touch a
hg add a
hg commit -m "Added a" -d "0 0"

touch main
hg add main
hg commit -m "Added main" -d "0 0"
hg checkout 0

echo Main should be gone
ls
}}}

Then run your test:
{{{#!/bin/sh

hg init touch a hg add a hg commit -m "Added a" -d "0 0"

touch main hg add main hg commit -m "Added main" -d "0 0" hg checkout 0

echo Main should be gone ls }}} Then run your test:
Line 94: Line 75:
Line 103: Line 83:
Line 105: Line 84:
Line 108: Line 86:
{{{
#!python
{{{#!python
Line 153: Line 130:
Line 155: Line 131:
Line 159: Line 134:
Line 166: Line 142:
Line 168: Line 143:
Line 173: Line 149:

There are some tricky points here that you should be aware of when
writing tests:
There are some tricky points here that you should be aware of when writing tests:
Line 181: Line 155:
{{{
cat <<'EOF' > merge
#!/bin/sh
echo merging for `basename $1`
EOF
chmod +x merge

env HGMERGE=./merge hg update -m 1
}}}
{{{#!/bin/sh cat <<'EOF' > merge echo merging for `basename $1` EOF chmod +x merge

env HGMERGE=./merge hg update -m 1 }}}
Line 192: Line 160:
  things like hg history output change - use -d:   . things like hg history output change - use -d:
Line 197: Line 165:
Line 203: Line 170:
Line 205: Line 171:

You also need to be careful that the tests are portable from one platform
to another. You're probably working on Linux, where the GNU toolchain has
more (or different) functionality than on MacOS, *BSD, Solaris, AIX, etc.
While testing on all platforms is the only sure-fire way to make sure that
you've written portable code, here's a list of problems that have been
found and fixed in the tests. Another, more comprehensive list may be
found in the GNU Autoconf manual, online here:

    http://www.gnu.org/software/autoconf/manual/html_node/Portable-Shell.html
You also need to be careful that the tests are portable from one platform to another. You're probably working on Linux, where the GNU toolchain has more (or different) functionality than on MacOS, *BSD, Solaris, AIX, etc. While testing on all platforms is the only sure-fire way to make sure that you've written portable code, here's a list of problems that have been found and fixed in the tests. Another, more comprehensive list may be found in the GNU Autoconf manual, online here:

 . http://www.gnu.org/software/autoconf/manual/html_node/Portable-Shell.html
Line 217: Line 176:

The Bourne shell is a very basic shell. /bin/sh on Linux is typically
bash, which even in Bourne
-shell mode has many features that Bourne shells
on other Unix systems don't have (and even on Linux /bin/sh isn't
guaranteed to be bash
). You'll need to be careful about constructs that
seem ubiquitous, but are actually not available in the least common
denominator. While using another shell (ksh, bash explicitly, posix shell,
etc.) explicitly may seem like another option, these may not exist in a
portable location, and so are generally probably not a good idea. You may
find that rewriting the test in python will be easier.
The Bourne shell is a very basic shell. On Linux, /bin/sh is typically bash, which even in Bourne-shell mode has many features that Bourne shells on other Unix systems don't have. (Note however that on Linux /bin/sh isn't guaranteed to be bash; in particular, on Ubuntu, /bin/sh is dash, a small Posix-compliant shell that lacks many bash features). You'll need to be careful about constructs that seem ubiquitous, but are actually not available in the least common denominator. While using another shell (ksh, bash explicitly, posix shell, etc.) explicitly may seem like another option, these may not exist in a portable location, and so are generally probably not a good idea. You may find that rewriting the test in python will be easier.
Line 232: Line 182:
 * don't use the "function" keyword to define functions; use the old-style form instead:

{{{
# DON'T USE THIS
function foo {
   ...
}

# USE THIS INSTEAD
foo () {
   ...
}
}}}
Line 233: Line 196:
Line 239: Line 201:
Line 248: Line 209:
Line 250: Line 210:
Line 254: Line 213:
Line 258: Line 216:
Line 262: Line 219:
Line 268: Line 224:
Line 274: Line 229:
Line 276: Line 230:
Line 282: Line 235:
Line 286: Line 238:
Line 288: Line 239:
  
Line 290: Line 240:

Mercurial contains a simple regression test framework that allows both Python unit tests and shell-script driven regression tests.

See also: DebuggingTests

Running the test suite

To run the tests, do:

$ make tests
cd tests && ./run-tests.py
............................................
Ran 44 tests, 0 failed.

This finds all scripts in the tests/ directory named test-* and executes them. The scripts can be either shell scripts or Python. Each test is run in a temporary directory that is removed when the test is complete.

You can also run tests individually:

$ cd tests/
$ ./run-tests.py test-pull test-undo
..
Ran 2 tests, 0 failed.

A test-<x> succeeds if the script returns success and its output matches test-<x>.out. If the new output doesn't match, it is stored in test-<x>.err.

Also, run-tests.py has some useful options:

$ ./run-tests.py --help
usage: run-tests.py [options] [tests]

options:
  -h, --help            show this help message and exit
  -v, --verbose         output verbose messages
  -t TIMEOUT, --timeout=TIMEOUT
                        kill errant tests after TIMEOUT seconds
  -c, --cover           print a test coverage report
  -s, --cover_stdlib    print a test coverage report inc. standard libraries
  -C, --annotate        output files annotated with coverage
  -r, --retest          retest failed tests
  -f, --first           exit on the first test failure
  -R, --restart         restart at last error
  -i, --interactive     prompt to accept changed output

One option that comes in handy when running tests repeatedly is --local. By default, run-tests.py installs Mercurial into its temporary directory for each run of the test suite. You can save several seconds per run with --local, which tells run-tests.py simply to use the local hg script and library. The catch: if you edit the code during a long test suite run, different tests will run with different code. It's best to use --local when you are running the same test script many times, as often happens during development.

Running tests under Windows is a bit harder; see WindowsTestingPlan for details.

Note that tests won't run properly with an egg based install of Mercurial; the system install of Mercurial will be used instead of the checked out version. Use a Mercurial installed from source instead to avoid conflicts.

Writing a shell script test

Creating a regression test is easy. Simply create a shell script that executes the necessary commands to exercise Mercurial.

Here's an example:

{{{#!/bin/sh

hg init touch a hg add a hg commit -m "Added a" -d "0 0"

touch main hg add main hg commit -m "Added main" -d "0 0" hg checkout 0

echo Main should be gone ls }}} Then run your test:

$ ./run-tests.py test-example
.
test-example generated unexpected output:
Main should be gone
a

Ran 1 tests, 1 failed.

Double-check your script's output, then save the output so that future runs can check for the expected output:

$ mv test-example.err test-example.out
$ ./run-tests.py test-example
.
Ran 1 tests, 0 failed.

Writing a Python unit test

A unit test operates much like a regression test, but is written in Python. Here's an example:

   1 #!/usr/bin/env python
   2 
   3 import sys
   4 from mercurial import bdiff, mpatch
   5 
   6 def test1(a, b):
   7     d = bdiff.bdiff(a, b)
   8     c = a
   9     if d:
  10         c = mpatch.patches(a, [d])
  11     if c != b:
  12         print "***", `a`, `b`
  13         print "bad:"
  14         print `c`[:200]
  15         print `d`
  16 
  17 def test(a, b):
  18     print "***", `a`, `b`
  19     test1(a, b)
  20     test1(b, a)
  21 
  22 test("a\nc\n\n\n\n", "a\nb\n\n\n")
  23 test("a\nb\nc\n", "a\nc\n")
  24 test("", "")
  25 test("a\nb\nc", "a\nb\nc")
  26 test("a\nb\nc\nd\n", "a\nd\n")
  27 test("a\nb\nc\nd\n", "a\nc\ne\n")
  28 test("a\nb\nc\n", "a\nc\n")
  29 test("a\n", "c\na\nb\n")
  30 test("a\n", "")
  31 test("a\n", "b\nc\n")
  32 test("a\n", "c\na\n")
  33 test("", "adjfkjdjksdhfksj")
  34 test("", "ab")
  35 test("", "abc")
  36 test("a", "a")
  37 test("ab", "ab")
  38 test("abc", "abc")
  39 test("a\n", "a\n")
  40 test("a\nb", "a\nb")
  41 
  42 print "done"

Writing Windows-only tests

Sometimes, it is necessary to write tests which will only run on Windows (for example, testing case sensitivity issues, or cases where os.sep is not '/'). The simplest way of doing this is to write the test as a .bat file. As usual, the output will be compared with the expected output, stored in a file with the .out extension.

Here is a simple example:

@echo off
call hg init
echo hello >a
call hg add a
call hg status

Some things to note:

  • Use @echo off at the start of the file to stop commands being echoed. (Although command echo could be useful to see what commands produce what output - just be careful to add the command echo into the expected output!)

  • You have to use call hg to run Mercurial. If you don't, the test will terminate after the first hg call!. This is because in the test environment, the hg command is implemented as a .bat file, and Windows won't allow one batch file to call another without an explicit call command. Without the call, control is transferred to the hg.bat file, and does not return.

  • Windows has a much more limited set of utilities available by default - so don't assume that things like sed exist. (If you're following WindowsTestingPlan, you will have many of the normal Unix tools, so in practice you can get away with this without too much bother).

Making Tests Repeatable

There are some tricky points here that you should be aware of when writing tests:

  • hg commit wants user interaction - use -m "text"
  • hg up -m wants user interaction, set HGMERGE to something noninteractive:

{{{#!/bin/sh cat <<'EOF' > merge echo merging for basename $1 EOF chmod +x merge

env HGMERGE=./merge hg update -m 1 }}}

  • changeset hashes will change based on user and date which make
    • things like hg history output change - use -d:

hg commit -m "test" -u test -d "0 0"
  • diff will show the current time - strip with sed:

hg diff | sed "s/\(\(---\|+++\) [a-zA-Z0-9_/.-]*\).*/\1/"

Making tests portable

You also need to be careful that the tests are portable from one platform to another. You're probably working on Linux, where the GNU toolchain has more (or different) functionality than on MacOS, *BSD, Solaris, AIX, etc. While testing on all platforms is the only sure-fire way to make sure that you've written portable code, here's a list of problems that have been found and fixed in the tests. Another, more comprehensive list may be found in the GNU Autoconf manual, online here:

1. sh

The Bourne shell is a very basic shell. On Linux, /bin/sh is typically bash, which even in Bourne-shell mode has many features that Bourne shells on other Unix systems don't have. (Note however that on Linux /bin/sh isn't guaranteed to be bash; in particular, on Ubuntu, /bin/sh is dash, a small Posix-compliant shell that lacks many bash features). You'll need to be careful about constructs that seem ubiquitous, but are actually not available in the least common denominator. While using another shell (ksh, bash explicitly, posix shell, etc.) explicitly may seem like another option, these may not exist in a portable location, and so are generally probably not a good idea. You may find that rewriting the test in python will be easier.

  • don't use pushd/popd; save the output of "pwd" and use "cd" in place of the pushd, and cd back to the saved pwd instead of popd.
  • don't use math expressions like let, (( ... )), or $(( ... )); use "expr" instead.
  • don't use the "function" keyword to define functions; use the old-style form instead:

# DON'T USE THIS
function foo {
   ...
}

# USE THIS INSTEAD
foo () {
   ...
}

2. grep

  • don't use the -q option; redirect stdout to /dev/null instead.
  • don't use extended regular expressions with grep; use egrep instead, and don't escape any regex operators.

3. sed

  • make sure that the beginning-of-line matcher ("^") is at the very beginning of the expression -- it may not be supported inside parens.
  • don't use the -i option; instead, redirect to a file:

sed -e 's/foo/bar/' a > a.new
mv a.new a

4. echo

  • echo may interpret "\n" and print a newline; use printf instead if you want a literal "\n" (backslash + n).

5. false

  • false is guaranteed only to return a non-zero value; you cannot depend on it being 1. On Solaris in particular, /bin/false returns 255. Rewrite your test to not depend on a particular return value, or create a temporary "false" executable, and call that instead.

6. diff

  • don't use the -N option. There's no particularly good workaround short of writing a reasonably complicated replacement script, but substituting gdiff for diff if you can't rewrite the test not to need -N will probably do.

7. wc

  • don't use it, or else eliminate leading whitespace from the output:

wc -l | sed -e 's/^ *//'

or use python:

python -c "print len(open('foo').readlines())"

  • don't use the -c option (not part of SUSv3, not supported on OpenBSD). Instead, use dd. the following are equivalent; the latter is preferred:

head -c 20 foo > bar

dd if=foo of=bar bs=1 count=20 2>/dev/null

9. ls

  • don't use the -R option. Instead, use find(1).


CategoryContributing CategoryTesting CategoryHowTo

WritingTests (last edited 2018-04-18 16:35:50 by GregorySzorc)