Parallel Testing with nose

Note

Use of the multiprocess plugin on python 2.5 or earlier requires the multiprocessing module, available from PyPI and at http://code.google.com/p/python-multiprocessing/.

Using the nose.plugins.multiprocess plugin, you can parallelize a test run across a configurable number of worker processes. While this can speed up CPU-bound test runs, it is mainly useful for IO-bound tests that spend most of their time waiting for data to arrive from someplace else and can benefit from parallelization.

How tests are distributed

The ideal case would be to dispatch each test to a worker process separately, and to have enough worker processes that the entire test run takes only as long as the slowest test. This ideal is not attainable in all cases, however, because many test suites depend on context (class, module or package) fixtures.

Some context fixtures are re-entrant – that is, they can be called many times concurrently. Other context fixtures can be shared among tests running in different processes. Still others must be run once and only once for a given set of tests, and must be in the same process as the tests themselves.

The plugin can’t know the difference between these types of context fixtures unless you tell it, so the default behavior is to dispatch the entire context suite to a worker as a unit. This way, the fixtures are run once, in the same process as the tests. (That, of course, is how they are run when the plugin is not active: All tests are run in a single process.)

Controlling distribution

There are two context-level variables that you can use to control this default behavior.

If a context’s fixtures are re-entrant, set _multiprocess_can_split_ = True in the context, and the plugin will dispatch tests in suites bound to that context as if the context had no fixtures. This means that the fixtures will execute multiple times, typically once per test, and concurrently.

For example, a module that contains re-entrant fixtures might look like:

_multiprocess_can_split_ = True

def setup():
    ...

A class might look like:

class TestClass:
    _multiprocess_can_split_ = True

    @classmethod
    def setup_class(cls):
        ...

Alternatively, if a context’s fixtures may only be run once, or may not run concurrently, but may be shared by tests running in different processes – for instance a package-level fixture that starts an external http server or initializes a shared database – then set _multiprocess_shared_ = True in the context. Fixtures for contexts so marked will execute in the primary nose process, and tests in those contexts will be individually dispatched to run in parallel.

A module with shareable fixtures might look like:

_multiprocess_shared_ = True

def setup():
    ...

A class might look like:

class TestClass:
    _multiprocess_shared_ = True

    @classmethod
    def setup_class(cls):
        ...

These options are mutually exclusive: you can’t mark a context as both splittable and shareable.

Example

Consider three versions of the same test suite. One is marked _multiprocess_shared_, another _multiprocess_can_split_, and the third is unmarked. They all define the same fixtures:

called = []

def setup():
print “setup called” called.append(‘setup’)
def teardown():
print “teardown called” called.append(‘teardown’)

And each has two tests that just test that setup() has been called once and only once.

When run without the multiprocess plugin, fixtures for the shared, can-split and not-shared test suites execute at the same times, and all tests pass.

Note

The run() function in nose.plugins.plugintest reformats test result output to remove timings, which will vary from run to run, and redirects the output to stdout.

>>> from nose.plugins.plugintest import run_buffered as run
>>> import os
>>> support = os.path.join(os.path.dirname(__file__), 'support')
>>> test_not_shared = os.path.join(support, 'test_not_shared.py')
>>> test_shared = os.path.join(support, 'test_shared.py')
>>> test_can_split = os.path.join(support, 'test_can_split.py')

The module with shared fixtures passes.

>>> run(argv=['nosetests', '-v', test_shared]) 
setup called
test_shared.TestMe.test_one ... ok
test_shared.test_a ... ok
test_shared.test_b ... ok
teardown called

----------------------------------------------------------------------
Ran 3 tests in ...s

OK

As does the module with no fixture annotations.

>>> run(argv=['nosetests', '-v', test_not_shared]) 
setup called
test_not_shared.TestMe.test_one ... ok
test_not_shared.test_a ... ok
test_not_shared.test_b ... ok
teardown called

----------------------------------------------------------------------
Ran 3 tests in ...s

OK

And the module that marks its fixtures as re-entrant.

>>> run(argv=['nosetests', '-v', test_can_split]) 
setup called
test_can_split.TestMe.test_one ... ok
test_can_split.test_a ... ok
test_can_split.test_b ... ok
teardown called

----------------------------------------------------------------------
Ran 3 tests in ...s

OK

However, when run with the --processes=2 switch, each test module behaves differently.

>>> from nose.plugins.multiprocess import MultiProcess

The module marked _multiprocess_shared_ executes correctly, although as with any use of the multiprocess plugin, the order in which the tests execute is indeterminate.

First we have to reset all of the test modules.

>>> import sys
>>> sys.modules['test_not_shared'].called[:] = []
>>> sys.modules['test_can_split'].called[:] = []

Then we can run the tests again with the multiprocess plugin active.

>>> run(argv=['nosetests', '-v', '--processes=2', test_shared],
...     plugins=[MultiProcess()]) 
setup called
test_shared.... ok
teardown called

----------------------------------------------------------------------
Ran 3 tests in ...s

OK

As does the one not marked – however in this case, --processes=2 will do nothing at all: since the tests are in a module with unmarked fixtures, the entire test module will be dispatched to a single runner process.

However, the module marked _multiprocess_can_split_ will fail, since the fixtures are not reentrant. A module such as this must not be marked _multiprocess_can_split_, or tests will fail in one or more runner processes as fixtures are re-executed.

We have to reset all of the test modules again.

>>> import sys
>>> sys.modules['test_not_shared'].called[:] = []
>>> sys.modules['test_can_split'].called[:] = []

Then we can run again and see the failures.

>>> run(argv=['nosetests', '-v', '--processes=2', test_can_split],
...     plugins=[MultiProcess()]) 
setup called
teardown called
...
test_can_split....
...
FAILED (failures=...)

Other differences in test running

The main difference between using the multiprocess plugin and not doing so is obviously that tests run concurrently under multiprocess. However, there are a few other differences that may impact your test suite:

  • More tests may be found

    Because tests are dispatched to worker processes by name, a worker process may find and run tests in a module that would not be found during a normal test run. For instance, if a non-test module contains a test-like function, that function would be discovered as a test in a worker process if the entire module is dispatched to the worker. This is because worker processes load tests in directed mode – the same way that nose loads tests when you explicitly name a module – rather than in discovered mode, the mode nose uses when looking for tests in a directory.

  • Out-of-order output

    Test results are collected by workers and returned to the master process for output. Since different processes may complete their tests at different times, test result output order is not determinate.

  • Plugin interaction warning

    The multiprocess plugin does not work well with other plugins that expect to wrap or gain control of the test-running process. Examples from nose’s builtin plugins include coverage and profiling: a test run using both multiprocess and either of those is likely to fail in some confusing and spectacular way.

  • Python 2.6 warning

    This is unlikely to impact you unless you are writing tests for nose itself, but be aware that under python 2.6, the multiprocess plugin is not re-entrant. For example, when running nose with the plugin active, you can’t use subprocess to launch another copy of nose that also uses the multiprocess plugin. This is why this test is skipped under python 2.6 when run with the --processes switch.