Python CLI program unit testing

Start from the user interface with functional tests and work down towards unit tests. It can feel difficult, especially when you use the argparse module or the click package, which take control of the application entry point.

The cli-test-helpers Python package has examples and helper functions (context managers) for a holistic approach on writing tests for your CLI. It's a simple idea, and one that works perfectly with TDD:

1. Start with functional tests (to ensure your user interface definition) and
2. Work towards unit tests (to ensure your implementation contracts)

Functional tests

NOTE: I assume you develop code that is deployed with a setup.py file or is run as a module (-m).

• Is the entrypoint script installed? (tests the configuration in your setup.py)
• Can this package be run as a Python module? (i.e. without having to be installed)
• Is command XYZ available? etc. Cover your entire CLI usage here!

Those tests are simplistic: They run the shell command you would enter in the terminal, e.g.

def test_entrypoint():
    exit_status = os.system('foobar --help')
    assert exit_status == 0

Note the trick to use a non-destructive operation (e.g. --help or --version) as we can't mock anything with this approach.

Towards unit tests

To test single aspects inside the application you will need to mimic things like command line arguments and maybe environment variables. You will also need to catch the exiting of your script to avoid the tests to fail for SystemExit exceptions.

Example with ArgvContext to mimic command line arguments:

@patch('foobar.command.baz')
def test_cli_command(mock_command):
    """Is the correct code called when invoked via the CLI?"""
    with ArgvContext('foobar', 'baz'), pytest.raises(SystemExit):
        foobar.cli.main()

    assert mock_command.called

Note that we mock the function that we want our CLI framework (click in this example) to call, and that we catch SystemExit that the framework naturally raises. The context managers are provided by cli-test-helpers and pytest.

Unit tests

The rest is business as usual. With the above two strategies we've overcome the control a CLI framework may have taken away from us. The rest is usual unit testing. TDD-style hopefully.

Disclosure: I am the author of the cli-test-helpers Python package.

So my question is, what is the best way to do testing with CLI program, can it be as easy as unit testing with normal python scripts?

The only difference is that when you run Python module as a script, its __name__ attribute is set to '__main__'. So generally, if you intend to run your script from command line it should have following form:

import sys

# function and class definitions, etc.
# ...
def foo(arg):
    pass

def main():
    """Entry point to the script"""

    # Do parsing of command line arguments and other stuff here. And then
    # make calls to whatever functions and classes that are defined in your
    # module. For example:
    foo(sys.argv[1])


if __name__ == '__main__':
    main()

Now there is no difference, how you would use it: as a script or as a module. So inside your unit-testing code you can just import foo function, call it and make any assertions you want.

I think it's perfectly fine to test functionally on a whole-program level. It's still possible to test one aspect/option per test. This way you can be sure that the program really works as a whole. Writing unit-tests usually means that you get to execute your tests quicker and that failures are usually easier to interpret/understand. But unit-tests are typically more tied to the program structure, requiring more refactoring effort when you internally change things.

Anyway, using py.test, here is a little example for testing a latin1 to utf8 conversion for pyconv::

# content of test_pyconv.py

import pytest

# we reuse a bit of pytest's own testing machinery, this should eventually come
# from a separatedly installable pytest-cli plugin. 
pytest_plugins = ["pytester"]

@pytest.fixture
def run(testdir):
    def do_run(*args):
        args = ["pyconv"] + list(args)
        return testdir._run(*args)
    return do_run

def test_pyconv_latin1_to_utf8(tmpdir, run):
    input = tmpdir.join("example.txt")
    content = unicode("\xc3\xa4\xc3\xb6", "latin1")
    with input.open("wb") as f:
        f.write(content.encode("latin1"))
    output = tmpdir.join("example.txt.utf8")
    result = run("-flatin1", "-tutf8", input, "-o", output)
    assert result.ret == 0
    with output.open("rb") as f:
        newcontent = f.read()
    assert content.encode("utf8") == newcontent

After installing pytest ("pip install pytest") you can run it like this::

$ py.test test_pyconv.py
=========================== test session starts ============================
platform linux2 -- Python 2.7.3 -- pytest-2.4.5dev1
collected 1 items

test_pyconv.py .

========================= 1 passed in 0.40 seconds =========================

The example reuses some internal machinery of pytest's own testing by leveraging pytest's fixture mechanism, see http://pytest.org/latest/fixture.html. If you forget about the details for a moment, you can just work from the fact that "run" and "tmpdir" are provided for helping you to prepare and run tests. If you want to play, you can try to insert a failing assert-statement or simply "assert 0" and then look at the traceback or issue "py.test --pdb" to enter a python prompt.

Maybe too little too late, but you can always use

import os.system
result =  os.system(<'Insert your command with options here'>
assert(0 == result)

In that way, you can run your program as if it was from command line, and evaluate the exit code.

(Update after I studied pytest) You can also use capsys. (from running pytest --fixtures)

capsys Enable text capturing of writes to sys.stdout and sys.stderr.

The captured output is made available via ``capsys.readouterr()`` method
calls, which return a ``(out, err)`` namedtuple.
``out`` and ``err`` will be ``text`` objects.