Python library 'unittest': Generate multiple tests programmatically

As often with Python, there is a complicated way to provide a simple solution.

In that case, we can use metaprogramming, decorators, and various nifty Python tricks to achieve a nice result. Here is what the final test will look like:

import unittest

# Some magic code will be added here later

class DummyTest(unittest.TestCase):
  @for_examples(1, 2)
  @for_examples(3, 4)
  def test_is_smaller_than_four(self, value):
    self.assertTrue(value < 4)

  @for_examples((1,2),(2,4),(3,7))
  def test_double_of_X_is_Y(self, x, y):
    self.assertEqual(2 * x, y)

if __name__ == "__main__":
  unittest.main()

When executing this script, the result is:

..F...F
======================================================================
FAIL: test_double_of_X_is_Y(3,7)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/xdecoret/Documents/foo.py", line 22, in method_for_example
    method(self, *example)
  File "/Users/xdecoret/Documents/foo.py", line 41, in test_double_of_X_is_Y
    self.assertEqual(2 * x, y)
AssertionError: 6 != 7

======================================================================
FAIL: test_is_smaller_than_four(4)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/xdecoret/Documents/foo.py", line 22, in method_for_example
    method(self, *example)
  File "/Users/xdecoret/Documents/foo.py", line 37, in test_is_smaller_than_four
    self.assertTrue(value < 4)
AssertionError

----------------------------------------------------------------------
Ran 7 tests in 0.001s

FAILED (failures=2)

which achieves our goal:

• it is unobtrusive: we derive from TestCase as usual
• we write parametrized tests only once
• each example value is considered an individual test
• the decorator can be stacked, so it is easy to use sets of examples (e.g., using a function to build the list of values from example files or directories)
• The icing on the cake is it works for arbitrary arity of the signature

So how does it work? Basically, the decorator stores the examples in an attribute of the function. We use a metaclass to replace every decorated function with a list of functions. And we replace the unittest.TestCase with our new magic code (to be pasted in the "magic" comment above) is:

__examples__ = "__examples__"

def for_examples(*examples):
    def decorator(f, examples=examples):
      setattr(f, __examples__, getattr(f, __examples__,()) + examples)
      return f
    return decorator

class TestCaseWithExamplesMetaclass(type):
  def __new__(meta, name, bases, dict):
    def tuplify(x):
      if not isinstance(x, tuple):
        return (x,)
      return x
    for methodname, method in dict.items():
      if hasattr(method, __examples__):
        dict.pop(methodname)
        examples = getattr(method, __examples__)
        delattr(method, __examples__)
        for example in (tuplify(x) for x in examples):
          def method_for_example(self, method = method, example = example):
            method(self, *example)
          methodname_for_example = methodname + "(" + ", ".join(str(v) for v in example) + ")"
          dict[methodname_for_example] = method_for_example
    return type.__new__(meta, name, bases, dict)

class TestCaseWithExamples(unittest.TestCase):
  __metaclass__ = TestCaseWithExamplesMetaclass
  pass

unittest.TestCase = TestCaseWithExamples

If someone wants to package this nicely, or propose a patch for unittest, feel free! A quote of my name will be appreciated.

The code can be made much simpler and fully encapsulated in the decorator if you are ready to use frame introspection (import the sys module)

def for_examples(*parameters):

  def tuplify(x):
    if not isinstance(x, tuple):
      return (x,)
    return x

  def decorator(method, parameters=parameters):
    for parameter in (tuplify(x) for x in parameters):

      def method_for_parameter(self, method=method, parameter=parameter):
        method(self, *parameter)
      args_for_parameter = ",".join(repr(v) for v in parameter)
      name_for_parameter = method.__name__ + "(" + args_for_parameter + ")"
      frame = sys._getframe(1)  # pylint: disable-msg=W0212
      frame.f_locals[name_for_parameter] = method_for_parameter
    return None
  return decorator

I had to do something similar. I created simple TestCase subclasses that took a value in their __init__, like this:

class KnownGood(unittest.TestCase):
    def __init__(self, input, output):
        super(KnownGood, self).__init__()
        self.input = input
        self.output = output
    def runTest(self):
        self.assertEqual(function_to_test(self.input), self.output)

I then made a test suite with these values:

def suite():
    suite = unittest.TestSuite()
    suite.addTests(KnownGood(input, output) for input, output in known_values)
    return suite

You can then run it from your main method:

if __name__ == '__main__':
    unittest.TextTestRunner().run(suite())

The advantages of this are:

• As you add more values, the number of reported tests increases, which makes you feel like you are doing more.
• Each individual test case can fail individually
• It's conceptually simple, since each input/output value is converted into one TestCase

Not tested:

class TestPreReqs(unittest.TestCase):
    ...

def create_test (pair):
    def do_test_expected(self):
        self.assertEqual(under_test(pair[0]), pair[1])
    return do_test_expected

for k, pair in enumerate ([(23, 55), (4, 32)]):
    test_method = create_test (pair)
    test_method.__name__ = 'test_expected_%d' % k
    setattr (TestPreReqs, test_method.__name__, test_method)

If you use this often, you could prettify this by using utility functions and/or decorators, I guess. Note that pairs are not an attribute of TestPreReqs object in this example (and so setUp is gone). Rather, they are "hardwired" in a sense to the TestPreReqs class.