Unit testing mathematical code

In my experience, you should use Unit-Testing as a sanity check and a possible regression check. Unit testing should be as thorough as possible of course, but it is sometimes very tedious to make it completely test the full functionality of the code.

Unit Tests are not a formal proof. They cannot and will not fortell future bugs and issues with your code. Test the common use cases of the code. If you need a large amount of reliability, then you'll need to create a large repository of regression tests. Fortunately, for common problems, there are some online databases for that kind of thing. TPLP for instance, is a database of problems (and solutions) to Theorem Provers.

One technique that sometimes works for me... typically in mathematical code, there are "easy but slow" methods, and "fast but hard to program" methods. When writing the code, you want to use the fast but hard to write (so you expect bugs). So... make the fast way the System Under Test (SUT). When you make a Unit Test, create 1000 random problems and solve them with the "easy but slow" method. Then, run the SUT and make sure the answers are analogous.

Assuming of course... that creating random problems is an easy problem to solve. Sometimes it is, sometimes it isn't. Its hard to say without you telling us about the mathematical code itself. Now... Does this get ALL the cases? No. But it will get the "common" cases. And if a corner case pops up in practice, wrap it up in a Unit Test and fix it in the next version of your code.

Don't let testing create a completely unusable and incomprehensible code. And don't overengineer functional being with object-oriented approach.

You're testing a function, i.e. stateless being that produces the same result for the same arguments. And I think that's how you should test it: Give it arguments from all possible equivalence classes and assert on the result.