Best practices for using the Testing Framework

Testing notebooks or .wlt test files? I use both. I create the tests in a notebook, but keep a .wlt file for automated test runs.

This is my workflow:

• I start by creating a new "Testing Notebook" from the file menu
• I add in my testing inputs. For each test, I put all inputs in one cell, and return a single expression.
• Select all the cells and use the "Convert Selection" button to create the tests.
• Assign TestIDs. Obviously having meaningful names here is better, but I tend to just use the randomly generated ones.
• Save the testing notebook in the Tests subdirectory of my package. Then select "Save As .wlt" from the menu, save with the same name as the notebook.

Now I have 2 files in the directory for every test, but that's fine. When I make some code changes, before I commit them I just run the command

TestReport /@ FileNames["~/Path_to_package/*.wlt"]

If the result looks like this,

then everything is good. If I get a failure, then I can copy that TestReportObject and query it directly, asking for "TestsFailed", which returns a list of TestResultObject(s), and I can find the failing test right there. In practice, when I get a test failure I tend to just reopen the notebook, and run the tests interactively to find the cause.

Then when I update the tests in the notebook, I just save as a .wlt file, overwriting the old file.

If you have a build script for any libraries, or paclet build, then you can add this line to the script

successFlag = SameQ[
    Total[
        Map[TestReport[#]["TestsFailedCount"]&,
            FileNames @ "~/Path_to_package/*.wlt"
        ]
    ],
    0
];

And then abort the build if successFlag is False.

With this method you aren't really getting around the issues of file duplication, large files in version control. For my purposes, this isn't a big deal since the notebooks I use aren't that large. This just seems like the workflow that the framework is designed for - we have easily editable testing notebooks, and .wlt files that are less readable, much smaller, and run faster.

I'm actually very new to testing in general, the above is what I've been able to make work so far.

After I have spent some time developing my package MeshTools and experimenting with testing workflows I am coming back to report my experience.

I have put tests (VerificationTest) in a .wl file and this currently satisfies my requirements. I can conveniently edit it in the FrontEnd (see figure below), code is saved in InputForm rather than FullForm and can be easily read in other text editors. Version Control Systems (Git) have no problem with automatic merging and diffs.

During development I regularly run this test file from another development notebook and inspect the results.

(* Assuming I have PacletInfo.m in added directory. *)
PacletDirectoryAdd@ParentDirectory[NotebookDirectory[]]

Get["MeshTools`"]
$testReport=TestReport@FileNameJoin[{NotebookDirectory[],"Tests.wl"}]

These 2 helper function give me a clear overview of the most important information about tests and highlight any failed tests.

getTestResults[tr_TestReportObject]:=Module[
  {fields,results,abbreviations},

   (* Add other querries to this list. *)
  fields={"TestIndex","Outcome","AbsoluteTimeUsed","MemoryUsed","TestID"};
  abbreviations={"TestIndex"->"Idx","AbsoluteTimeUsed"->"Time [s]"};

  results=ReplaceRepeated[
    Outer[#1[#2]&,Values[tr["TestResults"]],fields],
    {x_Quantity:>QuantityMagnitude[x],x_Real:>Round[x,0.001]}
  ];

  Join[{fields/.abbreviations},results]
]

printTestResults[tr_TestReportObject]:=Module[
  {list,indx,time,noTests},

  list=getTestResults[tr];
  indx=MapIndexed[
    If[
        MemberQ[#1,"Failure"|"MessagesFailure"|"Error"],
        First[#2],
        Nothing
    ]&,
    list
  ];
  time=Round[QuantityMagnitude[tr["TimeElapsed"]],0.01];
  noTests=Length[tr["TestResults"]];

  Print[noTests," tests run in ",time," seconds."];

  If[
    TrueQ@tr["AllTestsSucceeded"],
    Print["All tests succeeded!"],
    Print[tr["TestsFailedCount"]," tests failed!"];
  ];

  Print@Grid[list,
    Alignment->Left,
    Dividers->{None,{2->True}},
    Background->{None,Thread[indx->Pink]}
  ];
  tr
]



printTestResults[$testReport];

---

Integration with Git

I have included automatic (unit) testing before every Git commit via hooks. Besides the Tests.wl file the testing folder contains RunTests.wls script with the following content. If any test fails, then script returns value 1 and commit is rejected.

#!/usr/bin/env wolframscript
(* ::Package:: *)

Print["  Running \"MeshTools\" tests...","\n"];

(* Modify paths depending from which directory the script is started.
By default we assume script is started from git root directory. *)
PacletDirectoryAdd@Directory[];
Get["MeshTools`"];

Print["  Using Mathematica: ",$VersionNumber];

Module[
    {report,time,results,failIdx},

    report=TestReport[
        FileNameJoin[{Directory[],"Tests","Tests.wl"}]
    ];
    time=Round[
        QuantityMagnitude@report["TimeElapsed"],
        0.001
    ];
    results=report["TestResults"];
    failIdx=report["TestsFailedIndices"];

    Print["\n","  ",Length[results]," tests run in ",time," seconds."];

    If[
        TrueQ@report["AllTestsSucceeded"]
        ,
        Print["  All tests succeeded!"];
        Quit[0] (* exit code if all test pass *)
        ,
        Print["  ",Length[failIdx]," tests failed!"];
        Do[
            Print["  ",i," / ",results[i]["Outcome"]," / ",results[i]["TestID"]],
            {i,failIdx}
        ];
        Quit[1] (* exit code if any test fails *)
    ]
]

In pre-commit file I have put only the call to .wls script and test are automatically run before every commit.

#!/bin/sh
./Tests/RunTests.wls

In my experience this works well if you keep your unit tests as short as possible (e.g each test <0.1 second) otherwise you may lose patience waiting at each commit. It has already saved me a few times when I was not paying enough attention and some tests have unexpectedly failed.

I think @Jason B.'s answer is a good workflow. Here is an alternative that I have adopted over the years. Most of my code I write in *.m / *.mt / *.wl / *.wlt files using IntelliJ Idea with the Mathematica Plugin, so version control is straightforward.

My projects are written as paclets (see How to distribute Mathematica packages as paclets? for an introduction). Paclets allow you to specify resource files in the PacletInfo.m file. For every (major) function, I write a test-file <function-name>.wlt or <function-name>.mt and add it to the Paclet. Here is an example from my Multiplets paclet:

Paclet[
  Name -> "Multiplets",
  Version -> "0.1",
  MathematicaVersion -> "11.0+",
  Description -> "Defines multiplets and Young tableaux.",
  Thumbnail -> "multiplets.png",
  Creator -> "Johannes E. M. Mosig",
  Extensions -> {
    {"Kernel", Root -> ".", Context -> "Multiplets`"},
    {"Resource", Root -> "Testing", Resources -> {
        "MultipletDimension.mt",
        "MultipletQ.mt",
        "MultipletReduce.mt",
        "Tableau.mt",
        "TableauAppend.mt",
        "TableauClear.mt",
        "TableauDimension.mt",
        "TableauFirst.mt",
        "TableauFromMultiplet.mt",
        "TableauQ.mt",
        "TableauRest.mt",
        "TableauSimplify.mt",
        "TableauToMatrix.mt",
        "TableauToMultiplet.mt"
      }
    }
  }
]

In Mathematica, I can then load my paclet

<< Multiplets`

and check if any given function works as expected

TestReport@PacletResource["Multiplets", "MultipletReduce.mt"]

I can also open the test file within Mathematica, using

NotebookOpen@PacletResource["Multiplets", "MultipletReduce.mt"]

The front end opens *.mt and *.wlt files as plain text files, however, so if the notebook interface is important to you, then you may want to safe them as *.wl files instead.

In addition, one may add a test-script that just calls all other test scripts, for convenience.