How to compose all QtTestLib unit tests' results in a single file while using a single test project?
With this trick you can collect the individual test xml reports to temporary buffers/files; all from a single test binary. Lets employ QProcess to collect separate test outputs from within one binary; the test calls itself with modified arguments. First, we introduce a special command-line argument that harnesses the subtests proper - all still within your test executable. For our convenience, we use the overloaded qExec function that accepts a QStringList. Then we can insert/remove our "-subtest" argument more easily.
// Source code of "Test"
int
main( int argc, char** argv )
{
int result = 0;
// The trick is to remove that argument before qExec can see it; As qExec could be
// picky about an unknown argument, we have to filter the helper
// argument (below called -subtest) from argc/argc;
QStringList args;
for( int i=0; i < argc; i++ )
{
args << argv[i];
}
// Only call tests when -subtest argument is given; that will usually
// only happen through callSubtestAndStoreStdout
// find and filter our -subtest argument
size_t pos = args.indexOf( "-subtest" );
QString subtestName;
if( (-1 != pos) && (pos + 1 < args.length()) )
{
subtestName = args.at( pos+1 );
// remove our special arg, as qExec likely confuses them with test methods
args.removeAt( pos );
args.removeAt( pos );
if( subtestName == "test1" )
{
MyFirstTest test1;
result |= QTest::qExec(&test1, args);
}
if( subtestName == "test2" )
{
MySecondTest test2;
result |= QTest::qExec(&test2, args);
}
return result;
}
Then, in your script/commandline call:
./Test -subtest test1 -xml ... >test1.xml
./Test -subtest test2 -xml ... >test2.xml
and here you are - we have the means to separate the tests output. Now we can continue to use QProcess'es ability to collect stdout for you. Just append these lines to your main. The idea is to call our executable again, if no explicit tests are requested, but with our special argument:
bool
callSubtestAndStoreStdout(const String& subtestId, const String& fileNameTestXml, QStringList args)
{
QProcess proc;
args.pop_front();
args.push_front( subtestId );
args.push_front( "-subtest" );
proc.setStandardOutputFile( fileNameTestXml );
proc.start( "./Test", args );
return proc.waitForFinished( 30000 ); // int msecs
}
int
main( int argc, char** argv )
{
.. copy code from main in box above..
callSubtestAndStoreStdout("test1", "test1.xml", args);
callSubtestAndStoreStdout("test2", "test2.xml", args);
// ie. insert your code here to join the xml files to a single report
return result;
}
Then in your script/commandline call:
./Test -xml # will generate test1.xml, test2.xml
Indeed, hopefully future QTestLib versions makes this easier to do.
I've used this dirty workaround (works with Jenkins):
int main(int argc, char *argv[])
{
QCoreApplication a(argc, argv);
int result = 0;
freopen("MyAppTests_Test1.xml", "w", stdout);
result |= QTest::qExec(new Test1, argc, argv);
freopen("MyAppTests_Test2.xml", "w", stdout);
result |= QTest::qExec(new Test2, argc, argv);
return result;
}
Then in Jenkins I've added build action "execute shell": ./path_to_MyAppTests -xml
and added Post-build Actions "publish xUnit test result report" (QTestlib). QTestlib Pattern: MyAppTests*.xml
As I cannot comment here yet, I will post it here in addtition to muenalan's answer. There're few fixes that have to be applied to for it to work (at least with Qt5):
callSubtestAndStoreStdout has 3 bugs. First, the first arg has to be popped from front (this is the arg 0) before pushing new ones. Second, you have to redirect the output before starting the process. Third, it has to return some value ;)
QProcess proc; args.pop_front(); args.push_front(subtestId); args.push_front("-subtest"); proc.setStandardOutputFile(fileNameTestXml); proc.start("sportSystemTest.exe", args); return proc.waitForFinished(30000);
main also has some (obvious) mistakes. The main one is in the if statement:
if ((-1 != pos) && (pos + 1 < args.length()))
as the original one would never fire.
Anyway, thanks for the solution, it solved my big headache :)