Any way to get consistent test counts when parallel testing is used?
I just completed a parallel test run. Note that all these tests currently pass when run individually or in the synchronous test mode. After the parallel run I then ran the same SOQL query you are:
select Status, MethodsEnqueued, MethodsCompleted, MethodsFailed
from ApexTestRunResult
where AsyncApexJobId = '7071W00006mnz7p'
That came back with:
- Status: Completed
- MethodsEnqueued: 811
- MethodsCompleted: 797
- MethodsFailed: 13
This aligns somewhat with your findings. E.g. Why doesn't MethodsCompleted + MethodsFailed equal MethodsEnqueued?
A query against ApexTestResult for the Apex Job showed only 797 records.
select
Id, TestTimestamp, Outcome, ApexClassId, MethodName, AsyncApexJobId, QueueItemId, RunTime
from ApexTestResult
where AsyncApexJobId = '7071W00006mnz7p'
And that included the 13 marked as failures. So the first thing we can conclude is that ApexTestRunResult.MethodsFailed is actually included in the ApexTestRunResult.MethodsCompleted count and there are 14 test methods unaccounted for.
I think the important details are in ApexTestQueueItem with the Status Failed.
select
Id,ApexClassId, ApexClass.Name,Status,ExtendedStatus,ParentJobId,TestRunResultId,ShouldSkipCodeCoverage
from ApexTestQueueItem
where ParentJobId = '7071W00006mnz7p'
order by Status desc
There are three test classes that outright failed to run. They failed with the statuses:
- Status: Failed
- ExtendedStatus: "Could not run tests on class 01p40000000HDZA because: connection was cancelled here"
I checked the number of test methods in the classes that failed. They were 2, 1, and 11 respectively. So that explains my missing 14 test methods. Those test methods where neither Completed or Failed. They just didn't run.
For my scenario - There is a data isolation issue when running the tests in parallel that manifests as a "connection was cancelled here" error. As a result, the test methods aren't run and no results come back for them. As it is a concurrency error the number of failures will vary from test run to test run. I've gone into some of the painful details of this in Speeding up Salesforce unit testing performance.
Your scenario is likely similar. You could try adding some sort of guard statement that ApexTestRunResult.ClassesEnqueued equals ApexTestRunResult.ClassesCompleted to make sure entire classes aren't being excluded from the results.
With this being a concurrency issue it would also be useful to control the number of test cases that run at any one time. The current limit of 30 is way to high when there are issues siloing test data. Please consider voting for the idea - Control the degree of parallelism when running apex tests in parallel
I'll need to gather some more data to prove it, but my suspicion is that is caused by using the Steaming API to monitor and report on the results.
It will either be a timing issue between when the test job finishes and when the last CometD message is received or possibly an outright failure to receive a message. Either way, the SFDX CLI command is missing a number of ApexTestResult (07M) records.
I'd take your test ApexJobIds and directly query for the corresponding ApexTestResult records. E.g.
select
Id,TestTimestamp,Outcome,ApexClassId,MethodName,AsyncApexJobId,QueueItemId,RunTime
from ApexTestResult
where AsyncApexJobId = '7071W000052hsrXQAQ'
The other possibility is that the force:apex:test:run command isn't queuing up all the existing test classes. If it is somehow not creating the ApexTestQueueItem records for a subset of Apex classes then those tests would never run. I'd confirm that the expected number of records are created for the ApexJobId.
select Id,ApexClassId,Status,ExtendedStatus,ParentJobId,TestRunResultId,ShouldSkipCodeCoverage
from ApexTestQueueItem
where ParentJobId = '7071W000052hsrXQAQ'
There should be one record per Apex test class.
The underlying bug that appears to be the culprit in this issue has been published as Known Issue W-7834236.
Some Apex unit tests fail in a way that only shows in specific test UIs. The ApexTestQueueItem records for their parent classes will show Status = Failed and ExtendedStatus = "Could not run tests on class <01p class id> because: connection was cancelled here".
For these tests, ApexTestResult records will usually not be created, although some may exist. It's these tests that show as in progress forever in tools like Salesforce DX and the Developer Console, which are looking for ApexTestResult records. One consequence is that many testing tools will report a different number of unit tests completed from run to run, due to monitoring ApexTestResult records.