How can I run my performance tests more than ten times?
In the latest Xcode (11.0+) you don't need swizzling to change iterations count. Use the following function:
func measure(options: XCTMeasureOptions, block: () -> Void)
This will allow you to specify XCTMeasureOptions
which has iterationCount
property.
Interesting note from docs:
A performance test runs its block
iterationCount+1
times, ignoring the first iteration and recording metrics for the remaining iterations. The test ignores the first iteration to reduce measurement variance associated with “warming up” caches and other first-run behavior.
The short answer: No, there is no current interface exposed to allow a measure block more than ten times.
The longer answer: No, but there is an interface exposed to modify certain metrics of the measure block. The default metrics are in a String array returned from defaultPerformanceMetrics
. There appears to only be one metric supported right now: XCTPerformanceMetric_WallClockTime
. This only specifies a performance metric that records the time in seconds between calls to a performance test's startMeasuring()
and stopMeasuring()
methods, and not the number of times the block is run.
a class dump of XCTestCase exposes this method:
- (void)_recordValues:(id)arg1 forPerformanceMetricID:(id)arg2 name:(id)arg3 unitsOfMeasurement:(id)arg4 baselineName:(id)arg5 baselineAverage:(id)arg6 maxPercentRegression:(id)arg7 maxPercentRelativeStandardDeviation:(id)arg8 maxRegression:(id)arg9 maxStandardDeviation:(id)arg10 file:(id)arg11 line:(unsigned long long)arg12;
when this method is swizzled the first parameter (arg1) has the 10 durations:
["0.003544568",
"0.003456569",
"0.003198263",
"0.003257955",
"0.003508724",
"0.003454298",
"0.003461192",
"0.00423787",
"0.003359195",
"0.003335757"]
i added 4 new values (1.0, 2.0, 3.0, 4.0) to the end of this list before passing it back to the original implementation, but unfortunately a different class that observes, XCTestLog
, has an internal sanity check that gets tripped:
Assertion failure in +[XCTestLog _messageForTest:didMeasureValues:forPerformanceMetricID:name:unitsOfMeasurement:baselineName:baselineAverage:maxPercentRegression:maxPercentRelativeStandardDeviation:maxRegression:maxStandardDeviation:file:line:]
caught "NSInternalInconsistencyException", "Performance Metrics must provide 10 measurements."
once the XCTestLog
method is also overridden so it doesn't assert, the additional 4 values can be added without any complaints. unfortunately the view still only shows the 10 results.
it does however update the total time + standard deviation values in the mini view.
Before Swizzling
After Swizzling and adding 4 values
in order to view more than 10 results one would probably have to tweak the XCode runtime to tell the table to show more items.