Assert.Inconclusive and IgnoreAttribute

I've been doing some research into this, as well as trying it out at home. The end result is that I believe the [Ignore] attribute for MSTest does indeed leave out the test method completely. I tried looking at settings in Visual Studio to see if there was an override, but could not find anything.

Shame about that, as not seeing the ignored tests is bad since you may end up thinking you have a suite of 100 MSTest tests running nicely, but it turns out that there are 50 which are missing from the results that you never knew about due to the [Ignore] attribute! Urgh.

Classically, in when testing, unit tests should be very specific so that there is a (close to) one-to-one correspondence between features and tests for these features.

To test certain features, the system under test (SUT) first needs to be brought into a certain state. Once that state is reached, the test can actually test the feature that it is meant for. Now, what should be the status of such a test, if already the setup part fails. It cannot make a statement on the functioning of the feature under test, as the test never reached the status requires to exercise the feature.

In such cases, it is common to assign an inconclusive result, as we simply cannot know if the feature works as it should, and so we cannot pass or fail. The fact that the setup itself does not work as expected should be covered by a separate test.

So imagine, I want to test that a 'foo' that has been 'bar'ed will return a 0 when 'qux'ed. This test should not test if 'foo' can be 'bar'ed, so any failure to be 'bar'ed will return an inconclusive, whereas a failure to respond to 'qux' will be proper fail.

I like to think how you are describing Inconclusive is the proper usage.

Though in my experience, Inconclusive is treated more like a warning than an error. In fact, they are reported in the TRX separately from errors:

<TestRun>
   <ResultSummary outcome="Inconclusive">
      <Counters total="1" 
          executed="0" error="0" failed="0" 
          timeout="0" aborted="0" inconclusive="1" 
          passedButRunAborted="0" notRunnable="0" 
          notExecuted="0" disconnected="0" 
          warning="0" passed="0" completed="0" 
          inProgress="0" pending="0" />

I typically run the mstest executable from an <Exec> task in my msbuild script and then peek inside the TRX to determine whether it should fail the build.

i also see a dilemma in the current implementation.

• Inconclusive assertions are included in the TRX report, but mstest.exe (and also vstest.console.exe) will return 1 (meaning error) after execution.
• TestMethods with the Ignore attribute will not be reported as an error, but they are completely hidden from the TRX report.

my personal understanding is as follows:

use the [Ignore] attribute to (temporarily) disable / skip the method:

[TestMethod]
[Ignore] // <== disabled through "Ignore" attribute
public void Test001()
{
   //execute some stuff ...
   Assert.IsTrue(...);

   //execute some stuff ...
   Assert.AreEqual(...);
}

do not misuse the Inconclusive assertion for this purpose:

[TestMethod]
public void Test002()
{
    Assert.Inconclusive(); // <== misuse of "Inconclusive" to skip this test

    //execute some stuff ...
}

instead, Inconclusive should be used conditionally: only if we can't tell whether the component to be tested does work as expected or not.
for example in case an external resource we depend on is not available at time of test execution:

[TestMethod]
public void Test003()
{
    //check if the server is running,
    //otherwise can can't test our local client component!
    if (!WebServiceAvailable())
    {
        Assert.Inconclusive(); // <== skip remaining code because the resource is not available
    }

    //execute some stuff ...
    Assert.AreEqual(...);

    //execute some stuff ...
    Assert.AreEqual(...);
}

_ _

conclusion:
to disable / skip a test the logical way is to use the [Ignore] attribute.
i clearly see the current behaviour of mstest.exe not reporting on any ignored test as a bug that should be fixed.

feel free to up-vote the following bug reports:

• https://connect.microsoft.com/VisualStudio/feedback/details/779743/
• http://visualstudio.uservoice.com/forums/121579-visual-studio/suggestions/2088845