MemberData tests show up as one test instead of many

I spent a lot of time trying to figure this one out in my project. This related Github discussion from @NPadrutt himself helped a lot, but it was still confusing.

The tl;dr is this: [MemberInfo] will report a single group test unless the provided objects for each test can be completely serialized and deserialized by implementing IXunitSerializable.

---

Background

My own test setup was something like:

public static IEnumerable<object[]> GetClients()
{
    yield return new object[] { new Impl.Client("clientType1") };
    yield return new object[] { new Impl.Client("clientType2") };
}

[Theory]
[MemberData(nameof(GetClients))]
public void ClientTheory(Impl.Client testClient)
{
    // ... test here
}

The test ran twice, once for each object from [MemberData], as expected. As @NPadrutt experienced, only one item showed up in the Test Explorer, instead of two. This is because the provided object Impl.Client was not serializable by either interface xUnit supports (more on this later).

In my case, I didn't want to bleed test concerns into my main code. I thought I could write a thin proxy around my real class that would fool the xUnit runner into thinking it could serialize it, but after fighting with it for longer than I'd care to admit, I realized the part I wasn't understanding was:

The objects aren't just serialized during discovery to count permutations; each object is also deserialized at test run time as the test starts.

So any object you provide with [MemberData] must support a full round-trip (de-)serialization. This seems obvious to me now, but I couldn't find any documentation on it while I was trying to figure it out.

---

Solution

• Make sure every object (and any non-primitive it may contain) can be fully serialized and deserialized. Implementing xUnit's IXunitSerializable tells xUnit that it's a serializable object.

• If, as in my case, you don't want to add attributes to the main code, one solution is to make a thin serializable builder class for testing that can represent everything needed to recreate the actual class. Here's the above code, after I got it to work:

TestClientBuilder

public class TestClientBuilder : IXunitSerializable
{
    private string type;

    // required for deserializer
    public TestClientBuilder()
    {
    }

    public TestClientBuilder(string type)
    {
        this.type = type;
    }

    public Impl.Client Build()
    {
        return new Impl.Client(type);
    }

    public void Deserialize(IXunitSerializationInfo info)
    {
        type = info.GetValue<string>("type");
    }

    public void Serialize(IXunitSerializationInfo info)
    {
        info.AddValue("type", type, typeof(string));
    }

    public override string ToString()
    {
        return $"Type = {type}";
    }
}

Test

public static IEnumerable<object[]> GetClients()
{
    yield return new object[] { new TestClientBuilder("clientType1") };
    yield return new object[] { new TestClientBuilder("clientType2") };
}

[Theory]
[MemberData(nameof(GetClients))]
private void ClientTheory(TestClientBuilder clientBuilder)
{
    var client = clientBuilder.Build();
    // ... test here
}

It's mildly annoying that I don't get the target object injected anymore, but it's just one extra line of code to invoke my builder. And, my tests pass (and show up twice!), so I'm not complaining.

MemberData can work with properties or methods which return IEnumerable of object[]. You will see a separate test result for each yield in this scenario:

public class Tests
{ 
    [Theory]
    [MemberData("TestCases", MemberType = typeof(TestDataProvider))]
    public void IsLargerTest(string testName, int a, int b)
    {
        Assert.True(b>a);
    }
}

public class TestDataProvider
{
    public static IEnumerable<object[]> TestCases()
    {
        yield return new object[] {"case1", 1, 2};
        yield return new object[] {"case2", 2, 3};
        yield return new object[] {"case3", 3, 4};
    }
}

However, as soon as you will need to pass complex custom objects no matter how many test cases you will have the test output window will show just one test. This is not ideal behaviour and indeed very inconvenient while debugging which test case is failing. The workaround is to create your own wrapper which will derive from IXunitSerializable.

public class MemberDataSerializer<T> : IXunitSerializable
    {
        public T Object { get; private set; }

        public MemberDataSerializer()
        {
        }

        public MemberDataSerializer(T objectToSerialize)
        {
            Object = objectToSerialize;
        }

        public void Deserialize(IXunitSerializationInfo info)
        {
            Object = JsonConvert.DeserializeObject<T>(info.GetValue<string>("objValue"));
        }

        public void Serialize(IXunitSerializationInfo info)
        {
            var json = JsonConvert.SerializeObject(Object);
            info.AddValue("objValue", json);
        }
    }

Now you can have your custom objects as parameters to Xunit Theories and still see/debug them as independent results in the test runner window:

public class UnitTest1
{
    [Theory]
    [MemberData("TestData", MemberType = typeof(TestDataProvider))]
    public void Test1(string testName, MemberDataSerializer<TestData> testCase)
    {
        Assert.Equal(1, testCase.Object.IntProp);
    }
}

public class TestDataProvider
{
    public static IEnumerable<object[]> TestData()
    {
        yield return new object[] { "test1", new MemberDataSerializer<TestData>(new TestData { IntProp = 1, StringProp = "hello" }) };
        yield return new object[] { "test2", new MemberDataSerializer<TestData>(new TestData { IntProp = 2, StringProp = "Myro" }) };      
    }
}

public class TestData
{
    public int IntProp { get; set; }
    public string StringProp { get; set; }
}

Hope this helps.