Unit test, NUnit or Visual studio?
NUnit can be used in combination with visual studio. It is a framework not a separate program. So you could care an see if it suits you :).
alt text http://i3.codeplex.com/Project/Download/FileDownload.aspx?ProjectName=nunitit&DownloadId=61802
"After installing the plugin you'll find a new submenu under the tools menu."
See http://nunitit.codeplex.com/ for more information on importing it.
Also, a lot can be found using the search of SO. This topic lists advantages of NUnit over MS standard testing for instance.
From my current perspective (after 8 months of development with about 10 developers on average) I would advise against using MSTest for the following reasons
- The framework in itself is quite slow. I don't mean the test code that you write - that's under your control. I mean the framework running those tests is slow, whether it's running a test suite, single tests etc.
- The need to keep a Test-Metadata file which always leads to complications when several developers are working on it (recreating e.g. the metadata etc.). Every other test suite doesn't need a metadata file. It is kind of nice to organize your tests but you can achieve the same through namespaces, classes and method names.
- Doing Continuous Integration, if you want to run unit tests on your build machine you will need to install Visual Studio on that machine.
In other words, if I would have to decide again 8 months ago, I would probably take NUnit. I may not have the integrated test results report, but developers would have a more seamless testing experience.
Here is my experience with MS Test
- We are running MS Test with around 3800 Test.
- It takes very long for the tests just to start executing, which is painful when running single tests.
- It takes around 1GB Memory to execute the tests. No, it is not due to memory leaks in our tests. Frequently we run into OutOfMemoryExceptions.
- Because it uses that much resource, we are starting to execute the tests from batch-files. So what's the whole integration good for?
- It is buggy and unstable:
- For instance, if you remove the [Ignore] Attribute from a test, it does not recognize it, because it caches information about tests somewhere. You need to refresh the testlist, which sometimes solves the problem, or restart VS.
- It randomly does not copy reference assemblies to theout directory.
- Deployment Items (additional files to be used) just don't work properly. They are ignored randomly.
- There is hidden (not visible in the test code) information in vsmdi and testrunconfig files. If you don't care about it, it might not work.
- Functionally it might be comparable to NUnit, but it is very expensive if you consider using VS tester edition.
Addition: We have some more tests now, can't even say how many. It is impossible to run them all anymore from Visual Studio, because of OutOfMemoryExceptions and other instability problems. We run the tests from scripts. It would be easy to view test results in Visual Studio, but when the solution is open, VS crashes (every time). So we need to search the failing tests using text search. There is no advantage of an integrated tool anymore.
Another Update: We are using VS 2013 now. A lot of things changed. They rewrote the MS Test test runner for the third time since we started. This caused a lot of breaking changes, but neither new version was doing anything better. We are glad that we didn't use the fancy features of MS Test, because they are all not supported anymore. It's really a shame. We are still using scripts to build and run all unit tests, because it is handier. Visual Studio required a few minutes to start running tests (time measures after compilation until first test starts). They probably fixes it with an update and this might be a specific problem of our project. However, Resharper is much quicker when running the same tests.
Conclusion: At least in combination with Resharper, MS Test is useful. And I hope that they finally find out how the test runner should be written and won't do this kind of breaking changes when we update Visual Studio next time.
NUnit has few advantages over MS-Test
- Suite attribute - can aggregate tests and execute them separately (useful for large projects with fast and slow tests for example)
- Readable Assert method, e.g.
Assert.AreEqual(expected, actual)vsAssert.That(actual, Is.EqualTo(expected)) - NUnit has frequent version updates - MS-Test has only one per VS version.
- Many integrated runners including Resharper and TestDriven.NET
- Expected exception message assertion - can be done using attribute in NUnit but must be done using Try-Catch in MS-Test
[TestCase]! NUnit allows for parameter-ized tests.