Can I use mstest.exe without installing Visual Studio?

It is possible to run mstest.exe without visual studio.
Download one of the Agents for Visual Studio ISO's below and install the Test Agent on the server:

Visual Studio 2017 (127MB disk space, less than that for download)
Visual Studio 2015 (128MB setup, 2GB disk space required)
Visual Studio 2012 (224MB)
Visual Studio 2013 (287MB)
Visual Studio 2010 (515MB)

This installs everything needed for running mstest.exe from the command line and is much lighter weight than visual studio. ~500mb download and around ~300mb to install just the test agent if I remember correctly.

UPDATE

For versions older than VS 2017, look here:

https://www.visualstudio.com/vs/older-downloads/

This answer pertains specifically to Visual Studio 2017, and the answer is yes. Please be aware, however, that Microsoft (still) doesn't provide any sort of official API for locating the relevant executables (MSBuild.exe and MSTest.exe), so you are stuck reading registry keys and/or probing various directories to locate these files. You've been warned.

• If you only need to build your unit test project(s), install the package MSTest.TestFramework into those project(s) and remove the reference to Microsoft.VisualStudio.QualityTools.UnitTestFramework from them. Now all you need is to install the Visual Studio 2017 Build Tools and invoke msbuild.exe to perform the build.
• If you need to run your tests as well, things become trickier:
  • The simplest solution is to install VS2017 Community Edition (which includes both msbuild and mstest) - but I am unsure of the legality of this, and am not a lawyer, so be careful!
  • A legally safer solution (and far lighter, in terms of disk space) is to install the Visual Studio 2017 Test Agent and then Build Tools for Visual Studio 2017 (exact order is vital¹); this will give you MSTest.exe and vstest.console.exe which you can then call out to. Do note that actually figuring out where these executables reside is a pain, because they won't exist in the same directory structure as MSBuild.exe in Build Tools.

Finally, and very importantly: if you do use MSTest.TestFramework and still need to be able to discover and run tests from within the Visual Studio IDE, you'll also need MSTest.TestAdapter installed in your unit test project(s).

^{1: While VS2017 supports side-by-side installs, it uses a single registry key that only records the most recent install. Hence, if you install Test Agent last, the key will point to its install directory... but Test Agent doesn't include MSBuild.exe, so any code that relies on this registry key to figure out that executable's path, will fail. Why Microsoft couldn't have made the Test Agent an optional part of Build Tools (so that all the EXEs live in the same directory hierarchy) is anyone's guess.}

@crocpulsar, you need to install Visual Studio on your build server, but you do NOT need to buy an additional licence.

There are just way too many dependencies to getting build & MSTest to work without VS installed, and it is most definitely not supported.

As long as the person who starts the build has a license, you do not need one for the build server. This has been the case since the dark days of 2005, and as long as there is edition parity then you are OK.

If everyone in your team has Ultimate, then you are free to install it on the build server; but if one of your team member's has Premium, then you should ideally install Premium on the build server. This also enables lots of other bits like Code Coverage, Test Impact Analysis, and Architecture Validation among others.