How to run and publish .NETCore Xunit tests on VSTS (Vs2017)?

starain-MSFT's answer will work, unless you want/need the xunit tests to be logged using an xunit logger. In that case, you'll need to do two things.

1. Add https://www.nuget.org/packages/XunitXml.TestLogger/1.0.2-pre-rtm as package ref to your test project, either through 'Manage NuGet Packages' in VS, or by adding the ref in your csproj file manually, i.e.

<PackageReference Include="xunitxml.testlogger" Version="1.0.2-pre-rtm" />

2. Modify the VSTS dotnet test build step to use this logger: dotnet test -a:. -l:xunit
   The -a:. switch, which specifies the adapter path, is only necessary for CLI tools V15.0, in 15.1 that can be removed (as discussed here). As of today, the VS2017 Hosted Queue is using 15.0, so you'll need the -a:. on VSTS for now. The -l:xunit uses the friendlyname, which I think isn't so friendly since you have to dig into the source code for the particular logger to find the attribute where it is specified (as seen here for xunit and here for trx)

The docs for the -l switch are spotty to say the least, but in the github for vstest, there is a document which talks about test loggers and links to their repositories and nuget packages, which after you look at the source for the friendlyname, gets you all the way there for whichever logger you need. If you need a custom logger, those are great examples to help understand how to implement.

Finally, the publish step that you used originally should be fine, since the output file is still called TestResults.xml

Change "Test Result Format" to "VSTest of Publish Test" result step/task, it reads the result file correctly.

Use dotnet xunit instead of dotnet test. See Getting Started with .NET Core.