Should a service layer return view models for an MVC application?

Generally, no.

View models are intended to provide information to and from views and should be specific to the application, as opposed to the general domain. Controllers should orchestrate interaction with repositories, services (I am making some assumptions of the definition of service here), etc and handle building and validating view models, and also contain the logic of determining views to render.

By leaking view models into a "service" layer, you are blurring your layers and now have possible application and presentation specific mixed in with what should focused with domain-level responsibilities.

No, I don't think so. Services should care only about the problem domain, not the view that renders results. Return values should be expressed in terms of domain objects, not views.

As per the traditional approach or theory wise, ViewModel should be part of User interface layer. At least the name says so.

But when you get down to implementing it yourself with Entity Framework, MVC, Repository etc, then you realise something else.

Someone has to map Entity/DB Models with ViewModels(DTO mentioned in the end). Should this be done in [A] the UI layer (by the Controller), or in [B] the Service layer?

I go with Option B. Option A is a no no because of the simple fact that several entity models combine together to form a ViewModel. We may not pass unnecessary data to UI layer, whereas in option B, the service can play with data and pass only the required/minimum to the UI layer after mapping (to the ViewModel).

But still, let us go with option A, put ViewModel in the UI layer(and entity model in Service layer).

If the Service layer needs to map to the ViewModel, then the Service layer need to access ViewModel in UI layer. Which library/project? The Viewmodel should be in a separate project in the UI layer, and this project needs to be referenced by Service Layer. If the ViewModel is not in a separate project, then there is circular reference, so no go. It looks awkward to have Service layer accessing UI layer but still we could cope with it.

But what if there is another UI app using this service? What if there is a mobile app? How different can the ViewModel be? Should the Service access the same view model project? Will all UI projects access the same ViewModel project or they have their own?

After these considerations my answer would be to put the Viewmodel project in Service Layer. Every UI layer has to access the Service layer anyways! And there could be a lot of similar ViewModels that they all could use (hence mapping becomes easier for service layer). Mappings are done through linq these days, which is another plus.

Lastly, there is this discussion about DTO. And also about data annotation in ViewModels. ViewModels with data annotations(Microsoft.Web.Mvc.DataAnnotations.dll) cannot reside in service layer instead they reside in UI layer(but ComponentModel.DataAnnotations.dll can reside in service layer). If all projects are in one solution(.sln), then it doesn't matter which layer you put it. In enterprise applications, each layer will have its own solution.

So DTO actually is a ViewModel because mostly there will be one on one mapping between the two(say with AutoMapper). Again DTO still has the logic needed for the UI(or multiple applications) and resides in Service Layer. And the UI layer ViewModel(if we use Microsoft.Web.Mvc.DataAnnotations.dll) is just to copy the data from DTO, with some 'behavior'/attributes added to it.

[Now this discussion is about to take an interesting turn read on...:I]

And don't think data-annotation attributes are just for UI. If you limit the validation using System.ComponentModel.DataAnnotations.dll then the same ViewModel can also be used for front-end & backend validation(thus removing UI-residing-ViewModel-copy-of-DTO). Moreover attributes can also be used in entity models. Eg: using .tt, Entity Framework data models can be autogenerated with validation attributes to do some DB validations like max-length before sending to the back end. This saves round-trips from UI to backend for validation. It also enables back-end to re-validate for security reasons. So a model is a self-reliant validator and you can pass it around easily. Another advantage is that if backend validation changes in DB then .tt (reads DB specifics and create the attribute for entity class) will automatically pick that up. This can force UI validation unit tests to fail as well, which is a big plus(so we can correct it and inform all UIs/consumers instead of accidentally forgetting and failing). Yes, the discussion is moving towards a good framework design. As you can see it is all related: tier-wise validation, unit test strategy, caching strategy, etc.

Although not directly related to the question. 'ViewModel Façade' mentioned in this must watch channel 9 link is also worth exploring. It starts exactly at 11 minutes 49 seconds in the video. Because this would be the next step/thought once your current question given above is sorted out: 'How to organize ViewModels?'

And Lastly, many of these model vs logic issues could be resolved with REST. Because every client can have the intelligence to query the data and get only the data that it needs. And it keeps the model in UI, there is no server/service layer model/logic. The only duplication then will be on the automated tests that each client need to perform. Also if there are changes in data then some clients fail if they do not adapt to the change. The question then is, are you removing service layer altogether(and the models they carry) or pushing the service layer up to your UI project(so model issue still persists) which calls the REST API. But in terms of the responsibility of Service layer, they are the same regardless.

Also in your example "_repository.ListContacts()" is returning a ViewModel from repository. This is not a mature way. Repositories should provide entity models or DB models. This gets converted to view models and it is this view model that is returned by the service layer.