What's an appropriate HTTP status code to return by a REST API service for a validation failure?

If "validation failure" means that there is some client error in the request, then use HTTP 400 (Bad Request). For instance if the URI is supposed to have an ISO-8601 date and you find that it's in the wrong format or refers to February 31st, then you would return an HTTP 400. Ditto if you expect well-formed XML in an entity body and it fails to parse.

(1/2016): Over the last five years WebDAV's more specific HTTP 422 (Unprocessable Entity) has become a very reasonable alternative to HTTP 400. See for instance its use in JSON API. But do note that HTTP 422 has not made it into HTTP 1.1, RFC-7231.

Richardson and Ruby's RESTful Web Services contains a very helpful appendix on when to use the various HTTP response codes. They say:

400 (“Bad Request”)
Importance: High.
This is the generic client-side error status, used when no other 4xx error code is appropriate. It’s commonly used when the client submits a representation along with a PUT or POST request, and the representation is in the right format, but it doesn’t make any sense. (p. 381)

and:

401 (“Unauthorized”)
Importance: High.
The client tried to operate on a protected resource without providing the proper authentication credentials. It may have provided the wrong credentials, or none at all. The credentials may be a username and password, an API key, or an authentication token—whatever the service in question is expecting. It’s common for a client to make a request for a URI and accept a 401 just so it knows what kind of credentials to send and in what format. [...]

From RFC 4918 (and also documented at http://www.iana.org/assignments/http-status-codes/http-status-codes.xhtml):

The 422 (Unprocessable Entity) status code means the server understands the content type of the request entity (hence a 415 (Unsupported Media Type) status code is inappropriate), and the syntax of the request entity is correct (thus a 400 (Bad Request) status code is inappropriate) but was unable to process the contained instructions. For example, this error condition may occur if an XML request body contains well-formed (i.e., syntactically correct), but semantically erroneous, XML instructions.

A duplicate in the database should be a 409 CONFLICT.

I recommend using 422 UNPROCESSABLE ENTITY for validation errors.

I give a longer explanation of 4xx codes here: http://parker0phil.com/2014/10/16/REST_http_4xx_status_codes_syntax_and_sematics/