Is lack of User-Agent in HTTP request valid?
I guess many people use HTTP requests without a User-Agent mostly when they are using an API to perform the request.
As stated in RFC 7231 (but nearly the same paragraph can be found in RFC2616):
5.5.3 User-Agent
The "User-Agent" header field contains information about the user agent originating the request, which is often used by servers to help identify the scope of reported interoperability problems, to work around or tailor responses to avoid particular user agent limitations, and for analytics regarding browser or operating system use. A user agent SHOULD send a User-Agent field in each request unless specifically configured not to do so.
The keyword here is SHOULD. And yes, there's an RFC that defines what that word is supposed to mean, RFC 2119:
- SHOULD This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course.
So, although the agents that do not send User-Agent do not follow what can be considered best practice, they do not violate any rule (rfc). So, in my opinion, there's not really a valid technical reason to block them.