Why should you base64 encode the Authorization header?

Eventhough I can't find it in the w3 documentation, I believe that it is just protocol to encode the credentials of the Authorization header to base64, no matter what content it has. In the case of Twitter it doesn't make much difference as you said, but in other cases the credentials can contain these characters. To keep it uniform and prevent mistakes of whether it should be encoded or not, all credentials should be encoded.

Another reason could be, that browsers also encode the credentials the same way. Twitter probably also wants to accept that.

The Basic Authentication Scheme is described in the RFC7617 (and the old RFC2617).

This is a standard way to send password credentials to the server. The base64 encoding is used to encode credentials to allow non HTTP characters and multibytes strings to be sent.

By default, message header field parameters in Hypertext Transfer Protocol (HTTP) messages cannot carry characters outside the ISO- 8859-1 character set.

If user name and password contains incompatible charset than HTTP would not be able to carry those text. to prevent from this we encode user name and password with base64 to make sure we are sending HTTP compatible char over HTTP. for more information see this Basic_access_authentication