strange 401 error appears for some urls when using .htaccess to redirect http to https

Try using this instead. Not the L and R flag.

RewriteEngine On
RewriteCond %{HTTPS} !on
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]

Also clear your browsers cache first, to remove the old incorrect redirect.

If that doesn't work try using this.

RewriteCond %{HTTPS} !on
RewriteCond %{THE_REQUEST} ^(GET|HEAD)\ ([^\ ]+)
RewriteRule ^ https://%{HTTP_HOST}%2 [L,R=301]

I feel a bit bad about writing it, as it seems kind of hackish in my view.

EDIT Seems the 2nd option fixed the problem. So here is the explanation as to why it works.

The authentication module is executed before the rewrite module. Because the username and password is not send when first requesting the page, the authentication module internally 'rewrites' the request url to the 401 page's url. After this mod_rewrite comes and %{THE_REQUEST} now contains 401.shtml instead of the original url. So the resulting redirect contains the 401.shtml, and not the url you want.

The get to the original (not 'rewritten') url, you need to extract it from %{THE_REQUEST}. THE_REQUEST is in the form [requestmethod] [url] HTTP[versionnumber]. The RewriteCond extracts just the middle part ([url]).

For completeness I added the [L,R=301] flags to the second solution.

I think I found an even better solution to this!

Just add this to your .htaccess

ErrorDocument 401 "Unauthorized"

Solution found at:

http://forum.kohanaframework.org/discussion/8934/solved-for-reall-this-time-p-htaccess-folder-password-protection/

-- EDIT

I eventually found the root cause of the issue was ModSecurity flagging my POST data (script and iframe tags cause issues). It would try to return a 401/403 but couldn't find the default error document because ModSecurity had made my htaccess go haywire.

Using ErrorDocument 401 "Unauthorized" bypassed the missing error document problem but did nothing to address the root cause.

For this I ended up using javascript to add 'salt' to anything which was neither whitespace nor a word character...

  $("form").submit(function(event) {
    $("textarea,[type=text]").each(function() {
      $(this).val($(this).val().replace(/([^\s\w])/g, "foobar$1salt"));
    });
  });

then PHP to strip the salt again...

function stripSalt($value) {
  if (is_array($value)) $value = array_map('stripSalt', $value);
  else $value = preg_replace("/(?:foobar)+(.)(?:salt)+/", "$1", $value);

  return $value;
}
$_POST = stripSalt($_POST);

Very, Very, Very Important Note:
Do not use "foobar$1salt" otherwise this post has just shown hackers how to bypass your ModSecurity!

Regex Notes:
I thought it may be worth mentioning what's going on here...

(?:foobar)+ = match first half of salt one or more times but don't store this as a matched group;

(.) = match any character and store this as the first and only group (accessible via $1);

(?:salt)+ = match second half of salt one or more times but don't store this as a matched group.

It's important to match the salt multiple times per character because if you've hit submit and then you use the back button you will go back to the form with all the salt still in there. Hit submit again and more salt gets added. This can happen again and again until you end up with something like: foobarfoobarfoobarfoobar>saltsaltsaltsalt