Why is & special outside of tabular environments?

Let's assume that the format or the user doesn't set the catcode of & (or any other symbol) globally to 4.

What would be the consequences assuming that you still want to have locally such a character (and not use commands like \nc for "new column" all the time)?

1. Every tabular like environment would have to make sure to set a char to be the alignment tab -- and they must agree which one to use. This would involve a lot of patching. This is naturally not an argument why it hasn't be done when the format was written.
2. It wouldn't be possible to use this character if the tabular is used in the argument of some command. E.g. you would run into problems when using tabulars in footers and headers, in addresses of letters, in author lists, in beamer frames. One could naturally define a command as a replacement, but users would have to understand when this command is needed.

3. It would be rather difficult to disable locally the alignment tab. E.g. currently you can easily type a tabular with lots of & like this:

   \documentclass{article}
   
   \begin{document}
   \begingroup
   \catcode`\&=12
   \catcode`!=4
   \begin{tabular}{ll}
   a & b ! c & d
   \end{tabular}
   \endgroup    
   \end{document}
   

   But if tabular resets the catcode of & it will no longer work. So you would need some hooks to allow such manipulations -- and again in every tabular like environment.

On the whole a lot things would get much more difficult for package writers and for users. It is easer to learn to type \&.

You ask:

Why is & special outside of tabular environments?

I think the question is not well-posed. I would say that & is special everywhere, not just outside tabular environments. The character & has category code 4 -- "alignment tab" -- in all TeX and LaTeX formats I've ever encountered. This catcode setting allows it to be used as the alignment tab character not only in tabular-like environments (e.g., tabular*, tabularx, longtable, etc), but also array environments, the deprecated eqnarray environment, some of the display math environments of the amsmath package, the IEEEeqnarray environment of the IEEEtrantools package, and probably quite a few additional environments.

The virtue of assigning catcode 4 to & by default is that all package writers can (and, frankly, should ) write their code assuming that & has catcode 4. Without this default, you'd have some package writers choosing & as the alignment tab character, whereas others might choose @ or who knows what else. Such an outcome would be distinctly inferior for most users, since they'd have to memorize which character, or characters, serve as the alignment tab function in which environments.

Conversely, if one does not want & to be special -- and doesn't want to modify its catcode -- one must input it as \&.