Why is the sky *uniformly* blue?

First of all, the scheme of only single scattering seems to be an oversimplification: the light direction should be changed more than once. Can we prove that this is negligible by calculation, or is it not negligible?

This is an oversimplification, but for a clear sky at daytime it's not too wrong. See the following comparison of an atmosphere model computed with only single scattering and that including 4 orders of scattering (basically, 4 direction switches per light ray). The projection here is equirectangular, so you can see all the directions in one picture.

This becomes a much more problematic simplification when the sun is under the horizon, particularly noticeable under the belt of Venus, where the Earth's shadow is located:

Assuming the sun is in zenith, it follows from symmetry that the color of the sky in the directions having the same zenith angle must be the same, but closer to the horizon the way of the scattered light differs a lot from the rays coming near zenith — so is it possible to derive theoretically a formula which would predict the color of sky given the azimuth angle and the position of the Sun (at least in a simple geometrical setup when the Sun is in zenith)?

If we neglect non-uniformity of the atmosphere with latitude and longitude, this scenario will lead to the colors independent of azimuth. It's not quite clear what you mean by "position of the Sun" though, if you already put it into zenith. Also, if by "derive theoretically a formula" you mean some closed-form expression, then it's unlikely, given that the atmosphere is not a simple distribution of gases and aerosols. But it's possible to calculate the colors numerically, and the above pictures demonstrate this calculation done by my (work in progress) software, CalcMySky.

It's not clear why the color should not rapidly change from near blue at horizon to almost red near the Sun position: after all, the atmosphere is thicker along the lines going closer to horizon!

It shouldn't be bluer at the horizon than at the zenith. After all, you have relatively small thickness near zenith, which makes most of the light scattered to you not too extincted due to Beer-Lambert law, while near the horizon the thickness is much larger, and the light scattered into the observer, in addition to becoming bluer due to Rayleigh scattering depending on wavelength, becomes also redder due to extinction along this long path. The combination of this bluing and reddening effects gives a color closer to white (which you can see in the daytime simulation above), or reddish-orange (in the twilight).

Further, it follows from the usual explanation that blue light is partially reflected back into the space. Due to this, about half of all scattered light should be lost, so the total amount of red light coming from sun should be greater than the amount of blue light, which seems to contradict the observable reality.

Yes, the Earth indeed looks bluish from space, so the total radiation incoming from above should be redder at the ground level than at the top of atmosphere. But this is modified by the ozone layer, without which we'd have sandy color of twilight instead of blue. See for details the question Why is there a “blue hour” after the “golden hour”?

Short explanation is this. Red light comes directly from Sun almost un-scattered or scattered to small degree. And when blue light enters atmosphere it gets scattered by air molecules a lot in each direction, thus according to Huygens-Fresnel principle making each point in atmosphere as a secondary sources of blue light. These blue light sources adds-up along the direction of view, which in the end increases intensity of blue waves, compared to the red ones which reaches us only directly from Sun. So speaking by analogy, Earth atmosphere acts as a some sort of optical lens, focusing blue light towards direction of view. Schematics :

Of course this is a bit oversimplified, because blue light is scattered in ALL directions across the air. You can imagine a thousands of blue light bulbs turned-on in the sky. Maybe this would be better analogy, because each point in air acts as an ambient light source for blue waves.

Here are some answers, albeit back-of-the-envelope.

At a reasonably good site with a low amount of atmospheric aerosols and dust, the "extinction" is about 0.3 magnitudes per airmass at 400 nm, in astronomers units, compared to about 0.1 mag/airmass at 550 nm and about 0.04 mag/airmass at 700 nm.

What this means is that if light travels through the atmosphere at zenith, then a factor of $10^{-0.3/2.5}=0.758$ of blue light makes it to the ground, compared with a factor of 0.912 for green light and 0.963 for red light. Most of the remainder will be Rayleigh scattered (although there is some component from atmospheric absorption and scattering by aerosols in these numbers).

From this you can see that multiple scattering cannot be negligible for blue light, because at least a quarter of it is scattered by just travelling through the minuimum possible amount of air between space and the observer.

The next point: yes, it is possible to calculate the spectrum of the daylight sky given the appropriate atmospheric conditions (the run of density with height) and the aerosol content (the latter is important because the dependence of the scattering cross-section on wavelength is much more uniform than for Rayleigh scattering). Is there a simple formula - no. An example of where detailed calculations have been set out in great detail can be found here.

Then, why doesn't the sky become red near the Sun? Why would it? Red light is not effectively scattered, so red light that is emitted by the Sun does not get scattered towards the observer. On the other hand if you look directly towards the Sun (please do not do this) then blue light is preferentially scattered out of the direct sunlight, and indeed the Sun is "redder" than it would appear from space (plot below).

The only source of illumination from directions that are not towards the Sun are from scattered light. If we ignored multiple scattering and aerosols then that scattered light would have a spectrum that was proportional to the illuminating light multiplied by the Rayleigh scattering cross-section. The illuminating light does get progressively redder as the zenith angle increases (because the illuminating beam has to travel further and deeper through the atmosphere), so you would expect a whiter colour near the horizon, transitioning to a deeper blue higher above the horizon. However, this is not a very strong effect because only a quarter of blue light is scattered per airmass (and the eye has a pseudo-logarithmic response to spectral flux). Note though that in practice aerosols are not absent and that scattering from aerosols and particulates has some concentration in the forward scattering direction, which messes up this simple prediction, by making the sky whiter near to the Sun. Multiple scatterings also make the sky whiter near to the horizon because some of the blue light coming from that direction is then scattered out of the line of sight.

This is perfectly illustrated by a calculated sky image that shows the separate contribution of Rayleigh and aerosol (Mie) scattering (taken from this website, which does quantitative calculations, but which does not take account of multiple scattering). The sky is quite white near the horizon, then becomes a deeper blue at higher angles and is finally quite white again near to the Sun because of Mie scattering.

Your final point, I've already covered. Yes, direct sunlight arriving at the surface of the Earth is "redder" than received at the top of the atmosphere. Plot below from https://commons.wikimedia.org/wiki/File:Solar_Spectrum.png