Ambiguity in applying Newton's shell theorem in an infinite homogeneous universe
The problem lies in the boundary conditions. Ignoring factors of $G$ and $\pi$, gauss's law of gravitation relates the gravitational potential $\Phi$ to the mass density $\rho$ by $$\rho=-\nabla^2 \Phi. $$ In order to have a unique, well-defined solution, we need to specify boundary conditions for $\Phi$. Usually, we assume that $\rho$ dies off sufficiently quickly at spatial infinity that a reasonable choice of boundary condition is $\Phi(|\vec x|\to\infty)=0$ is. The shell theorem relies on this assumption. However in your example $\rho$ does not die off at infinity and is instead non-zero everywhere and therefore the shell theorem fails.
Often when a given scenario in physics doesn't, but almost, satisfies the 'if' part of a theorem, it can be helpful to try and modify the problem so that it does. Therefore we can use a window function $W_\epsilon(x-x_0)$ that dies off quickly as $x\to\infty$ but $\lim_{\epsilon\to0} W_\epsilon =1$ to regulate the charge density. [e.g. take $W_\epsilon(x-x_0)=e^{-\epsilon (\vec x-\vec x_0)^2}$.] Then we can replace your uniform charge density $\rho$ by $$\rho\to\rho_{\epsilon,x_0}\equiv \rho W_\epsilon(x-x_0) .$$ In this case, the shell theorem does hold. However, the result we get is not regulator-independent, that is if we solve for $\Phi_{\epsilon,x_0}$ using the charge distribution $\rho_{\epsilon,x_0}$ and then send $\epsilon \to0$, we find that our answer still depends on choice of $x_0$. This is the mathematically rigorous way to see that there really is an ambiguity when applying the shell theorem to such a situation!
Edit: There seems to be some debate in the comments as to whether the shell theorem should be proved with forces or with Gauss's law. In reality, it doesn't matter, but I will address what goes wrong if you just use forces. Essentially, Newton's laws are only guaranteed to be valid if there is a finite amount of matter in the universe. Clearly if there is uniform mass density throughout all of space, then there is an infinite amount of matter, so the shell theorem fails. The requirement that $\rho(|\vec x|\to \infty)\to 0$ 'sufficiently quickly' from above is more precisely that $\int d^3 x \rho(x) <\infty$, which is just the condition that there is a finite amount of matter in the universe.
Updated 07.11
We can chose the model to discuss the problem and so let us chose:
Model: Newtonian mechanics/Newtonian gravity, with the Universe filled with uniformly dense matter, interacting only gravitationally (in cosmology this called “dust matter”), and at the initial time of our spaceship journey all this matter is at rest.
Hence my spaceship should start accelerating toward ×. By choosing the sphere large enough, I should be able to make it accelerate arbitrarily fast, and by choosing the location of × I can make it accelerate in any direction.
Absolutely!
Of course this doesn't work, but why?.
It does work. If we assume that initially the spaceship was at rest together with the whole universe it will reach the point × in time needed for the ship to fall into a point mass equal to the mass of the pink sphere.
The problem is that by that time all of the pink sphere also falls toward that same point as well, as do all other colored spheres and the rest of the universe also. If our astronaut checks its distance to the point × before the spaceship falls into it she would notice that this distance has decreased, but at the same time is she checks her surroundings she would notice that the spaceship is surrounded by precisely the same matter particles that when the journey started only they are closer to each other and to the spaceship. This distance contraction is simply a Newtonian version of Big Crunch event.
If the universe is filled with matter interacting only gravitationally and we assume that the density of matter will stay uniform throughout the universe, then the only conclusion would be that such universe is not static. It has either (Newtonian version of) Big Bang in its past or Big Crunch in its future (or in our model, since we chose initial moment as a turning point from expansion to contraction, it has both).
It may seem that the whole Universe falling toward our chosen point × is an absurdity, since we have chosen this point arbitrarily. But in this situation there is no paradox, the acceleration of all matter toward this point is due to the fact that in our setup there is no “absolute space”, no set of outside stationary inertial observers which could give us absolute accelerations, instead we can only choose a reference point × (or rather specify an observer located at this point and at rest with respect to surrounding matter) and calculate relative accelerations toward this point.
Recall, that the first principle of Newtonian mechanics states that every particle continues in its state of rest or uniform motion in a straight line unless it is acted upon by some exterior force. For an isolated system, for example collection of gravitating objects of finite total mass we could (at least in principle) place an observer at rest so far away that it could be considered an inertial object. This would allow us to define a reference frame with respect to which we would measure accelerations. But in our Newtonian cosmology matter is filling the whole Universe, there is no observer on which gravity is not acting, so there is no set of reference frames defined by observers “at infinity” only observers inside the matter concentrations that are affected by the gravitational forces.
While there is no absolute accelerations, the relative positions ($\mathbf{d}_{AB}(t)= \mathbf{x}_A(t)-\mathbf{x}_B(t)$ between objects $A$ and $B$ comoving with the matter of the universe) do have a meaning independent of the choice of reference point. This relative positions, relative velocities ($\dot{\mathbf{d}}_{AB}$), relative accelerations, etc. constitute the set of unambiguously defined quantities measurable within our universe.
then my intuition tells me that I can just choose a sufficiently static universe.
This intuition is wrong, if there is a gravitational force that would accelerate your spaceship toward ×, then it would also be acting on a nearby matter (call them dust particles or planets or stars) producing the same acceleration, so all of the universe would be falling toward ×.
Note on Newtonian cosmology it may seems that Newtonian theory of gravitation is ill suited to handle homogeneous spatially infinite distributions of matter. But one can try to separate the physics of the situation from the deficiencies of particular formalism and possibly to overcome them. As a motivation we could note that over large, cosmological distances our universe to a high degree of accuracy could be considered spatially flat, and the velocities of most massive objects relative to each other and to the frame of CMB are very small compared with the speed of light, meaning that Newtonian approximation may be appropriate. While we do know that general relativity provides a better description for the gravitation, Newtonian gravity is computationally and conceptually much simpler. This seems to suggest that it is worthwhile to “fix” whatever problems one encounters while attempting to formalize cosmological solutions of Newtonian gravity.
The most natural approach is to “geometrize” Newtonian gravity and instead of “force” consider it a part of geometry, dynamical connection representing gravity and inertia. This is done within the framework of Newton–Cartan theory.
As a more detailed reference, with an emphasis on cosmology, see this paper (knowledge of general relativity is required):
- Rüede, C., & Straumann, N. (1996). On Newton-Cartan cosmology. arXiv:gr-qc/9604054.
Newton–Cartan theory underscores conceptual similarities between Newtonian gravity and general relativity, with Galilei group replacing the Lorentz group of GR. The general approach is coordinate-free and is closely related to the machinery of general relativity, but a specific choice of local Galilei coordinates would produce the usual equations for acceleration ($\mathop{\mathrm{div}} \mathbf{g} = - 4\pi \rho$), with gravitational acceleration now being part of Newtonian connection. Homogeneous and isotropic cosmological solutions are a straightforward lifts of FLRW cosmologies.
While equations are the same, we may already answer some conceptual questions.
Since gravitational acceleration is part of the connection, there is no reason to expect it to be an “absolute” object, there would be gauge transformations that would alter it. We can have multiple charts on which we define the physics with the normally defined transition maps between.
We can have a closed FRW cosmology, the “space” does not has to be a Euclidean space, it could be torus $T_3$ (field equations require that locally the space is flat). Since the spatial volume of a closed universe varies, and tend to zero as the universe approaches the Big Crunch, this asserts that not just matter but space itself collapses during the Big Crunch (to answer one of the comments).
It is quite simple to include the cosmological constant / dark energy thus making the models more realistic.
Note on answer by user105620: If we formulate a regularization procedure by introducing a window function $W(\epsilon,x_0)$ that would make potential well behaved. This provides us with an another way to “fix” problems of our cosmological model. The acceleration of our spaceship computed with this regularization is indeed dependent on the choice of $x_0$ in the limit $\epsilon\to 0$, which is the consequence of the same freedom in choosing the reference point ×. But he/she just should not have stopped there. Divergences requiring the use of regulators and ambiguities remaining after regularization are quite normal features in developing physical models. The next step would be identifying the physically meaningful quantities and checking that those are independent on the regulator artifacts. In our case neither potential $\Phi$ nor gravitational acceleration $\mathbf{g}$ are directly observable in this model. Relative positions, relative velocities and relative accelerations are observable and those are turning to be independent of the regulator parameter $x_0$.
by choosing the location of × I can make it accelerate in any direction.
This freedom of choice is the key to the puzzle. I'll assume Newtonian gravity in a static universe filled with a homogeneous dust.
Let the ship be at the origin. The ship feels a force proportional to $x$ towards the centre of the sphere of radius $x$ centered at $\pmb{x}$, but it also feels the exact opposite force towards the centre of the identical but disjoint sphere centered at $\pmb{-x}$, so these two forces cancel exactly. In each case, I'm only considering the mass inside the ball and ignoring the mass outside it, per the shell theorem.
The same logic applies to any arbitrary $\pmb{x}$.