Why Galilean spacetime is not $\mathbb{E}^4$?

My argument would be something along the following lines: assume that your spacetime is $\mathbb E^4$ and determine its isometry group. This turns out to be $O(4)\ltimes\mathbb R^4$, which doesn't coincide with the Galilei group (indeed in special relativity $O(1,3)\ltimes\mathbb R^4$ is the full Poincaré group, which is also the isometry group of flat Minkowski space-time). So the decomposition $\mathbb E^1\times\mathbb E^3$ is evocative of this fact.

In a four dimensional euclidean space $\newcommand{\ef}{\mathbb{E}^4}$$\ef$, you have no preferred time direction, so this is not equivalent to the four dimensional affine space $\newcommand{\af}{\mathbb{A}^4}$$\af$ together with its time function $\pi$. In short, you don't have enough structure.

Now if you take your $\ef$ and add a time function, then you actually have too much structure. One of the key facts about galilean spacetime is that you should not be able to say event at $t_1$ occured at the same spatial point as some later event at $t_2$. However, $\ef$ with the time function allows you to do this. If you have an event at $t_1$, you associate with the closest event at the time $t_2$, with closeness being measure by the metric in $\ef$. (Notice this trick does not work in $\af$ because there is not metric in $\af$. To measure distances of simultaneous points, you added by hand that each $\pi^{-1}({t})$ was in fact a Euclidean space.) Since $\ef$ with a time function allows you to associate points at different times, it is not a good model of galilean spacetime.

Clearly $\mathbb{E}^1 \times \mathbb{E}^3$ fails in the same way, but much more obviously. In this case you can clearly associate the event $(t_1,\mathbf{r})$ with the event $(t_2,\mathbf{r})$, and so you clearly have a notion of events at different times occuring at the same point in space.