Why is a straight line the shortest distance between two points?
I think a more fundamental way to approach the problem is by discussing geodesic curves on the surface you call home. Remember that the geodesic equation, while equivalent to the Euler-Lagrange equation, can be derived simply by considering differentials, not extremes of integrals. The geodesic equation emerges exactly by finding the acceleration, and hence force by Newton's laws, in generalized coordinates.
See the Schaum's guide Lagrangian Dynamics by Dare A. Wells Ch. 3, or Vector and Tensor Analysis by Borisenko and Tarapov problem 10 on P. 181
So, by setting the force equal to zero, one finds that the path is the solution to the geodesic equation. So, if we define a straight line to be the one that a particle takes when no forces are on it, or better yet that an object with no forces on it takes the quickest, and hence shortest route between two points, then walla, the shortest distance between two points is the geodesic; in Euclidean space, a straight line as we know it.
In fact, on P. 51 Borisenko and Tarapov show that if the force is everywhere tangent to the curve of travel, then the particle will travel in a straight line as well. Again, even if there is a force on it, as long as the force does not have a component perpendicular to the path, a particle will travel in a straight line between two points.
Also, as far as intuition goes, this is also the path of least work.
So, if you agree with the definition of a derivative in a given metric, then you can find the geodesic curves between points. If you define derivatives differently, and hence coordinate transformations differently, then it's a whole other story.
Let me start by saying that on a gut level, I agree with everything you said. But I feel like I should make this argument anyway, since it might help you (and me!) sort out ideas on the matter.
It doesn't seem inconsistent to argue that the model of Euclidean space (defined by, say, the Hilbert axioms) as $\Bbb R^n$ really gets around all the philosophical questions. We can ask why $\Bbb R$ and such, but taken as an object in its own right, the standard inner product defines everything from the geometry to the topology to the notion of size.
In this view, the integral you mentioned can be taken as the definition of "length" of a curve (in $\Bbb R^2$, I think), observing that it matches with the Lebesgue measure when the curve under consideration is given by an affine transformation (although this is formally irrelevant). The definition is motivated not as being broken down into straight lines, but rather into vectors, which have a different definition of length (this does not trouble me much: it is only wishful thinking that we use the same term for each). The notion of a "line" per se arises as a fairly natural question: what is the infimum of the length between two points and if so, is there actually a curve that achieves it? Once you see that not only is the answer "yes" but also "and it's unique", it's not much of a stretch to think these objects are worth adding to our basic understanding of the space.
As for the remark choosing the Manhattan distance: nothing prevents you from doing this, but if you prefer this to be your norm (which you well might, for the reasons you described above), then you lose all aspects of geometry relating to angles. You also lose the uniqueness of minimal-length curves, and perhaps you then become less interested in the question. From the omniscient perspective, we might see this as a tragedy, an acceptable loss, or even as a gain. This objection, as well as Will Jagy's comment, only appear to highlight the flexibility we have in terms of which formalisms to use.
Your other question is of course much harder to answer, but I think a nice reduction of the question is "What makes $\Bbb R^3$ the most physical-feeling model?" The question is particularly interesting in light of the fact that $\Bbb R^3$ is certainly not a complete model of space for actual physics! But I do not think you would be taken seriously if you tried to argue that the universe is not a manifold. For some reason, (open subsets of) $\Bbb R^n$ is locally "almost right".
This is just me talking out of you-know-where: It could be that the reason we have such strong intuitions about straightness and distance is because of evolutionary pressures. People who could intuit how to get from place to place efficiently would not burn the unnecessary calories, and in a less sheltered world this could help them reach the age of sexual viability. Once we began thinking inductively then we would be allowed to think of the notion of straightness as going on forever, and as an general construct rather than a situational feature. But by then it would be too late to straighten out the conflation of straightness and linearity, and we would need to wait a long time before we could do so with any rigor.