Why does shrink_to_fit (if the request is fulfilled) cause reallocation?

The underlying memory management system defines what is possible, and typically, they don't allow to return parts of the allocated memory: if you got n bytes, you either return n bytes, or nothing.
Returning the last m bytes (with m < n), or worse, returning m bytes in the middle of the n bytes, would of course be possible to offer, but consider the extra complexity needed to handle this correctly.
Of course, there could be some out there that do offer it, but your C++ compiler and the language definition doesn't necessarily know which ones run below it in the OS, so they have to accept the possibility that a reallocation is needed. Note that they don't guarantee that it will be needed - they just expect it.

The container does not allocate/deallocate the memory on itself, but it is it's allocator who does it.

For the (vector's) allocator to be able to deallocate the memory, it needs to be provided the exact the same pointer as the pointer to the memory it has allocated for the vector's data.

At this is the begin of the vector data and not the begin of the "not more used" data.

Basically, we are talking about this allocator's allocation/deallocation methods:

pointer allocate( size_type n, const void * hint = 0 );
void deallocate( T* p, std::size_t n );

The argument T* p of the deallocate will be the same as the pointer returned from allocate ( == begin of vector's data). This is what the vector's implementation will pass to the deallocate.

It is surely imaginable to have a custom vector implementation, who would be able to pass any pointer in range [data, data+size] to allocators deallocate method. One could construct such an allocator to be able to deal with it. But then all other allocators would need to conform to this API, also the standard allocator.

Then something like this would need to be able to "work":

int* p = new int[100];
delete [] (p + 50);  // imagine making this work

This would add extra complexity, performance and other issues.