Why isn't there an atom with an electron at the center and a proton on the outside?

the proton is 1800 times more massive than the electron. when the two of them are in a bound state (attracting each other via the electrostatic force) the proton's inertia makes it sit almost still and it lets the lighter electron do all the orbiting.

Please note that in actuality, the electron and the proton are both orbiting the center of mass that they share, but because the proton is so much heavier, that center of mass is so close to the proton's own center of mass that the proton's orbital radius around it is very small.

And how would that element be different from "normal" Hydrogen?

The Coulomb interaction would look the same as the separation $r$ is unchanged by the particle exchange.

Plus, if I were an observer in the rest frame of the electron, I would see the proton orbiting me.

You need another interaction to "break" the "symmetry", i.e. a neutron (strong force) or another atom (molecular stuff). Otherwise these two "states" are indistinguishable.

Why isn't there an atom with an electron at the center and a proton on the outside?

Because a proton has 1836 times more mass than an electron does. Thinking classically, this means its orbit around the center of mass is about 1836 times smaller than the electron’s. In other words, it basically stays fixed.