Is it in general true that a space is not homeomorphic to the punctured version of this space?

Another example: $\mathbb C \setminus \mathbb Z$. This is connected and homeomorphic to the punctured version of itself.

Any infinite discrete space is homeomorphic to itself minus any point. For example, the map $n\mapsto (n+1)$ is a homeomorphism $\mathbb{N}\to \mathbb{N}\setminus\{0\}$ (where $\mathbb{N}$ has the discrete topology).

The same example works if you give $\mathbb{N}$ the trivial topology or the Alexandrov topology (where open sets are upwards-closed sets).

$\mathbb{Q}$ is homeomorphic to its punctured version (all countable metric spaces without isolated points are). Same for the irrationals $\mathbb{P}$.