If $X$ is homeomorphic to dense subspace $Y\subseteq X$, then $X=Y$

It's not true. Consider $Y=\mathbb N$ and $X=\mathbb Z$, both with the trivial (indiscrete) topology.

Then $Y$ is dense in $X$ (every non-empty subset is dense in this topology) and homeomorphic to $X$

(there is a bijection between $Y$ and $X$, and every map to an indiscrete space is continuous),

but clearly $X\ne Y$.

For a more interesting example, let $X=\Bbb Q$ with the usual topology, and let

$$Y=\left\{\frac{m}{2^n}\in\Bbb Q:m,n\in\Bbb Z\text{ and }n\ge 0\right\}$$

be the set of dyadic rationals. Then $Y$ is a proper subset of $X$ that is both dense in $X$ and homeomorphic to $X$.