Why is a metric space an open subset of itself?

What you’re missing is that in the space $A$ the points $a$ and $b$ are interior points of $A$, even though they are not interior points of $A$ in the space $\Bbb R$. Let $r=b-a$. By definition the open ball of radius $r$ centred at $a$ in the space $A$ is $\{x\in A:|x-a|<r\}$, the set of points in $A$ that are less than distance $r$ from $a$. And

$$\{x\in A:|x-a|<r\}=[a,b)\subseteq A\;,$$

so $a$ is in the interior of $A$ in the space $A$: the open ball of radius $r$ around $a$ is contained in $A$.

You can do the same thing at $b$.

Given a non-empty set $A$ endowed with a metric $d$, we define the metric space as $X:=(A,d)$.

For each $p\in A$, the open ball centred at point $p$ with radius $r$ is given by $N_p(r):=\{ q\in A:d(p,q)<r,r\in\mathbb{R}_{>0} \}\subset A$, implying that $A$ is open relative to itself.