What are relative open sets?

Forget your definition above. The general notion is:

Let $X$ be a topological space, $A\subset X$ any subset. A set $U_A$ is relatively open in $A$ if there is an open set $U$ in $X$ such that $U_A=U\cap A$.

I think that in your definition $N_\epsilon(x)$ is meant to denote an open neighborhood of radius $\epsilon$ of $x$, ie $(x-\epsilon,\ x+\epsilon)$. As you can see, this would agree with the definition I gave you above.

Recall that generally, $O$ is open if for every $x\in O$ there exists some $\varepsilon>0$ such that $N_\varepsilon(x)\subseteq O$.

Being open relative to $X$ means that there is some open set $O'$ such that $O=O'\cap X$, and equivalently for every $x\in O$ there is some $\varepsilon>0$ such that $N_\varepsilon(x)\cap X\subseteq O$.

For example $O=\{0\}$ is not open in $\mathbb R$, but if we consider $X=\{0\}$ then for $\varepsilon=1$ we have that $N_\varepsilon(0)\cap X\subseteq O$, and therefore it is open relative to $X$.