Could someone explain the concept of a set being "open relative" to another set?

In order to know what "open relative to" means, you have to first know what "open" means:

A subset S of a metric space X is called open if, for every point p of S, there is a positive real number $\epsilon$ such that every point in X of distance less than $\epsilon$ from p lies in S.

Now, if we're working in some metric space and consider a subset A of that space, you can, if you wish, disregard the rest of the space and think of A as a metric space in itself. The notion of "open relative to A" is what you get when you realize, looking at the above definition, that some subsets of A that are not open subsets of the larger space may still be open subsets of A.

For example, suppose you look at the interval I=[0,1] as a metric space in itself, and disregard the rest of the real line. Then the subset S=(0.5,1] may seem at first not to be an open set, because you think, "Hey, there are points arbitrarily close to 1 that are not in S, so 1 is actually a boundary point!" Well, yes... except that none of those arbitrarily-close points are actually in I, so from the point of view of I as a space in itself, they don't exist. Every point in I sufficiently close to 1 is also in S, so if we want to work in I alone, we have to concede that S is open.

The moral of this story is that sets aren't inherently open: the definition depends on what metric space they are considered as subsets of. If you are only talking about one space it's harmless to just write "open," but otherwise you may have to be more specific. If you are considering both $\mathbb{R}$ and I, for example, writing "(0.5,1] is open" is ambiguous. So instead you write "(0.5,1]" is open relative to I."

We say that a set $S$ is open relative to $A$ if for all $x\in S$, we can find an open ball (open with the topology/metric of $A$) centered at $x$ that is contained in $S,$ i.e. there exists $\epsilon>0$ such that $B(x,\epsilon)\subset S.$

For your example, take $(0,1)$ in $\mathbb{R}.$ The open balls in $\mathbb{R},$ with standard metric are the open intervals. Take any point in $(0,1),$ you can always find $\epsilon$ such that $0<|x-\epsilon|<1$, because of the density of $\mathbb{R}.$

Now take the same subset in $\mathbb{R}^2.$ The open balls are now the open disks of the form $$ (x-a)^2+(y-b)^2<r^2, $$ that is, an open disk of radius $r$ centered at $(a,b)$. Now for any point $(x,y)$ in $(0,1),$ a line segment, and for any $\epsilon$, the open ball $B(x,\epsilon)$ is going to leave the line segment. To see it, any point in $(0,1)$ will be of the form $(x,0)$ (or $(0,y)$), and so a disk centered at these kind of point is bound to have a $y$ coordinate (or a $x$) other than $0$, leaving it impossible for $B$ to be in $(0,1)$ entirely.

Let $X$ denote a topological space. Let $A$ denote a subset of $X$. The set $B\subset A$ is open in $A$ if $B= A \cap U$ for some open set in $X$. When $X ={\mathbb R}$ then $B$ is a union of open intervals intersected with $A$. For example, suppose that $A$ denotes the rational numbers between $0$ and $1$. The an open interval in $A$ is an open interval of rationals. If $A$ is a closed interval, then an open interval is either open in the interior, or it is a half open interval in $A$. I specify intervals in these examples, because an open set of reals is a union of open intervals.