Is there a maximum value between open $(0,1)$ set?

Your answer doesn't make much sense. It's equal to $1$ if anything (see Is it true that $0.999999999\ldots = 1$?).

You have to look at the definition: $m$ is the maximum if $m\in(0,1)$ and $\forall x\in(0,1),\ x\le m$. Clearly, this fails for any element $m\in(0,1)$ because $m<(m+1)/2<1$.

That's a very creative answer, but infinity is tricky. If $$ 1 - \frac{1}{10^\infty} < 1 $$ would you not have to admit that $$ 1 - \frac{1}{10^\infty} < 1 - \frac{1}{2} \cdot \frac{1}{10^\infty} < 1 $$ as well? This would contradict your assertion that $1 - \frac{1}{10^\infty}$ was the maximum value of $(0,1)$.

You are right that this seems fishy, and this is the whole problem with treating infinity like a number. The only reasonable real number value of $\frac{1}{10^\infty}$ would be zero, so your number is $1$, and $1$ is not in the set.

A maximum must be in the set. In the case of $(0,1)$ if $a \in (0,1)$ then the point $(1+a)/2$ is larger than $a$ and still in the set $(0,1).$ So there can be no max. There is a sup, namely $1,$ of the set $(0,1).$