Prove that the given set is bounded above in $\mathbb{Q}$ but does not have a Supremum in $\mathbb{Q}$
Let me give you an example to show you why talk about the real numbers and the supremum in the reals is really not a good idea in general.
Consider the set $\mathfrak{Q}=\mathbb{Q}\cap \Bigl((-\infty,0)\cup[1,\infty)\Bigr)$. That is, $\mathfrak{Q}$ consists of all rationals that are either negative, or greater than or equal to $1$.
Now let $\mathcal{S}=\{q\in\mathbb{Q}\mid q\lt 0\}$, the negative rationals.
Notice that $\mathcal{S}$ is a subset of $\mathfrak{Q}$, and is bounded above in $\mathfrak{Q}$, since $1\in\mathfrak{Q}$, and for all $s\in \mathcal{S}$, $s\leq 1$.
Now, let me take a parallel argument to the one you are attempting to "show" that $S$ does not have a supremum in $\mathfrak{Q}$: Take $m=0$; then $m$ is the supremum of $\mathcal{S}$ in $\mathbb{R}$. Now suppose that $t\in\mathfrak{Q}$ is an upper bound on $\mathcal{S}$, and that $t\lt 0$. Then because $t\lt 0$ and $0$ is the supremum of $\mathcal{S}$ in $\mathbb{R}$, there exists $s\in \mathcal{S}$ such that $t\lt s\leq 0$. So $t$ is not an upper bound for $\mathcal{S}$ in $\mathfrak{Q}$, a contradiction. "Therefore", $\mathcal{S}$ does not have a supremum in $\mathfrak{Q}$.
Of course, the argument is completely false: because $1\in\mathfrak{Q}$ is the supremum of $\mathcal{S}$ in $\mathfrak{Q}$: note that $s\leq 1$ for all $s\in \mathcal{S}$; and if $t\in\mathfrak{Q}$ is strictly smaller than $1$, then because it is in $\mathfrak{Q}$ it must also be strictly smaller than $0$, so there exists $s\in \mathcal{S}$ with $t\lt s\leq 1$. That is:
- For all $s\in \mathcal{S}$, $s\leq 1$.
- For all $t\in\mathfrak{Q}$, if $t\lt 1$ then there exists $s\in \mathcal{S}$ such that $t\lt s$.
These two properties show that $1$ is the supremum of $\mathcal{S}$ in $\mathfrak{Q}$. Notice that $1$ is not the supremum of $\mathcal{S}$ in $\mathbb{R}$, or even in $\mathbb{Q}$. But it is the supremum of $\mathcal{S}$ in $\mathfrak{Q}$.
So you should really not use the supremum of your $S$ in $\mathbb{R}$, except perhaps as a "behind the scenes" guide to what you want. It could be, at least in principle, possible for a subset of $\mathbb{Q}$ to have a supremum in $\mathbb{Q}$ which is different from its supremum in $\mathbb{R}$ (the supremum in $\mathbb{Q}$ would have to be larger than the supremum in $\mathbb{R}$, just as it is in my example above).
First: Since you are saying what $m$ is, you should not then follow it up with "There exists an $m$ that belongs to the real numbers." You would just say "Then $m$ belongs to the real numbers and..."
Second: That said, that part of the argument is incorrect.
To show that a subset $S$ of $\mathbb{Q}$ is bounded above in $\mathbb{Q}$, you need to exhibit an element $m$ of $\mathbb{Q}$ such that $s\leq m$ for all $s\in \mathbb{Q}$. Showing an element not in $\mathbb{Q}$ does not show the set is bounded above in $\mathbb{Q}$. So your argument about $m$ doesn't work.
Third: You cannot conclude that an upper bound of $S$ in $\mathbb{Q}$ will be strictly smaller than $\sqrt{3}$. And if you assume it, then you are assuming your own contradiction:
You know that if $t$ is an upper bound to $S$ in $\mathbb{Q}$, then by virtue of being in $\mathbb{Q}$, $t$ is also in $\mathbb{R}$ and an upper bound to $S$. And therefore $t$ will be greater than or equal to the supremum of $S$ in $\mathbb{R}$, since $S$ is also bounded above in $\mathbb{R}$. So you know that $t\geq \mathrm{sup}_{\mathbb{R}}(S)$. Now, $\mathrm{sup}_{\mathbb{R}}(S)$ happens to be $\sqrt{3}$, but you haven't proven that this is the case, so you cannot invoke that. And you don't explain how you get that $\sqrt{3}\leq t$ (presumably from the supremum property, but you haven't shown that $\sqrt{3}$ is the supremum in $\mathbb{R}$, so you cannot invoke the supremum property for $\sqrt{3}$).
Fourth: So if you know anything about a possible supremum of $S$ in $\mathbb{Q}$, it is that it is greater than $\sqrt{3}$ (the supremum of $S$ in $\mathbb{R}$), since it must be greater than or equal, and cannot be equal. So even if your argument about what happens if $t\lt\sqrt{3}$ were correct (well, technically it is because your assumption is an impossibility, which is why you get a contradiction), you would not be done; you would still need to consider the possibility that the supremum $t$ of $S$ in $\mathbb{Q}$ is greater than $\sqrt{3}$, instead of smaller.
Added. Comment on 2nd attempt.
To show that $S$ is bounded above in $\mathbb{Q}$, you should produce an element of $\mathbb{Q}$ that bounds $S$ above. Going around to $\mathbb{R}$ does not do it. Your argument is "gappy", in that it assumes without stating that you can take $m$ and produce a rational bigger than $m$ in order to conclude that there is a rational that bounds $S$ above. Well, why not produce one?
I could argue that the set $\{1.5\}$ of rationals is bounded above in the rationals by saying "Well, $\pi\in\mathbb{R}$ is greater than all elements of $\{1.5\}$, and so $\{1.5\}$ is bounded above in $\mathbb{R}$, and therefore $\{1.5\}$ is bounded above in $\mathbb{Q}$." But isn't it so much better, clearer, and simpler, to just say "$2\in\mathbb{Q}$, and $2$ is an upper bound for ${1.5}$"?
You merely assert that $m$ is greater than or equal to all $r\in\mathbb{Q}$; it should be proven.
Even assuming the rest of the argument were correct (it isn't), you cannot simply assume that $m=\sup(S)$ in $\mathbb{R}$. If you don't prove it, then your argument is contingent on that extra assumption. To complete the proof, you would have to also consider the case in which $m$ is not the supremum of $S$ in $\mathbb{R}$.
Next: You continue to introduce your own contradiction when you assume that $t$ is both an upper bound to $S$ in $\mathbb{Q}$ and that $t\lt m$. That is impossible to begin with because you are assuming that $m$ is the supremum of $S$ in $\mathbb{R}$: any upper bound to $S$ has to be greater than or equal to the supremum of $S$ (any upper bound in $\mathbb{Q}$ is also in $\mathbb{R}$). So you would still need to consider the possibility that $t\geq m$, which you never do. So even if your argument about $t$ were correct (it's not), it would still be incomplete.
Finally: you merely assert "there would exist an $r\in S$ such that $r\gt t$. Just saying so doesn't prove it. You have to prove that there is such a thing.
Here's the issue here: you know that $S$ has a supremum in $\mathbb{R}$; call it $m$. If $t$ is any element of $\mathbb{Q}$ that is an upper bound to $S$ in $\mathbb{Q}$, then it is also an upper bound to $S$ in $\mathbb{R}$, so by definition of the supremum, you must have $m\leq t$. If $m\lt t$, then by the properties of the rationals, you should show or justify that there has to be a number $t'$ which (i) is an upper bound to $S$; (ii) is in $\mathbb{Q}$; and (iii) is strictly between $m$ and $t$, $m\lt t' \lt t$. That means that the only number that has a shot at being the supremum of $S$ in $\mathbb{Q}$ is $m$ itself, the supremum of $S$ in $\mathbb{R}$ (why? You need to explain this). So then you should prove that the supremum of $S$ in $\mathbb{R}$ is not a rational number, and that will give you a correct argument.
There is no need to mention the real numbers.
As Pacciu says, we have $S=\{ r\in \mathbb{Q}:\ r\leq 0 \text{ or } r^2\leq 3\}$. So $\frac{2}{1}$ is an upper bound for $S$ since $\frac{2}{1} \gt 0$ and if $ r \gt \frac{2}{1}$ then $r^2 \gt \left(\frac{2}{1}\right)^2 = 4 \gt 3$.
Now suppose $\frac{p}{q}$ with $p$ and $q$ positive integers is an upper bound for $S$, i.e. $p^2 \ge 3 q^2 $. Then $\frac{3q^2+4pq-p^2}{4q^2}$ is a smaller upper bound for $S$ and so there is no lowest upper bound or supremum for $S$ in $\mathbb{Q}$.
Incidentally, this gives the sequence $\frac{2}{1}, \frac{7}{4}, \frac{111}{64}, \frac{28383}{16384}, \cdots$, approaching $\sqrt{3}$ from above.