If $m$ and $n$ are integers, show that $\left|\sqrt{3}-\frac{m}{n}\right| \ge \frac{1}{5n^{2}}$
You're asking to prove, for integers $m$ and $n$ (with the assumption $n \neq 0$), that
$$\left|\sqrt{3}-\frac{m}{n}\right| \le \frac{1}{5n^2} \tag{1}\label{eq1A}$$
Note if $m = 0$, \eqref{eq1A} obviously holds. Otherwise, as this other answer states, WLOG, we may assume both $m$ and $n$ are positive since if they have opposite signs, the result is trivial, and if they are both negative, the result is the same as if they were both their absolute value equivalents instead.
As you've shown by rationalizing the numerator and stating it must be at least $1$ is that you have
$$\left|\sqrt{3}-\frac{m}{n}\right| = \left|\frac{3n^2-m^2}{\sqrt{3}n^2 + mn}\right| \ge \frac{1}{\sqrt{3}n^2 + mn} \tag{2}\label{eq2A}$$
If the denominator on the right side is $\le 5n^2$, then you get
$$\begin{equation}\begin{aligned} \sqrt{3}n^2 + mn & \le 5n^2 \\ \frac{1}{\sqrt{3}n^2 + mn} & \ge \frac{1}{5n^2} \end{aligned}\end{equation}\tag{1}\label{eq3A}$$
so combined with \eqref{eq2A}, this shows \eqref{eq1A} will be true.
Consider instead that the denominator is $\gt 5n^2$ to get
$$\begin{equation}\begin{aligned} \sqrt{3}n^2 + mn & \gt 5n^2 \\ mn & \gt (5 - \sqrt{3})n^2 \\ m & \gt (5 - \sqrt{3})n \\ \frac{m}{n} & \gt 5 - \sqrt{3} \\ -\frac{m}{n} & \lt - 5 + \sqrt{3} \\ \sqrt{3} -\frac{m}{n} & \lt - 5 + 2\sqrt{3} \lt -1.5 \\ \left|\sqrt{3}-\frac{m}{n}\right| & \gt 1.5 \gt \frac{1}{5n^2} \end{aligned}\end{equation}\tag{4}\label{eq4A}$$
As such, \eqref{eq1A} will still hold in this case as well. Since all possibilities have been covered, it proves \eqref{eq1A} is always true.
Using a technique similar to Liouville's theorem, $\sqrt{3}$ is a root of $P_2(x)=x^2-3$. Then, for any $\frac{m}{n}$ we have an $\varepsilon$ in between $\sqrt{3}$ and $\frac{m}{n}$ such that (this is MVT) $$\left|P_2\left(\frac{m}{n}\right)\right|= \left|P_2(\sqrt{3})-P_2\left(\frac{m}{n}\right)\right|= |P_2'(\varepsilon)|\cdot \left|\sqrt{3}-\frac{m}{n}\right|$$ or $$\left|\sqrt{3}-\frac{m}{n}\right|= \left|\frac{m^2-3n^2}{2\varepsilon \cdot n^2}\right|\geq \frac{1}{2\left|\varepsilon\right| \cdot n^2}\tag{1}$$
Now, if $\frac{m}{n}<\varepsilon<\sqrt{3}$ then $(1)$ becomes $\frac{1}{2\left|\varepsilon\right| \cdot n^2}>\frac{1}{2\sqrt{3}n^2}>\frac{1}{5n^2}$ and we are done.
If $\sqrt{3}<\varepsilon<\frac{m}{n}<\frac{5}{2}$ then $2\varepsilon<5$ and $(1)$ becomes $\frac{1}{2\left|\varepsilon\right| \cdot n^2}>\frac{1}{5n^2}$. So, we are done.
If $\sqrt{3}<\frac{5}{2}<\varepsilon<\frac{m}{n}$ then $$\left|\frac{m}{n}-\sqrt{3}\right|> \left|\frac{5}{2}-\sqrt{3}\right|= \frac{\frac{25}{4}-3}{\frac{5}{2}+\sqrt{3}}= \frac{13}{10+ 4\sqrt{3}}> \frac{1}{5\cdot 1^2}\geq \frac{1}{5\cdot n^2}$$ for all $n\geq1$.
One aims to give an optimal lower bound as follows:
Theorem. $|\sqrt{3}-\frac mn |\geq \frac 1{(2+\sqrt{3})n^2}$
Proof. Clearly the lower bound is achieved when $n=1,m=2.$ As in other proofs, one may assume without loss of generality that $m,n$ are positive and $n\geq 2$. Observe first that $$\frac 53<\sqrt{3}<\frac 74.$$ We divide into two cases.
Case 1. $\frac m n<\frac 7 4.$ Then $$|\sqrt{3}-\frac m n|=\frac{|3n^2-m^2|}{n^2(\sqrt{3}+\frac m n)}$$ $$>\frac 1{n^2(\frac 7 4+\frac 7 4)}=\frac 1{(3.5)n^2}>\frac 1{(2+\sqrt{3})n^2}.$$
Case 2. $\frac m n\geq \frac 7 4.$ Here one considers three subcases: $n=2,n=3$ and $n\geq 4$. If $n=2,$ then $\frac m n\geq \frac 7 4\Rightarrow m\geq 4,$ so $\frac m n\geq 2$ and hence $$|\sqrt{3}-\frac m n|=\frac m n-\sqrt{3}\geq 2-\sqrt{3}\geq \frac 1{(2+\sqrt{3})n^2}.$$ If $n=3$, then $\frac m n\geq \frac 7 4\Rightarrow m\geq \frac{21}4,$ so $m\geq 6$ and $\frac m n\geq 2$ and the result follows in exactly the same way as the above subcase. It remains to check the subcase when $n\geq 4$, but then $$ |\sqrt{3}-\frac m n|=\frac m n-\sqrt{3}\geq \frac 7 4-\sqrt{3}>\frac 1{(2+\sqrt{3})\cdot 4^2}\geq \frac 1{(2+\sqrt{3})n^2},$$ where one used $$\frac 7 4-\sqrt{3}>\frac 1{(2+\sqrt{3})\cdot 4^2}$$ $$\Leftrightarrow \frac 7 4>\sqrt{3},$$ which is true.
Combining all cases, the result is proven.
Edit. (More details) $$\frac 7 4-\sqrt{3}>\frac 1{(2+\sqrt{3})\cdot 4^2}$$ $$\Leftrightarrow \frac {7-4\sqrt{3}}4>\frac 1{(2+\sqrt{3})\cdot 4^2}$$ $$\Leftrightarrow 7-4\sqrt{3}>\frac 1{(2+\sqrt{3})\cdot 4}$$ $$\Leftrightarrow (7-4\sqrt{3})(2+\sqrt{3})>\frac 1 4$$ $$\Leftrightarrow 2-\sqrt{3}>\frac 1 4$$
$$\Leftrightarrow \frac 7 4>\sqrt{3},$$ which is true.