Small-angle approximation of $ \frac{\sin^2 x}{x^2 \sqrt{1-\frac{\sin^2 x}{3}}} $
Suppose for simplicity you had two polynomials $P(z) = 1+ z + z^2 + 4z^4 + 7z^5$ and $Q(z) = 1 + 2z + 3z^2 + 4z^3$, and I asked you to calculate the product $P(z)Q(z)$... but not the entire thing. Suppose I only want the terms up to quadratic term; i.e if we write \begin{align} P(z)Q(z) &= a_0 + a_1z + a_2z^2 + a_3z^3 + a_4z^4 + a_5z^5 +a_6z^6 + a_7z^7 +a_8z^8 \end{align} then I'm asking you to find the coefficients $a_0,a_1,a_2$ (but for now, let's just say for some reason I'm interested in what happens when $|z|$ is very small up to an accuracy of quadratic order, so I don't really care about the rest of the terms ). Well, we just multiply everything out: \begin{align} P(z)Q(z) &= (1+ z + z^2 + 4z^4 + 7z^5)(1 + 2z + 3z^2 + 4z^3) \\ &= 1 + (1\cdot 2z + z \cdot 1) + (1\cdot 3z^2 + z\cdot 2z + z^2 \cdot 1) \\ &+ \text{(terms involving $z^3$ or higher, which I don't care about for now)} \\ &= 1 + 3z + 6z^2 + O(z^3) \end{align} In other words, because in my final product, I'm only interested in calculating up to the quadratic term, I can simply truncate the polynomials $P$ and $Q$ to quadratic order, and then multiply them out (and then again only keep terms up to quadratic order): \begin{align} P(z)Q(z) &= (1 + z + z^2 + \cdots)(1 + 2z + 3z^2 + \cdots) \\ &= 1 + 3z + 6z^2 + O(z^3) \end{align}
Again, because I'm only interested up to quadratic order, there's no need for me to keep any terms beyond that for $P(z)$ and $Q(z)$, because if I approximate $P(z) \approx 1+ z + z^2 + \color{red}{4z^4}$ (i.e I keep the $4^{th}$ order term) and I multiply with $Q(z) = 1+2z+3z^2 + 4z^3$, then the red term multiplied with anything in $Q(z)$ will yield terms which are $4^{th}$ order or higher (which I don't care about).
But what you should not do is truncate $P(z)$ and $Q(z)$ up to linear order, and say that \begin{align} P(z)Q(z) & \approx (1+z)(1+2z) = 1 + 3z + 2z^2 \end{align} Because in this way, you're missing out other second order contributions (by multiplying constant term of $P$ with quadratic term of $Q$ and vice-versa).
This is how you know how many terms you need to use in your approximation. In your case, you want to approximate \begin{align} f(x) &= \dfrac{\sin^2x}{x^2\sqrt{1 - \frac{\sin^2x}{3}}} \end{align} up to $2^{nd}$ order. So, what do we do? We write things as a product first: \begin{align} f(x) &= \left(\dfrac{\sin x}{x}\right)\cdot\left(\dfrac{\sin x}{x}\right) \cdot \left(\dfrac{1}{\sqrt{1- \frac{\sin^2x}{3}}}\right)\tag{$1$} \end{align} Now, we have to expand each bracketed term up to atleast second order in $x$ and then multiply the result together. First: \begin{align} \dfrac{\sin x}{x} &= \dfrac{x - \dfrac{x^3}{6} + O(x^4)}{x} = 1 - \dfrac{x^2}{6} + O(x^3) \tag{$2$} \end{align} Next, we recall that \begin{align} \dfrac{1}{\sqrt{1-z}} &= 1+ \dfrac{z}{2} + \dfrac{3z^2}{8} + O(z^3) \end{align} Now, plug in $z= \frac{\sin^2x}{3} = x + O(x^3)$, to get \begin{align} \dfrac{1}{\sqrt{1-\frac{\sin^2x}{3}}} &= 1+ \dfrac{1}{2}\left(\dfrac{\sin^2x}{3}\right) + \dfrac{3}{8}\left(\frac{\sin^2x}{3}\right)^2 + O((\sin^2 x)^3) \\ &= 1 + \dfrac{1}{2}\left(\dfrac{x^2 + O(x^4)}{3}\right) + O(x^4) + O(x^6) \\ &= 1 + \dfrac{1}{6}x^2 + O(x^4)\tag{$3$}, \end{align} where in the second line, hopefully it's clear how I got the various terms: for example $\sin x = x + O(x^3)$, so $\left(\frac{\sin^2x}{3}\right)^2 = \frac{1}{9}\sin^4x = \frac{1}{9} (x + O(x^3))^4 = O(x^4)$. Therefore, the final answer is obtained by plugging $(2)$ and $(3)$ into $(1)$ : \begin{align} f(x) &= \left(1 - \dfrac{x^2}{6} + O(x^3)\right)^2 \cdot \left(1 + \dfrac{1}{6}x^2 + O(x^4)\right) \\ &= 1 - \dfrac{x^2}{6} + O(x^4) \end{align}
Long story short, if your end goal is to calculate up to second order, then at each stage of your algebra make sure you're keeping terms atleast up to $x^2$.
$${\sin x\over x}\approx1-{1\over6}x^2$$
so
$${\sin^2x\over x^2}\approx\left(1-{1\over6}x^2\right)^2\approx1-{1\over3}x^2$$
not just $1$. We get
$${\sin^2x\over x^2\sqrt{1-{\sin^2x\over3}}}\approx\left(1-{1\over3}x^2\right)\left(1+{1\over6}x^2\right)\approx1-{1\over6}x^2$$