Solving equation $\tan(5π\cos\alpha) = \cot(5π\sin\alpha)$

Core approach

And then used the solution of Trigonometric Equation $\tan(θ)=\tan(β)$…

That sounds like a good approach to me. So what you're saying is that you got to

$$ 5\pi\cos\alpha = n\pi+\tfrac\pi2-5\pi\sin\alpha,\quad n\in\mathbb Z $$

and then solved this for $\alpha$? How exactly?

Tangent half-angle approach

Personally I would use the tangent half-angle formulas to turn this trigonometric equation into a polynomial one:

$$ t:=\tan\frac\alpha2\quad \sin\alpha=\frac{2t}{1+t^2}\quad \cos\alpha=\frac{1-t^2}{1+t^2}\\ 5\frac{1-t^2}{1+t^2}=n+\frac12-5\frac{2t}{1+t^2}\\ 10-10t^2=2n+2nt^2+1+t^2-20t\\ $$

So what values of $n$ should you be considering? Let'suse the fact that $\sin\alpha\in[-1,1]$ and the same for $\cos\alpha$.

$$5[-1\ldots 1]=n+\tfrac12-5[-1\ldots 1]\\n=5[-1\ldots 1]+5[-1\ldots 1]-\tfrac12$$

So a conservative estimate would be $n\in\{-10,-9,-8,\ldots,7,8,9\}$. Since you can't have both $\sin\alpha$ and $\cos\alpha$ be close to $\pm1$ at the same time, not all of these $n$ will have solutions, but this is good enough for now. Take each $n$ and compute the resulting $t$ (at most two for each $n$). You get $28$ different values.

$$ \begin{array}{rl|rr|r} t && \alpha && n \\\hline -18.88819 = & -\sqrt{79} - 10 & -3.035805 = & -173.93882° & -6 \\ -5.18925 = & -\frac{1}{3} \, \sqrt{31} - \frac{10}{3} & -2.760848 = & -158.18495° & -7 \\ -1.47741 = & \frac{1}{3} \, \sqrt{31} - \frac{10}{3} & -1.951541 = & -111.81505° & -7 \\ -1.11181 = & \sqrt{79} - 10 & -1.676584 = & -96.06118° & -6 \\ -0.90871 = & -\sqrt{119} + 10 & -1.475215 = & -84.52361° & -5 \\ -0.76274 = & -\frac{1}{3} \, \sqrt{151} + \frac{10}{3} & -1.303204 = & -74.66809° & -4 \\ -0.64575 = & -\sqrt{7} + 2 & -1.146765 = & -65.70481° & -3 \\ -0.54575 = & -\frac{1}{7} \, \sqrt{191} + \frac{10}{7} & -0.999154 = & -57.24732° & -2 \\ -0.45630 = & -\frac{1}{9} \, \sqrt{199} + \frac{10}{9} & -0.856168 = & -49.05481° & -1 \\ -0.37334 = & -\frac{1}{11} \, \sqrt{199} + \frac{10}{11} & -0.714628 = & -40.94519° & 0 \\ -0.29387 = & -\frac{1}{13} \, \sqrt{191} + \frac{10}{13} & -0.571642 = & -32.75268° & 1 \\ -0.21525 = & -\frac{1}{3} \, \sqrt{7} + \frac{2}{3} & -0.424031 = & -24.29519° & 2 \\ -0.13460 = & -\frac{1}{17} \, \sqrt{151} + \frac{10}{17} & -0.267592 = & -15.33191° & 3 \\ -0.04783 = & -\frac{1}{19} \, \sqrt{119} + \frac{10}{19} & -0.095581 = & -5.47639° & 4 \\ 0.05294 = & -\frac{1}{21} \, \sqrt{79} + \frac{10}{21} & 0.105787 = & 6.06118° & 5 \\ 0.19271 = & -\frac{1}{23} \, \sqrt{31} + \frac{10}{23} & 0.380745 = & 21.81505° & 6 \\ 0.67686 = & \frac{1}{23} \, \sqrt{31} + \frac{10}{23} & 1.190052 = & 68.18495° & 6 \\ 0.89944 = & \frac{1}{21} \, \sqrt{79} + \frac{10}{21} & 1.465009 = & 83.93882° & 5 \\ 1.10046 = & \frac{1}{19} \, \sqrt{119} + \frac{10}{19} & 1.666377 = & 95.47639° & 4 \\ 1.31107 = & \frac{1}{17} \, \sqrt{151} + \frac{10}{17} & 1.838389 = & 105.33191° & 3 \\ 1.54858 = & \frac{1}{3} \, \sqrt{7} + \frac{2}{3} & 1.994827 = & 114.29519° & 2 \\ 1.83233 = & \frac{1}{13} \, \sqrt{191} + \frac{10}{13} & 2.142438 = & 122.75268° & 1 \\ 2.19152 = & \frac{1}{11} \, \sqrt{199} + \frac{10}{11} & 2.285425 = & 130.94519° & 0 \\ 2.67853 = & \frac{1}{9} \, \sqrt{199} + \frac{10}{9} & 2.426964 = & 139.05481° & -1 \\ 3.40290 = & \frac{1}{7} \, \sqrt{191} + \frac{10}{7} & 2.569951 = & 147.24732° & -2 \\ 4.64575 = & \sqrt{7} + 2 & 2.717562 = & 155.70481° & -3 \\ 7.42940 = & \frac{1}{3} \, \sqrt{151} + \frac{10}{3} & 2.874000 = & 164.66809° & -4 \\ 20.90871 = & \sqrt{119} + 10 & 3.046012 = & 174.52361° & -5 \end{array} $$

All of these look like valid solutions to me: they satisfy the initial equation. Since the tangent half-angle formulas can't represent $\alpha=\pi$ (it corresponds to $t=\infty$), we also need to check that this is not a solution. And of course these $\alpha$ are arguments to trigonometric functions, so adding any multiple of $2\pi$ will be a solution, too. The above are all the solutions in the $\alpha\in(-\pi,+\pi]$ range.

Trigonometric identities instead of tangent half-angle formulas

Update: After reading some other answers, and seeing how they avoid the tangent half-angle formulas, I wanted to look up the computation for that using well-established identities. Starting from the equation

\begin{align*} 5\pi\cos\alpha &= n\pi+\tfrac\pi2-5\pi\sin\alpha,\quad n\in\mathbb Z \\ \sin\alpha+\cos\alpha &= \frac{2n+1}{10} \end{align*}

the sum on the left hand side is the most interesting part. Wikipedia list of trigonometric identities lists your $\tan\left(\tfrac\pi2-\theta\right)=\cot\theta$ under Reflections and also some formulas you can use to tackle that sum.

One approach uses shifts to turn $\cos$ into $\sin$ and product to sum identities in reverse to turn the sum into a product:

\begin{align*} \cos\alpha &= \sin(\alpha+\tfrac\pi2) \\ \sin(\theta+\varphi)+\sin(\theta-\varphi)&=2\sin\theta\cos\varphi \qquad\text{with } \theta:=\alpha+\tfrac\pi4, \quad \varphi:=\tfrac\pi4 \\ \sin\alpha+\cos\alpha = \sin\alpha + \sin(\alpha+\tfrac\pi2) &= 2\sin(\alpha+\tfrac\pi4)\cos\tfrac\pi4 = \sqrt2\sin(\alpha+\tfrac\pi4) \end{align*}

You might also start from a formula for angle sums:

\begin{align*} \sin\alpha\cos\beta + \cos\alpha\sin\beta &= \sin(\alpha+\beta) \\ \beta := \tfrac\pi4 \qquad & \cos\beta=\sin\beta=\tfrac1{\sqrt2} \\ \tfrac1{\sqrt2}\left(\sin\alpha+\cos\alpha\right) &= \sin\left(\alpha+\tfrac\pi4\right) \end{align*}

Either way you get

$$ \sin\alpha+\cos\alpha = \sqrt2\sin(\alpha+\tfrac\pi4) = \frac{2n+1}{10} \\ \sin(\alpha+\tfrac\pi4) = \frac{2n+1}{10\sqrt2} \\ \alpha = \arcsin\frac{2n+1}{10\sqrt2}-\frac14\pi \qquad\text{or}\qquad \alpha = \frac34\pi-\arcsin\frac{2n+1}{10\sqrt2} \qquad\pmod{2\pi} $$

where the second solution accounts for the fact that $\arcsin$ should be considered a multi-valued function, and I'd like to get all solution angles in some $2\pi$-wide interval. You'd consider any $n\in\mathbb Z$ for which

$$ -1\le\frac{2n+1}{10\sqrt2}\le1\\ -7.57\approx\frac{-10\sqrt2-1}2\le n\le\frac{10\sqrt2-1}2\approx6.57 $$

which matches the list in my original table of solutions.

Your range considerations

But the basic condition of using the above result is that $\beta$ lies between $\left(-\frac π2,\frac π2\right)$.

I'm not sure where you get this condition from. Neither the move from $\cot$ to $\tan$ nor the approach for solving $\tan\theta=\tan\beta$ does warrant such a restriction, as far as I can reason about it.

And so gives $\sin \alpha $ lies between $\left(0,\frac 15\right)$

Since some of the solutions in the above table are outside that range and appear to be valid, that's not the case.

Using the formula in the question, we get $$5\pi\cos\alpha=n\pi+\frac \pi2-\sin\alpha$$Where n is an integer. Simplifying, we get $$\sin\alpha+\cos\alpha=\frac{2n+1}{10}$$ Now, there are many ways to show that $\sin\alpha+\cos\alpha=\sqrt2\sin(\alpha+\frac\pi4)$. I'm not going to prove that here. So, we have$$\sin(\alpha+\frac\pi4)=\frac{2n+1}{10\sqrt2}$$Now, moving the sine to the other side and subtracting $\frac\pi4$ on both sides, we get$$\alpha=\arcsin(\frac{2n+1}{10\sqrt2})-\frac\pi4$$However, this only holds when the argument of the arcsine lies between 1 and -1. Or,$$-1\leq\frac{2n+1}{10\sqrt2}\leq1$$solving this, we get$$\frac{-10\sqrt2-1}{2}\leq n\leq \frac{10\sqrt2-1}{2}$$Combining this with the original restraint that n is an integer, we get $n=0, \pm1, \pm2, \pm3, \pm4, \pm5, \pm6,-7$. Therefore, our final answer is$$\alpha=\arcsin(\frac{2n+1}{10\sqrt2})-\frac\pi4,n=0, \pm1, \pm2, \pm3, \pm4, \pm5, \pm6,-7 $$Its my first time writing an answer here, so I omitted a few simple steps. Hope you don't mind.