How to solve $8n^2=64n\cdot \log_2(n)$?

You may simplify $n$ from both sides ($n=0$ is not an acceptable solution) to get $$8n=64\log_2{n}\iff \frac{8n}{64}=\log_2{(n)}\iff2^{\frac{n}8}=2^{\log_2(n)}\iff 2^{\frac{n}{8}}=n\iff2^n=n^8$$ So, you are looking for the zeros of $f(n)=2^n-n^8$. This function has two solutions, which are approximately $n_1=1.0999$ and $n_2=43.559$. However, I do not think that there are methods other than numerical that can solve this problem.

We certainly cannot have $n=0$, so a division is safe. Upon dividing by $8n$ we obtain

$$ n = 8 \log_2 n $$

which is a nonlinear equation. You can try fixed-point methods or even Newton's Method to solve this. As Jimmy says, I don't think this can be done analytically.

Instead of numerical methods, you can also express the solution in terms of the Lambert W function. In some sense, you can think of this idea as someone has tabulated the solution to the Lambert W function numerically and you are simply reusing his work. The W function is defined by the relation, $W(z e^z) = z$. Another way to understand this relation is that if $W(k) = z$ then $k = z e^z$.

Going back to your question,

$n = 8\log_2(n) \Leftrightarrow \frac{n}{8} = \frac{\ln(n)}{\ln(2)} \Leftrightarrow -\frac{\ln(2)}{8} = -\ln(n)\frac{1}{n} \Leftrightarrow -\frac{\ln(2)}{8} = \ln(\frac{1}{n})e^{\ln(\frac{1}{n})}$

Thus by noting that $z = \ln(\frac{1}{n})$ and $k = -\frac{\ln(2)}{8}$, we get

$W(-\frac{\ln(2)}{8}) = \ln(\frac{1}{n}) \Leftrightarrow n = e^{-W(-\frac{\ln(2)}{8})} $