Finding integral $\int_{0}^{\infty} \frac{x^{\alpha}\log{x}}{1-x^2}dx$ using complex analysis - residues

OK, you have done all of the contour integration correctly. Thus, all you really are missing is the fact that the "non-integrable" piece about which you are concerned is a Cauchy Principal Value, which is finite! So, let

$$I = \int_0^{\infty} dx \, \frac{x^{\alpha} \log{x}}{1-x^2} $$ $$J = PV \int_0^{\infty} dx \, \frac{x^{\alpha}}{1-x^2} $$

where the $PV$ denotes a Cauchy Principal Value. Now, the other piece you would need - and I assume you know how to do this - comes from the residue theorem. Because I assume you know what you are doing, I will skip this evaluation and just write the equation resulting from the application of the theorem:

$$\left ( 1-e^{i 2 \pi \alpha} \right ) I - i 2 \pi e^{i 2 \pi \alpha} J + \pi^2 e^{i 2 \pi \alpha} = \pi^2 e^{i \pi \alpha} $$

Now, we just need to equate real and imaginary parts and we get two equations in two unknowns $I$ and $J$:

$$(1-\cos{2 \pi \alpha}) I + 2 \pi \sin{ 2 \pi \alpha} \, J = \pi^2 (\cos{\pi \alpha} - \cos{ 2 \pi \alpha} )$$ $$\sin{2 \pi \alpha}\, I + 2 \pi \cos{ 2 \pi \alpha} \, J = \pi^2 (\sin{2 \pi \alpha} - \sin{ \pi \alpha} )$$

You can then eliminate $J$ and solve for $I$. The result is

$$I = \int_0^{\infty} dx \, \frac{x^{\alpha} \log{x}}{1-x^2} = -\pi^2 \frac{1-\cos{\pi \alpha}}{1-\cos{2 \pi \alpha}} = -\frac{\pi^2}{4 \cos^2{\left ( \frac{\pi}{2} \alpha \right )}}$$

You can also get $J$ if you want.