Determine the Taylor series for $f(x) = x^3 \cdot \ln{\sqrt{x}}$ around the point $a = 1$ and determine its radius of convergence.

In fact a lot of the work needed is already done. Let's recapitulate. We have $f(x)=x^3\ln\left(\sqrt{ x}\right)$ and the Taylor expansion at $a=1$ is given as \begin{align*} f(x)=\sum_{n=0}^\infty \frac{1}{n!}f^{(n)}(a)(x-a)^n \end{align*}

We obtain \begin{align*} f(x)&=x^3\ln\left(\sqrt{ x}\right)=\frac{1}{2}x^3\ln(x)\\ f^{\prime}(x)&=\frac{3}{2}x^2\ln(x)+\frac{1}{2}x^2\\ f^{\prime\prime}(x)&=3x\ln(x)+\frac{5}{2}x\\ f^{\prime\prime\prime}(x)&=3\ln(x)+\frac{11}{2}\\ f^{(4)}(x)&=\frac{3}{x}\\ f^{(5)}(x)&=-\frac{3}{x^2},\qquad f^{(6)}(x)=3\cdot\frac{2!}{x^3},\qquad f^{(7)}=-3\cdot \frac{3!}{x^4}\\ &\vdots\\ f^{(n)}(x)&=3(-1)^n\frac{(n-4)!}{x^{n-3}}\qquad\qquad n\geq 4\tag{1} \end{align*}

From the fourth derivative $\frac{3}{x}$ we can relatively easy obtain higher derivatives and assume the general formula (1) which can be shown by induction.

Evaluated at $a=1$ we have \begin{align*} f(1)=0, f^{\prime}(1)=\frac{1}{2}, f^{\prime\prime}(1)=\frac{5}{2}, f^{\prime\prime\prime}(1)=\frac{11}{2}, f^{(n)}(1)=3(-1)^n(n-4)!\qquad n\geq 4 \end{align*}

We obtain from the derivatives above the Taylor series \begin{align*} \color{blue}{f(x)}&\color{blue}{=x^3\ln\sqrt{ x}}\\ &\color{blue}{=\frac{1}{2}(x-1)+\frac{5}{4}(x-1)^2+\frac{11}{12}(x-1)^3+3\sum_{n=4}^\infty\frac{(-1)^n}{n(n-1)(n-2)(n-3)}(x-1)^n} \end{align*}

The radius $R$ of convergence is \begin{align*} R&=\lim_{n\to \infty}\left|\frac{a_{n}}{a_{n+1}}\right|=\lim_{n\to \infty}\left|\frac{3(-1)^n(n+1)n(n-1)(n-2)}{3(-1)^{n+1}n(n-1)(n-2)(n-3)}\right|\\ &=\lim_{n\to\infty}\left|\frac{n+1}{n-3}\right|\\ &=1 \end{align*}

$$f(x+1)=(x+1)^3\ln(\sqrt{x+1})=\frac{(x+1)^3}{2}\ln(x+1)$$ Now write $$\ln(x+1) = \sum\limits_{n=1}^\infty \frac{(-1)^{n+1}x^n}{n} $$ And multiply the infinite sum by the polynomial. Then you can switch back to $f(x)$. Your approach is good in some cases, but often it's just guessing.

Easiest way to derive the Taylor series for $\ln(1+x)$ is, i think, as follows. We know that $$\int\limits_0^x\frac{1}{1-t}dt = -\ln(1-x) $$ But we also know the Taylor series for $\frac{1}{1-t}$. $$ \frac{1}{1-t}=\sum\limits_{n=0}^\infty t^n $$ We can integrate the sum, and we can do it term by term, and we get: $$ -\ln(1-x)=\sum\limits_{n=1}^\infty \frac{x^n}{n} $$ So that $$ \ln(1+x)=\sum\limits_{n=1}^\infty \frac{(-1)^{n+1}x^n}{n} $$ You can also try your guessing method to derive the Taylor series for $\ln(x+1)$, that will be more straightforward.

Write $x=1+t$, so you need to find the Taylor series of $$ \frac{1}{2}(1+t)^3\ln(1+t) $$ at $t=0$. Since $$ \ln(1+t)=\sum_{n>0}\frac{(-1)^{n+1}t^n}{n} $$ your Taylor series is $$ \frac{1}{2}\sum_{n>0}\frac{(-1)^{n+1}t^n}{n}+ \frac{3}{2}\sum_{n>0}\frac{(-1)^{n+1}t^{n+1}}{n}+ \frac{3}{2}\sum_{n>0}\frac{(-1)^{n+1}t^{n+2}}{n}+ \frac{1}{2}\sum_{n>0}\frac{(-1)^{n+1}t^{n+3}}{n} $$ that can be rewritten as $$ \frac{1}{2}\sum_{n>0}\frac{(-1)^{n+1}t^n}{n}+ \frac{3}{2}\sum_{n>1}\frac{(-1)^{n}t^{n}}{n-1}+ \frac{3}{2}\sum_{n>2}\frac{(-1)^{n-1}t^{n}}{n-2}+ \frac{1}{2}\sum_{n>3}\frac{(-1)^{n-2}t^{n}}{n-3} $$ Now isolate the only term for $n=1$, the two terms for $n=2$ and the three terms for $n=3$: $$ \frac{t}{2}+\frac{1}{2}\left(-\frac{t^2}{2}+3t^2\right)+ \frac{1}{2}\left(\frac{t^3}{3}-\frac{3t^3}{2}+3t^3\right)+\\ \frac{1}{2}\sum_{n>3}\left(\frac{(-1)^{n+1}}{n}+\frac{(-1)^n3}{n-1}+\frac{(-1)^{n-1}3}{n-2}+\frac{(-1)^{n-2}}{n-3}\right)t^n $$ The last summation can be rewritten as $$ \frac{1}{2}\sum_{n>3}\left(-\frac{1}{n}+\frac{3}{n-1}-\frac{3}{n-2}+\frac{1}{n-3}\right)(-1)^nt^n= \sum_{n>3}\frac{3(-1)^nt^n}{n(n-1)(n-2)(n-3)} $$ so the end result is $$ \frac{t}{2}+\frac{5}{4}t^2+\frac{11}{12}t^3+\sum_{n>3}\frac{3(-1)^nt^n}{n(n-1)(n-2)(n-3)} $$ and you can check that the coefficient for $t^4$ is indeed $1/8$; the coefficient for $t^5$ is $-1/40$ as you computed.

The Taylor series for $f(x)=x^3\ln\sqrt{x}$ around $1$ is obtained by substituting $t$ with $x-1$. The radius of convergence is the same as for $\ln(1+t)$, that is, $1$. You can also compute it with the ratio test, as further check.