Prove that $\sum_{n\leq x}d^2(n)=O(x\log^3 x)$

Here is a different argument. Note that $d(ab)\leq d(a)d(b)$ for all integers $a$ and $b$. We thus have

$$\sum_{n\leq x} d(n)^2 = \sum_{ab\leq x} d(ab) \leq \sum_{ab\leq x} d(a)d(b) = \sum_{a\leq x} d(a) \sum_{b\leq x/a} d(b) \leq 2\sum_{a\leq x} d(a) {x\over a} \log x,$$ where in the final inequality I used your bound on $\sum_{n\leq x} d(n)$. Now note $$\sum_{a\leq x} {d(a)\over a} = \sum_{cd\leq x} {1\over cd} \leq \sum_{c\leq x}\sum_{d\leq x} {1\over cd} \leq (2\log x)^2.$$

I can give you a slightly weaker bound easily. Note that, $$\sum_{n=1}^\infty\frac{d^2(n)}{n^s}=\prod_p\sum_{r=0}^\infty\frac{(r+1)^2}{p^{rs}}=\prod_{p}\left(1-\frac{1}{p^{2s}}\right)\left(1-\frac{1}{p^s}\right)^{-4}=\frac{\zeta^4(s)}{\zeta(2s)}.$$ Second equality comes from the identity,$$\sum_{n=1}^\infty(n+1)^2x^n=\frac{(1-x^2)}{(1-x)^4}.$$ Now using well-known formula for sum of coefficients of a Dirichlet series, $$D(x):=\sum_{n\le x}d^2(n)={1\over 2\pi i}\int_{(c)}\frac{\zeta^4(s)}{\zeta(2s)}x^s\frac{\,ds}{s},$$ where $(c)=\{c+it:t\in \mathbb{R},c>1 \ \text{fixed}\}.$ So, $$D(x)={1\over 2\pi}\int_{-\infty}^{\infty}\frac{\zeta^4(c+it)}{\zeta(2c+2it)}\frac{x^{c+it}}{c+it}\,dt\le\frac{x^c}{2\pi}\int_{-\infty}^{\infty}\left|{\frac{\zeta^4(c+it)}{\zeta(2c+2it)}\frac{x^{it}}{c+it}}\right|\,dt=O(x^c)$$ for any $c>1$

Now for your bound, note that, $$\sum_{n=1}^\infty\frac{d_3(n)}{n^s}\sum_{n=1}^\infty\frac{|\mu(n)|}{n^s}=\zeta^3(s)\frac{\zeta(s)}{\zeta(2s)}=\sum_{n=1}^\infty\frac{d^2(n)}{n^s},$$ where $d_3(n)$ is number of ways to write $n$ as product of $3$ numbers. So,$$d^2(n)=\sum_{r|n}d_3(r)\left|\mu\left( {n\over r}\right)\right|.$$ Hence, $$D(x):=\sum_{n\le x}d^2(n)=\sum_{n\le x}\sum_{r|n}d_3(r)\left|\mu\left( {n\over r}\right)\right|$$ $$=\sum_{ab\le x}d_3(a)|\mu(b)|\le x\sum_{a\le x}{d_3(a)\over a}=O(x\log^3x)$$.

Here, I have used $$\sum_{n\le x}d_k(x)=O(x(\log x)^{k-1}),$$ which would be easy for you to prove.