Find positive $K$ such that $\int_0^\infty\left(\frac{1}{\sqrt{2x^2+1}}-\frac{K}{x+1}\right)dx$ converges

Your bounds are correct but too weak. Here we need a more precise asymptotic analysis.

Note that as $x\to +\infty$, $$\begin{align*}\frac{1}{\sqrt{2x^2+1}}-\frac{K}{x+1}&=\frac{x+1-\sqrt{2}Kx\sqrt{1+\frac{1}{2x^2}}}{(x+1)\sqrt{2x^2+1}}\\ &=\frac{x+1-\sqrt{2}Kx(1+\frac{1}{4x^2}+o(\frac{1}{x^2}))}{\sqrt{2}x^2+O(x)}\\ &=\frac{(1-\sqrt{2}K)x+1+O(1/x)}{\sqrt{2}x^2+O(x)}.\end{align*}$$ What may we conclude?


Hint

I suggest to have a look at what happens at the bounds.

For $x$ close to $0$, the Taylor expansion of the integrand is $$\frac{1}{\sqrt{2x^2+1}}-\frac{K}{x+1}=(1-K)+K x-(K+1) x^2+O\left(x^3\right)$$

Now, for infinitely large values of $x$, $$\frac{1}{\sqrt{2x^2+1}}-\frac{K}{x+1}=\frac{\frac{1}{\sqrt{2}}-K}{x}+\frac{K}{x^2}+O\left(\frac{1}{x^3}\right)$$

Does this tell you something ?