Is it possible to prove unboundedness of 3rd order ODE?

Actually, no matter what $A$ is, there will be nonzero solutions that will converge to zero, so you can't prove unboundedness, even though it may be true that the 'generic' solution is unbounded. Here is why: First, convert the system into a first order system in $\mathbb{R}^3$ by setting $x = x_0$, $\dot x = x_1$, and $\ddot x = x_2$. Then the equation becomes the first order system $$ \begin{aligned} \dot x_0 &= x_1\,,\\ \dot x_1 &= x_2\,,\\ \dot x_2 &= -x_0 + {x_1}^2 - A\,x_2\,. \end{aligned} $$ This vector field (i.e., ODE) has one singular point, $(x_0,x_1,x_2) = (0,0,0)$, and its linearization at this point has the matrix $$ \begin{pmatrix} 0 & 1 & 0\\ 0 & 0 & 1\\ -1 & 0 & -A \end{pmatrix} $$ The eigenvalues of this matrix are the roots of $\lambda^3+A\lambda^2 + 1 = 0$, and there is always at least one negative real root. Hence the stable manifold of $(0,0,0)$ has dimension at least $1$, so there will always be a nonzero solution that decays to zero (in infinite time).

I suspect there may be periodic solutions. For $A = 1$, numerically plotting the solution with initial conditions $$x(0)=0, \dot{x}(0) = 0.442091320614410, \ddot{x}(0) = 0.774949154843236$$ I get this:

This looks to me like an approximation of an unstable periodic orbit.