$A^2+B^2 +AB=36. B^2+C^2+BC=49. C^2+A^2+AC=64.$ Find $(A+B+C)^2$

If $A$, $B$, and $C$ are assumed to be positive real numbers, then consider a triangle $PQR$ with $p:=QR=7$, $q:=RP=8$, and $r:=PQ=6$. Then, the (internal) angles of this triangle are all less than $\frac{2\pi}{3}$. If $X$ is the Fermat point of the triangle $PQR$, then $XP=A$, $XQ=B$, and $XR=C$. Let $R'$ be a point on the opposite side of $QR$ with respect to $P$ such that $QRR'$ is an equilateral triangle. Then, $A+B+C=PR'$. This shows that $$(A+B+C)^2=\left(PR'\right)^2=(PQ)^2+\left(QR'\right)^2-2\,(PQ)\,\left(QR'\right)\,\cos\left(\angle PQR'\right)\,,$$ where $PQ=6$, $QR'=QR=7$, and $\angle PQR'=\angle PQR+\frac{\pi}{3}$. You are left to determine the values of $\cos(\angle PQR)$ and $\sin(\angle PQR)$, as $$\cos\left(\angle PQR'\right)=\cos\left(\angle PQR+\frac{\pi}{3}\right)=\frac{1}{2}\,\cos(\angle PQR)-\frac{\sqrt{3}}{2}\,\sin(\angle PQR)\,.$$

The answer is $(A+B+C)^2=\frac{149+63\sqrt{5}}{2}$.

Here is a purely algebraic way. Subtract the equations in pairs to get$$(C-B)(A+B+C)=28,$$$$(C-A)(A+B+C)=13,$$$$(A-B)(A+B+C)=15.$$Let us write $x=(A+B+C)^2$ and $y=BC+CA+AB$. Then squaring and adding the above three equations yields$$x(x-3y)=589.$$Summing the original equations gives$$2x-3y=149.$$Eliminating $3y$ between these equations results in the quadratic equation$$x^2-149x+589=0.$$Solving this, we get$$x=\tfrac12(149\pm63\surd5),$$where the larger root corresponds to the case when $A$, $B$, and $C$ are all positive.

Solution without words

Assume that $A,B,C>0.$