Maximize area of a quadrilateral given three sides
Your assumption of being maximal happens when it is a cyclic quadrilateral seems right. Consider $ABCD$ your maximal quadrilateral. Let's say $AB=3$, $BC=4$, $CD=5$. If $\angle ACD\neq 90$, then you can rotate $CD$ around $C$ so that this angle $\angle ACD$ becomes $90$, and your area would increase, contradicting the maximality of your quadrilateral. So, $\angle ACD=90$.
Similar analysis shows that $\angle ABD=90$. So, your maximal quadrilateral has to be cyclic. Even more, your other side $AD=2x$ has to be the diameter of the circumcircle. I think this should be enough info to actually find the value of $x$. And probably you'll get the same cubic formula to solve.
Your approach is fine and correct. If four segments $a_1,a_2,a_3,a_4$ satisfy the inequalities $$ a_i<\frac12\sum_i a_i,\tag1 $$ which is necessary for construction of any quadrilateral, it is possible to construct a cyclic quadrilateral out of these segments as well. There are up to congruence two such quadrilaterals, both having the same area, which is by the Bretschneider's formula larger than the area of any other quadrilateral constructible of the same sides.
Though the solution of your problem admits a simple geometric interpretation (the fourth side is the diameter of the circumscribed circle), the construction of the side by geometric means (i.e. with compass and unmarked straightedge) is impossible.
Indeed the length in question is the (only) positive solution of the cubic equation: $$ x^3-(a^2+b^2+c^2)x-2abc=0\tag2 $$ with integer values $a,b,c$.
By the well-known Lemma the roots of the equation $(2)$ are constructible if and only if at least one of them is rational.
Assume the equation $(2)$ has a rational root $x=\frac pq$, with $p,q\in\mathbb Z, (p,q)=1, q>0$. Substituting the value into $(2)$ one obtains: $$p^3=[(a^2+b^2+c^2)pq+2abc q^2]q\implies q\mid p\implies q=1.$$
On the other hand the same equation can be rewritten as: $$ [p^2-(a^2+b^2+c^2)q^2]p=2abcq^3\implies p\mid 2abc. $$
The direct substitution of all 32 (including negative) divisors of $2abc=120$ into $(2)$ verifies that none of them satisfies the equation. This proves that the fourth side of the quadrilateral maximizing its area is not constructible.