Simplifying an expression containing inverse trigonometric functions
expr = 2/3 (J+2 Sqrt[3 B^2+4 J^2] Cos[1/3 ArcCos[-((J (9 B^2+8 J^2))/Sqrt[(3 B^2+4 J^2)^3])]]);
expr2 = Simplify[expr /. {B->r Cos[t]/√3, J->r Sin[t]/2 },{r>0, 0<t<2Pi}]
expr3 = expr2 /. e:Cos[ArcCos[_]/3 ] :> First[y/.Solve[Reduce[{y==e, 0<t<2Pi},t,Reals]/.c:_Cos:>TrigExpand[c],y,Reals]//Simplify//Normal//Union]//Simplify
Simplify[TrigExpand //@ expr3 /. {r->Sqrt[3 B^2 + 4 J^2], t->ArcTan[√3B, 2J]}, _∈ Reals]
Output
$\frac{1}{3} r \left(4 \cos \left(\frac{1}{3} \cos ^{-1}\left(-\frac{1}{4} (5+\cos (2 t)) \sin (t)\right)\right)+\sin (t)\right)$
$\frac{r \sqrt{\cos (2 t)+7}}{\sqrt{6}}$
$2 \sqrt{B^2+J^2}$
Another way to reduce expression.
expr = 2/3 (J +
2 Sqrt[3 B^2 + 4 J^2] Cos[
1/3 ArcCos[-((J (9 B^2 + 8 J^2))/Sqrt[(3 B^2 + 4 J^2)^3])]]) //
Expand;
rule = Cos[
1/3 ArcCos[-((J (9 B^2 + 8 J^2))/Sqrt[(3 B^2 + 4 J^2)^3])]
] -> cc;
Solve for unknown simplified expression x
sol = Solve[x == expr /. rule, cc, Reals][[1, 1]] // Normal
(* cc -> (-2 J + 3 x)/(4 Sqrt[3 B^2 + 4 J^2]) *)
eq1 = Equal @@ sol /. Reverse[rule]
(* Cos[1/3 ArcCos[-((J (9 B^2 + 8 J^2))/
Sqrt[(3 B^2 + 4 J^2)^3])]] == (-2 J + 3 x)/(
4 Sqrt[3 B^2 + 4 J^2]) *)
eq2 = ArcCos[#] & /@ eq1 //
Simplify[#,
0 < ArcCos[-((J (9 B^2 + 8 J^2))/Sqrt[(3 B^2 + 4 J^2)^3])] < Pi] &
(* ArcCos[-((J (9 B^2 + 8 J^2))/Sqrt[(3 B^2 + 4 J^2)^3])] ==
3 ArcCos[(-2 J + 3 x)/(4 Sqrt[3 B^2 + 4 J^2])] *)
eq3 = TrigExpand[Cos /@ eq2]
(* -((9 B^2 J Sqrt[(3 B^2 + 4 J^2)^3])/(3 B^2 + 4 J^2)^3) - (
8 J^3 Sqrt[(3 B^2 + 4 J^2)^3])/(3 B^2 + 4 J^2)^3 == -(J^3/(
2 (3 B^2 + 4 J^2)^(3/2))) + (3 J)/(2 Sqrt[3 B^2 + 4 J^2]) + (
9 J^2 x)/(4 (3 B^2 + 4 J^2)^(3/2)) - (9 x)/(
4 Sqrt[3 B^2 + 4 J^2]) - (27 J x^2)/(8 (3 B^2 + 4 J^2)^(3/2)) + (
27 x^3)/(16 (3 B^2 + 4 J^2)^(3/2)) *)
Solve[eq3, x, Reals] // Simplify
(* {{x -> ConditionalExpression[2 J, B != 0]},
{x -> ConditionalExpression[-2 Sqrt[B^2 + J^2], B != 0]},
{x -> ConditionalExpression[2 Sqrt[B^2 + J^2], B != 0]}} *)
I didn't examine further, why 2 J is given as solution for B!=0. It is the solution for B==0.
expr /. B -> 0 // Simplify[#, J \[Element] Reals] &