Nested equilateral triangles

Here I will show how you can draw the figure. The calculations that follow are somewhat similar to other answers.

We can build triangle $DEF$ by scaling down triangle $ABC$ and rotating it around its centroid $G$ by some angle $\alpha$.

Notice the hatched triangle $DHG$. This is a right triangle with sides: $$GH = \frac13 AH =\frac{4}{12} AH$$ $$DG = \frac23 DY = \frac23 \times \frac58 AH = \frac{5}{12} AH $$ $$HD = \sqrt{DG^2 - GH^2} = \frac{3}{12} AH$$ It seems the designer of the problem intentionally used the ratio $\frac58$ to lead us to the $3-4-5$ right triangle!

Anyway, Now we are able to locate point $D$ on $BC$. It is at distance $\frac14 AH$ from point $H$ , the midpoint of BC. We can similarly locate points $E$ and $F$ , hence constructing triangle $DEF$.

Now, as for calculations, note that $\widehat{DEC}=120^o - \alpha$ and that $\widehat{DGH}=60^o - \alpha$. So $\widehat{DEC}=60^o+\widehat{DGH}$ . Therefore: $$\cos \widehat{DEC} = \cos (60^o+\widehat{DGH}) = \cos60^o \cos \widehat{DGH} - \sin60^0 \sin \widehat{DGH}$$ $$= \frac12 \times \frac45 - \frac{\sqrt 3}{2} \times \frac35 = \frac{4-3\sqrt3}{10}$$ Note that when locating point $D$ we have two options, say, above and below $H$. The above calculations are for the option shown in the figure, where $D$ is selected below $H$. Alternatively, we could select $D$ to be above $H$, and respectively have $E$ below $K$. The interested reader can verify that in that case $\widehat{DEC}$ would be $\alpha$. And $\; \cos\alpha \;$ can be calculated similar to above. $$\cos \alpha = \cos (60^o-\widehat{DGH}) = \cos60^o \cos \widehat{DGH} + \sin60^0 \sin \widehat{DGH} = \frac{4+3\sqrt3}{10}$$

Note that there are two distinct solutions given the conditions in the problem. This is because there are two admissible orientations of $\triangle DEF$: one in which $AF > BF$, and one in which $AF < BF$. We can understand this by noting that since $\triangle DEF$ and $\triangle ABC$ share the same center, the circumcircle of $\triangle DEF$ will intersect $\triangle ABC$ at six points, unless the side length of $\triangle DEF$ is less than or equal to half the side length of $\triangle ABC$ (in which case there will be either exactly three intersection points in the tangent case, or zero), or greater than or equal to the side length of $\triangle ABC$.

A simple way to solve for the required angle is to place the triangle on a coordinate system, say with $A = (0,0)$, $B = (8,0)$, $C = (4, 4 \sqrt{3})$, and let $F = (x,0)$ for some $0 \le x \le 8$. Then $$E = \left(\frac{8-x}{2}, \frac{(8-x)\sqrt{3}}{2}\right).$$ We then require $EF = 5$, or $$25 = \left(x - \frac{8-x}{2}\right)^2 + \left(\frac{(8-x)\sqrt{3}}{2}\right)^2 = 3x^2 - 24x + 64.$$ Consequently $$x = 4 \pm \sqrt{3}$$ and depending on which root is chosen, $\angle DEC$ can be determined through a straightforward application of the Law of Sines.

WLOG, $AB = 8, DF = 5$. Say $\angle DEC = \theta$

If $CD = x$ then $CE = 8 - x$

Applying sine law in $\triangle CDE$,

$\displaystyle \frac{\sin 60^0}{5} = \frac{\sin \theta}{x} = \frac{\sin(60^0+\theta)}{8-x}$

From first two, we have $\sin \theta = \frac{x \sqrt3}{10}$

From first and third, $\sin 60^0 \cos \theta + \cos 60^0 \sin\theta = \frac{4\sqrt3}{5} - \frac{x \sqrt3}{10}$

$\implies \cos\theta = \frac{8}{5} - \frac{3x}{10}$

Applying $sin^2\theta + \cos^2\theta = 1$, we get

$x^2 - 8x+\frac{39}{3} = 0 \implies x = 4 \pm \sqrt3$

So, $\cos\theta = \frac{4 \mp 3\sqrt3}{10}$ (please note $\pm$ for $x$ and corresponding $\mp$ for $\cos \theta$)