Area of sub-triangle inside a triangle
Hint: use ratio of areas of $2$ triangles. Let's use your diagram to calculate the red area and you can generalize it easily. The red area is half of the area of the triangle whose vertices at $0,2,2$ whose area is again is $\frac{2}{5}$ area of the triangle whose vertices are at $0,2,5$ and in turn whose area is half the area of the triangle whose vertices are at $0,4,5$. Hope it helps.
Without loss of generality we can assume that $F$ lies between $A$ and $E$. Recall that an area of a triangle is a half of a product of length of adjacent sides by the sinus of the angle between these sides. Therefore, since triangles $DEF$ and $DEA$ have a common angle $\angle E$ and common side $DE$, their areas relates as the sides adjacent to the angle $\angle E$, that is $\frac{\operatorname{area}(\triangle DEF)}{\operatorname{area}(\triangle DEA)}=\frac{EF}{EA}$. Also since triangles $DAE$ and $BAC$ have a common angle $\angle A$, $\frac{\operatorname{area}(\triangle DAE)}{\operatorname{area}(\triangle BAC)}=\frac{DA}{AB}\cdot\frac{AE}{AC}$. Finally, $$b=\operatorname{area}(\triangle DEA)=\frac{EF}{EA}{\cdot\operatorname{area} (\triangle DEA)}=\frac{EF}{EA}\cdot\frac{DA}{AB}\cdot\frac{AE}{AC}\cdot {\operatorname{area}(\triangle BAC)}= \frac{EF}{EA}\cdot \frac{DA}{AB}\cdot\frac{AE}{AC}\cdot a.$$
The points are canonically labeled such that $B \le D \le C$. This means if $D > C$, we swap the labels before applying the following logic.
Using Side-angle-side formula, we calculate area of $\triangle ABD$ as
$$x = { AB \times BD \times sin(\theta) \over 2}$$
Area of $\triangle ABC$:
$$y = { AB \times BC \times sin(\theta) \over 2}$$
Area of $\triangle ADC = y – x$, given by
$$y - x = { { AB \times BC \times sin(\theta) \over 2} - { AB \times BD \times sin(\theta) \over 2} } = {{AB sin(\theta)} \times (BC - BD) \over 2}$$
Ratio of Areas ($\triangle ADC$ / $\triangle ABC$) is given by
$$BC - BD \over BD$$
This is true when $BC - BD > 0$. When $BC - BD = 0$, the $\triangle ADC \cong \triangle ABC$ and the ratio is $1$.
I notice that I have used a different nomenclature for the triangles than what the OP had used in the problem statement and also the fact that the largest triangle has area $a$. For consistency in proof, let me call the outer triangle as $\triangle A'BC'$ (area $= a$). Using the same principles as outlined above for calculating the ratio of areas of triangles, the ratio Area of $\triangle A'BC'$ and Area of $\triangle ABC$ is given by
$$A'B \times BC' \over AB \times BC$$
Ratio of Areas ($\triangle ADC$ / $\triangle A'BC'$) is given by
$${BC - BD \over BD} \div {A'B \times BC' \over AB \times BC}$$
$\therefore$, Area $\triangle ADC$ is given by
$${BC - BD \over BD} \times {AB \times BC \over A'B \times BC'} \times a$$
Substituting $A'B = 4$ and $BC' = 5$, we get
Area $\triangle ADC$
$$ = \big( {BC - BD \over BD}\big) \times {AB \times BC \over 4 \times 5} \times a$$
As mentioned earlier, this is true when $BC - BD > 0$. When $BC - BD = 0$, the $\triangle ADC \cong \triangle ABC$ and the ratio (given inside the parenthesis) is $1$.
Note: Just a reminder, the sides are as denoted in the figure in this response (that are slightly different from the nomenclature used by the OP).