How to find the radius of this smaller circle?

See the figure below. One unit on the paper is six units in your problem. $AB=48,AC=40,BC=40$. Circle $D$ has radius $12$ as you say. $HI$ is tangent to both circles and parallel to $AB$, so $ABC$ is similar to the small triangle cut off by $HI$. $EC=32$ by Pythagoras, $EG=24$ from the circle, so $CG=8$ and the small triangle is $\frac 14$ the size of the large one. That says the radius of the small circle is $\frac 14 \cdot 12=3$

$$\frac{R}{12}=\frac{2x}{48}=\frac{40-(24+x)}{40} \implies \frac{R}{12}=\frac{2x+2(16-x)}{48+2(40)}$$