The first, fifth, and eighth terms of an arithmetic progression are the first, second, the third terms of a geometric progression ...

The series starts with a, and increases with d, so you get the numbers to be

$a,a+d,a+2d,a+3d,a+4d,a+5d,a+6d,a+7d$ as your 8 terms.

$a, a+4d, a+7d$ are your terms in your geometric one. The multiplication from the first two terms is $\frac{a+4d}{a} $ or $ 1 +\frac{4d}{a}$, and the second multiplications is $\frac{a+7d}{a+4d} $ or $ 1 +\frac{3d}{a+4d}$

For a geometric progression, the multiplication is always the same so $$ 1 +\frac{4d}{a} =1 +\frac{3d}{a+4d}$$ $$ \frac{4d}{a} =\frac{3d}{a+4d}$$ $$ 4d =\frac{3d*a}{a+4d}$$ $$ 4da+ 16d^2 =3da$$ $$ da+ 16d^2 =0$$ $$ 16d^2 =-da$$ $$ -16d =a$$

This means that we can rewrite those geometric progression ratios to

$$ 1 +\frac{4d}{-16d} $$ $$ 1 +-\frac{1}{4} = 3/4$$

The first, fifth, and eight terms of the arithmetic progression are $$ a, \quad a+4d, \quad a+7d. $$ The first three terms of the geometric progression are $$ a, \quad ar, \quad ar^2. $$ So we have \begin{align} a+4d & = ar, \\ a+7d & = ar^2. \end{align} Subtracting $a$ from both sides of both equations and then doing some routine algebra, we get \begin{align} 4d & = a(r-1), \\ 7d & = a(r^2-1) = a(r-1)(r+1). \end{align} Dividing the two sides of the second equation respectively by the two sides of the first, we get $$ \frac 7 4 = r+1, $$ so $r=\dfrac 3 4$. Hence $a+4d=3a/4$, and so $d=-a/16$.