find the sum of the series with given first two terms.

As you stated, let

$$c = (a+b)\%10 \tag{1}\label{eq1}$$

Next, let $f_i$, for $i \ge 1$, be the elements of the series, so $f_1 = a$, $f_2 = b$, $f_3 = c$ and, in general, for $i \ge 3$, you get

$$f_i = \left(2^{i-3}c\right)\%10 \tag{2}\label{eq2}$$

Next, note that $2^{5+j} = 32 \times 2^j \equiv 2 \times 2^j \equiv 2^{j+1} \pmod{10}$ for $j \ge 0$. Thus, this shows the values in \eqref{eq2} start repeating, with a period of $4$, beginning at $i = 4$. You can therefore determine the sum $S_n$ based on the value of $n$ being less than $4$ (i.e., $S_1 = a$, $S_2 = a + b$, $S_3 = a + b + c$), else its value based on its quotient and remainder modulo $4$.

Note: this assumed that the whole sum was to be reduced $\bmod 10$. It appears not, but I will leave it anyway.

The whole system is linear, so we can compute the values starting with $1,0$ and $0,1 $. If we ignore the mod $10$ we get $$1,0,1,2,4,8,16,\ldots 2^{n-3}\\ 0,1,1,2,4,8,16,\ldots,2^{n-3}$$ Now we can multiply the first by $a$, the second by $b$, add them, and take the result $\bmod 10$. For $n \ge 3$ the result is $$(a+b)2^{n-3} \bmod 10$$ Now if we add these up, because there were two $1$s at the start we just get $$(a+b)2^{n-2} \bmod 10$$