Two series relations, each one implies the other - from Andrews' partition book

Consider the formal series $$ \begin{array}{}f(x)=\sum_{n=1}^\infty a_nx^n&\text{and}&g(x)=\sum_{n=1}^\infty b_nx^n\end{array} $$ Then \eqref{first_eq} is equivalent to $$ \begin{align} f(x) &=\sum_{n=1}^\infty\sum_{j=0}^{n-1}\binom{r-n+j}{j}b_{n-j}x^n\\ &=\sum_{n=1}^\infty\sum_{j=1}^{n}\binom{r-j}{n-j}b_jx^n\\ &=\sum_{j=1}^\infty\sum_{n=j}^{\infty}\binom{r-j}{n-j}b_jx^n\\ &=\sum_{j=1}^\infty\sum_{n=0}^{\infty}\binom{r-j}{n}b_jx^{n+j}\\ &=\sum_{j=1}^\infty b_j(1+x)^{r-j}x^{j}\\ &=(1+x)^rg\left(\frac{x}{1+x}\right) \label{a}\tag{a} \end{align} $$ and \eqref{second_eq} is equivalent to $$ \begin{align} g(x) &=\sum_{n=1}^\infty\sum_{j=0}^{n-1}\binom{r-n+j}{j}(-1)^ja_{n-j}x^n\\ &=\sum_{n=1}^\infty\sum_{j=1}^{n}\binom{r-j}{n-j}(-1)^{n-j}a_jx^n\\ &=\sum_{j=1}^\infty\sum_{n=j}^{\infty}\binom{r-j}{n-j}(-1)^{n-j}a_jx^n\\ &=\sum_{j=1}^\infty\sum_{n=0}^{\infty}\binom{r-j}{n}(-1)^na_jx^{n+j}\\ &=\sum_{j=1}^\infty a_j(1-x)^{r-j}x^{j}\\ &=(1-x)^rf\left(\frac{x}{1-x}\right) \label{b}\tag{b} \end{align} $$ Simple algebraic substitutions yield that \eqref{a} and \eqref{b} are equivalent. Thus, \eqref{first_eq} and \eqref{second_eq} are equivalent.

As noted in the comments, this is a variation of the binomial inversion formula $\sum_{k=m}^n (-1)^{k-m} \binom{k}{m} \binom{n}{k} = \delta_{mn}$. For proofs of this formula (which I will use), see this math.SE question. It can also be proved easily using the trinomial revision formula I cite below. From another perspective, binomial inversion says that the Pascal matrix is (up to sign) its own inverse. For more on that see "Combinatorial interpretation of binomial inversion" and my answer to "Stirling numbers and inverse matrices."

For your problem, I'll just prove that \eqref{second_eq} $\Rightarrow$ \eqref{first_eq}, since, as you note, that is sufficient.

First, reindex \eqref{first_eq} and \eqref{second_eq} to get the simpler forms $$a_n = \sum_{j=1}^n \binom{r-j}{n-j} b_j, \tag{1}$$ $$b_n = \sum_{j=1}^n \binom{r-j}{n-j} (-1)^{n-j} a_j. \tag{2}$$

Assuming that (2) is true, and starting with the right-hand side of (1), we have $$\sum_{j=1}^n \binom{r-j}{n-j} b_j = \sum_{j=1}^n \binom{r-j}{n-j} \sum_{k=1}^j \binom{r-k}{j-k} (-1)^{j-k} a_k = \sum_{k=1}^n \sum_{j=k}^n \binom{r-j}{n-j} \binom{r-k}{j-k} (-1)^{j-k} a_k.$$ Now, apply the trinomial revision formula $\binom{r}{m} \binom{m}{k} = \binom{r}{k} \binom{r-k}{m-k}$. (This is easy to prove; just write out the binomial coefficients in factorial form and regroup. For a reference, see Concrete Mathematics, 2nd edition, p. 168.) We get $$\sum_{k=1}^n \sum_{j=k}^n \frac{\binom{r}{n} \binom{n}{j}}{\binom{r}{j}} \frac{\binom{r}{j} \binom{j}{k}}{\binom{r}{k}} (-1)^{j-k} a_k = \sum_{k=1}^n \frac{\binom{r}{n}}{\binom{r}{k}} a_k \sum_{j=k}^n \binom{n}{j} \binom{j}{k} (-1)^{j-k} .$$ Then, binomial inversion yields $$=\sum_{k=1}^n \frac{\binom{r}{n}}{\binom{r}{k}} a_k \delta_{nk} = \frac{\binom{r}{n}}{\binom{r}{n}} a_n = a_n.$$

It's pretty easy to brute-force this proof, just by substitution and two basic identities. It's actually true for arbitrary $r\in\mathbb{R}$, as long as you define $z\choose k$ for arbitrary real $z$ and non-negative integers $k$.

Given (1), replace the $a_{n-j}$ in the right hand side of (2) to get that you want:

$$b_n = \sum_{j=0}^{n-1} \sum_{k=0}^{n-j-1} (-1)^j {{r-n+j}\choose j}{{r-n+j+k}\choose k} b_{n-j-k}$$

Gathering terms, the coefficient of $b_{n-m}$ in the right hand side is (noting that j+k=m, or that k=m-j):

$$\sum_{j=0}^{m} (-1)^j{{r-n+j}\choose j}{r-n+m \choose m-j}$$

Note this is a function of $r-n$, so we can write:

$$f_m(z) = \sum_{j=0}^m (-1)^j{z+j \choose j}{z+m \choose m-j}$$

So, the right hand side of (2), after substitution, is:

$$\sum_{m=0}^{n-1} f_m(n-r) b_{n-m}$$

But ${z+m \choose m-j}{z+j \choose j} = {z+m \choose m} {m \choose j}$. You can see this algebraically pretty easily, or you can prove it combinatorially for non-negative integers $z$, and then use that both sides are polynomials of degree $m$ in $z$, so they must agree everywhere.

So

$$f_m(z) = {z+m \choose m} \sum_{j=0}^m (-1)^j {m \choose j} = {z+m \choose m} (1+(-1))^m$$

And therefore $f_m(z)=0$ if $m>0$, and $f_0(z)=1$.

But that means that the right hand side of (2) is equal to $b_n$.