Error in predicting remainders

Let's figure out what the textbook's method here is actually doing. You have a polynomial $f$ with integer coefficients and you want to divide it by some monic polynomial $g$ with integer coefficients and find the remainder. That is, you are finding the unique polynomials $q$ and $r$ with $\deg r<\deg g$ such that $$f(x)=q(x)g(x)+r(x)\tag{1}$$ (it turns out that the assumptions here guarantee that $q$ and $r$ have integer coefficients).

Now what the textbook says to do is pick some integer value $x=a$ and plug it in and do ordinary integer division with remainder of $f(a)$ and $g(a)$. That is, you find the unique integers $s$ and $t$ with $0\leq t<g(a)$ and $$f(a)=sg(a)+t.\tag{2}$$ The textbook's method would then claim that $t=r(a)$, so if you can compute $t$ and have multiple choice options for $r$, you can test your options to see which one satisfies $r(a)=t$.

Now, why would this be true? Well, plugging in $x=a$ in equation (1) above we have $$f(a)=q(a)g(a)+r(a),$$ where $q(a)$ and $r(a)$ are integers (since $a$ is an integer and $q$ and $r$ have integer coefficients). Now if $r(a)$ happens to satisfy $0\leq r(a)<g(a)$, then $q(a)$ and $r(a)$ will satisfy the requirements of $s$ and $t$ in equation (2). Since the $s$ and $t$ satisfying these requirements are unique, this means that actually $q(a)=s$ and $r(a)=t$.

So the textbook's method works as long as you choose $a$ such that $0\leq r(a)<g(a)$. In general, however, there's no reason that this will be true for arbitrary $a$. Since $g$ has degree strictly larger than $r$ and the leading coefficient of $g$ is $1$ (in particular, it is positive), $g(a)>r(a)$ for all sufficiently large $a$. If the leading coefficient of $r$ is also positive, then $r(a)\geq 0$ for sufficiently large $a$, and so $0\leq r(a)<g(a)$. So in the case that the leading coefficient of $r$ happens to be positive, this method will work as long as you pick $a$ to be big enough. But, as you have found by experimentation, the method typically won't work for small $a$.

However, all is not lost! While it's not always true that $r(a)=t$, it is always true that that $$r(a)\equiv t\pmod{g(a)}.$$ Indeed, since $$f(a)=sg(a)+t=q(a)g(a)+r(a),$$ $r(a)-t=(s-q(a))g(a)$ so $r(a)-t$ is divisible by $g(a)$. So even when $t$ does not give the correct value for $r(a)$, it still tells you what the remainder of $r(a)$ must be mod $g(a)$. For instance, in your example where you plugged in $x=3$, you learn that $r(3)\equiv 5\pmod{15}$. From the given multiple choice options, the correct answer is actually the only one that satisfies this congruence. So you could still use this variant on the textbook's method to quickly find the right answer in this case using $x=3$.

Much easier (and deterministic) is the following. Dividing $\,f(x)\,$ by $\,x^2+2x= x(x+2)\,$ yields

$$ f(x)\, =\, q(x)\, x(x+2) + r(x) $$

thus $\ f(0) = r(0)\ $ and $\, f(-2) = r(-2)\ $ and these values uniquely determine the remainder $\,r(x)\,$ since it has degree $\le 1.$

First you need to clearly know what the remainder theorem is. So if you know that, then you will realise that what you are doing is not in accordance with the theorem. As for the second problem, the application of remainder theorem strictly implies putting x = 3 in the given polynomial and the value of the polynomial for x = 3 give the remainder required. Further you need to realise that what you are doing is actually treating the division as some kind of an equation. The first problem will be solved just as I have shown below:

$$x^5 + 2x^4 -3x^2 + 2x -4= (x^2+2x)Q (x)+R (x)$$

Your job is to find the remainder $R (x) $. As per the remainder theorem, we observe that for x = $- 2$ and $x = 0$ the quotient term becomes $0$. Hence we get, $$R (0)=-4$$ $$R (-2)=-2^5 + 2\cdot 2^4 -3\cdot 2^2 - 2\cdot 2-4=-20$$

One thing that I missed to say earlier is that the degree of the remainder is always at least 1 less than that of the divisor so in this case the remainder will be linear in nature.

Say $R (x)=Ax+B$

This implies $$B=-4$$ $$-2A-4=-20 \implies A=8$$

So $$\boxed {R (x)=8x-4}$$

Thus I would directly recommend you to apply the remainder theorem as I have done to obtain the required result instead of following what your text book has asked you to do.

Hope this helps you.