Why isn't $f(x) = x\cos\frac{\pi}{x}$ differentiable at $x=0$, and how do we foresee it?

One way to "foresee" it is that there are clearly two lines in the first image you posted that serve as an envelope to $f(x)$. These two lines crossing at the origin make it impossible to approximate $f$ near $x=0$ as a linear function. This is the criterion of differentiability you want to keep in mind when trying to make this kind of judgement.

On the other hand, in the second image, the envelope is two parabolas touching at the origin. Since the parabolas are tangent at the origin, they force $y=0\cdot x$ to be the only way to approximate $f(x)$ as a linear function near $x=0$.

In the end, the criterion for differentiability of functions squeezed inside an envelope $$e_-(x)\leq f(x)\leq e_+(x)$$ is: no matter how wildly $f(x)$ oscillates inside the envelope, $f(x)$ will be differentiable at $x=0$ if (i) the envelopes touch each other: $$e_-(0)=e_+(0)$$ that is, they do squeeze $f(x)$ appropriately; and (ii) they are both differentiable with equal derivatives: $${e'}_{\!-}(0)={e'}_{\!+}(0)$$ thus forcing $f(x)$ to be differentiable with the same derivative.

Normally the rules of differentiation show that any elementary function is continuous and differentiable wherever it is defined. The problems arise when the function's formula is undefined at some point and then we need to check the differentiability via the definition of derivative. Another case is when we define functions via multiple formulas in different parts of the domain. Then we need to check for existence of derivative at the boundary points.

The point which your book has highlighted is very important but perhaps been left out by other answers. It says that most common functions are differentiable except for exceptional points. Thus $f(x) = x\cos (\pi/x)$ is an example where $f$ is differentiable at all $x \neq 0$ and $$f'(x) = \cos (\pi/x) + \frac{\pi}{x}\sin(\pi/x)$$ When we define $f(0) = 0$ then we get continuity. But $f$ is still not differentiable at $0$.

Now your book mentions a very deep idea. Normally people try to look at the formula for $f'(x)$ and try to see if it tends to a particular value as $x \to 0$. In the above example the limit of $f'(x)$ does not exist and hence we conclude that $f$ is not differentiable at $0$.

The book says that this is not the right way to go and gives another example which sort of is a failure case for above technique. The example is $f(x) = x^{2}\cos (\pi/x), f(0) = 0$. We have $$f'(x) = 2x\cos(\pi/x) + \pi\sin(\pi/x)$$ and again we see that $f'(x)$ does not tend to a limit as $x \to 0$. And hence we conclude that $f'(0)$ does not exist. This is wrong.

We have the following theorem:

Theorem: If $f$ is continuous at $a$ and $\lim_{x \to a}f'(x) = L$ then $f'(a) = L$.

But the converse of the theorem does not hold. Thus if $f'(a) = L$ it does not mean that $\lim_{x \to a}f'(x)$ exists necessarily.

Thus the method of using limit of $f'(x)$ works only when the limit exists. Better not to try this approach and rather use the definition of derivative. I must say that your book has done a great job to highlight this fact about limit of derivative at a point and its relation to existence of derivative at that point (although not in the formal manner which I have given in my answer).

The enveloping curves define differentiability. During infinite oscillations first curve tangent cannot decide between two slopes. But for the second, slopes are both equal to zero, makes it differentiable with its coinciding slope of tangent.