Prove that $f'(0)$ exists and $f'(0) = b/(a - 1)$
This is a tricky question and the solution is somewhat non-obvious. We know that $$\lim_{x \to 0}\frac{f(ax) - f(x)}{x} = b$$ and hence $$f(ax) - f(x) = bx + xg(x)$$ where $g(x) \to 0$ as $x \to 0$. Replacing $x$ by $x/a$ we get $$f(x) - f(x/a) = bx/a + (x/a)g(x/a)$$ Replacing $x$ by $x/a^{k - 1}$ we get $$f(x/a^{k - 1}) - f(x/a^{k}) = bx/a^{k} + (x/a^{k})g(x/a^{k})$$ Adding such equations for $k = 1, 2, \ldots, n$ we get $$f(x) - f(x/a^{n}) = bx\sum_{k = 1}^{n}\frac{1}{a^{k}} + x\sum_{k = 1}^{n}\frac{g(x/a^{k})}{a^{k}}$$ Letting $n \to \infty$ and using sum of infinite GP (remember it converges because $|a| > 1$) and noting that $f$ is continuous at $x = 0$, we get $$f(x) - f(0) = \frac{bx}{a - 1} + x\sum_{k = 1}^{\infty}\frac{g(x/a^{k})}{a^{k}}$$ Dividing by $x$ and letting $x \to 0$ we get $$f'(0) = \lim_{x \to 0}\frac{f(x) - f(0)}{x} = \frac{b}{a - 1} + \lim_{x \to 0}\sum_{k = 1}^{\infty}\frac{g(x/a^{k})}{a^{k}}$$
The sum $$\sum_{k = 1}^{\infty}\frac{g(x/a^{k})}{a^{k}}$$ tends to $0$ as $x \to 0$ because $g(x) \to 0$. The proof is not difficult but perhaps not too obvious. Here is one way to do it. Since $g(x)\to 0$ as $x \to 0$, it follows that for any $\epsilon > 0$ there is a $\delta > 0$ such that $|g(x)| < \epsilon$ for all $x$ with $0 <|x| < \delta$. Since $|a| > 1$ it follows that $|x/a^{k}| < \delta$ if $|x| < \delta$ and therefore $|g(x/a^{k})| < \epsilon$. Thus if $0 < |x| < \delta$ we have $$\left|\sum_{k = 1}^{\infty}\frac{g(x/a^{k})}{a^{k}}\right| < \sum_{k = 1}^{\infty}\frac{\epsilon}{|a|^{k}} = \frac{\epsilon}{|a| - 1}$$ and thus the sum tends to $0$ as $x \to 0$.
Hence $f'(0) = b/(a - 1)$.
BTW the result in question holds even if $0 < |a| < 1$. Let $c = 1/a$ so that $|c| > 1$. Now we have $$\lim_{x \to 0}\frac{f(ax) - f(x)}{x} = b$$ implies that $$\lim_{t \to 0}\frac{f(ct) - f(t)}{t} = -bc$$ (just put $ax = t$). Hence by what we have proved above it follows that $$f'(0) = \frac{-bc}{c - 1} = \frac{b}{a - 1}$$ Note that if $a = 1$ then $b = 0$ trivially and we can't say anything about $f'(0)$. And if $a = -1$ then $f(x) = |x|$ provides a counter-example. If $a = 0$ then the result holds trivially by definition of derivative. Hence the result in question holds if and only if $|a| \neq 1$.