$f''(x) = g(x)$ and $g''(x) = f(x).$ Suppose also that $f(x)g(x)$ is linear in $x$ on $(a,b).$ Show that $f(x) = g(x) = 0$ for all $x ∈ (a,b).$

$x=0$ is not a logically correct way to solve these types of functional equations, since $x$ is a variable which is supposed to vary from $a$ to $b$.You cannot fix its value any way whatsoever.


Better to proceed like this:

Assume $f(x)g(x) = kx$ where $k$ is some constant (because of the linearity condition).

Therefore, you get $$f'(x)g(x) + f(x)g'(x) = k$$ $$\Rightarrow f''(x)g(x) + 2f'(x)g'(x) + f(x)g''(x)= 0$$ $$\Rightarrow g^2(x) + 2f'(x)g'(x) + f^2(x)= 0 \tag1$$ Differentiating again gives, $$\Rightarrow 2g(x)g'(x) + 2f''(x)g'(x) + 2f'(x)g''(x) + 2f(x)f'(x)= 0$$ $$\Rightarrow 4g(x)g'(x) + 4f(x)f'(x)= 0$$ $$\Rightarrow g(x)g'(x) + f(x)f'(x)= 0 \tag{2}$$ If you differentiate two more times, you will get $$\Rightarrow f(x)g'(x) + g(x)f'(x)= 0 \tag3$$

Now can you solve $(2)$ and $(3)$ for $f(x)$ and $g(x)$?


Since $f(x)g(x)$ is linear then its second derivative is 0.So,$f''(x)g(x)+2f'(x)g'(x)+g''(x)f(x) =0$ that is $g^{2}(x)+f^2(x) =-2f'(x)g'(x)$ and as $f,g$ are twice differentiable and non-deacreasing,so $f'(x)$ and $g'(x)$ can not be negative.So $f(x)=g(x)=0$.