Assume that $\int_{a}^{ab} f(x) dx$ is independent of $a$. Prove $f(x)=\frac{c}{x}$
Most probably you mean that $f$ is continuous.
Let $F$ be an antiderivative of $f$.
Hence, for any $t>0$ we have
$$\int_a^{at}f(x)\;dx = F(at)-F(a)$$
Since the integral is independent of $a$ you have
$$\partial_a(F(at)-F(a))=tf(at) - f(a) = 0 \Rightarrow f(at)=\frac{f(a)}{t}$$
Now, seting $x=at$ you get
$$f(x) = \frac{af(a)}{x} =\frac{c}{x}$$
Hint. Let $F$ be the antiderivative of $f$, then the condition is $F(ab)-F(a)=g(b)$ for some $g$. Differentiating with respect to $a$, we get $bf(ab)-f(a)=0$, hence $f(ab)/f(a)=1/b$.
Define $$g(t)=\int_t^{tb}f(x)\operatorname dx$$
Then $g(t)=c$, so $g'(t)=0$.
By FTC, $$g'(t)=bf(tb)-f(t)$$ $$\implies f(tb)=\frac1{bf(t)}$$ Let $t=1$. Then $f(b)={f(1)\over b}$. $$\implies \boxed{f(x)=\frac cx}$$ with $c=f(1)$.