Find all continuous functions such that $f(x)f(2x)\dots f(nx) \le an^k$

The condition is equivalent to saying $$\sum ^n _{j=1} \ln {f(jx)} \leqslant \ln a + k \ln{n}, $$ For real number $x$ and positive integer $n$.

Taking $\alpha > 0$ and $x = \frac{\alpha}{n}$, we have

$$\sum ^n _{j=1} \ln {f\left(j\frac{\alpha}{n}\right)} \leqslant \ln a + k \ln{n}$$

$$\sum ^n _{j=1}\frac{\alpha}{n} \ln {f\left(j\frac{\alpha}{n}\right)} \leqslant \frac{\alpha\ln a + \alpha k \ln{n}}{n}$$

The left-hand side is a Riemann sum for the function $\ln {f}$ on the interval $[0, \alpha]$. Because $f$ is continuous, so is $\ln {f}$ , and thus $\ln {f}$ is integrable. Letting $n$ tend to infinity, we get

$$ \int_0^\alpha \ln{ f (x)}\ \mathrm dx \leqslant \lim_{n \to \infty} \frac{\alpha\ln a + \alpha k \ln{n}}{n} = 0$$

The fact that $f (x) \geqslant 1$ implies that $\ln {f (x)} \geqslant 0$ for all $x$. Hence $\ln f (x) = 0$ for all $x \in [0, \alpha]$. Since $\alpha$ is an arbitrary positive number, $f (x) = 1$ for all $x \geqslant 0$. A similar argument yields $f (x) = 1$ for $x < 0$. Therefore there is only one such function.

If $f$ is not constant, by continuity there is some $m>1$ and an interval $[b,c]$ such that $f\geqslant m$ on $[b,c]$.

We can assume without loss of generality that $0<b$ and $b=\frac{p_1}{q}$ while $c=\frac{p_2}{q}$ for some natural numbers $p_1,p_2,q$ (making the interval smaller if necessary). Taking $p=p_1q$, we can actually assume that $b=\frac{p}{q^2}$ and $c=\frac{p+q}{q^2}$.

For $x=q^{-2l}$ and $n=(p+q)q^{2l-2}$ we have $$ f(x)f(2x)\dots f(nx) \geqslant f(pq^{2l-2}q^{-2k})\cdot f((pq^{2l-2}+1)q^{-2l})\cdots f((p+q)q^{2l-2}q^{-2l})\geqslant m^{q\cdot q^{2l-2}}=m^{q^{2l-1}}$$

Now $n$, as well as any $an^k=a(p+q)^kq^{2lk-2k}$, grows only exponentially in $l$, while this expression grows superexponentially, so it cannot be bounded (for all $l$) by any $an^k$.