Are there two functions $f, g$ such that $f(g(x)) = x^3$ and $g(f(x)) = x^5$?

Yes, there exist such $f,g$. Let us define$$ g(x)=\begin{cases}0,\quad x=0\\ 1,\quad x=1\\ \exp\left[5\exp\left(\frac{\log 5}{\log 3}\log \log x\right)\right],\quad x>1\\ \exp\left[-5\exp\left(\frac{\log 5}{\log 3}\log \log \frac{1}{x}\right)\right],\quad 0<x<1\\ -g(-x),\quad x<0 \end{cases} $$ and $$ f(x)=\begin{cases}0,\quad x=0\\ 1,\quad x=1\\ \exp\left[\exp\left(\frac{\log 3}{\log 5}\log \log x\right)\right],\quad x>1\\ \exp\left[-\exp\left(\frac{\log 3}{\log 5}\log \log \frac{1}{x}\right)\right],\quad 0<x<1\\ -f(-x),\quad x<0 \end{cases}. $$ Then we can check that $f,g$ are continuous, odd, strictly increasing bijection on $\mathbb{R}$. And we can also see that $g((0,1))=(0,1)$ and $g((1,\infty))=(1,\infty)$, and the same holds for $f$. Finally, observe that $$ f(g(x))= \begin{cases}\exp\left[\exp\left(\frac{\log 3}{\log 5}\left(\frac{\log 5}{\log 3}\log \log x+\log 5\right)\right)\right]=\exp[3\log x]=x^3,\quad x>1\\\exp\left[-\exp\left(\frac{\log 3}{\log 5}\left(\frac{\log 5}{\log 3}\log \log \frac{1}{x}+\log 5\right)\right)\right]=\exp[-3\log \frac{1}{x}]=x^3,\quad 0<x<1\\ 1,\quad x=1\\ 0,\quad x=0. \end{cases} $$ Since $f,g$ are odd, this shows $f(g(x))=x^3$. Similarly, it holds that $g(f(x))=x^5$.

Note: I guess similar construction works for general odd pairs $(i,j)$ where $i \neq 1,j \neq 1$ by modifying parameters. When $(i,j)$ is an even pair, I guess we can construct even $f,g$ also by modifying paramters.