Fourier transform of function composition

There is no such rule in general. The key here is variable substitution: If $g$ is a bijection and smooth enough then, if all integrals exist: $$ (\widehat{f\circ g})(\xi) = \int f(g(x))\exp(ix\xi)dx = \int f(y)\exp(ig^{-1}(y)\xi)|\det g'(y)|^{-1}dy.$$ This does only rarely lead to something interesting, e.g. in the case of scaling (i.e. linear transformation of the variable): Working in $\mathbb{R}^d$ with $A\in\mathbb{R}^{d\times d}$ invertible: $$ (\widehat{f\circ A})(\xi) = |\det A^{-1}|\widehat{f}(A^{-T}\xi). $$

Have a look at Bergner et al. 2006 "A Spectral Analysis of Function Composition and Its Implications for Sampling in Direct Volume Visualization" IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 12, NO. 5. Sec. 3.

Unfortunately it's not quite as simple as the transform of a convolution or even the derivative. The short answer is, in one dimension let \begin{equation} P(k,l) = \int_{x \in \mathcal{R}} e^{i2\pi(lg(x) - kx)}\,dx. \end{equation} The FT of the composite function is \begin{equation} \text{FT}\left[f\left(g(x)\right)\right](k) = \int_{l\in\mathcal{R}} \hat{f}(l)P(k,l) \,dl, \end{equation} where $\hat{f}(l)$ is the Fourier transform of $f(x)$. As you can see the transformation involves the inner product of $\hat{f}(l)$ with a slightly awkward two dimensional function. In the discrete case this would be implemented as a matrix multiplication.

This is the Lee that gave the answer in 2014, now posting from a registered account. That answer is a simplified version of the multidimensional case. If you want to understand the multidimensional case I suggest a Google search for the paper I referenced.

Best of luck.