Forward FFT an image and backward FFT an image to get the same result
One important thing to note when you do forward FFT followed by inverse FFT is that this normally results in a scaling factor of N being applied to the final result, i.e. the resulting image pixel values will need to be divided by N in order to match the original pixel values. (N being the size of the FFT.) So your output loop should probably look something like this:
//Overwrite the pixelcolors with the result.
for (int y = 0; y < height; y++) {
for (int x = 0; x < width; x++) {
int currentIndex = ((y * width) + (x)) * 3;
pixelColors[currentIndex] = resultR[y * width + x][0] / (width * height);
pixelColors[currentIndex + 1] = resultG[y * width + x][0] / (width * height);
pixelColors[currentIndex + 2] = resultB[y * width + x][0] / (width * height);
}
}
Also note that you probably want to do a real-to-complex FFT followed by a complex-to-real IFFT (somewhat more efficient in terms of both memory and performance). For now though it looks like you're doing complex-to-complex in both directions, which is fine, but you're not filling your input arrays correctly. If you're going to stick with complex-to-complex then you probably want to change your input loop to something like this:
//Fill in arrays with the pixelcolors.
for (int y = 0; y < height; y++) {
for (int x = 0; x < width; x++) {
int currentIndex = ((y * width) + (x)) * 3;
inR[y * width + x][0] = (double)pixelColors[currentIndex];
inR[y * width + x][1] = 0.0;
inG[y * width + x][0] = (double)pixelColors[currentIndex + 1];
inG[y * width + x][1] = 0.0;
inB[y * width + x][0] = (double)pixelColors[currentIndex + 2];
inB[y * width + x][1] = 0.0;
}
}
i.e. the pixel values go into the real parts of the complex input values and the imaginary parts need to be zeroed.
One more thing to note: when you eventually get this working you'll find that performance is terrible - it takes a long time to create a plan relative to the time taken for the actual FFT. The idea is that you create the plan just once, but use it to perform many FFTs. So you'll want to separate out the plan creation from the actual FFT code and put it in an initialisation routine or constructor or whatever.
But if you use the realToComplex or the ComplexToRealFunction pay attention to the fact that the image will be stored in a matrix of dimensions [height x (width/2 +1)] and if you want to do some intermediate calculations in the frequency domain, they will become a bit harder...
The reason it didn't work is that fftw_plan_dft_2d() does some bench-marking to find the best algorithm and changes input data in the process, so you have to fill the input data after fftw_plan_dft_2d(), not before it.