Why do I get this particular color pattern when using rand()?

Many of the calculations that you're doing in your code won't lead to truly random values. Those sharp lines that you're seeing correspond to places where the relative values of your x and y coordinates trade with one another, and when that happens you're using fundamentally different formulas. For example, computing (x + y) % rand() will generally give you back the value x + y, since rand() will (usually) return a number much, much bigger than x + y given that RAND_MAX is usually a fairly large number. In that sense, you shouldn't expect to get back white noise, since the algorithm you're using to generate things is biased away from generating white noise. If you want white noise, just set each pixel to rand(). If you'd like a nice pattern like the one you have above, but with a little bit of randomness tossed in here and there, keep using the code that you have written.

Additionally, as @pm100 has noted in the comments, the rand function doesn't return truly random numbers, and instead uses a pseudorandom function to produce it values. The default implementation of rand on many systems uses a type of pseudorandom number generator called a linear congruential generator that produces numbers that in short bursts can appear random, but which are decidedly nonrandom in practice. For example, here's an animation from Wikipedia showing how random points in space chosen with a linear congruential generator end up falling into a fixed number of hyperplanes:

If you replace x, y, and z coordinates with R, G, and B coordinates, this looks remarkably similar to the output being produced by your program. I suspect that this is probably not the core issue here, since the other aspect mentioned above will probably be much more pronounced.

If you're looking for higher-quality random numbers, you'll need to use a higher-quality random source. In C, you could consider reading bytes from /dev/urandom/ (on a Linux-like system), which gives fairly uniformly random values. C++ now has a number of good random-number generation primitives in its standard libraries, if that's available to you.

I was initially going to have the same answer as everyone else had and chalk this up to the issues with rand(). However, I thought better of doing so and instead analyzed the distribution your math is actually producing.

TL;DR: The pattern you see has nothing to do with the underlying random number generator and instead is simply due to the way your program is manipulating the numbers.

I'll stick to your blue function since they're all similar.

uint8_t blue(uint32_t x, uint32_t y) {
    return (rand() % 2)                  ? (x + y) % rand() :
           ((x * y % 1024) % rand()) % 2 ? (x - y) % rand() :
                                           rand();
}

Each pixel value is selected from one of three functions: (x + y) % rand(), (x - y) % rand(), and rand();

Let's look at images produced by each of these alone.

• rand()

This is what you would expect, just noise. Call this "Image C"

---

• (x + y) % rand()

Here you're adding the pixel coordinates together and taking the remainder from dividing by a random number. If the image is 1024x1024 then the sum is in the range [0-2046]. The random number you're diving by is in the range [0,RAND_MAX], where RAND_MAX is at least 32k and on some systems is 2 billion. In other words there's at best a 1 in 16 chance that the remainder isn't just (x + y). So for the most part this function will just produce a gradient of increasing blue toward the +x +y direction.

However you're only using the lowest 8 bits, because you return a uint8_t, so you'll have stripes of gradients 256 pixels wide.

Call this "Image A"

---

• (x - y) % rand()

Here you do something similar, but with subtraction. As long as x is greater than y you'll have something similar to the previous image. But where y is greater, the result is a very large number because x and y are unsigned (negative results wrap around to the top of the unsigned type's range), and then the the % rand() kicks in and you actually get noise.

Call this "Image B"

Each pixel in your final image is taken from one of these three images using functions rand() % 2 and ((x * y % 1024) % rand()) % 2. The first of these can be read as choosing with 50% probability (ignoring issues with rand() and its low order bits.)

Here's a closeup of where rand() % 2 is true (white pixels) so Image A is selected.

The second function ((x * y % 1024) % rand()) % 2 again has the issue where rand() is usually greater than the thing you're dividing, (x * y % 1024), which is at most 1023. Then (x*y%1024)%2 doesn't produce 0 and 1 equally often. Any odd number multiplied by any even number is even. Any even number multiplied by any even number is also even. Only an odd number multiplied by an odd number is odd, and so %2 on values that are even three quarters of the time will produce 0 three quarters of the time.

Here's a closeup of where ((x * y % 1024) % rand()) % 2 is true so that Image B could be selected. It's selecting exactly where both coordinates are odd.

And here's a closeup of where Image C could be selected:

Finally combining the conditions here's where Image B is selected:

And where Image C is selected:

The resulting combination can be read as:

With 50% probability use the pixel from Image A. The rest of the time pick between Image B and Image C, B where both coordinates are odd, C where either one is even.

Finally, since you're doing the same for three different colors, but with different orientations the patterns are oriented differently in each color and produce the crossing strips or grid pattern you're seeing.