The randomness of modular squaring

Plotting things on a circle like this focuses attention on dihedral symmetry, which is implemented on the integers $\bmod n$ by translation $t \mapsto t+1$ and negation $t \mapsto -t$. Addition $t \mapsto t + k$ is itself a translation, so of course it has translational symmetry (it commutes with translations). Scaling $t \mapsto kt$ always at least commutes with negation, and it has a simple behavior with respect to translation: the result of translating and then scaling is

$$t \mapsto k_1(t + k_2) = k_1 t + k_1 k_2$$

and so scaling sends translations to different translations. In the example you plotted where $k_1 = 30$ and $n = 120$ we see that taking $k_2 = 4$ gives $30(t + 4) = 30t + 120 \equiv 30t \bmod 120$ which explains some of the regularity of that graph, and $30(t + 30) = 30t + 900 \equiv 30t + 30 \bmod 120$ which explains some more of it.

For powers $t \mapsto t^k$ we can argue as follows. If $k$ is odd then taking powers comutes with negation so we have a reflection symmetry. If $k = p$ is a prime dividing the modulus $n$ then $(t + k_2)^p \equiv t + k_2 \bmod p$ so we have a translation symmetry $\bmod p$ which becomes a partial translation symmetry $\bmod n$ by the Chinese remainder theorem; in particular it implies a symmetry with respect to the translation $t + \frac{n}{p}$ which explains some of the regularity in the graphs for $p = 3, 5$.

When $k = 2$ we no longer have a reflection symmetry, and the moduli you picked were either odd or divisible by $4$, so so we don't have any obvious translation symmetry either. In fact we don't have any translation symmetry at all: if squaring commuted with $t \mapsto t + k$ then we must have

$$(t + k)^2 \equiv t^2 + k \bmod n$$

for all $t$, which gives $2kt + (k^2 - k) \equiv 0 \bmod n$ for all $t$. Taking $t = 0, 1$ and subtracting gives $2k \equiv 0 \bmod n$ and $k^2 - k \equiv 0 \bmod n$ which has no nonzero solutions if $n$ is odd, and if $4 \mid n$ the only nonzero possibility is $k \equiv \frac{n}{2} \bmod n$, but then $\frac{n}{2} - 1$ is odd and so $k^2 - k$ is divisible by one less power of $2$ than $n$. We only get at best a translation symmetry by $\frac{n}{2}$ if $n \equiv 2 \bmod 4$.

With most such maps you should get some sort of uniform distribution result. For example, J. Beck and I did this for the map x^{-1} mod n. See "On the uniform distribution of inverses modulo n", Periodica Math. Hungarica, Vol 44 (2) 2002, 147-155. (Forgive the self-serving PR, but as one gets older one loses all sense of propriety or modesty! Also this is the age of Trump!) It follows from estimates for Kloostermann sums.

As long you have good estimates for the relevant exponential sum, you should get something interesting.

In reply to your question in the comments it depends what you mean by "randomness". All I am saying that if you look at pictures of the set of points (x/n,(x^{-1} mod n)/n) in the unit square, then, as you let n go to infinity, it fills up the square in an uniform way. However, it is rare to get 3 or more points to lie on a line. This never happens when n is prime. Composite is a different matter. So one can "argue" that the points are distributed uniformly in a "random" way.

In 1963 Renyi and Sulanke published about random polytopes. The paper is frequently cited. It was about the properties of polytopes that are the convex hull of n points where the points are chosen with regard to a uniform distribution function. Something like that. Barany has a 2007 preprint online about this. You may want to look at these articles.