Orbits in modular arithmetic

You can first notice that $fgfgfg(x)=x$ and conclude that most orbits have size $6$. It is easy to show that there is one orbit of size $3$ which you found and there is an orbit of size $2$ whenever there is a solution to $$x(1-x)\equiv 1\pmod{p}$$ this happens if $\binom{-3}{p}=1$ so if $p\equiv 1\pmod{3}$. You can easily show that there can't be orbits of size $4$ or $5$. So the answer is $\frac{p+1}{6}$ if $p\equiv -1\pmod{3}$ and $\frac{p+5}{6}$ if $p\equiv 1\pmod{3}$.

Your transformations act on the projective line $\mathbb Z_p \cup\{\infty\}$, preserving the three points $0$, $1$, $\infty$. The group they generate is isomorphic to the group $\mathfrak S_3$ of permutations on these three points.

Anyway, you can enumerate the orbit of a point $x\in\mathbb Z_p\setminus\{0,1\}$ and obtain [ { x, 1-x, \frac 1x, \frac1{1-x}, \frac x{x-1} , \frac{x-1}x}. ] This orbit usually has cardinality 6, unless if the stabilizer of $x$ is nontrivial. When this happens, you have $x=1-x$, or $x=1/x$, or $x=1/(1-x)$, or $x=x/(x-1)$, or $x=(x-1)/x$.

Assume now that $p>3$. (The cases $p=2$ and $3$ are trivial.)

The preceding analysis shows that there is exaclty one orbit with cardinality $3$, namely $\{-1,2,1/2\}$, and one orbit of cardinality $2$, $\{\alpha,1-\alpha\}$, where $\alpha$ is an element of $\mathbb Z_p$ satisfying $\alpha^2-\alpha+1=0$. Such an $\alpha$ exists if and only if $-3$ is a square in $\mathbb Z_p$.

Let $k$ be the number of orbits with cardinality $6$. The total number of orbits is $k+2$ if $-3$ is a square, and $k+1$ otherwise. Counting the number of elements in $\mathbb Z_p\setminus\{0,1\}$ gives $p-2=6k+3+2$ in the former case, and $p-2=6k+3$ in the latter, that is: $p=6k+7$, resp. $p=6k+5$.

Finally, we obtain that for $p\equiv 1\pmod 6$, $-3$ is a square modulo $p$ and there are $(p+5)/6$ orbits, while for $p\equiv -1\pmod 6$, $-3$ is not a square and there are $ (p+1)/6$ orbits.

NB. In both cases, the number of orbits is equal to $\lfloor(p+5)/6\rfloor$.