$fgf = f$, $gfg = g$, $fg$ not necessarily identity, what is this called?

It is called "generalized inverse". In that case $fg$ and $gf$ are idempotents. In particular, if you have a semigroup of maps $X\to X$ (i.e. a set of maps closed under composition) such that every $f$ has a generalized inverse, the semigroup is called regular. If the generalized inverse is unique, the semigroup is called inverse. See Clifford and Preston "Algebraic theory of semigroups".

Linear case

In the linear case, these identities are part of the definition of the Moore-Penrose pseudo-inverse, which exists and is unique. Given $A\in M_{p\times q}(\mathbb C)$, its MPpi is the matrix $A^\dagger\in M_{q\times p}(\mathbb C)$ that satisfies $$AA^\dagger A=A,\qquad A^\dagger AA^\dagger=A^\dagger,\qquad(AA^\dagger)^H=AA^\dagger,\qquad(A^\dagger A)^H=A^\dagger A,$$ where the superscript $H$ stands for the Hermitian adjoint.

If $A\in GL_n(\mathbb C)$, then $A^\dagger=A^{-1}$. But otherwise, $AA^\dagger$ and $A^\dagger A$ are only unitary projections.

Nonlinear case

The situation where $f=g$ is amazing: one looks at functions $h$ such that $h\circ h\neq {\rm id}$, whereas $h\circ h\circ h=h$. Then we have $h^{(2k)}=h^2$ and $h^{(2k-1)}=h$ for every $k\ge1$.

Such an $h$ can be obtained by the following construction, when we are given $f,g$ such that $fgf=f$, $gfg=g$ and at least one of $fg$ or $gf$ is not the identity. Just define $h(x,y)=(f(x),g(y))$ on the cartesian product.

Application: take for $f$ the backward shift on $\ell^p({\mathbb N})$ and for $g$ the forward shift.

This is also naturally related to adjointness:

if you consider both $X$ and $Y$ to be poset categories, and add the conditions

• both $f$ and $g$ order-preserving
• $x \leq_X g(f(x))$ and $f(g(y)\leq_Y y$ for all $(x,y)\in X\times Y$

to your conditions (this is admittedly a little less general, but satisfied in many natural situations), then:

the conditions are met

$\Longleftrightarrow$

$f:X\leftrightarrow Y:g$

is an adjunction.