Functional equation: what function is its inverse's reciprocal?

No, it is impossible.

If $f: (0,\infty) \to (0,\infty)$ is continuous and $f^{-1}$ exists, then $f$ is either increasing or decreasing. If $f$ is increasing, $f^{-1}$ is increasing but $1/f$ is decreasing. If $f$ is decreasing, $f^{-1}$ is decreasing but $1/f$ is increasing.

EDIT: However, for $f: \mathbb R \backslash \{0\} \to \mathbb R \backslash \{0\}$ it is possible. Take

$$ f(x) = \cases{ -x & if $x > 0$\cr -1/x & if $x < 0$\cr} $$

Call $g=\ln\circ f\circ \exp:\Bbb R\to\Bbb R$. Then, $g^{-1}=\ln\circ f^{-1}\circ \exp=\ln\frac1{f\circ \exp}=-g$.

So, we want the homeomorphisms $g:\Bbb R\to\Bbb R$ such that $g^{-1}=-g$. But the homeomorphism $\Bbb R\to\Bbb R$ are strictly monotone continuous functions. And, if $g$ is strictly monotone, $g^{-1}$ must be monotone of the same sign. This is not consistent with $g^{-1}=-g$. Thus, there is no such $g$ and no such $f$.