If $(f\circ g)(x)=x$ does $(g\circ f)(x)=x$?
Take, for example, $f(x) = \ln(x)$ and $g(x) = e^x$.
Then, for every $x\in\mathbb R$, you have $f(g(x))=\ln(e^x) = x$. But that is not true the other way around, i.e. the statement:
For every $x\in \mathbb R: g(f(x)) = x$
is a false statement, because $f(x)$ is not defined for $x\leq0$.
If you want $f$ to be defined everywhere, you can define
$$f(x)=\begin{cases}\ln|x| & \text{if } x\neq 0\\ 0&\text{if } x = 0\end{cases}.$$
You then have, for every $x\in\mathbb R$: $f(g(x)) = f(e^x)$ which, because $e^x\neq 0$, is equal to $\ln(|e^x|) = \ln(e^x)=x$.
On the other hand, you have, for every $0\neq x\in \mathbb R$,$g(f(x)) = e^{\ln|x|} = |x|$, so if $x<0$, $g(f(x))$ is not equal to $x$.
Let $ dom(f)$ be the domain of the function $f$ and $im(f)$ be its image. It's easier to understand this if you write the question in the form
"If $f \circ g(x) = x$ for all $x \in dom(g)$, then does if follow that $g \circ f(y) = y$ for all $y \in dom(f)$?"
To get the general answer, note that $f \circ g(x) = f(g(x)) = x$ implies $g(f(g(x))) = g(x)$. Therefore, $g \circ f(y) = y$ for all $y$ that can be written as $y = g(x)$ for some $x$, i.e. for all $y \in im(g)$. But if there are $y \in dom(f)$ that are not in $im(g)$, then $g \circ f(y)$ need not equal $y$.
So the answer is yes if $im(g) = dom(f)$, but it may be no if that is not the case.
In the case discussed by @5xum, $g(x) = e^x$ and $im(g) = (0, \infty)$. Now if $f(x) = \ln x$, then $dom(f) = (0,\infty)$. Both $f\circ g(x) = x$ and $g \circ f(y) = y$ are correct for all $x \in dom(g), y \in dom(f)$. But if $f(x) = \ln |x|, f(0) = 0$, then $dom(f) = \mathbb{R}$ which is larger than $im(g)$. Therefore there are $y \in dom(f)$ such that $g \circ f(y) \ne y$ although $f \circ g(x) = x$ for all $x \in dom(g)$.
Let $f(x)=\begin{cases}\tan(x),&\text{if }x \in (-\pi/2,\pi/2)\\0,&\text{otherwise}\end{cases}$
and $g(x)=\tan^{-1}(x)$ with image $(-\pi/2,\pi/2)$.
Then $f(g(x))=x$ but $g(f(2\pi))=0\ne 2\pi$.
The hypothesis is indeed true if both $f$ and $g$ are continuous. $g(\mathbb{R})$ is an interval, and $f|_{g(\mathbb{R})}$ has inverse* $g$ and hence is injective, so must be strictly monotone. If $g(\mathbb{R})$ has a finite endpoint, e.g. $(-\infty,t)$, then $\lim_{x\to t^-} f(x)=\pm\infty$, otherwise $f$ cannot have image $\mathbb{R}$. This contradicts the assumption that $f$ is continuous. Hence $g$ is surjective, and so $f$ and $g$ must be inverses of each other.
*To show that $\forall x\in g(\mathbb{R}), g(f(x))=x$:
Let $y\in \mathbb{R}$ s.t. $g(y)=x$.
$g(f(x))=g(f(g(y)))=g(y)=x$.
I think you can come up with a counterexample if we have only $f$ continuous.