Affine GIT is an open map?
Categorical quotients are in general very far away from being open. In fact, it is a theorem of Chevalley (I think) that a morphism $\pi:X\to Y$ onto a normal variety $Y$ is open if and only it is equidimensional. Unless $G$ is finite, this is a very rare condition for quotient morphisms.
The idea behind this is the following: Assume that the fiber $X_y$ has bigger dimension than the generic fiber dimension $\dim X-\dim Y$. Then there is a curve $C\subseteq Y$ passing through $y$ such that the fiber dimension jumps over $C$. Let $Z$ be the closure of $\pi^{-1}(C)\setminus X_y$ in $X$. Then for dimension reasons $X_y\not\subseteq Z$. Consider the open subset $U:=X\setminus Z$ of $Z$. Then $\pi(Z)=(Y\setminus C)\cup\{y\}$ is only constrictible but not open.
More formally, the openness of a morphism is expressed by the so-called going-down property of Cohen-Seidenberg.
Let's try this out for the quotient of $G=\mathbf G_m$ acting on $\mathbf A^3$ by $(tx,ty,t^{-1}z)$. Then $$ \pi:\mathbf A^3\to\mathbf A^2:(x,y,z)\mapsto (u,v):=(xz,yz) $$ is the quotient morphism mapping the plane $\{z=0\}$ to $(0,0)$. Take $C=\{u=0\}$. Then $\pi^{-1}(C)=\{x=0\}\cup\{z=0\}$ and the image of $U=\mathbf A^3\setminus\{x=0\}$ is $\mathbf A^2\setminus\{u=0\}\cup\{(0,0)\}$ which is not open.
PS: Quotient maps have some properties not shared by other morphism:
- Images of closed $G$-stable subsets are closed.
- A subset of $Y$ is open iff its preimage in $X$ is open.
- The morphism is surjective.
These properties hold universally, i.e., even after base change.
Preamble:
The original answer below is wrong. The mistake is that when restricting the open set $U$ to $U\cap X^{ps}$, the image of the GIT projection (call the map $\pi$) of $U$ may not equal the image of the quotient map $p:X^{ps}\to X^{ps}/G$. Certainly, $p(U\cap X^{ps})\subset \pi(U)$ but the point is that $U$ can contain some non-polystable points that get removed when intersecting and can change the topology of the image. This in fact was my original intuition with my attempted counter-example, but the example I choose was in fact open. So instead of trying another example, I decided to challenge my "gut" by trying to prove myself wrong and surprisingly came up with a "proof".
Anyway, Friedrich Knop's answer is right. I feel like I should delete this "answer," but on the other hand, sometimes failed attempts are instructive to others so I am not sure I will. I added a remark at the end that might be useful to the OP since the OP expressed interest in understanding the strong topology of $X//G$.
Failed Counter Example:
Any open orbit maps to a point, so generally the GIT quotient is not an open map (see comments for the mistake).
Failed Proof of Openness:
We work over $\mathbb{C}$. Take an open set $U\subset X$ then $U\cap X^{ps}$ is open in $X^{ps}$ (with respect to the relative topology) where $X^{ps}$ is the set of polystable points (points with closed orbits). Therefore $U\cap X^{ps}$ maps to an open set in $X^{ps}/G$ since $p:X^{ps}\to X^{ps}/G$ is an open map (this follows from the definition of the quotient topology and the fact that $G$ acts by homeomorphisms). But that set is equal to the image of $U$ under the GIT projection (this step is the mistake). Hence it is open in $X//G$ since $X//G\cong X^{ps}/G$ (see for example Theorem 2.1 here).
Weak Correction:
Whenever $p(U\cap X^{ps})\supset\pi(U)$ for all open $U,$ then $\pi$ is an open map in the strong topology. This follows from the above failed proof, since it fills the gap with an assumption.
Remark:
On the other hand, if one wants to understand the space $X//G$ in the strong topology one can replace $\pi:X\to X//G$ by $p:X^{ps}\to X^{ps}/G$. The latter is open while we now know the former might not be, but as a space in the strong topology $X//G$ remains homeomorphic to $X^{ps}/G.$ More still, as per Proposition 3.4 here, the usual quotient $X/G$ is homotopic to $X//G$.