How can I understand the "groupoid" quotient of a group action as some sort of "product"?
Let $X$ and $Y$ be groupoids. An action of $Y$ on $X$ is a functor $\rho: Y \to B\operatorname{Aut}(X)$, where $B\operatorname{Aut}(X)$ is the one-object 2-groupoid such that $\operatorname{Hom}(\ast, \ast)$ is the 2-group of autoequivalences of $X$.
We define $X \rtimes Y$ as follows. Its objects are simply $\operatorname{Ob}(X) \times \operatorname{Ob}(Y)$. An element of $(X \rtimes Y)((x_1, y_1), (x_2, y_2))$ consists of a pair $(f, g)$, where $g \in Y(y_1, y_2)$, and $f \in X(x_1, \rho(g)x_2)$. Given $(f, g) \in (X \rtimes Y)((x_1, y_1), (x_2, y_2))$, and $(f', g') \in (X \rtimes Y)((x_2, y_2), (x_3, y_3))$, we define $(f', g') \circ (f, g) \in \operatorname{Hom}((x_1, y_1), (x_3, y_3))$ as $(\rho(g)f' \circ f, g' \circ g)$. It is straightforward to check that in the case that $Y$ is $1 // G$, $X \rtimes Y \cong X // G$.
The notion of semidirect product $\Gamma \rtimes G$ where $G$ is a group acting on a groupoid $\Gamma$ is set up in Chapter 11, Section 11.4, of my book "Topology and Groupoids".
It is used there in connection with studying orbit groupoids, and their relevance to the fundamental groupoid of an orbit space by a group action.
One nice point is that this semidirect product includes the case $\Gamma$ is a discrete groupoid, i.e. essentially a set, when you get what is commonly called the action groupoid. In this case the morphism $p: \Gamma \rtimes G \to G$ is known as a covering morphism of groupoids, and all covering morphisms of $G$ arise in this way.
I feel the use of covering morphisms of groupoids makes for a nice exposition, base point free, of the theory of covering spaces. Such an idea was pointed out for the simplicial case in the 1967 book on simplicial theory by Gabriel and Zisman, was used in the first 1968 edition of my book, and is partially used in Peter May's 1999 book "A concise course in algebraic topology".
Update: I should also add that the notion of action of a groupoid on a groupoid is given, following C. Ehresmann, in my paper
[11] ``Groupoids as coefficients'', Proc. London Math Soc. (3) 25 (1972) 413-426.
available here. The aim was cohomology with coefficients in a groupoid. One of the methods exploited is fibrations of groupoids.
A notion of "double product" for groups which act on other "compatibly" is discussed in paper 22 of this list. Another relevant paper on groupoids and actions is this paper.