Homotopy group action and equivariant cohomology theories

From modern perspective this is much more straightforward than the "genuine" version you described above the question. Naive $G$-spaces are just functors $BG\to \cal{S}$ among infinity categories. $G$-spectra are just functors $BG\to \mathrm{Sp}$. You can think of a $G$-spectrum as a functor on $G$-spaces by $E \mapsto (X\mapsto \mathrm{Map}_{\mathrm{Sp}^{BG}}(\mathbb{S}[X],E))$ where $\mathbb{S}[-]= \Sigma^{\infty}$ is the stabilization functor, applied pointwise to functors from $BG$. Hence after accepting some notions like functors and stabilization in infinity category theory you immediately get a theory of equivariant stuff of this "up to homotopy" flavour. In particular, if $E$ has trivial $G$-action then by trivial-colimit adjunction and colimits preservation of the stabilization we get $$\mathrm{Map}_{\mathrm{Sp}^{BG}}(\mathbb{S}[X],E)\simeq \mathrm{Map}_{\mathrm{Sp}}(\mathbb{S}[X]_{hG},E)\simeq \mathrm{Map}_{\mathrm{Sp}}(\mathbb{S}[X_{hG}],E)$$ and you indeed get the cohomology of the homotopy quotient.

In some sense, the surprising thing from this modern perspective is the existence of the "strict" version, which is slightly harder to define internally to modern homotopy theory, even though it is doable.

Much has already been said in the other answers and comments, but let me summarize a few points.

One way to obtain from a category a 'homotopy theory' (aka an $\infty$-category) is to specify a notion of weak equivalence. On the category of $G$-spaces (i.e. topological spaces with strict $G$-action), two of the major notions of weak equivalences are the following:

• A map $X \to Y$ of $G$-spaces is a weak equivalence if the underlying map of spaces is a weak homotopy equivalence, or

• a map $X \to Y$ of $G$-spaces is a weak equivalence if the maps $X^H \to Y^H$ are weak homotopy equivalences for all subgroups $H\subset G$.

More generally, you could specify a family $\mathcal{F}$ of subgroups of $G$ and you demand that you have a weak equivalence on $H$-fixed points for all $H\in \mathcal{F}$, but let's focus on the two cases above and call them underlying and genuine.

(Edit: Reacting to Denis's comment a clarification: Why should we consider these two kinds of equivalences? Geometrically, $G$-homotopy equivalences (i.e. we have an equivariant homotopy inverse and the homotopies are also equivariant) are maybe the most relevant notion. As in non-equivariant topology, there is a Whitehead theorem showing that genuine weak equivalences between $G$-CW complexes are $G$-homotopy equivalences. Illman's theorem shows that every compact $G$-manifold has the structure of a $G$-CW complex, so one can say that most nice $G$-spaces have the structure of a $G$-CW complex. If we want a Whitehead theorem for underlying equivalences instead, we must demand that the $G$-action is free though. Sometimes we are happy to do this, but often this is too restrictive. The different families $\mathcal{F}$ correspond to allowing different families of isotropy. )

It is the $\infty$-category associated with the underlying equivalences that can be modelled by homotopy coherent actions. Taking the coherent nerve of the simplicial category of spaces $\mathcal{S}$, we obtain the $\infty$-category of spaces and the $\infty$-category of spaces with homotopy coherent $G$-action is then modelled/defined as simplicial set maps (aka functors) from $BG$ into this coherent nerve. (If we fix $X$, this is the same as simplicial set maps from $BG$ into $B$ of the homotopy automorphisms of $X$.) This $\infty$-category is equivalent to that associated with $G$-spaces and underlying equivalences. (It is nothing special about starting with a group here. We can instead take functors from an arbitrary small category $\mathcal{C}$ into topological spaces and have a similar story using $B\mathcal{C}$. See e.g. Proposition 4.2.4.4 of Higher Topos Theory.)

We cannot, however, recover from the homotopy coherent action the data of the fixed points $X^H$. If we want to model this homotopy-coherently, we need not only $X$ with a homotopy coherent $G$-action, but we also need all spaces of fixed points $X^H$ with their residual actions and all the restriction maps between them. This can be modelled as a functor from the (nerve of the) orbit category $\mathrm{Orb}_G$ of $G$ into $\mathcal{S}$. In the background is Elmendorf's theorem that shows that there is a Quillen equivalence between $G$-spaces with genuine equivalences and functors from $\mathrm{Orb}_G$ to $\mathrm{Top}$ with underlying equivalences (the Quillen equivalence being given by associating to $G/H$ the fixed points $X^H$). Then one can apply e.g. Proposition 4.2.4.4 of HTT again.

As already remarked by others, some equivariant cohomology theories are only sensitive to underlying equivalences (Borel theories), while others are only invariant under genuine equivalences. The latter are actually more frequent (Bredon cohomology, equivariant K-theory, equivariant bordism...).

The story for spectra is a bit more complicated because there are even more types of weak equivalences one can put on, say, orthogonal spectra with a $G$-action. In Shachar's answer, he describes the case corresponding to underlying equivalences. Genuine equivalences (with respect to a complete universe) require more work. For finite groups, one can consider functors from the Burnside category -- this is the perspective of viewing $G$-spectra as spectral Mackey functors. But this is maybe leading too far here.

Since the OP asked for references, and about $(-)_{hG}$ in particular, I'll mention a few.

Schwede has very clear lecture notes about the basics of equivariant (stable) homotopy theory.

Many more references can be found in this syllabus, including specific references to Lurie's work.

Lastly, Paul VanKoughnett created a wonderful series of lectures, and this one carefully goes through homotopy fixed points and homotopy orbits, i.e. $(-)^{hG}$ and $(-)_{hG}$.