Yoga of six functors for group representations?

Allow me to answer not in stacks but rather in spaces (which may be appropriate if the answer is about getting intuition; e.g. for $B\mathbb{Z}/2\mathbb{Z}=\mathbb{RP}^\infty$ there is probably nothing that cannot be said without stack language). The main point I want to make is that setting up the dictionary is more subtle and doesn't work in the generality suggested by the question.

One of the relevant references here would appear to be

• [BL] J. Bernstein and V. Lunts. Equivariant sheaves and functors. Lecture Notes in Math 1578, Springer, 1994.

They are mostly interested in topological groups with some mild conditions (and what I say below applies in this generality), but also discuss in particular discrete groups. Some of the stuff can be done similarly for algebraic groups.

First there is an issue with the actual category we are interested in. If the group is discrete, then it's ok to consider the derived category of representations (but that requires proof, cf. Section 8 [BL]). If the group is not discrete, one would rather like to look at the equivariant derived category (which is not the derived category of equivariant sheaves). So the $D(BG):= D(Rep(G))$ is already a bit problematic. Moreover, if we only consider the space $BG$ then $D(BG)$ is not really the category one would be interested in (too big); one would want objects which have locally constant cohomology. (I think these issues with already finding the appropriate definition doesn't go away if we just say stack.)

Now come the issues with the six functors: well, there is no issue for $\pi^\ast$. This exists and corresponds to restriction. There is an issue with $\pi_\ast$: if we look at the equivariant derived category, this is not the full derived category of $BG$ but only those with locally constant cohomology. The right adjoint functors will typically not preserve this property. This is why there is an extensive discussion on how to actually construct the functor $\pi_\ast$ in the book of [BL]. There is another issue with $\pi_\ast$: if we are only looking at the derived category of bounded complexes, then $\pi_\ast$ may not actually be defined. For example if $G$ which is not of finite cohomological dimension then the ordinary pushforward of the constant sheaf along $BG\to {\rm pt}$ will not land in the bounded derived category.

There are further significant issues with the exceptional functors. First, they are not constructed in the situation for $BG\to BH$, [BL] only do this for morphisms of $G$-spaces where $G$ is fixed. One of the issues for $\pi_!$ is that the morphism $BG\to BH$ may not be compactifiable. For instance, for $B\mathbb{Z}/2\mathbb{Z}\cong \mathbb{RP}^\infty$ the space is not locally compact. So what should $\pi_!$ for $\pi:\mathbb{RP}^\infty\to{\rm pt}$ be? We can use the definition as colimit of cohomology relative to compact subsets, in which case $\pi_!$ will be trivial. One could use a definition via compactification but that also seems problematic for $\mathbb{RP}^\infty$. In particular, translating $\pi_!$ into group cohomology doesn't seem appropriate.

Now there is a similar issue for $\pi^!$. For $\mathbb{RP}^\infty$ we can approximate it by $\mathbb{RP}^n$. For the map $\pi_n:\mathbb{RP}^n\to{\rm pt}$ we can use Poincaré-Verdier duality to tell us that $\pi_n^\ast=\pi_n^![-2n]$. To get the result for $\mathbb{RP}^\infty$ we should take the limit which again means $\pi^!=0$.

So these are some reasons why the exceptional functors do not quite appear in this setting. What exists in any case are two induction functors (Section 3.7 of [BL]), left and right adjoint to restriction for the map $BG\to BH$ induced by a closed subgroup $G\subset H$. (This implies that the fiber of $BG\to BH$ is the homogeneous space $H/G$ and there is no infinite-dimensional contribution like the classifying space of the kernel.) [BL] denote these functors $Ind_!$ (left adjoint to Res) and $Ind_\ast$ (right adjoint to Res). In the setting of discrete groups, Section 8 of [BL] discusses the relation between the left adjoint and usual induction.

I'm not sure if these problems go away if we say "stack" everywhere. As far as I understand the six-functor formalism for stacks is also quite problematic to set up.

I assume all groups are discrete and ignore Q3.

Also I assume that π comes from a group homomorphism from H to G.

By Poincaire-Verdier duality π^! is the same as π^*. Both of these are identified with restriction.

Then π_* is the right adjoint of restriction, hence is what I would call coinduction, while π_! is the left adjoint of restriction, hence induction. In some contexts (e.g. p-adic groups), I've seen the right adjoint be called induction.

The dualising sheaf is the same as the constant sheaf, hence corresponds to the trivial representation.

I'm more familiar with sheaf cohomology than group cohomology, but I think this is at least partly an issue of notation, as Peter McNamara suggests. I will also assume that all groups here are discrete.

To expand on his answer a bit, your notation for the adjoint triple induction $\dashv$ restriction $\dashv$ coindunction (or extension $\dashv$ restriction $\dashv$ coextension) appears to follow the common algebraic geometry convention for ring maps, but they do that because affine schemes are the opposites of commutative rings, and left adjoints are swapped for right adjoints when you take the opposite category.

So to make group cohomology look more like sheaf cohomology, the notation should be $\pi_! \dashv \pi^* \dashv \pi_*$. Then group cohomology is also right derived pushforward $\mathbf{R}\pi_*$, just like sheaf cohomology, and group homology is derived lower shriek, just like compactly supported sheaf cohomology.

Hence, Verdier duality should be given by an adjunction $\pi_! \dashv \pi^!$, but for group reps $\pi_!$ aleady has a right adjoint $\pi^*$, so for discrete group representations we are always in the Wirthmüller context where $\pi^!=\pi^*$.