Text for studying group representations in the context of (abstract) harmonic analysis

As you've seen, "representation theory" is a broad subject, and means different things to different people.

Varadarajan's delightful book "An introduction to harmonic analysis on semi-simple Lie groups" is one of the most readable books on "harmonic analysis" in the repn-theoretic sense, while taking analytic issues seriously. A. Deitmar has a readable book on harmonic analysis, and Echterhoff-Deitmar. Many people have on-line notes about the more-analytic, harmonic-analysis aspects of "repn theory". (E.g., see my various course notes, at http://www.math.umn.edu/~garrett/m/repns/, the latter trying to make a transition from finite-group repn theory to more genuinely analytical issues, and also applications to modular forms and number theory, at http://www.math.umn.edu/~garrett/m/mfms/)

The classics by Serre and Fulton-Harris are almost entirely aimed at finite-dimensional repns of finite groups, or of finite-dimensional Lie algebras. The issues there are very different from the issues of infinite-dimensional repns of not-finite, especially non-abelian, non-compact groups on Hilbert spaces and other topological vector spaces. The latter context was first investigated for reasons coming from physics, where unitary repns on (infinite-dimensional) Hilbert spaces have significance. Bargmann and Wigner did the earliest examples, and then Gelfand-Naimark et al. (Their invocation of asymptotics of special functions appearing as solutions of ODEs was an antecedent of Harish-Chandra's subquotient theorem, and then Casselman's subrepn theorem.)

But by now the utility of repn theory of all sorts in the theory of automorphic forms, number theory is well known. But, equally, the extremes of the "sort" of repn theory are wildly different from each other. Indeed, it has started to seem to me that "repn theory" is not a reasonable name for the vast collection of things that might fall under that label, since it is hilariously non-descriptive.

The courses I've given under the name "repn theory" have invariably displeased at least half the audience on every occasion, by being either "too algebraic" or "too analytic"... or both. Some people want Lie algebras and their finite-dimensional repns, but not Lie groups, no infinite-dimensional repns, and no analysis whatsoever.

One reason that the situation is invariably confusing for beginners is that, for example, Fulton-Harris do not explain very strongly that they are ignoring infinite-dimensional repns. Of course, any author would spend a crazy amount of time if they were obliged to explain in detail all the things they're not doing. (!) Still, it leaves a beginner wondering.

From the opposite end, "harmonic analysis" sometimes (e.g., E. Stein's book of that title) has very little to do with group repns, dealing perhaps exclusively with the abelian groups $\mathbb R^n$ and products of circles. ... or ragged chunks of the group $\mathbb R^n$!?! :) I think that this use of "harmonic analysis" is so extreme that it is not connected to repn theory any more than the rest of mathematics may be.

The contemporary repn theory of finite groups has no analytical issues, naturally, but in effect is always in the "bad" situation, that the characteristic of the field on whose vector spaces the (finite) group acts divides the order of the group. Thus, Maschke's theorem tends to fail, and the homological algebra of the situation is much more complicated than "direct sum". But/and the various non-vanishing (co-)homologies are "new" repn spaces.

There is also a failure of "semi-simplicity" in the "harmonic analysis" version of repn theory, e.g., principal series repns of nice groups like $SL_2(\mathbb R)$ can be reducible, but do not decompose as direct sums. Analogous to the finite-group case, this means that some derived functors will be non-zero, and thus give "new" repn spaces. Borel-Wallach's book pursues this.

Another example is the "oscillator repn", or "Segal-Shale-Weil repn", which began as a Lie algebra repn, then a real Lie group repn, then abstracted considerably by Weil. Discussion of this is made less crazy in the context of various incarnations of the Stone-vonNeumann theorem (roughly asserting the uniqueness of models of quantum thingies) about repns of the Heisenberg group. The finite-field analogues of these things are relatively elementary, since there's no analysis, but/and the real Lie and p-adic cases require some non-trivial analysis to have something more than a mere heuristic. (My notes mentioned above do some of this.)

I'd like to make a couple of recommendations at a slightly lower or comparable level than Varadarajan's excellent treatise that I wished I had read before that book.

First, Folland's 'A course in abstract harmonic analysis' gives a clear road map from functional analysis to the representation theory of topological groups. It treats compact groups carefully with many worked examples, it emphasizes the peculiarities of non-unimodular groups (and even non type I groups, which I found very illuminating), it has a complete treatment o systems of imprimitivity (finally!) and in general touches upon all the subtle analytical issues sometimes glossed over in representation theory textbooks. This is why I recommend it before Varadarajan.

Secondly, the book by Howe and Tan Non-abelian harmonic analysis has a completely different feel: it is driven by the representation theory of a single group, but because of that it has the luxury of going pretty deep into the analytical aspects of its representation theory. It is much more readable (but definitely not as complete) than Lang's $\textrm{SL}(2,\mathbb{R})$ and chapter 4 is a spectacular demonstration of the power of representation theory in analysis! (although the one I found most useful was chapter 5 on decay of matrix coefficients).

I recommend these two books, the first for the solid analytical foundations it gives you for further study, and the second for a remarkable example with all sorts of applications that gives an honest, if rank $1$, glimpse into the general theory of representations of semisimple groups.

As Paul says in his very good answer, there is no simple answer, but I would think that if you are starting with Serre and Fulton/Harris, then Barry Simon's book may be a reasonable next step, content wise (Simon is generally a pretty good writer, too, though this is obviously subjective).