Mathematical rigor for engineers

I think that what you're really looking for is Representation Theory. Now the problem is that it can be quite advanced for an engineer trying to approach this subject from a mathematical point of view. But maybe it can be done from a more "physical" point of view.

I would suggest to you "Lectures on Linear Algebra" from Gelfand that if I rember correctly also adresses Fourier Series, and then the second volume of "Principles of Advanced Mathematical Physics Volume 2" of Richtmeyer which has in the first 10 chapter the basis of representation theory in a very comprehensible way. I'm not sure on this last one since it might be too advanced, but I would give it a try if I were you.

For the Fourier series<->measure theory connection, you can perform some operations using infinite series as a black box but in order to describe a Fourier series corresponding to a discontinuous function you will (eventually) begin to move into measure theory. Please remind me to respond after this semester is over, as I am enrolling in a course on this exact question which starts next week (my undergrad didn't emphasize the "applied" side outside of the Physics department). Group theory is connected to eigenvalues because the square matrices form a group and eigenvalues allow you to detect properties of an individual matrix (such as whether it can be decomposed in any of several ways, what the "long-term behavior" of a Markov chain will be, whether the matrix will be stable or unstable under exponentiation even if it lacks the Markov property, etc.) I would be happy to answer further questions on this subject next week.

See Roman's 2-volume book [1] below. Although it's possibly the best reference I know for what you're asking (which has been asked fairly often over the years in various online math forums), his 2-volume book is almost never mentioned. The books [2] and [3] are rather well known and possibly others will mention them, the book [4] is fairly advanced but sufficiently reader-friendly to be worth looking at from time-to-time, and [5] is a bit less known (and a bit idiosyncratic).

[1] Paul Roman, Some Modern Mathematics for Physicists and Other Outsiders, Volume 1 (1975): An Introduction to Algebra, Topology, and Functional Analysis (Volume 1 contents) AND Volume 2 (1975): Functional Analysis with Applications (Volume 2 contents -- click on amazon.com's "look inside" for Volume 1; Volume 2 contents are on pp. x-xi)

Review in Physics Today Volume 30 #5 (May 1977), pp. 72 & 74; review by Andrew Lenard (1927-2020)

Review in Computers and Mathematics with Applications Volume 3 #1 (1977), pp. 83-84; review by Wilhelm Ornstein (1905-2002)

[2] George F. Simmons, Introduction to Topology and Modern Analysis (1963)

[3] Thomas A. Garrity and Lori Pedersen, All the Mathematics You Missed: But Need to Know for Graduate School (2001)

[4] Charalambos D. Aliprantis and Kim C. Border, Infinite Dimensional Analysis. A Hitchhiker's Guide (2006)

[5] Robert Hermann, Lectures in Mathematical Physics Volume 1 (1970) and Volume 2 (1970)