Application of representation theory

Your question seems to ask more for a reference. There are many good papers for this that I read through while looking at this for my undergraduate thesis. Here are a few good references to get you started at why the two are connected:

Representation Theory, Symmetry, and Quantum Mechanics

Quantum Mechanics and Representation Theory

and this which is not as related but easier to read.

Scientists tend to be more concerned with inspecting maps from a group to vector space automorphisms rather than properties of groups themselves.

Here are some concrete examples for you:

For simple-enough molecules, representations of dihedral groups are a good place to start. They can describe discrete mirror/rotational transformations that leave regular polygons invariant.

Example. An example for a representation of the group $$D_4:=\langle x,y; x^4=1=y^2, y^{-1}xy=x^{-1}\rangle$$ is the group homomorphism $\rho:D_4\rightarrow GL(2,\mathbb{C})$. The group itself is the group of symmetries of the square in the Cartesian plane (keeping the center fixed). To find a representation $\rho$, you can write some generic matrices with complex entries and brute force it or use some simple geometric intuition to recall what types of matrices give you mirror/rotation transformations, then apply the right ones.

I’m no expert on chemistry, but both the topology and symmetries of molecules are very important to their functions and properties.

As far as quantum theory, the basic example is actually not a normal (faithful) representation, but a projective one onto ray space (see projective Hilbert space). The intuition is quantum states are only required to be the same up to a global $U(1,\mathbb{C})$ phase factor. The group associativity induces a 2-cocycle condition which imposes a restriction on the particular form of phase difference the two states may have. The canonical reference for this is Weinberg’s Quantum Theory of Fields, Volume 1, $\S 2.7$. So yes it applies to quantum mechanics and the Schrodinger equation, but representation theory tends to appear much more in other facets of quantum theory such as gauge theory and quantum field theory.

A lot of credit for incorporating representation theory into quantum mechanics and gauge theory was Hermann Weyl and his book on Groups and Quantum Mechanics is very worth checking out.

I Googled representation theory in quantum mechanics and found:

• Quantum Field Theory and Representation Theory

To quote the source: In 1928 Weyl published a book called "Theory of Groups and Quantum Mechanics", which had alternate chapters of group theory and quantum mechanics.

I think that you will find the rest of the the source most enlightening.