Why do eigenvalues correspond to observable quantities?

Suppose we don't know quantum mechanics yet and we want to calculate the expecatation value of an observable $A$. Could be momentum, spin whatever. It is given by $$\mathbb E(A)=\sum_i a_i\,p(a_i)$$ Where $a_i$ are the possible outcomes and $p(a_i)$ are the probabilities of those outcomes. When the outcome is continuous this becomes an integral.

To each of these states we can associate a vector $|a_i\rangle$ and it is possible to make these states orthonormal such that $\langle a_i|a_j\rangle=\delta_{ij}$. Quantum mechanics is linear so if we have two solutions $|a_1\rangle,|a_2\rangle$ then the state $|\psi\rangle=\alpha|a_1\rangle+\beta|a_2\rangle$ is also a valid solution. How do we interpret this new state? It is a postulate (Born rule) that the probability of finding $a_1$ is given by $p(a_1)=|\alpha|^2$. This means we have to normalize $|\psi\rangle$ such that $|\alpha|^2+|\beta|^2=1$ in order for it to be a valid state.

If we then define Dirac notation as usual we get $\alpha=\langle a_1|\psi\rangle$ and $\alpha^*=\langle \psi|a_1\rangle$ which you can check using orthonormality. After some manipulation we can get the expectation value in the following form \begin{align} \mathbb E(A)&=|\alpha|^2a_1+|\beta|^2a_2\\ &=\alpha^*\alpha\ a_1+\beta^*\beta\ a_2\\ &=\langle \psi|a_1\rangle \langle a_1|\psi\rangle a_1+\langle \psi|a_2\rangle \langle a_2|\psi\rangle a_2\\ &=\langle \psi|\left(\sum_i |a_i\rangle\langle a_i|a_i\right)|\psi\rangle \end{align} If we then define $\hat A=\sum_i |a_i\rangle\langle a_i|a_i$ then we get $\mathbb E(A)=\langle \psi|\hat A|\psi\rangle$.

So what's the link with eigenvectors/eigenvalues? It turns out that according to the spectral theorem that any Hermitian matrix can be written as $\hat A=\sum_i |\lambda_i\rangle\langle \lambda_i|\lambda_i$ where $\lambda_i$ are its eigenvalues and $|\lambda_i\rangle$ its eigenvectors. Notably these eigenvectors form an orthonormal basis. This implies that only the eigenvectors of $\hat A$ can give the outcome of a measurement. This is because $|a_i\rangle\langle a_i|$ is a projection along $|a_i\rangle$. Any vectors that are orthogonal to $|a_i\rangle$ will be projected out. If a state is orthogonal to all eigenvectors of $\hat A$, which means it can't be written as a sum of eigenvectors, then it will automatically give zero contribution in the expectation value because it is projected out.

As a final note I would like to add that my reasoning has been a bit backwards from how you would usually do it but I hope this made it more clear why this eigenvalue/eigenvector construction actually makes a lot of sense.

I think the conceptual issue is that you've got it backwards. Given any set of eigenfunctions, you can have operators with those eigenfunctions, whose corresponding eigenvalues are anything you want. For example, if you're in a three-dimensional Hilbert space and the eigenvectors are the standard unit vectors, then $$A = \begin{pmatrix} 1 & 0 & 0 \\ 0 & 2 & 0 \\ 0 & 0 & 3 \end{pmatrix}$$ is an operator that assigns the eigenvalues $1$, $2$, and $3$ to those eigenvectors. But $$B = \begin{pmatrix} 83 & 0 & 0 \\ 0 & - \pi^e & 0 \\ 0 & 0 & 10^{100} \end{pmatrix}$$ is an operator with the same eigenvectors but totally different eigenvalues.

Suppose you have a measuring apparatus that yields the values $x$, $y$, and $z$ on those three eigenvectors. Then we define the operator that corresponds to it to be $$\mathcal{O} = \begin{pmatrix} x & 0 & 0 \\ 0 & y & 0 \\ 0 & 0 & z \end{pmatrix}.$$ That is, the eigenvalues don't magically match the measured values. Instead, the measured values tell you what the eigenvalues have to be, and that defines what the operator is.

If a system is described by the eigenfunction $\psi$ of an operator $A$ then the value measured for the observable property corresponding to $A$ will always be the eigenvalue $a$ which can be calculated from the eigenvalue equation

$\hat {A} | \Psi \rangle = a | \Psi \rangle \tag 1$

This is a mathematical description relating the eigenvalue and eigenfunction of quantum systems. Why this works in reality is probably a philosophical question, but it is true. The fact that this is true is just how nature works.

So, even though equation (1) is a mathematical formulation, it certainly works in reality.

In fact, quantum theory and its mathematical framework, is spectacularly successful at describing nature.