The reasoning behind doing series expansions and approximating functions in physics

The key reason is that we want to understand the behavior of the system in the neighborhood of the state rather than at the state itself.

Take the equation of motion for a simple pendulum, for example:

$$\ddot{\theta} = -\frac{g}{\ell}\sin(\theta)$$

If we take the limit where $\theta \rightarrow 0$, we find $\ddot{\theta}= 0$, and we would conclude that the pendulum angle increases or decreases linearly with respect to time.

If we however take a Taylor expansion and truncate at the linear term, we find $\ddot{\theta} = -\frac{g}{\ell}\theta$, which is a simple harmonic oscillator! This expansion shows us that in the neighborhood of $0$, the system returns back to $0$ as if it was a simple harmonic oscillator: completely unlike what we could state in the limit approximation above.

In fact, you could consider the limiting behavior around a state to be the zeroth-order component of a local expansion, which holds true straightforwardly for the example above since the limit term contributes no terms to the dynamics of the pendulum (but correctly notes that the angle increases/decreases linearly very close to $0$).

The idea behind any expansion is to express a "complicated" function in terms of simpler ones. In the case of a series expansion, the simpler ones are polynomials. Thus for instance the function $$ \frac{1}{a+x}-\frac{1}{a-x} \tag{1} $$ is a difference of two approximately equal quantities when $x/a\to 0$ and so appears to be $0$ when $x/a\to 0$ but that's not really useful information so it is convenient to reexpress it as $$ \frac{1}{a(1+x/a)}-\frac{1}{a(1-x/a)} \approx -\frac{2 x}{a^2}-\frac{2 x^3}{a^4} $$ which gives some additional information in this limit.

There are also multiple circumstances where some equations - say a differential equation - cannot be solved exactly but can be solved in some limit (often yielding a linearized equation or systems of equations), which still allows some qualitative understanding of the features of the solutions: this is the basis for perturbation theory. For instance, solving the Schrodinger equation for the Lennard-Jones potential $$ V(r)= 4\epsilon\left[\left(\frac{\sigma}{r}\right)^{12} -\left(\frac{\sigma}{r}\right)^6\right] $$ cannot be done analytically, but near the minimum $r_0=2^{1/6}\sigma$ one can expand to obtain $$ V(r)\approx -\epsilon+ \frac{18\ 2^{2/3} \epsilon (r-r_0)^2}{\sigma ^2} $$ which, up to an unimportant constant and shift in $r$, is a harmonic oscillator potential for which the solutions are known. Thus, we can get some approximate insight (or at least some orders of magnitude) into the appropriate molecular transitions.

Of course great care must be taken to insure that the assumptions behind the expansion are not ultimately violated by the solutions, i.e. one must understand that the resulting solutions are approximate and may fail badly in some cases.

Consider the function $f(x)$ defined by $$ f(x)\equiv \int^\infty_{-\infty} ds\ \big(\exp(-s^2-xs^4) - \exp(-s^2)\big). \tag{1} $$ When $x=0$, we get $f(0)=0$. What if we want to know the value of $f(x)$ when $x$ is a very small positive number? We don't know how to evaluate this integral exactly and explicitly, and just saying that the result will be "close to zero" is not very enlightening.

We could evaluate the integral numerically, but that requires a computer (or a very patient person with a lot of time), and if we do the calculation that way, then we have to re-do it for each new value of $x$ that we care about.

An alternative is to expand in powers of $x$: $$ f(x) = \int^\infty_{-\infty} ds\ (-xs^4)\exp(-s^2) + \int^\infty_{-\infty} ds\ \frac{(-xs^4)^2}{2!}\exp(-s^2) + \int^\infty_{-\infty} ds\ \frac{(-xs^4)^3}{3!}\exp(-s^2) +\cdots \tag{2} $$ Each term in this expansion is an elementary integral, which can be evaluated explicitly, so we end up a series in powers of $x$ with explicit numeric coefficients. The expansion doesn't converge (it's an asymptotic expansion), but if $x$ is small enough, then the first few terms give a good approximation, and we don't have to re-compute the coefficients every time we want to try a new value of $x$.

Examples like this are everywhere in physics. This particular example is the single-variable version of an integral that shows up in the simplest type of non-trivial quantum field theory (the "$\phi^4$ model").