What kind of book would show where the inspiration for the Laplace transform came from?

I think it is easier to start with the theory of Fourier series. The corresponding integral can be motivated in various ways but roughly the idea, once you conceive of a periodic function as being a sum of sine waves, is to find a way to filter out all of the frequencies except one, and integrating against a particular wave turns out to accomplish this (the idea being that integrating waves of two different frequencies against each other causes "destructive interference").

The Fourier transform can then be thought of as the "limit," in a suitable sense, of the theory of Fourier series. If you want to go this route, I recommend Stein and Shakarchi's Fourier Analysis. I haven't read it in awhile, but from what I recall it was very well-written and well-motivated.

(The Laplace transform can also be motivated using probability, the idea being that it can be thought of as the moment generating function of an appropriate random variable. Apparently it is in fact named after Laplace's work in probability. The moment generating function is in turn a natural thing to study because it converts sums of independent random variables to products of functions.)

As far as I know an early reference for a thorough mathematical theory (in terms of today's mathematical language) of the Laplace transform and its inversion are the books by Gustav Doetsch. (There are several: A three volume handbook, some books on applications, a practical guide...). Probably one of this books also provide insight about the history (i.e. early references, Heaviside formal treatment,...) but I don't know which of them is available in English.

Moreover, there are two articles called "The development of the Laplace transform" by Deakin here and here that could be helpful.

You probably already found the Monthly article "What is the Laplace transform?"?