Amateur moon laser ranging
Per BarsMonsters's request I'll expand my comment into an answer.
This same calculation has been done in a post on the blog Built on Facts. His conclusion is that amateur lunar ranging isn't feasible with a the lasers at his disposal. (Specs are given in the quote.)
“Amateur Lunar Ranging? Hmm.”
Typical values for the lasers we use might be somewhere in the 1 milliradian range. The distance to the moon is roughly 400,000 km, so the portion of the lunar surface illuminated by our laser will be about (0.001)*(400,000 km) = 400 kilometers in diameter. This is an area of about 125 billion square meters. If the retroreflector is one square meter, only about 1 part in 10^11 of our emitted light even makes it to the reflector. Now the reflected light has to make it back to the earth. If we’re extremely optimistic, we might say that the reflector introduces no extra angular spread, and its reflected light is spread over a 400 km diameter of the earth’s surface. To a first approximation this just means the total there-and-back efficiency is about 1 part in (10^11)^2, or one photon in 10^22.
...
The energy of one photon of wavelength λ is:
$$\frac{1}{A}\frac{dp}{dt}=\frac{S}{c}$$
The laser we’d likely use has a wavelength of 532 nanometers, and plugging into the equation we find that each photon has an energy of about 3.7 x 10^-19 joules. Therefore we’d need about 1000 joules per pulse to get up to the neighborhood of 10^22 photons per pulse. And we need 10^22 photons just to get one photon back per pulse on average.
We have a few compact 15 mJ/pulse q-switched Nd:YLF lasers with 1 KHz rep rates, and we even have a few not-so-compact 2 J/pulse with a 10 Hz rep rate. With a 2 second round trip time, the repetition rate isn’t so relevant since we can effectively only use one pulse per round-trip time. Even 2 joules won’t cut it unless we’re willing to do many thousands of shots worth of statistics. And that’s without even thinking about noise, which will not be inconsequential even with good filtering.
I can't answer your second question, but I can help with a few references. Linked from the above post, there is an article on The Lunar Ranging Experiment (PDF). The introduction gives some history of the retroreflector measurements, as well as the ranging experiments before the retroreflectors were placed.
In 1962, Smullin and Fiocco (3) at Massachusetts Institute of Technology succeeded in observing laser light pulses reflected from the lunar surface using a laser with millisecond pulse length. Additional measurements of this kind were reported by Grasyuk et al. (4) from the Crimean Astrophysical Observatory, and later Kokurin et al. reported successful results (5) using a Q-switched ruby laser.
...
(3): L. D. Smullin and G. Fiocco, Institute of Electrical and Electronics Engineers Proceedings 50, 1703 (1962).
(4): A. Z. Grasyuk, V. S. Zuev, Yu. L. Kokurin, P. G. Kryukov, V. V. Kurbasov, V. F. Lobanov, V. M Mozhzherin, A. N. Sukhanovskii, N. S. Chernykh, K. K. Chuvaev, Soviet Physics Doklady 9, 192 (1964).
(5): Yu. L. Kokurin, V. V. Kurbasov, V. F. Lobanov, V. M. Mozhzherin, A. N. Sukhanovskii, N. S. Chernykh, Journal of Experimental and Theoretical Physics Letters 3, 139 (1966). (link)
First, you said that the telescope would be > 150 mm so that atmosphere is the limit, not diffraction. But the question was, will a 114 mm scope work? Not only are you diffraction-limited, but actually coupling all of the laser power into the scope so as to get that level of collimation on the output beam is not trivial. It is true, though, that this isn't quite as bad as the answer asserted - it ignored the stipulation that the laser beam would be run through a telescope to reduce the divergence. The answer's calculations assume a 1 mrad divergence, which is reasonable for a raw laser beam, but does not reflect the effects of a beam expander.
More importantly, the return from the retros is not good to 1 arc-second. The divergence is set, not by the size of the array, but by size of the individual retros. In the case of existing arrays, the divergence of the return beam will be at least 8 arc-seconds, so your photon count must be reduced by a factor of 64.
Another, minor, effect is Rayleigh scattering. At 532 nm, a 45 degree elevation will give about a 25% loss in power (for sea level).
Finally, on an operational note, aiming the system is non-trivial, to say the least. You have to aim the beam well away from the array to allow for lead effects, and there is no good way to tell what correction is needed if you miss. This was a major problem during the early years of the LLR program. On the plus side, the larger your beam footprint, the easier this gets.