Why is signal on XTAL1 and XTAL2 a sine wave (not square)?
This circuit is not a digital circuit. In fact, it is a fairly mathematically complicated non-linear analog circuit with automatic gain control with self-sustainable oscillation mode. It is called a "Pierce oscillator".
The frequency of oscillations is defined by a sharp slope of the electromechanical resonator (crystal), while the gain control is based on dependence of the input on the DC bias voltage - if the DC bias (at C1) is too low to ground or too close to Vcc, the gain is low. The linear gain is highest somewhere in between the ground and power rail.
The (usually internal) bias resistor R1 plays a crucially important role in the oscillator. Typical value of it in CMOS implementations is about 1 MOhm. Together with C1 it forms a low-pass filter, which integrates the output and provides a variable DC offset depending on slight asymmetry of the output signal, even if the output gets to saturation (rail limiting).
As result, there could be a variety of signal shapes with more or less non-linear distortion on Xout and Xin, depending on the inverter's raw gain and parameters of the crystal resonator and loading capacitors. With a very low gain and at the verge of self-oscillations, the signals will be nearly sinusoidal, while at higher gain the output will hit the voltage rail and can be nearly rectangular. The art of making Pierce oscillators is to provide some golden trade-off between rectangular output and sinusoidal one, with good stability of the entire circuit to temperature and voltage variations.
This article deals with a MEMS resonator, not quartz crystal, but the ideas are the same. This is an example of how the circuit starts and drifts to steady state:
The crystal (+C1/C2) is a very narrow bandwidth resonator/filter. Only the fundamental frequency can fit through it.
Sine waves are a single pure frequncy, So it is a sine wave.
Square waves are made square, by all the odd harmonics filling out the hump until the sine becomes square. No Harmonics = Not Square
[Note crystals do in fact have "harmonics" called overtones, but they are slightly off frquency from each other, so the harmonics of the fundamental do not quite hit the 3rd overtone etc]
Another view is that the crystal is like the wheels of a bicycle rolling down the road. The CMOS invertor driving it, is like your feet and legs. Now you could "stab" at the pedals, and try to make the movement a square wave if you wanted. But the pedals are just going to go round and round smoothly regardless, because the flywheel effect is so large. The crystal is like an enormous flywheel smoothly and sinusoidally rolling around.
The crystal really is like a heavy flywheel. If you suddenly disconnect the drive, the signal will take thousands of cycles to die away. When you turn the oscillator on, it takes thousands of cycles to start, slowly building up amplitude. This is why your processor has an "oscillator startup timer"
A crystal will convert electrical energy to mechanical energy and vice versa. It is able to do this efficiently when driven with a sinusoidal waveform of a particular frequency. Driving it with anything else will result in it converting a larger fraction of the applied energy into heat or mechanical degradation.
While it would be possible for a processor to output a square wave to the crystal, this would result in the crystal generating more heat and being subject to more stress than driving it with something closer to a sinusoidal waveform. Further, if the purpose of a pin is to serve as the output of a crystal oscillator, a small transistor which isn't strong enough to force the voltage on the pin to change instantly may be quite cheap compared with a transistor that's powerful enough to forcibly drive a square wave.
Note, incidentally, that in most cases the processor won't be putting much power into the crystal, and the sinusoidal shape isn't dominated by the energy that's flowing from the processor into the crystal, but rather by the energy that's repeatedly flowing from the crystal into the attached caps and back again.