Does this VCO produce a sine wave or square wave

This VCO outputs a frequency between 5.5 and 6.1 GHz.

It is sort of "implied" that the output signal will be a sinewave because:

• a "decent" square wave at that frequency would need a signal bandwidth of several hundreds of GHz. That's because a square wave relies on harmonic frequencies (multiples of the base frequency) to become "square". Read up on Fourier analysis to understand this

• this chip is designed to be used in RF transceivers, usually as a local oscillator for mixing up/down RF signals. This works best using sinewaves as RF designers want one frequency to deal with, not one frequency and all its harmonics (like a square wave would have).

Neither. The second and third harmonics are specified in the part of the datasheet page which you cut off. Writing it like this clearly indicates that the harmonics are considered to be spurious to a sine wave. So even if you would actually design a circuit layout which let's say is able to transport up to e.g. 20GHz BW, you would not see a "rectangle", since the second harmonic is already significantly suppressed.