Why does this comparator not output a square wave?

It will be one of two things & more than likely both:

1. The probe you are using is not suitable, be it in frequency or in its compensation (the little screw on the side of the probe).

   

2. A 10 MHz scope is too slow for a 4.5 MHz signal

Here is the buildup of a squarewave up to the 100th harmonic (4.43 MHz fund):

import numpy as np
from matplotlib import pylab
F= 4.43e6
t = np.arange(0, 2/F, 1e-12)
x = np.sin(2*np.pi*F*t) 
pylab.subplot(3,1,1)
pylab.title('Sinewave of increasing frequency: Fourier content of a squarewave')
pylab.plot(t,x)
pylab.grid(True)

for i in range(3,100,2):
    a = (1/i)*np.sin(2*np.pi*F*i*t)
    pylab.plot(t,a)
    x +=a

pylab.subplot(3,1,2)
pylab.title('Equivelent squarewave for summation of its harmonics')
pylab.plot(t,x)
pylab.grid(True)

y= np.zeros(len(t))

A= 10e6*2*np.pi*t[1]/(10e6*2*np.pi*t[1]+1)
for i in range(1,len(t)):
    y[i] = y[i-1] + A*(x[i] - y[i-1])
pylab.subplot(3,1,3)
pylab.plot(t,y,label='4.43MHz through 1 filter')
x = y
y= np.zeros(len(t))
A= 10e6*2*np.pi*t[1]/(10e6*2*np.pi*t[1]+1)
for i in range(1,len(t)):
    y[i] = y[i-1] + A*(x[i] - y[i-1])
pylab.plot(t,y)
pylab.plot(t,y,label='4.43MHz through 2 cascaded filters')

pylab.title('Result of passing a 4.43MHz squarewave through 1 & two 10MHz 1st order filters')
pylab.legend()

pylab.grid(True)
pylab.show()

If the acquisition is only capable of 10 MHz, the contributors will be attenuated and phase shifted producing a distorted waveform similar to the one you are seeing.

Cascading two 10MHz "filters" (one in the probe, one on the input of the scope) will further distort the waveform resulting in a signal closer to that seen on the scope.

The mean of a 0-5V squarewave is 2.5V. if your scope as an "average input" it will also produce a similar waveform and tend towards 2.5V. I have been caught out a number of times looking at PWM only to see a very strange walking waveform ONLY to find someone messed with my scope and had enable "16sample averaging"

You should realize that a 4.43 MHz square wave has a much larger bandwidth than 10 MHz.

A "proper" 4.43 MHz square wave will contain frequencies up to and beyond 50 MHz. That is because a square wave is made up of a whole sum of frequencies (as opposed to a sinewave which is only one frequency, this is why EEs use it a lot).

If you had an ideal 4.43 MHz square wave but looked at it through a 10 MHz bandwidth system (like your scope) then you'd see a distorted triangle wave. Which is what you see here.

Try again but at a 10x lower frequency (or even 100 x lower) and see what you get.

A 10 MHz scope rise time should be 0.35 * 1000 / 10 = 35 nS.

Half cycle time at 4.43 MHz is 500 / 4.43 = 113 nS which is over 3 times the scope rise time showing the scope should be adequate for displaying the full excursion of the output signal. However the scope trace provided looks CR / rise-time limited in excess of this. Therefore the first thing to look at is output loading and as the LM393 data sheet shows a parameter for Output Sink Current I would suggest in the first instance you try a 4.7k pull up resistor between +5 volt and SQ_OUT. When working correctly outputting a clean square wave I would expect the scope output waveform to be similar to the bottom one simulated by JonRB - due to the scope bandwidth limit - although the voltage scales will differ. Whilst scope probe tuning is important for digital work - I believe it to be a red herring in this instance.

UPDATE

@Batperson in your comment following ovirt's answer you stated that you had substituted a LM393 which has an open collector output, hence the pullup suggestion. However this is a trivial circuit and shouldn't be difficult to nail. First a word of advice. When there are problems and you find yourself answering 'should' rather than 'does' - you need to check as there is an element of doubt. There is often a big difference between should and what is actually happening. e.g. this circuit SHOULD be producing a square-wave output.

What you describe does not make sense. You have a 0.5 Vp-p input signal that biased at +2.5V to ground connected to the comparator input and you are shifting the comparator ref between gnd and +5V. Once the reference voltage exceeds the oscillator bias plus about 0.25V the output should flatline near gnd. Conversely once the ref drops below the bias minus about 0.25V it should flatline near +5V. e.g. the output should flatline whenever the ref is outside of the input signal range. After you investigate this hang an 0.1uF ceramic C between ref and ground close to the IC pins and try again. Next replace the oscillator input with two 10k R's in series and connect between gnd and +5V the comparator input connected to the mid point. Look for the output changing between flatline +5V and gnd as the ref passes through the mid point. You have just proved/disproved the comparator functioning at DC.

THOUGHTS FURTHER

@Batperson having though about some more I realise your scope traces do not make sense. The only way (other than -ve feedback) the circuit shown can have an output bias near mid point is for the output to be spending equal time at +5V and gnd (The resulting level being the average). This is not evident in your scope pictures 1 & 2 - it looks more what the input should be - almost as if the ground IC gnd was not connected. The tests I suggested yesterday should help to resolve this. It would be helpful if you titled pictures 2 & 3 with voltage reference points and scale or frequency as it is not clear from your text. Also maybe a picture of your breadboard.