Why is my understanding of unbalanced forces wrong?

The gravitational force would not be getting bigger, but the upward normal force would be getting smaller. You can try this out by putting an object on a scale, and then pull up on it a bit. The object won't be lifted, but the reading on the scale will go down.

Your equations assume the reaction force of the floor with the box is constant. However, what should happen is $R=Mg-F$ where R is the reaction force, Mg is the weight of the box and F is the force you exert, I.e. R decreases as you exert an upwards force but the box still remains in contact with the floor. In order to lift the box of the floor, your force F has to excceed Mg such that $R=0$.