Why does work depend on distance?

Often it is important to know if a given formula is a simplification of a more general equation and, when you encounter a conceptual problem, check the general formula. In this case it is a simplification of this formula: $$W=\int_S F\cdot ds $$ Where $S$ is the path over which we are interested in the work and $ds$ is an infinitesimally small segment of $S$.

So back to your question, wherever $F=0$ the integrand is $0$ regardless of how long that segment of the path is. So it is only that first segment where you are applying the 1N that work is done. Once you stop pushing the distance increases but the work does not.

You have to put in the distance on which the force acts. If you release the force, there will be no work done since there is no force acting on the body.

If I'm in a vacuum, and I push a block with a force of 1N, it will move forwards infinitely

and accelerate the block ie change the block's velocity and hence change the kinetic energy of the block.

The longer you apply the force, the n more work the force does resulting in a greater change in the kinetic energy of the block.