What is the difference between jQuery off() and unbind()

Using off is the modern way, unbind is the old way. Use off.

See jquery documentation:

As of jQuery 3.0, .unbind() has been deprecated. It was superseded by the .off() method since jQuery 1.7, so its use was already discouraged.

What's the modern way to do this?

Use a named function expression:

$(window).on('scroll', function handler(){
  ... code ...
  if(code_was_successful){
    $(window).off('scroll', handler);
  }
});

.off() requires a selector to unbind a specific handler

No it does not. Just like .on doesn't require a selector. You only need the selector if you want to unbind a delegated event handler.

As you can read in the documentation of .off about the selector argument:

A selector which should match the one originally passed to .on() when attaching event handlers.

So if you didn't use one in .on, you don't use one in .off.

you can use .on() and .off() like so:

function scrollHandler(e){
    if (myCondition) $(e.target).off('scroll', scrollHandler);
}

$(window).on('scroll', scrollHandler);