How does Node.js choose random ports?

The OS assigns the port number. See https://github.com/joyent/node/blob/v0.6.11/lib/net.js#L780-783

On OS X, the assignment is sequential, userland and does not check the port to verify it is not in use.

On Ubuntu 11.04, the assignment is random, userland and also does not check if port is in use.

The script below can be used to test on other platforms. To verify the ports are userland, I ran the script 10,000 times via bash piped to grep -c "port: [0-9]{1,3}" with zero matches.

var net = require('net'),
    firstPort;

(function createServer(port) {
  var server = net.createServer();
  server.listen(port, function() {
    address = server.address();
    if (port === 0) { 
      if (firstPort === undefined) {
        firstPort = address.port;
        // cause a EADDRINUSE in 10 more sockets for sequential platforms
        // without this, will run out of fd's before hitting EADDRINUSE
        createServer(firstPort + 10); 
        console.log('addr in use port trap: ', firstPort + 10);
      } else {
        // on OS X (sequential) this will increment the OS's sequential counter
        // and not cause EADDRINUSE
        createServer(address.port + 1);
      }
      createServer(0);
    }
    console.log("requested port:", port, " binded port:",address.port);
  });  
})(0);

Does Node.js select a port that isn't in use?

Yes, the port will be an available port. The operating system selects a port that isn't in use rather than Node.js, but from the end-user perspective, that's more-or-less the same thing.

The documentation no doubt had different wording at the time this question was originally posted in 2012, but as of now (January 2019), it is explicit about this: "If port is omitted or is 0, the operating system will assign an arbitrary unused port...".

Do I have to check that myself and retry if Node.js happens to pick a port that is already open and bound to another application?

No, you do not. You should handle errors anyway as any number of things can go wrong. But writing extra code to test for port availability is not something you need to do.

Does it pick any old port, or only userland ports (>1024)?

As far as I know, it will always be an unprivileged port.

---

For operating systems like macOS that assign available ports sequentially, here is code that shows that the operating system will skip a port if it is unavailable.

// Testing for macOS, which supplies available ports sequentially.

var net = require('net');

createServer(0, function () {
  var port = this.address().port;
  console.log('server was assigned port ' + port);
  createServer(port+1, function () {
    var port = this.address().port;
    console.log('server was assigned port ' + port);
    createServer(0, function () {
      var port = this.address().port;
      // This line will show that the OS skipped the occupied port and assigned the next available port.
      console.log('server was assigned port ' + port);
    });
  });
});

function createServer(port, callback) {
  console.log('create server with port ' + port);
  var server = net.createServer();
  server.listen(port, callback).unref();
}