What is the smartest way to handle robots.txt in Express?

Use a middleware function. This way the robots.txt will be handled before any session, cookieParser, etc:

app.use('/robots.txt', function (req, res, next) {
    res.type('text/plain')
    res.send("User-agent: *\nDisallow: /");
});

With express 4 app.get now gets handled in the order it appears so you can just use that:

app.get('/robots.txt', function (req, res) {
    res.type('text/plain');
    res.send("User-agent: *\nDisallow: /");
});

1. Create robots.txt with following content :

User-agent: *
Disallow: # your rules here

2. Add it to public/ directory.

3. If not already present in your code, add:

app.use(express.static('public'))

Your robots.txt will be available to any crawler at http://yoursite.com/robots.txt