Apache - listen only on specific domain, not IP

You cannot have it refuse connections, since the hostname (or IP) that the user is trying to use as their HTTP host is not known to the server until the client actually sends an HTTP request. The TCP listener is always bound to the IP address.

Would an HTTP error response be acceptable instead?

<VirtualHost *:80>
    ServerName catchall
    <Location />
        Order allow,deny
        Deny from all
    </Location>
</VirtualHost>

<VirtualHost *:80>
    ServerName example.com
    DocumentRoot /var/www/
    <Directory /var/www/>
        AllowOverride All
        Order allow,deny
        allow from all
    </Directory>
</VirtualHost>