Can the Host Header be used to hide the existence of a service?

The only way to determine the other name based virtual host on the same IP address is to perform forward lookups to all domains and see the A records with the same IP addess. Only after that it's possible to see whether the server is responding to all these hostnames through the Host header or not, and is the reponse same for every hostname or are there different services.

Knowing that, it's indeed possible to hide a service like this, but it would still be security through obscurity. While someone might be using this, it has some disadvantages:

• The secret hostname may leak by accident, which makes "the Host: header" a rather bad place for a password. This may be caused by:
  • A request through a 3rd party DNS service. It doesn't make this less bad at all that DNS is typically unencrypted and that it may be cached on multiple recursive servers.
  • If the hostname is on a public authoritative DNS server, it may be vulnerable to insufficient zone transfer restrictions or DNSSEC zone walking (possible with older NSEC records, fixed by NSEC3, RFC 5155 & RFC 6781, 5).
  • Server Name Indication (SNI, RFC 6066, 3) reveals the hostname, as it's part of unencrypted ClientHello on all TLS versions prior 1.3 (how encrypted SNI works).
• Depending on how the hostname leaked you might still need to know which IP address to connect to, but that's merely a slow down.
  • The IP address would be hidden if the A record was only available on a DNS server on a private network, but that would make it hard to use the service outside the network: in that case it would be better to have the service on the private network, too.
  • If the IP address was on the local hosts file, it's still available on the IP packet header for anyone who could see your traffic on the network layer (L3). That would include anyone who already got the hostname by monitoring your DNS requests or SNI.
• Difficulties with TLS implementation.
  • You won't get a certificate for an arbitrary fake address.
  • If it was a subdomain of your real domain...
    • A certificate issued for the subdomain would reveal the hostname through Certificate Transparency (e.g. https://crt.sh/?q=example.com)
    • e.g. Let's Encrypt requires it to be publicly available for the ACME challenge.
  • Suitable ways to circumvent this would be
    • an own internal certificate authority
    • a wildcard certificate *.example.com covering the "secret" subdomain.

Because it creates more problems than it solves, I wouldn't recommend it. Password protection with multifactor authentication and possible client certificates are better options, and hiding the location of the service doesn't really add much security over that.

It can work - for a while...

Having a Host header for a domain that is not directly resolvable means anyone not knowing the correct header and IP will not be able to open the service. This will work until both the secret domain and IP leaks, and that can take a long time.

Besides external DNS and certificate transparency that Esa Jokinen already pointed out, there's the Referer header:

If you have any external links or resources on the hidden site (images, Javascript), the client browser will set the Referer with your hidden domain, so any site you link will know the existence of it.

But it's only part of the problem. Knowing the domain means nothing if the IP address of the server is not known, and that is more difficult to accidentally leak.

Client-side leaks will leak the secret domain, but the IP address of the server must be leaked from the server side (or by someone publishing both the IP and secret domain somewhere). And the methods for uncovering this are more difficult to pull of.

In the case of Tor secret services, you don't need to know the server IP to be able to attack it: Tor connects you to the server without the need of IP. But on your hidden service, one would need to connect to the server first to later be able to attack it somehow and get its IP. But you cannot connect to it without the IP, and you need the IP to get the IP.

How to find your server? Bruteforce... Connect to every public IP alive, send the Host header with the secret domain and see what gets back. Can take a while but will find it.

I would not rely only on this for security, as it's security thru obscurity, but in this case is very obscure.

It's not exactly the situation you describe, but domain fronting is an example of abusing the host header to hide the true nature of your communication - typically to evade firewalls.

The basic setup for domain fronting is:

1. You make a DNS request for innocuous.example.com
2. You open an SSL connection to the IP address returned, specifying innocuous.example.com as the address for SNI
3. Over the encrypted connection, you make a HTTP request, for the host suspicious.example.com

As far as the firewall is aware, this is a connection to innocuous.example.com, which is allowed. But if the server is configured to serve traffic for suspicious.example.com, as well as innocuous.example.com (which might be the case if they're both served by the same CDN, for example), it will serve a response for suspicious.example.com.

Note that in most real-world use cases, the existence of suspicious.example.com is unlikely to be secret. Domain fronting is generally used to access sites that are legitimate but blocked by a firewall.

If you were designing a system to hide a suspicious web site in plain sight, it would be much simpler to host it at https://innocuous.example.com/cx6wdffpuik997eljf6d878i6f3np4207ne30vyjsvhpra69, since it avoids the need to worry about DNS or SNI leaks.