How to have Windows Server DNS use hosts file to resolve specific host names
Solution 1:
The latter part of this post is wrong. I was under the impression, based on some stuff I had read on the web (if it's on the web, it must be true!) that part of the Windows DNS Server Service's tasks for creating its cache was to also load its host file into cache along with its local zone data. I searched around and couldn't find hard evidence of this. I tested the theory on my own Server 2008 R2 machine and found that the hosts file was not used to build the DNS Server's Cache.
However, I believe I have a slightly more elegant solution that Massimo. Instead of creating an authoritative zone for the entire nlscan.com zone, simply create a zone named mailserver.nlscan.com and place a nameless A record in that zone. The nameless A record will have the same name as the zone itself and you can give it the IP address that you want. All other domains underneath nlscan.com as well as nlscan.com itself will resolve by public DNS.
I just tested this out on my own Server 2008 R2 DNS server and was able to make my friend's website (nessus.nl) resolve via public DNS servers but the specific subdomain (blog.nessus.nl) resolve to an Apple.com IP address. Try it and see if it works for you.
Older, wrong post commences:
If my understanding is correct (EDIT: and it is not), when the DNS cache is built in the Server 2003 machine it pulls in entries from the hosts file as well as it's zone data. Placing 172.16.0.10 mailserver.nlscan.com in your Server 2003 machine's hosts file should solve the problem. Restart your DNS services after changing your hosts file.
Use ipconfig /displaydns on any Windows machine (specifically, your Server 2003 DNS machine) to see your host file entries. Also keep in mind that negative responses are cached in your clients so always run ipconfig /flushdns on the clients that you're experimenting with. Otherwise you'll end up abusing yourself against various hard objects as you wonder why your clients can't resolve a name you just entered into a zone / hosts file. =)
Have you tried this and failed?
Solution 2:
The desire to have internal users get internal IPs for resources while external users get external IPs for those same resources is common. It is referred to as split brain DNS. You have one DNS server that faces the internet and another internal DNS server for local users. Internal users use DHCP on your network and an your DHCP server you advertise the internal DNS server. When your users are away from the office, their DHCP server will assign them to a DNS server that will only know about the external zone.
You seem to want split brain DNS without actually hosting the zone internally. You suggest that hosting the zone internally is problematic because you don't want users to get the internal IP when they are working from home but that doesn't make sense because when they are at home they are getting their IP from a different DHCP server that isn't going to advertise your internal DNS server. It is going to advertise their ISP's DNS server which will only know about your external zone and will thus only provide them with external IP addresses.
Finally, I don't think you'll have success with asking a DNS server to serve records from the hosts file on the DNS server. A DNS server serves records from its zone files. The local hosts file on that DNS server propagates entries into the local client resolve cache which is applicable only to lookups on that machine. Those entries aren't served by the DNS server which is a different mechanism.
Read up on split brain DNS - it is the normal way to handle this situation.
Solution 3:
Wes: I'm not sure who dinged you but I'd like to clarify the use of the hosts file: The hosts file is used by the DNS client resolver component, not the DNS server component. An entry in the hosts file on a DNS server will be used by the DNS server when it's acting as a DNS client. For instance, an entry in my W2K8 DNS server's hosts file like this:
1.1.1.1 test.test.com
is loaded into the DNS server's DNS client cache (not it's server cache). If I ping test.test.com from my DNS server it returns 1.1.1.1 as expected. If I then run nslookup on the DNS server and ask it for test.test.com it returns the correct public ip address registered for test.test.com as the DNS client component on the DNS server is now asking the DNS server component for resolution (just the way any other DNS client would). It's a confusing idea to wrap your head around, but the DNS server is also a DNS client and when the DNS client component is called into action it acts as any other DNS client does by looking at it's own DNS client cache, including any entries pre-loaded from the hosts file. Only when the DNS client component uses the DNS server component (by querying the DNS server(s) configured in it's TCP\IP properties, which should be pointed to itself) will the DNS server's cache get populated with the correct information.
Any DNS client querying the DNS server will always get the "real" answer and not the hosts entry because the DNS server's DNS client cache is used by the server itself (as a DNS client) and not by the DNS server component.