Why does adding both private and public DNS servers in Windows cause unexpected behavior?

1) It doesn't work that way.

I learned a hard lesson that Windows does not use this DNS server list in order. You cannot assume it will simply walk down the list when it can't resolve a name.

There's a quite lengthy explanation on TechNet, which you can find here. The problem I personally experienced was caused by this little snippet here:

The DNS Client service keeps track of which servers answer name queries more quickly, and it moves servers up or down on the list based on how quickly they reply to name queries.

We had five DNS servers listed in our DHCP settings, the last two were the local ISP's. Our other three were very old machines, and we had figured it would be best to add the local ISP's servers as backup. Guess whose were faster?

It's not fun when servers stop resolving internal names.

_{This prompted me to threaten a switch to Linux.}

2) So why did it work in the command prompt?

I'm assuming you used nslookup, which will always use the first server in your DNS list unless you specify otherwise. Since you were always checking your private DNS server, it always worked. Once you open Internet Explorer and start actually trying to resolve names, the behavior is different (as specified by the lengthy flowchart above.)

_{Isn't that wonderfully confusing? I chased name resolution problems for a week wondering why it always worked using nslookup yet for some reason Lotus Notes could never find its mail server. I didn't figure it out until using Wireshark to check how it was really resolving names.}

3) How can I fix it?

You can fix your specific problem by configuring your DNS server with a forwarder. When your private DNS server can't find a name, it will forward the request to the server you specify. This way, you can simply use your DNS server and nothing else, preventing the odd behavior that's showing up.

Rather than setting up two DNS servers, what you would want to do is use just your private DNS and then have that DNS server check results it does not know with the external DNS. The exact way of configuring that is going to depend on your private DNS server, but that should give more uniform results as your alternate DNS server may be used when it thinks the primary is busy.