Unable to run DNS queries when response is bigger than 512 Bytes and truncated

"Can anyone please provide some insight about why this works like this and help with some solution, if any?"

SHORT ANSWER:

A default Azure VM is created with broken DNS: systemd-resolved needs further configuration. sudo systemctl status systemd-resolved will quickly confirm this. /etc/resolv.conf points to 127.0.0.53- a local unconfigured stub resolver.

The local stub resolver systemd-resolved was unconfigured. It had no forwarder set so after hitting 127.0.0.53 it had nobody else to ask. Ugh. Jump to the end to see how to configure it for Ubuntu 18.04.

If you care about how that conclusion was reached, then please read the Long Answer.

LONG ANSWER:

Why DNS Responses Truncated over 512 Bytes:

TCP [RFC793] is always used for full zone transfers (using AXFR) and is often used for messages whose sizes exceed the DNS protocol's original 512-byte limit.

Source: https://tools.ietf.org/html/rfc7766

ANALYSIS:

This was trickier than I thought. So I spun-up an Ubuntu 18.04 VM in Azure so I could test from the vantage point of the OP:

My starting point was to validate nothing was choking-off the DNS queries:

sudo iptables -nvx -L
sudo apparmor_status

All chains in the iptables had their default policy set to ACCEPT and although Apparmor was set to "enforcing", it wasn't on anything involved with DNS. So no connectivity or permissions issues observed on the host at this point.

Next I needed to establish how the DNS queries were winding through the gears.

cat /etc/resolv.conf 

# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "systemd-resolve --status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 127.0.0.53
options edns0
search ns3yb2bs2fketavxxx3qaprsna.zx.internal.cloudapp.net

So according to resolv.conf, the system expects a local stub resolver called systemd-resolved. Checking the status of systemd-resolved per the hint given in the text above we see it's erroring:

sudo systemctl status systemd-resolved

● systemd-resolved.service - Network Name Resolution
   Loaded: loaded (/lib/systemd/system/systemd-resolved.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-10-08 12:41:38 UTC; 1h 5min ago
     Docs: man:systemd-resolved.service(8)
           https://www.freedesktop.org/wiki/Software/systemd/resolved
           https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
           https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
 Main PID: 871 (systemd-resolve)
   Status: "Processing requests..."
    Tasks: 1 (limit: 441)
   CGroup: /system.slice/systemd-resolved.service
           └─871 /lib/systemd/systemd-resolved

Oct 08 12:42:14 test systemd-resolved[871]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
<Snipped repeated error entries>

/etc/nsswitch.conf set the order sources of sources used to resolved DNS queries. What does this tell us?:

hosts:          files dns

Well, the DNS queries will never hit the local systemd-resolved stub resolver as it's not specified in /etc/nsswitch.conf.

Are the forwarders even set for the systemd-resolved stub resolver?!?!? Let's review that configuration in /etc/systemd/resolved.conf

[Resolve]
#DNS=
#FallbackDNS=
#Domains=
#LLMNR=no
#MulticastDNS=no
#DNSSEC=no
#Cache=yes
#DNSStubListener=yes

Nope: systemd-resolved has no forwarder set to ask if a local ip:name mapping is not found.

The net result of all this is:

• /etc/nsswitch.conf sends DNS queries to DNS if no local IP:name mapping found in /etc/hosts

• The DNS server to be queried is 127.0.0.53 and we just saw this is not configured from reviewing its' config file /etc/systemd/resolved.conf. With no forwarder specified in here, there's no way we'll successfully resolve anything.

TESTING:

I tried to override the stub resolver 127.0.0.53 by directly specifying 168.63.129.16. This failed:

dig aerserv-bc-us-east.bidswitch.net 168.63.129.16

; <<>> DiG 9.11.3-1ubuntu1.9-Ubuntu <<>> aerserv-bc-us-east.bidswitch.net 168.63.129.16
;; global options: +cmd
;; connection timed out; no servers could be reached
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 24224
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;168.63.129.16.         IN  A

;; Query time: 13 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Tue Oct 08 13:26:07 UTC 2019
;; MSG SIZE  rcvd: 42

Nope: seeing ;; SERVER: 127.0.0.53#53(127.0.0.53) in the output tells us that we've not overridden it and the local, unconfigured stub resolver is still being used.

However using either of the following commands overrode the default 127.0.0.53 stub resolver and therefore succeeded in returning NOERROR results:

sudo dig aerserv-bc-us-east.bidswitch.net @168.63.129.16

or

dig +trace aerserv-bc-us-east.bidswitch.net @168.63.129.16 

So any queries that relied on using the systemd-resolved stub resolver were doomed until it was configured.

SOLUTION:

My initial- incorrect- belief was that TCP/53 was being blocked: the whole "Truncated 512" was a bit of a red-herring. The stub resolver was not configured. I made the assumption- I know, I know, "NEVER ASSUME ;-) - that DNS was otherwise configured.

How to configure systemd-resolved:

Ubuntu 18.04

Edit the hosts directive in /etc/nsswitch.conf as below by prepending resolve to set systemd-resolved as the first source of DNS resolution:

hosts:          resolve files dns

Edit the DNS directive (at a minimum) in/etc/systemd/resolved.confto specify your desired forwarder, which in this example would be:

[Resolve]
DNS=168.63.129.16

Restart systemd-resolved:

sudo systemctl restart systemd-resolved

RHEL 8:

Red Hat almost does everything for you in respect to setting up systemd-resolved as a stub resolver, except they didn't tell the system to use it!

Edit the hosts directive in /etc/nsswitch.conf as below by prepending resolve to set systemd-resolved as the first source of DNS resolution:

hosts:          resolve files dns

Then restart systemd-resolved:

sudo systemctl restart systemd-resolved

Source: https://www.linkedin.com/pulse/config-rhel8-local-dns-caching-terrence-houlahan/

CONCLUSION:

Once systemd-resolved was configured my test VM's DNS behaved in the expected way. I think that about does it....