possible to make samba4's internal dns server listen on non-standard port?

The reason you're not finding the option is that OS resolver libraries cannot use ports other than 53. While it's possible to have a nameserver forwarder use ports other than 53, it is unlikely that the authors of this software support and/or anticipate your particular use case. You would need ask them why this feature is absent.

You could try something like

• add a secondary IP address to your system.
  • configure DNSmasq to listen on that address only.
• Configure Samba to listen on the primary IP address
• Configure samba to forward requests it can't handle to the IP DNSmasq is listening on.

I guess you could do it the other way around too.