apt-get command does not work on Ubuntu-On-Windows / Windows Subsystem for Linux

This might be a problem of /etc/resolve.conf. The latest Windows Update fixes this problem, so you can try updates.

To fix the issue, I ran the following command:

apt-get update && apt-get upgrade

And now it works fine, even if I close and re-open the terminal.

Found out that AVG explicitly has a toggle for allowing/disallowing firewall access to Ubuntu. Turns out you have to go into Firewall > Advanced Settings > Expert Settings > Advanced Settings > Allow any traffic of Windows subsystem for Linux.

Just documenting this in case someone else is looking for this one little checkbox, like I spent hours doing.