When and why should I use apt-get update?

apt-get update downloads the list of available packages.

The list of packages can change over time. New packages are added, and old packages are removed. Thus if you have a really old cache, and you try to do an apt-get install, it might try to download a package which no longer exists.
How long an old package is kept in a repository is up to the repo maintainer (your distribution). As such, if you're using something like docker, where the cache might be very out of date, you should always run apt-get update before installing any packages.

The reason for removing and adding packages is mostly bug fixes & security updates. Though if you're using 3rd party repos like PPA, anything goes.

When using something like docker for containerization in a corporate environment, you should build the container once, and then move that container through your various release environments (development, staging, production), and not rebuild the container each time. This will make sure you don't get a different container that hasn't been tested.

To answer your question of where the cache files live, /var/lib/apt/lists.