What is the Difference Between `apt-get purge` and `apt-get remove`?

As the man apt-get page says:

remove - remove is identical to install except that packages are removed instead of installed. Note that removing a package leaves its configuration files on the system. If a plus sign is appended to the package name (with no intervening space), the identified package will be installed instead of removed.

purge - purge is identical to remove except that packages are removed and purged (any configuration files are deleted too).

This of course, does not apply to packages that hold configuration files inside the user's home folder (e.g. /home/SexyNoJutsuUser), these files will not be touched ( Why does "Purge" not remove everything related to an app? )

So for example, if you were to purge Chrome, Firefox, Kodi or any other that holds some configuration files inside your /home folder, these configuration files will stay there.

On the other hand if you were to install Apache, squid, MySQL or any other services similar that save their configuration files in /etc, these configuration files will be deleted if you use purge.

An excerpt from my answer on another question:

• apt-get remove packagename
  will remove the binaries, but not the configuration or data files of the package packagename.
• apt-get purge packagename, or
  apt-get remove --purge packagename
  will remove about everything regarding the package packagename, [...]
  Particularly useful when you want to 'start all over' with an application because you messed up the configuration.

apt-get purge removes configuration files, while apt-get remove does not.