This ADC code works, but I don't understand why

Generally the output of an ADC is \$\frac{v_{in}}{V_{ref}}N\$, where \$N\$ is the number of counts you get from the ADC. So if it's a 12-bit ADC the count is \$N = 4096\$. If your reference voltage is \$3.3\mathrm{V}\$, then each ADC count represents a voltage increase of \$\frac{3.3\mathrm{V}}{4096} \simeq 806\mu\mathrm{V}\$ This is close to multiplying by 0.8 to get millivolts.

It's best to make this calculation explicit in your code. Modern C compilers will let you do this as a series of #defines, or possibly const float expressions and then optimize to the actual value; modern C++ compilers will let you do the same thing with constexpr float expressions, only with better type checking than C #defines.

Something like the following would do, and would eliminate the magic number 0.8 from your code:

#define ADC_REF 3.3     // volts
#define ADC_COUNT 4096
#define ADC_LSB_MV (1000.0 * ADC_REF / ADC_COUNT)

Giving it a 5V reference voltage is violating the absolute maximum parameters. The data sheet asks you to limit this to VDD - 0.6V. I'm not sure why the -0.6V part, but generally chips have input protection diodes to the highest voltage (\$\mathrm{V_{DDANA}}\$ in this case), so the chip is probably (unhappily) yanking the +5V reference down to around 4V, which would give you a multiplier of about 1 -- and do all sorts of strange and possibly bad things to the chip.

For that matter, giving it a 3.3V reference is violating the maximum absolute parameters, too, only not as bad. Table 37-24 in the datasheet lists the maximum reference voltage as \$\mathrm{V_{DDANA} - 0.6V}\$. So, properly, if you're using a 3.3V analog supply, you should use no higher than a 2.7V reference (2.5V would be convenient, because there's precision 2.5V references).

The value coming from the DAC is 12-bit, i.e. an integer between 0 and 4095 inclusive. The highest value is seen when the input voltage is equal to the full-scale reference voltage.

3.3V ref, 3.3V in yields 4095; 4095 * 0.8 = 3276, which, if interpreted as mV, is pretty close.

5V ref, 5V in yields 4095; 4095 * 0.8 = 3276. You'll want a multiplier of about 5000/4095 = 1.22 to get an accurate report with a 5V reference -- though as pointed out in comments, this part might not actually support an external reference over 3.3V!