Why does pgffor estimate that 0.1-0=0.100005 in a \foreach loop?

0.1 in binary has infinite representation. See, for instance, Why 0.1 Does Not Exist In Floating-Point? So there will be rounding errors. Also, as @DavidCarlisle pointed out in the comment, TeX is good at integer arithmetic, which adds another layer of error.

\documentclass{article}
\usepackage{tikz}
\begin{document}
\begin{tikzpicture}[scale=16]
\foreach \x in {0,0.1,...,0.5}
    \node at (\x,0.1) {\x};
\draw (0,0.05) -- (0.5,0.05);
\foreach \x[evaluate=\x as \y using \x/10] in {0,1,...,5}
    \node at (\y,0) {\y};
\draw (0,-0.05) -- (0.5,-0.05);
\foreach \x[evaluate=\x as \y using \x/10] in {0,1,...,5}
    \node at (\y,-0.1) {\pgfmathprintnumber[fixed,precision=1]{\y}};
\end{tikzpicture}
\end{document}

Added: The binary representation of 0.1 is

0.00011001100110011... (0011 on repeat)

If the computer stores the above using 16 binary decimals, then it will become

0.0001100110011010

which converts back to 3277/32768 = 0.100006103515625. Then you would end up with 0.20001220703125, 0.300018310546875 and 0.4000244140625. There is no subtraction involved here. This is just how computer stores numbers.

Second edit: As @egreg pointed out, the actual “floating-point arithmetic” in TeX is implemented via dimension register. So you would then be subjected to the rounding error that comes with how pt is converted to sp.

Coincidentally, since 3277/32768 = 6554/65536, this matches perfectly with my illustrated conversion above: 0.1 through 0.5 are being accumulated precisely as 0.100006103515625, 0.20001220703125, 0.300018310546875, 0.4000244140625, and 0.500030517578125.

How does \foreach \x in { a, b, ..., z } { <code> } works?

Basically, PGF does

\start=a pt
\step=b pt
\advance\step by -\start

Here a, b and z should be decimal numbers, \start and \step are dimen registers (not the real names). Then, as the manual explains, the <code> is executed with \start having the initial value; then \advance\start by \step is executed and <code> follows. The loop goes on until \start>z pt (when the <code> will not be executed). This for \step>0pt, but the case \step<0pt is similar.

Now, how does arithmetic with dimen registers work in TeX?

A dimen register value is shown in points, but actually stored in “scaled points”, where one point is 65536 scaled points.

In your case, \step is set to 0.1pt, which corresponds to 6554sp. Adding again 0.1pt makes the internal value to be 13108sp. When 13108sp is converted back to points, the value is 0.20001pt.

However, \dimen0=0.2pt corresponds to an internal value of 13107sp, because 65536 multiplied by 2 is 131072. And 0.00001pt is 1sp, the least nonzero TeX dimension.

With further additions, rounding errors accumulate.

Using \foreach with decimal values should be avoided when possible; when the values are integer, no rounding takes place.