Why triangle's corners are sticking out instead of aligning perfectly with adjacent triangles?

As the error message makes clear, TikZ looks for circle and there is no such shape. In some cases, spaces get trimmed away, but not always and not here. The parsing is, to say the least, complex, and the result cannot always be what you'd intuitively expect since that would cause less intuitive behaviour elsewhere. Generally, it does a pretty good job, but sometimes you need to remember that it is, after all, code and precise syntax matters.

The peculiar spy result is a function of the placement method used by TikZ and the non-zero line widths, combined with a disjunctively-drawn border path.

That is, the trouble is that your plan for tessellation does not take account of the line widths. Either you get gaps or bits jut out. Moreover, imprecision will make for raggedness anyway if you don't draw closed paths. You have closed paths around each shape, but not the borders of the combination.

One way around this is to draw the borders after creating the nodes. For example,

\documentclass[border=10pt]{standalone}
\usepackage{tikz}
\usetikzlibrary{shapes.geometric,spy}
\begin{document}
\begin{tikzpicture}
  [
    every isosceles triangle node/.style={%
      inner sep=0mm, isosceles triangle apex angle=60, minimum size=20mm, outer sep=0mm, shape border uses incircle
    },
    spy using outlines={%
      circle, magnification=20,  size=2cm, connect spies,
    },
  ]
  \begin{scope}[every node/.append style={isosceles triangle}]
    \node [shape border rotate=90] (T_1) {};
    \node [anchor=right corner, shape border rotate=-90] (T_3) at (T_1.left corner) {};
    \node [anchor=apex, shape border rotate=90] (T_2) at (T_1.left corner) {};
  \end{scope}
  \draw (T_1.apex) -- (T_1.right corner) -- (T_3.apex) -- (T_2.left corner) -- (T_2.apex) edge (T_3.apex) edge (T_1.right corner) -- cycle;
  \spy[red]on(T_3.right corner) in node at(3,0);
\end{tikzpicture}
\end{document}