tikz: place a mark on a straight line

Not sure where you got that syntax for the mark from, you need

mark = at position 0.1 with {\fill [red] circle (0.5cm);}

As Schrödinger's cat mentioned in a comment, you don't need any \pgfdecorationpathlength at all, a number is interpreted as a fractional distance along the path. (If you do want to use \pgfdirectionpathlength, just 0.1*\pgfdecoratedpathlength is enough, no \dimexpr needed.) And the actual mark needs to be a proper path.

\documentclass[border=1cm]{standalone}
\usepackage{tikz}
\usetikzlibrary{decorations.markings}
\begin{document}
\begin{tikzpicture}
  \coordinate (A) at (0, 0);
  \coordinate (B) at (1, 3);
  \draw (A) -- (B);
  \path [
    postaction={decorate},
    decoration={
      markings,
      mark = at position 0.1 with {\fill [red] circle [radius=0.1cm];}
    }
  ] (A) -- (B) ;
\end{tikzpicture}
\end{document}

The same result you can obtain without decorations.markings:

\documentclass[tikz, border=1cm]{standalone}

\begin{document}
\begin{tikzpicture}
  \coordinate (A) at (0, 0);
  \coordinate (B) at (1, 3);
\draw  (A) -- (B) node[pos=0.1, circle,fill=red,inner sep=1.5pt] {};
\end{tikzpicture}
\end{document}