How to get the length of a segment in TikZ in a macro

You are loading calc, so

\documentclass[tikz,margin=3]{standalone}
\usetikzlibrary{calc}
\begin{document}
\begin{tikzpicture}
  \draw (0,0) coordinate (b) -- (5,0) coordinate (c) -- (1,4) coordinate (a) -- cycle;
  \draw let \p1=($(b)-(a)$),\n1={veclen(\x1,\y1)} in
  (0,0) node {\n1}  circle [radius=\n1];
\end{tikzpicture}
\end{document}

It is, however, possible to create a function distance that returns the distance between two named coordinates. It can be used, and gets parsed, like any other function. It does not create a path. HOWEVER, MY PREVIOUS EXAMPLE USING IT IN A PATH WAS DANGEROUS AND IS IN GENERAL WRONG. I thank JouleV top reporting this to me! It even works outside of tikzpictures. (This function here is tailored to resemble your original function.)

\documentclass[tikz,margin=3]{standalone}
\makeatletter
\pgfmathdeclarefunction{distance}{2}{%
\begingroup%
\pgfextractx{\pgf@xa}{\pgfpointanchor{#1}{center}}%
\pgfextracty{\pgf@ya}{\pgfpointanchor{#1}{center}}%
\pgfextractx{\pgf@xb}{\pgfpointanchor{#2}{center}}%
\pgfextracty{\pgf@yb}{\pgfpointanchor{#2}{center}}%
\pgfmathparse{veclen(\pgf@xa-\pgf@xb,\pgf@ya-\pgf@yb)}%
\pgfmathsmuggle\pgfmathresult\endgroup%
}%
\makeatother
\begin{document}
\begin{tikzpicture}
  \draw (0,0) coordinate (b) -- (5,0) coordinate (c) -- (1,4) coordinate (a) -- cycle;
  \pgfmathsetmacro{\mydist}{distance("a","b")}
  \path node {\mydist};
  \draw (0,0) circle [radius=\mydist pt];
\end{tikzpicture}
\end{document}

ADDENDUM: I agree with Alain Matthes that veclen is imprecise. This is just to mention that you do not need xfp to solve the problem, computing the Euclidean distance is sufficient. And I also think that tkz-euclide is great, but one does not need it in order to convert the result to cm, a simple \pgfmathparse{<whatever>/1cm} is sufficient.

\documentclass[tikz,margin=3]{standalone}
\makeatletter
\pgfmathdeclarefunction{distance}{2}{%
\begingroup%
\pgfextractx{\pgf@xa}{\pgfpointanchor{#1}{center}}%
\pgfextracty{\pgf@ya}{\pgfpointanchor{#1}{center}}%
\pgfextractx{\pgf@xb}{\pgfpointanchor{#2}{center}}%
\pgfextracty{\pgf@yb}{\pgfpointanchor{#2}{center}}%
\pgfmathparse{sqrt((\pgf@xa-\pgf@xb)*(\pgf@xa-\pgf@xb)+(\pgf@ya-\pgf@yb)*(\pgf@ya-\pgf@yb))}%
\pgfmathsmuggle\pgfmathresult\endgroup%
}%
\makeatother
\begin{document}
\begin{tikzpicture}
  \draw (0,0) coordinate (b) -- (40pt,0)
              coordinate (c) -- (40pt,30pt)
              coordinate (a) -- cycle;
  \pgfmathsetmacro{\mydistance}{distance("a","b")}            
  \path node 
  {$\pgfmathprintnumber{\mydistance}\,\mathrm{pt}=
  \pgfmathparse{\mydistance/1cm}\pgfmathprintnumber{\pgfmathresult}\,\mathrm{cm}$};
  \draw (0,0) circle [radius={\mydistance pt}];
\end{tikzpicture}
\end{document}

There is, though an advantage of xfp: it does not collapse when one has large distances. The same is true for the fpu library, which is made for this and used e.g. in pgfplots. One can use this in "ordinary TikZ, too. This yields a version that is immune to large values, and (rather) precise.

\documentclass[tikz,margin=3]{standalone}
\usetikzlibrary{fpu}
\newcommand{\pgfmathparseFPU}[1]{\begingroup%
\pgfkeys{/pgf/fpu,/pgf/fpu/output format=fixed}%
\pgfmathparse{#1}%
\pgfmathsmuggle\pgfmathresult\endgroup}
\makeatletter
\pgfmathdeclarefunction{distance}{2}{%
\begingroup%
\pgfextractx{\pgf@xa}{\pgfpointanchor{#1}{center}}%
\pgfextracty{\pgf@ya}{\pgfpointanchor{#1}{center}}%
\pgfextractx{\pgf@xb}{\pgfpointanchor{#2}{center}}%
\pgfextracty{\pgf@yb}{\pgfpointanchor{#2}{center}}%
\pgfmathparseFPU{sqrt((\pgf@xa-\pgf@xb)*(\pgf@xa-\pgf@xb)+(\pgf@ya-\pgf@yb)*(\pgf@ya-\pgf@yb))}%
\pgfmathsmuggle\pgfmathresult\endgroup%
}%
\makeatother
\begin{document}
\begin{tikzpicture}
  \draw (0,0) coordinate (b) -- (40pt,0)
              coordinate (c) -- (40pt,30pt)
              coordinate (a) -- cycle;
  \pgfmathsetmacro{\mydistance}{distance("a","b")}            
  \path node 
  {$\pgfmathprintnumber{\mydistance}\,\mathrm{pt}=
  \pgfmathparse{\mydistance/1cm}\pgfmathprintnumber{\pgfmathresult}\,\mathrm{cm}$};
  \draw (0,0) circle [radius={distance("a","b")}];
\end{tikzpicture}
\end{document}

The problem is that \pgfgetlastxy extracts the x and y coordinates as dimensions so you need to first define these dimensions. I think it is also good to define another dimension for the actual result. If you define these new dimensions then your code pretty much works.

For no good reason, I prefer defining a node to a path, so I rewrote your code so that \getlength{a}{b} sets the length \mylength, which you can use where you wish:

\documentclass[tikz,border=3]{standalone}
\usetikzlibrary{calc}
\newdimen\mypointx
\newdimen\mypointy
\newdimen\mylength
\def\getlength#1#2{
  \node (#1#2) at ($ (#1)-(#2) $){};% define a point at (#1)-(#2_
  \pgfgetlastxy{\mypointx}{\mypointy}% extract the coordinates
  \pgfmathsetlength\mylength{veclen(\mypointx, \mypointy)}% compute the length
}
\begin{document}
\begin{tikzpicture}
  \draw (0,0) coordinate (b) -- (5,0) coordinate (c) -- (1,4) coordinate (a) -- cycle;
  % It should work in both, but sadly it is not working in any of these
  \getlength{a}{b};
  \draw (0,0) circle (\mylength);
\end{tikzpicture}
\end{document}

This produces the expected:

I was not sure what \path node {\getlength{a}{b}}; was meant to do because, for example, \path node 1; is not valid syntax. Did you mean something like \path node (\getlength{a}{b},0){};?

Finally, it is not strictly necessary to use another length like \mylength above. You could instead use a standard macro with

\pgfmathsetmacro\mymacro{veclen(\mypointx, \mypointy)}

or even

\pgfmathparse{veclen(\mypointx, \mypointy)}

that you use in commands like

\draw (0,0) circle (\mymacro);

and

\draw (0,0) circle (\pgfmathresult);

respectively. However, as far as I can see you cannot put \pgfmathresult at the end of the \getlength and then use this inside a command like

\draw (0,0) circle (\getlength{a}{b});

as this always returns an error. I suspect is an expansion issue.

The only problem is the result with veclen from pgfmath.

\documentclass[tikz,margin=3]{standalone}
\usetikzlibrary{calc}
\usepackage{xfp} 
\makeatletter
\pgfmathdeclarefunction*{veclen}{2}{%
\begingroup%
  \pgfmath@x#1pt\relax%
  \pgfmath@y#2pt\relax%
  \pgf@xa=\pgf@x%
  \pgf@ya=\pgf@y%
  \edef\tkz@temp@a{\fpeval{\pgfmath@tonumber{\pgf@xa}}}
  \edef\tkz@temp@b{\fpeval{\pgfmath@tonumber{\pgf@ya}}}
  \edef\tkz@temp@sum{\fpeval{%
    (\tkz@temp@a*\tkz@temp@a+\tkz@temp@b*\tkz@temp@b)}}
  \edef\tkzFPMathLen{\fpeval{sqrt(\tkz@temp@sum)}}
  \pgfmath@returnone\tkzFPMathLen pt%
\endgroup%
}
\makeatother
\begin{document}
\begin{tikzpicture}
  \draw (0,0) coordinate (b) -- (40pt,0)
              coordinate (c) -- (40pt,30pt)
              coordinate (a) -- cycle;
  \draw let \p1=($(b)-(a)$),\n1={veclen(\x1,\y1)} in
  (0,0) node {\n1}  circle [radius=\n1];
\end{tikzpicture}
\end{document}

With tkz-euclide you have the macro \tkzCalcLength

\documentclass[border=.25cm]{standalone}
\usepackage{tkz-euclide}

\begin{document}
\begin{tikzpicture}
    \tkzDefPoint(0,0){B}
    \tkzDefPoint(1,4){A}
    \tkzCalcLength(A,B)\tkzGetLength{dAB}
    \node {\dAB pt};
    \tkzCalcLength[cm](A,B)\tkzGetLength{dAB}
    \node at (2,0) {\dAB cm};
\end{tikzpicture}
\end{document}