Test of integers? Or, round the number if the first two decimal numbers are sufficiently close to 0 or 1?

One cannot say from its floating point representation whether the output of an arithmetic operation involving division or non rational operations is actually an integer.

You can consider the l3fp module of expl3, available through the package xfp.

\documentclass{article}
\usepackage{xfp}

\begin{document}

$\fpeval{5/3}$ is a decimal number, I would like to round it to
$\fpeval{round(5/3,1)}$ or to $\fpeval{round(5/3,2)}$

Another difficulty is that $\fpeval{(1/3)*3}$ is an integer in fact, 
and should be printed as $\fpeval{round((1/3)*3,1)}$ or
$\fpeval{round((1/3)*3,2)}$.

\end{document}

If the accuracy of your numbers is important you might consider farming that out to a computer algebra system (CAS). The sagetex package relies on the CAS Sage; the documentation can be found on CTAN right here. Documentation on Sage is found here .Sage is not part of the LaTeX distribution (it's big) so it needs to be installed on your computer or, even easier, accessed through a free Cocalc account.

\documentclass{article}
\usepackage{sagetex}
\usepackage{tikz}
\usetikzlibrary{math}
\begin{document}
\begin{sagesilent}
a = 4/2
b = 5/3
c = 1/3*3
\end{sagesilent}
$\sage{a}$ is an integer, and it should be printed as     $\sage{a}$. 
And $\sage{b}$ is not an integer. As a decimal it is     approximately
$\sage{b.n(digits=6)}$. I would like to round it to     $\sage{b.n(digits=1)}$. 
Another difficulty is that $\sage{c}$ is an integer in fact, 
and should be printed as $\sage{c}$. 
\end{document}

The output, running in Cocalc, gives:

Notice that Sage interprets your numbers correctly: 4/2 is recognized as 2 and 1/3*3 is recognized as 1. It does need to know the format you want of non integers; but it recognizes that 5/3 is a fraction that can't be reduced and leaves it as a fraction. To force it into a decimal and to specify the number of digits we append .n(digits=6); the documentation is here.

Assuming that this is a question on how to do this with tikzmath, I'd do

\documentclass{article}
\usepackage{tikz}
\usetikzlibrary{math}
\begin{document}
\tikzmath{
function myint(\x) {
    if abs(\x-round(\x)) < 0.1 then { print{\x\ is an integer\newline};
      return int(round(\x));
    } else { print{\x\ is not an integer\newline};
       return \x;
}; };
\integer = myint(4/2); 
\decimal = myint(5/3);
\integerB= myint(1/3*3);
}


$\integer$ is an integer, and it should be printed as 2. 
And $\pgfmathprintnumber[precision=1]{\decimal}$ is a decimal number, I would like to round it to 1.7. 
Another difficulty is that $\integerB$  is an integer in fact, 
and should be printed as 1. 
\end{document}

Note that I used \pgfmathprintnumber to round to one digit after the dot. Of course, you can remove the prints.