How to color randomly drawn dots within a circle?

Mathematically, a circle with radius r with center at (a,b) is defined by the equation

(x-a)^2 + (y-b)^2 = r^2

So, to check if a point lies within this circle, you can check if the random points (x,y) satisfy

(x-a)^2 + (y-b)^2 <= r^2

i.e. they are less than r away from the center of the circle. If you use <=, then points on the border still count as "inside", if you use <, they won't.

To do that in TikZ, you can use the ifthenelse function provided by PGF Math. That is:

\pgfmathparse{ ifthenelse(condition, "value A", "value B")}

If the condition is satisfied, \pgfmathresult will contain "value A", if not it will contain "value B". We can use this to set the color either to "black" or "red":

\documentclass[tikz]{standalone}
\begin{document}
  \begin{tikzpicture}

    % Random seed for RNG
    \pgfmathsetseed{\number\pdfrandomseed}

    % Define circle parameters
    \newcommand{\cX}{2}
    \newcommand{\cY}{2}
    \newcommand{\cR}{1}

    \draw (0,0) -- (4,0) -- (4,4) -- (0,4) -- cycle;

    \foreach \x in {1,...,40}
    {
      % Find random numbers
      \pgfmathrandominteger{\a}{10}{390}
      \pgfmathrandominteger{\b}{10}{390}

      % Scale numbers nicely
      \pgfmathparse{0.01*\a}\let\a\pgfmathresult
      \pgfmathparse{0.01*\b}\let\b\pgfmathresult

      % Check if numbers are inside circle
      \pgfmathparse{ifthenelse((\a-\cX)^2 + (\b-\cY)^2 <= \cR^2,%
          "red",
          "black")}
        \fill[\pgfmathresult] (\a,\b) circle (0.1);
    };
    \draw (\cX,\cY) circle (\cR);
  \end{tikzpicture}
\end{document}

If you don't mind working in points, then the TikZ math library may be helpful. Here, points are rejected if they would lie on the border of the circle.

\documentclass[tikz,border=5]{standalone}
\usetikzlibrary{math}
\begin{document}
\begin{tikzpicture}[x=1pt,y=1pt]
\tikzmath{%
  coordinate \c, \p, \q;
  \q = (100, 100);
  \c = (30, 30);
  \R = 50;
  \r = 1;
  {
    \draw (-\qx, -\qy) rectangle (\qx, \qy);
    \draw [red] (\c) circle [radius=\R];
  };
  \x = \qx - \r; \y = \qy - \r;
  for \i in {0,...,100}{
    \p = (rand * \x, rand * \y);
    \v = veclen(\cx - \px, \cy - \py);
    if \v > (\R + \r) then {
      { \fill [black] (\p) circle [radius=\r]; };
    } else {
      if \v < (\R - \r) then {
        { \fill [red] (\p) circle [radius=\r]; };
      };
    };
  };
}
\end{tikzpicture}
\end{document}

Here is one "clipping mask" solution:

\documentclass[varwidth, border=7pt]{standalone}
\usepackage{tikz}
\usetikzlibrary{fadings}

\begin{document}
  % define clip mask with random points
  \begin{tikzfadingfrompicture}[name=rndpts]
    \fill[transparent!0] foreach ~ in {1,...,100}{(rand,rand) circle (0.02)};
  \end{tikzfadingfrompicture}
  % use it to clip rectangle and circle
  \begin{tikzpicture}
    \begin{scope}
      \fill[scope fading=rndpts, fit fading=true] (0,0) rectangle (4,4);
      \fill[red] (2,2) circle (1cm);
    \end{scope}
    \draw (0,0) rectangle (4,4);
    \draw[red] (2,2) circle (1cm);
  \end{tikzpicture}
\end{document}