big circle in math mode

The MnSymbol package provides a \bigcircle command, however, it is a little smaller than you want. You can use \scalebox from the graphicx to make the circle a little bigger. It is still a little high, so \raisebox can be used to adjust this. Putting this together into a macro:

\newcommand\Bigcircle{\raisebox{-0.5mm}{\scalebox{1.7}{$\bigcircle$}}}

produces:

Another, arguably more flexible, approach is to use tikz to draw the circle. Again, you need to adjust the baseline of the circle, but it is much easier to change the size of the circle and the width of line. It is also very easy to draw other shapes, such as squares, this way. The macro

\newcommand\TikCircle[1][2.5]{\tikz[baseline=-#1]{\draw[thick](0,0)circle[radius=#1mm];}}

shows one way of doing it this way to use tikz. (The \TikCircle command takes an optional argument that gives the radius of the circle in milimeters. By default, the radius is 2.5mm so \TikCircle is the same as \TikCircle[2.5].) This produces:

Here is the full code as a minimal working example:

\documentclass{article}
\usepackage{amsmath}

% mnsymbol approach
\usepackage{MnSymbol,graphicx}
\newcommand\Bigcircle{\raisebox{-0.5mm}{\scalebox{1.7}{$\bigcircle$}}}

% tikz approach
\usepackage{tikz}
\newcommand\TikCircle[1][2.5]{\tikz[baseline=-#1]{\draw[thick](0,0)circle[radius=#1mm];}}

\begin{document}

    MnSymbol:\bigskip

    $\frac34 \Bigcircle\frac14$

    \bigskip

    Tikz:\bigskip

    $\frac34 \TikCircle\frac14$

\end{document}

Finally, you probably want to add some space around these symbols. I would treat these as "mathematics operators" and use \mathop:

\newcommand\TikCircle[1][2.5]{{\mathop{\tikz[baseline=-#1]{\draw[thick](0,0)circle[radius=#1mm];}}}}

How about using a "traditional" square instead:

\documentclass{article}

\newcommand{\fillrel}{\mathrel{\fbox{$\phantom{\bigcirc}$}}}

\begin{document}

\[
  \frac{3}{4} \fillrel \frac{1}{2} \fillrel \frac{5}{10}
\]

\end{document}