Circle symbol compatible with \square and \triangle

You can scale \bigcirc down:

\documentclass{article}
\usepackage{amsmath}
\usepackage{amsthm}
\usepackage{amssymb}
\usepackage{graphicx}

\makeatletter
\DeclareRobustCommand{\iscircle}{\mathord{\mathpalette\is@circle\relax}}
\newcommand\is@circle[2]{%
  \begingroup
  \sbox\z@{\raisebox{\depth}{$\m@th#1\bigcirc$}}%
  \sbox\tw@{$#1\square$}%
  \resizebox{!}{\ht\tw@}{\usebox{\z@}}%
  \endgroup
}
\makeatother

\begin{document}

$\iscircle AB$, $\square ABCD$,  $\triangle ABC$,

\end{document}

Some other possibilities: use ocircle from wasysym, raised slightly (the regular symbol is rendered below the baseline), or use TikZ and print a tight circle around a lower case letter (but don't print the letter).

The commands can be defined in a simple way that scales with the font but not with sub- and superscript, or as a more math-aware version using \mathchoice. The solution in the other answer is also included, note that the circle is a little flat in the subscript of the normal size example, but round in the \Huge version.

\documentclass{article}
\usepackage{amsmath}
\usepackage{amsthm}
\usepackage{amsfonts}
\usepackage{wasysym}
\usepackage{tikz}
\makeatletter
\DeclareRobustCommand{\iscircle}{\mathord{\mathpalette\is@circle\relax}}
\newcommand\is@circle[2]{%
  \begingroup
  \sbox\z@{\raisebox{\depth}{$\m@th#1\bigcirc$}}%
  \sbox\tw@{$#1\square$}%
  \resizebox{!}{\ht\tw@}{\usebox{\z@}}%
  \endgroup
}
\makeatother
\newcommand{\wacircsimple}{\raisebox{0.2ex}{$\ocircle$}}
\newcommand{\tkzcircsimple}{\tikz{\node[circle,inner sep=0,outer sep=0,draw]{$\phantom{o}$}}}
\newcommand{\wacircscale}{\mathchoice%
{\raisebox{0.2ex}{$\displaystyle\ocircle$}}%
{\raisebox{0.2ex}{$\textstyle\ocircle$}}%
{\raisebox{0.2ex}{$\scriptstyle\ocircle$}}%
{\raisebox{0.2ex}{$\scriptscriptstyle\ocircle$}}%
}
\newcommand{\tkzcircscale}{\mathchoice%
{\tikz{\node[circle,inner sep=0,outer sep=0,draw]{$\displaystyle \phantom{o}$}}}%
{\tikz{\node[circle,inner sep=0,outer sep=0,draw]{$\textstyle \phantom{o}$}}}%
{\tikz{\node[circle,inner sep=0,outer sep=0,draw]{$\scriptstyle \phantom{o}$}}}%
{\tikz{\node[circle,inner sep=0,outer sep=0,draw]{$\scriptscriptstyle \phantom{o}$}}}%
}

\begin{document}
$\square\iscircle BA_{\iscircle AB_{\iscircle AB}}$

$\square\wacircsimple BA_{\wacircsimple AB_{\wacircsimple AB}}$

$\square\wacircscale BA_{\wacircscale AB_{\wacircscale AB}}$

$\square\tkzcircsimple BA_{\tkzcircsimple AB_{\tkzcircsimple AB}}$

$\square\tkzcircscale BA_{\tkzcircscale AB_{\tkzcircscale AB}}$

\Huge
$\square\iscircle BA_{\iscircle AB_{\iscircle AB}}$

$\square\wacircsimple BA_{\wacircsimple AB_{\wacircsimple AB}}$

$\square\wacircscale BA_{\wacircscale AB_{\wacircscale AB}}$

$\square\tkzcircsimple BA_{\tkzcircsimple AB_{\tkzcircsimple AB}}$

$\square\tkzcircscale BA_{\tkzcircscale AB_{\tkzcircscale AB}}$
\end{document}

In my opinion the TikZ version looks better in the normal font (considering line width) but in the \Huge version the wasysym symbol looks better.