How to make a small $\varhexagon$ in latex?

Indeed, \tiny is a text mode command and thus is not allowed in math mode.

What you may be looking for is

\newcommand\tinyvarhexagon{\vcenter{\hbox{\scalebox{0.5}{$\varhexagon$}}}}

where \scalebox is a macro provided by the graphic package and \vcenter and \hbox are TeX "primitive". \tiny corresponds to a linear reduction in font size of 50%; hence the 0.5 scaling factor. With this definition, the tiny hexagon is centered vertically on the math axis. If you would rather have the resized symbol placed on the baseline, just omit the \vcenter{\hbox{...}} "wrapper".

\documentclass{article}
\usepackage{wasysym} % for "\varhexagon" macro
\usepackage{graphicx}% for "\scalebox" macro
\newcommand\tinyvarhexagon{\vcenter{\hbox{\scalebox{0.5}{$\varhexagon$}}}}
   % ("\vcenter" and "\hbox" are TeX primitives)

\begin{document}
$a\varhexagon a$ $a\tinyvarhexagon a$ $a\scalebox{0.5}{$\varhexagon$} a$ 

$A_{\varhexagon}$ $A_{\tinyvarhexagon}$ $A_{\scalebox{0.5}{$\varhexagon$}}$
\end{document} 

You can load the font with a different scaling factor:

\documentclass{article}

\usepackage{wasysym} % just for comparison

\DeclareFontFamily{U}{wasysmall}{}
\DeclareFontShape{U}{wasysmall}{m}{n}{
  <-5.5> s*[0.75] wasy5
  <5.5-6.5> s*[0.75] wasy6
  <6.5-7.5> s*[0.75] wasy7
  <7.5-8.5> s*[0.75] wasy8
  <8.5-9.5> s*[0.75] wasy9
  <9.5-> s*[0.75] wasy10
}{}
\DeclareFontShape{U}{wasysmall}{b}{n}{
  <-5.5> s*[0.75] wasyb5
  <5.5-6.5> s*[0.75] wasyb6
  <6.5-7.5> s*[0.75] wasyb7
  <7.5-8.5> s*[0.75] wasyb8
  <8.5-9.5> s*[0.75] wasyb9
  <9.5-> s*[0.75] wasyb10
}{}
\DeclareFontShape{U}{wasysmall}{bx}{n}{ <-> sub * wasysmall/b/n}{}

\DeclareSymbolFont{wasysmall}{U}{wasysmall}{m}{n}
\SetSymbolFont{wasysmall}{bold}{U}{wasysmall}{b}{n}

\DeclareMathSymbol{\smallhexagon}{\mathord}{wasysmall}{57}

\begin{document}

{\tiny\varhexagon} $\scriptstyle\smallhexagon$ $A_{\smallhexagon}$

\end{document}

Note that loading wasysym is not required, I did just for the comparison.

You can avoid wasting a symbol font:

\documentclass{article}
\usepackage{amsmath}

\DeclareFontFamily{U}{wasysmall}{}
\DeclareFontShape{U}{wasysmall}{m}{n}{
  <-5.5> s*[0.75] wasy5
  <5.5-6.5> s*[0.75] wasy6
  <6.5-7.5> s*[0.75] wasy7
  <7.5-8.5> s*[0.75] wasy8
  <8.5-9.5> s*[0.75] wasy9
  <9.5-> s*[0.75] wasy10
}{}

\newcommand{\smallhexagon}{\text{\usefont{U}{wasysmall}{m}{n}\symbol{57}}}

\begin{document}

$A_{\smallhexagon}$

\end{document}

If you just need the symbol to change size in subscripts, you can do in a different way:

\documentclass{article}
\usepackage{amsmath}
\usepackage{wasysym}

\DeclareMathSymbol{\mhexagon}{\mathord}{wasy}{57}

\begin{document}

$A_{\varhexagon}$ (wrong)

$A_{\mhexagon}$

\end{document}

The amsmath package also provides the \text command which allows text mode commands inside math mode without warnings (see also the answer by @egreg above).

MWE:

\documentclass{article}
\usepackage{amsmath}
\usepackage{wasysym}

\begin{document}
A{\tiny\varhexagon}

$A_{\text{\tiny\varhexagon}}$
\end{document}

Result: