pgfplots: how to nest and scale axis environments

The problem is in the line \begin{axis}[scale=0.2][ (apart from the missing \documentclass).

First, the axis environment has only one set of options, so you have to replace ][ by ,.

Second, scale transforms only the drawing, not the labels etc. Therefore use transform canvas={scale=0.2} instead.

With

\begin{axis}[transform canvas={scale=0.2},

instead of

\begin{axis}[scale=0.2][

you get

Here is a solution using \savebox to avoid peculiar interferences arising from nesting and scaling; this also allows us to position the inserted axis in the scope of the outer axis with axis cs coordinates. Note that for asymmetric scaling (rectangular plots) we cannot use xscale and yscale as offered by tikz, since label positions don't end up where one would expect. Instead, we use unit vector ratio and then do symmetric canvas scaling (= scaling including also the text).

A sketch of the construction:

% First construct the graph to be inserted and save it
\newsavebox\insertedgraph
\savebox\insertedgraph
  {\begin{tikzpicture}
     \begin{axis}[unit vector ratio=..., xmin=..., xmax=...]
     ...
     \end{axis}
   \end{tikzpicture}%
  }

% Now for the main graph
\begin{tikzpicture}
  \begin{axis}
  ... \addplot ... whatever ...
  % Now we insert the other graph at a reduced size
  \node at (axis cs: ...position ...)
    {\scalebox{... scaling factor ...}{\usebox\insertedgraph}};
  \end{axis}
\end{tikzpicture}

Here are two examples: a simple one with plots defined by expressions, and the example of the original post reading the data from tables.

\documentclass{standalone}
\usepackage{pgfplots}\pgfplotsset{compat=1.14}
\begin{document}
\newsavebox\insertedgraph
\savebox\insertedgraph
  {\begin{tikzpicture}
     \begin{axis}[unit vector ratio=1 2,xmin=0,xmax=5]
       \addplot[domain=0.1:5] {1/x};
     \end{axis}
   \end{tikzpicture}%
  }
\begin{tikzpicture}
  \begin{axis}
    \addplot[domain=-5:5] {x^2};
    \node at (axis cs:-0.2,15) {\scalebox{0.5}{\usebox\insertedgraph}};
  \end{axis}
\end{tikzpicture}
\end{document}

\begin{filecontents}{delay.txt}
    10  83.70616734
    20  87.78888284
    30  89.30183444
    40  90.08903274
    50  90.58099978
    60  90.92872104
\end{filecontents}
\begin{filecontents}{hit.txt}
    10  90.98347396
    20  95.0861077
    30  96.60898943
    40  97.3979899
    50  97.89094057   
    60  98.23992643
\end{filecontents}
\documentclass{standalone}
\usepackage{pgfplots}
\pgfplotsset{compat=1.14}
\begin{document}
\newsavebox\mybox
\savebox\mybox
  {\begin{tikzpicture}
    \begin{axis}[
    unit vector ratio=3 2,
    xlabel style={align=center},
    xlabel=Expiration Time (s)\\(b),
    xlabel style={align=center},
    ylabel=Cache\ Hit Rate (\%),
    xmin=10, xmax=60,
    ymin=80, ymax=100,
    legend style={nodes=right},
    legend pos= south east]
    \addplot table {hit.txt};
    \addlegendentry{10000}
    \end{axis}
  \end{tikzpicture}%
  }

\begin{tikzpicture}
\begin{axis}[
    xlabel style={align=center},
    xlabel=Expiration Time (s)\\(a),
    ylabel style={align=center},
    ylabel=Reduced RTT Ratio\ (\%),
    xmin=10, xmax=60,
    ymin=75, ymax=95,
    legend style={nodes=right},
    legend pos= south east]
\addplot table {delay.txt};
\addlegendentry{10000}
\node at (axis cs:35,81) {\scalebox{0.4}{\usebox\mybox}}; 
\end{axis}
\end{tikzpicture}

\end{document}