Placing Subfigures vertically

To have three independent figures with the desired layout, you can use minipages:

\documentclass{article}
\usepackage[demo]{graphicx}

\begin{document}

\begin{figure}
\begin{minipage}[c][11cm][t]{.5\textwidth}
  \vspace*{\fill}
  \centering
  \includegraphics[width=5cm,height=10cm]{image1}
  \caption{test figure one}
  \label{fig:test1}
\end{minipage}%
\begin{minipage}[c][11cm][t]{.5\textwidth}
  \vspace*{\fill}
  \centering
  \includegraphics[width=5cm,height=4.5cm]{image1}
  \caption{test figure two}
  \label{fig:test2}\par\vfill
  \includegraphics[width=5cm,height=4.5cm]{image1}
  \caption{test figure three}
  \label{fig:test3}
\end{minipage}
\end{figure}

\end{document}

To have the three images as subfigures, you can simply replace \caption with \subcaption (from the subcaption package) in the previous code:

\documentclass{article}
\usepackage[demo]{graphicx}
\usepackage{caption,subcaption}

\begin{document}

\begin{figure}
\begin{minipage}[c][11cm][t]{.5\textwidth}
  \vspace*{\fill}
  \centering
  \includegraphics[width=5cm,height=10cm]{image1}
  \subcaption{test figure one}
  \label{fig:test1}
\end{minipage}%
\begin{minipage}[c][11cm][t]{.5\textwidth}
  \vspace*{\fill}
  \centering
  \includegraphics[width=5cm,height=4.5cm]{image1}
  \subcaption{test figure two}
  \label{fig:test2}\par\vfill
  \includegraphics[width=5cm,height=4.5cm]{image1}
  \subcaption{test figure three}
  \label{fig:test3}
\end{minipage}
\end{figure}

\end{document}

The demo option for graphicx simply replaces actual figures with black rectangles; do not use that option in your actual document.

A subfig solution:

\documentclass{article}
\usepackage[demo]{graphicx}
\usepackage{subfig}

\newsavebox{\bigleftbox}

\begin{document}

\begin{figure}
\centering
\sbox{\bigleftbox}{%
  \begin{minipage}[b]{.5\textwidth}
  \centering
  \vspace*{\fill}
  \subfloat[test figure one]
    {\includegraphics[width=5cm,height=10cm]{image1}\label{fig:test1}}
  \end{minipage}%
}\usebox{\bigleftbox}%
\begin{minipage}[b][\ht\bigleftbox][s]{.5\textwidth}
\centering
\subfloat[test figure two]
  {\includegraphics[width=5cm,height=4.5cm]{image1}\label{fig:test2}}

\vfill

\subfloat[test figure three]
  {\includegraphics[width=5cm,height=4.5cm]{image1}\label{fig:test3}}
\end{minipage}
\end{figure}

\end{document}

The big left subfigure is measured and typeset, so we can set the height of the right box to be exactly the same as the left box's.

You could also use minipages if you don't want to use subfigures, so the result looks more like the example in the question:

\documentclass{article}
\usepackage[demo]{graphicx}

\begin{document}

\begin{figure}[htbp]
\begin{minipage}{0.5\linewidth}
\centering
\includegraphics[width=5cm,height=10cm]{image1}
\caption{A Circle}
\label{fig:circle}
\end{minipage}%
\begin{minipage}{0.5\linewidth}
\centering
\includegraphics[width=5cm,height=4.5cm]{image1}
\caption{test subfigure two}
\label{fig:test2}\par \medskip \vfill
\includegraphics[width=5cm,height=4.5cm]{image1}
\caption{test subfigure three}
\label{fig:test3}
\end{minipage}
\end{figure}

\end{document}