Octave does not plot
And if you need Aqua terminal instead of X11 - this might help:
brew uninstall --force gnuplot
brew install gnuplot --with-aquaterm --with-qt4
You don't need to add setenv("GNUTERM","X11") or add GNUTERM env.var to octaverc file.
setenv("GNUTERM","qt")
in your octave command prompt, it should solve the problem.
The following worked out for me
Uninstall any existing gnuplot on your OSX
brew uninstall gnuplot
Install gnuplot with either X or X11
brew install gnuplot --with-x
Finally, set the GNUTERM to X11
setenv("GNUTERM","X11")
I think your problem comes from using the CLI.
As of January 2, 2014. If you're using OS/X 10.9 (Maverics) then you can opt to use integrated GUI from the binary package by downloading the Octave 3.8.0 .dmg. Use Octave-gui not Octave-cli, or otherwise start the GUI from the CLI.
A note about the gui from The Octave Home Page
But because it is not quite as polished as we would like, we have decided to wait until the 4.0.x release series before making the GUI the default interface (until then, you can use the --force-gui option to start the GUI).