How can I open a help file in Vim on a new buffer in an existing window?

You can open a new tab for help with :tab help. This will give you a full screen help. Also look at :help :tab.

The :help will usually open a new window unless the active window's buffer buftype is already help. So to truly reuse a window you must open a new empty buffer in that window with :enew, change the buftype with :set buftype=help and then issue the :help <whatever>.

For convenience you could define a command to do that in your .vimrc:

command! -nargs=1 -complete=help H :enew | :set buftype=help | :h <args>

And then use :H {subject} from any window.

Using this method you truly reuse the window and that allows you to use C-^ to go to the alternate for example. It will also respect your window layout (split windows, etc) unlike the other answers.

If I understand the question correctly, all you need to do is to chain the help command call with the only command:

:help <subject> | only

You might be looking for :only or CTRL-W o (the same command). This makes the current window the only one on the screen. All other windows are closed.

You can also vertically split the help window with:

:vert help {subject}

BTW, :help actually does open in a new buffer, it's just "unlisted". To list all buffers, including the unlisted ones:

:buffers!