Select an item in vim autocomplete list without inserting line break
It depends on which popup menu state you are in (see :help popupmenu-completion). I understand from your question that you're in state 2 (since you've pressed arrow keys to find a completion). However, the default behavior for Enter in state 2 is to insert the completion without newline; what you describe is normally the behavior of state 1 (which is when you use Ctrl+N/Ctrl+P.)
A way that works consistently in all states is to use Ctrl+Y. I like to remember the Y as standing for "yes, accept that word." It's also possible to just start typing the text that should come after the completed word, unless you've remapped things as in geedoubleya's answer.
In the same context, you can press Ctrl+E to cancel the menu and leave your text as it was before you invoked it. If you're used to the pairings of Ctrl+E and Ctrl+Y in other contexts (e.g. to scroll up or down in normal mode, or to insert the character below or above the cursor in insert mode), that's one way to remember it here. I guess you could also think of it as "exiting" the menu or similar.
See :help popupmenu-keys for more.
Personally I use this one:
inoremap <expr> <TAB> pumvisible() ? "\<C-y>" : "\<CR>"
inoremap <expr> <Esc> pumvisible() ? "\<C-e>" : "\<Esc>"
inoremap <expr> <C-j> pumvisible() ? "\<C-n>" : "\<Down>"
inoremap <expr> <C-k> pumvisible() ? "\<C-p>" : "\<Up>"
Anyone who uses the CtrlP plugin might find this mapping convenient.
As an alternative to using your arrow keys, enable your j & k keys to scroll through the autocomplete list.
Doing this changes the current line to match the selected word as you scroll.
Therefore you do not have to press enter as the cursor is still in insert mode at the end of the substituted word.
To enable this add this to your .vimrc (Thanks to others at stackoverflow):
inoremap <expr> j ((pumvisible())?("\<C-n>"):("j"))
inoremap <expr> k ((pumvisible())?("\<C-p>"):("k"))
Separately, instead of using the arrow keys, you could just repeat the Ctrl-n which will curse through the options (Ctrl-p to go backwards) and substitute on the current line as it moves, no need for Enter or vim key mappings.