How can I convert a decimal number to hex in VIM?

Use printf (analogous to C's sprintf) with the \= command to handle the replacement:

:%s/\d\+/\=printf("0x%04x", submatch(0))

Details:

• :%s/\d\+/ : Match one or more digits (\d\+) on any line (:%) and substitute (s).
• \= : for each match, replace with the result of the following expression:
• printf("0x%04x", : produce a string using the format "0x%04x", which corresponds to a literal 0x followed by a four digit (or more) hex number, padded with zeros.
  
• submatch(0) : The result of the complete match (i.e. the number).

For more information, see:

:help printf()
:help submatch()
:help sub-replace-special
:help :s

Select (VISUAL) the block of lines that contains the numbers, and then:

:!perl -ne 'printf "0x\%x\n", $_'

another way, to pass it to awk

awk '{printf "0x%x\n",$1}' file

Yet another way:

:rubydo $_ = '0x%x' % $_

Or:

:perldo $_ = sprintf '0x%x', $_

This is a bit less typing and you avoid a level of quoting / shell escaping that you'd get if you did this via :!. You need Perl / Ruby support compiled into your Vim.