match space or newline in sed

sed works on a line at a time and it will strip the newlines when processing each line.

So, in order to do what you want, you should match the end of line anchor ($) rather than a literal newline character.

This should work:

sed 's:\(\s\)\(00011\)\(\s\|$\):\1$03\3:g'

$ sed 's/\<00011\>/$03/g' file
ADD    00000 00001 $03
LSH    $03 00100 01111
ADD    $03 10100 00010
JSR    00011101000111010101100010

The \< and \> matches the zero-width word boundaries at the start and end of a word, respectively. BSD sed would also recognise [[:<:]] and [[:>:]], and GNU sed also understands \b as a word boundary.

sed will never see the newlines in the input data. Also, \s is specific to GNU sed. To match a space character in standard sed just use a literal space (to match a space-or-tab, use [[:blank:]]).