How do I escape special characters for a substitution in a Perl one-liner?

You have several problems:

1. You are using \b incorrectly
2. You are replacing code with shell variables
3. You need to quote metacharacters

From perldoc perlre

A word boundary ("\b") is a spot between two characters that has a "\w" on one side of it

Neither of the characters @ or & are \w characters. So your match is guaranteed to fail. You may want to use something like s/(^|\s)\@d\&(\s|$)/${1}new text$2/

(^|\s) says to match either the start of the string (^)or a whitespace character (\s).

(\s|$) says to match either the end of the string ($) or a whitespace character (\s).

To solve the second problem, you should use %ENV.

To solve the third problem, you should use the \Q and \E escape sequences to escape the value in $ENV{a}.

Putting it all together we get:

#!/bin/bash

export a='@d&'
export b='new text'

echo 'param5 @d&' | 
    perl -pe 'next if /^#/; s/(^|\s)\Q$ENV{a}\E(\s|$)/$1$ENV{b}$2/ if /param5/' 

Which prints

param5 new text

As discussed at perldoc perlre:

...Today it is more common to use the quotemeta() function or the "\Q" metaquoting escape sequence to disable all metacharacters' special meanings like this:

/$unquoted\Q$quoted\E$unquoted/

Beware that if you put literal backslashes (those not inside interpolated variables) between "\Q" and "\E", double-quotish backslash interpolation may lead to confusing results. If you need to use literal backslashes within "\Q...\E", consult "Gory details of parsing quoted constructs" in perlop.

You can also use a ' as the delimiter in the s/// operation to make everything be parsed literally:

my $text = '@';
$text =~ s'@'1';
print $text;

In your example, you can do (note the single quotes):

perl -pe 's/\b\Q@f&\E\b/new_value/g if m/param5/ and not /^ *#/'