How can I stop code from being run in the compiler phase?

Use the ${^GLOBAL_PHASE} variable. It contains "START" in the first case, but "RUN" in the second one.

# RUN
perl -wE'say ${^GLOBAL_PHASE}'

# START
perl -wE'BEGIN {say ${^GLOBAL_PHASE}}'

# RUN
perl -wE'eval q{BEGIN {say ${^GLOBAL_PHASE}}}'

See perlvar for details.