How can I catch the output from a carp in Perl?

Another way how to catch warnings and also all STDERR output:

my $stderr = '';
{
    local *STDERR;
    open STDERR, '>', \$stderr;
    do_stuf_here();
}
like( $stderr, qr/my result/, 'test stderr output' );

One can make fancy test function:

sub stderr_test (&$$) {
    my ( $code, $pattern, $text ) = @_;
    my $result = '';
    {
        local *STDERR;
        open STDERR, '>', \$result;
        $code->();
    }
    if ( UNIVERSAL::isa( $pattern, 'Regexp' ) ) {
        like( $result, $pattern, $text );
    }
    else {
        is( $result, $pattern, $text );
    }
}

# usage
stderr_test {do_stuf_here} qr/my expected STDERR output/,
    'stderr is like';
stderr_test {do_stuf_here} 'my expected STDERR output',
    'stderr is exactly';

From this page, http://perldoc.perl.org/perlvar.html, it looks like you want to set the local $SIG{__WARN__} to a subroutine that will turn warnings into fatal errors for your test script. The example they give is:

local $SIG{__WARN__} = sub { die $_[0] };
eval $proggie;