Why is this System.IO.Pipelines code much slower than Stream-based code?

This is perhaps not exactly the explanation you look for but I hope it give some insight:

Having a glance over the two approaches you have there, it shows in the 2nd solution is computationally more complex than the other, by having two nested loops.

Digging deeper using code profiling showcases that the 2nd one (GetLineNumberUsingPipeAsync) is almost 21.5 % more CPU intensive than the one uses the Stream (please check the screenshots, ) And it is close enough to the benchmark result I got:

• Solution#1: 683.7 ms, 365.84 MB

• Solution#2: 777.5 ms, 9.08 MB

I believe the reason is implementaiton of SequenceReader.TryReadTo. Here is the source code of this method. It uses pretty straightforward algorithm (read to the match of first byte, then check if all subsequent bytes after that match, if not - advance 1 byte forward and repeat), and note how there are quite some methods in this implementation called "slow" (IsNextSlow, TryReadToSlow and so on), so under at least certain circumstances and in certain cases it falls back to some slow path. It also has to deal with the fact sequence might contain multiple segments, and with maintaining the position.

In your case you can avoid using SequenceReader specifically for searching the match (but leave it for actually reading lines), for example with this minor changes (this overload of TryReadTo is also more efficient in this case):

private static bool TryReadLine(ref SequenceReader<byte> bufferReader, out ReadOnlySpan<byte> line) {
    // note that both `match` and `line` are now `ReadOnlySpan` and not `ReadOnlySequence`
    var foundNewLine = bufferReader.TryReadTo(out ReadOnlySpan<byte> match, (byte) '\n', advancePastDelimiter: true);

    if (!foundNewLine) {
        line = default;
        return false;
    }

    line = match;
    return true;
}

Then:

private static bool ContainsBytes(ref ReadOnlySpan<byte> line, in ReadOnlySpan<byte> searchBytes) {
    // line is now `ReadOnlySpan` so we can use efficient `IndexOf` method
    return line.IndexOf(searchBytes) >= 0;
}

This will make your pipes code run faster than streams one.