Parsing files over 2.15 GB in Java using Kaitai Struct

This library provides a ByteBuffer implementation which uses long offset. I haven't tried this approach but looks promising. See section Mapping Files Bigger than 2 GB

http://www.kdgregory.com/index.php?page=java.byteBuffer

public int getInt(long index)
{
    return buffer(index).getInt();
}

private ByteBuffer buffer(long index)
{
    ByteBuffer buf = _buffers[(int)(index / _segmentSize)];
    buf.position((int)(index % _segmentSize));
    return buf;
}

public MappedFileBuffer(File file, int segmentSize, boolean readWrite)
throws IOException
{
    if (segmentSize > MAX_SEGMENT_SIZE)
        throw new IllegalArgumentException(
                "segment size too large (max " + MAX_SEGMENT_SIZE + "): " + segmentSize);

    _segmentSize = segmentSize;
    _fileSize = file.length();

    RandomAccessFile mappedFile = null;
    try
    {
        String mode = readWrite ? "rw" : "r";
        MapMode mapMode = readWrite ? MapMode.READ_WRITE : MapMode.READ_ONLY;

        mappedFile = new RandomAccessFile(file, mode);
        FileChannel channel = mappedFile.getChannel();

        _buffers = new MappedByteBuffer[(int)(_fileSize / segmentSize) + 1];
        int bufIdx = 0;
        for (long offset = 0 ; offset < _fileSize ; offset += segmentSize)
        {
            long remainingFileSize = _fileSize - offset;
            long thisSegmentSize = Math.min(2L * segmentSize, remainingFileSize);
            _buffers[bufIdx++] = channel.map(mapMode, offset, thisSegmentSize);
        }
    }
    finally
    {
        // close quietly
        if (mappedFile != null)
        {
            try
            {
                mappedFile.close();
            }
            catch (IOException ignored) { /* */ }
        }
    }
}

There are two separate issues here:

1. Running Pcap.fromFile() for large files is generally not a very efficient method, as you'll eventually get all files parsed into memory array at once. A example on how to avoid that is given in kaitai_struct/issues/255. The basic idea is that you'd want to have control over how you read every packet, and then dispose of every packet after you've parsed / accounted it somehow.

2. 2GB limit on Java's mmaped files. To mitigate that, you can use alternative RandomAccessFile-based KaitaiStream implementation: RandomAccessFileKaitaiStream — it might be slower, but it should avoid that 2GB problem.