Image buffer display order with VTDecompressionSession

In my case, the problem wasn't with VTDecompressionSession, it was a problem with the demuxer getting the wrong PTS. While I couldn't get VTDecompressionSession to put out the frames in temporal (display) order with the kVTDecodeFrame_EnableAsynchronousDecompression and kVTDecodeFrame_EnableTemporalProcessing flags, I could sort the frames myself based on PTS with a small vector.

First, make sure you associate all of your timing information with your CMSampleBuffer along with the block buffer so you receive it in the VTDecompressionSession callback.

// Wrap our CMBlockBuffer in a CMSampleBuffer...
CMSampleBufferRef sampleBuffer;

CMTime duration = ...;
CMTime presentationTimeStamp = ...;
CMTime decompressTimeStamp = ...;

CMSampleTimingInfo timingInfo{duration, presentationTimeStamp, decompressTimeStamp};

_sampleTimingArray[0] = timingInfo;
_sampleSizeArray[0] = nalLength;

// Wrap the CMBlockBuffer...
status = CMSampleBufferCreate(kCFAllocatorDefault, blockBuffer, true, NULL, NULL, _formatDescription, 1, 1, _sampleTimingArray, 1, _sampleSizeArray, &sampleBuffer);

Then, decode the frame... It is worth trying to get the frames out in display order with the flags.

VTDecodeFrameFlags flags = kVTDecodeFrame_EnableAsynchronousDecompression | kVTDecodeFrame_EnableTemporalProcessing;
VTDecodeInfoFlags flagOut;

VTDecompressionSessionDecodeFrame(_decompressionSession, sampleBuffer, flags,
                                      (void*)CFBridgingRetain(NULL), &flagOut);

On the callback side of things, we need a way of sorting the CVImageBufferRefs we receive. I use a struct that contains the CVImageBufferRef and the PTS. Then a vector with a size of two that will do the actual sorting.

struct Buffer
{
    CVImageBufferRef imageBuffer = NULL;
    double pts = 0;
};

std::vector <Buffer> _buffer;

We also need a way to sort the Buffers. Always writing to and reading from the index with the lowest PTS works well.

 -(int) getMinIndex
 {
     if(_buffer[0].pts > _buffer[1].pts)
     {
         return 1;
     }

     return 0;
 }      

In the callback, we need to fill the vector with Buffers...

 void decompressionSessionDecodeFrameCallback(void *decompressionOutputRefCon, void *sourceFrameRefCon, OSStatus status, VTDecodeInfoFlags infoFlags, CVImageBufferRef imageBuffer, CMTime presentationTimeStamp, CMTime presentationDuration)
 {
    StreamManager *streamManager = (__bridge StreamManager     *)decompressionOutputRefCon;

    @synchronized(streamManager)
    {
    if (status != noErr)
    {
        NSError *error = [NSError errorWithDomain:NSOSStatusErrorDomain code:status userInfo:nil];
        NSLog(@"Decompressed error: %@", error);
    }
    else
    {
        // Get the PTS
        double pts = CMTimeGetSeconds(presentationTimeStamp);

        // Fill our buffer initially
        if(!streamManager->_bufferReady)
        {
            Buffer buffer;

            buffer.pts = pts;
            buffer.imageBuffer = imageBuffer;

            CVBufferRetain(buffer.imageBuffer);

            streamManager->_buffer[streamManager->_bufferIndex++] = buffer;
        }
        else
        {
            // Push new buffers to the index with the lowest PTS
            int index = [streamManager getMinIndex];

            // Release the old CVImageBufferRef
            CVBufferRelease(streamManager->_buffer[index].imageBuffer);

            Buffer buffer;

            buffer.pts = pts;
            buffer.imageBuffer = imageBuffer;

            // Retain the new CVImageBufferRef
            CVBufferRetain(buffer.imageBuffer);

            streamManager->_buffer[index] = buffer;
        }

        // Wrap around the buffer when initialized
        // _bufferWindow = 2
        if(streamManager->_bufferIndex == streamManager->_bufferWindow)
        {
            streamManager->_bufferReady = YES;
            streamManager->_bufferIndex = 0;
        }
    }
}
}

Finally we need to drain the Buffers in temporal (display) order...

 - (void)drainBuffer
 {
      @synchronized(self)
      {
         if(_bufferReady)
         {
             // Drain buffers from the index with the lowest PTS
             int index = [self getMinIndex];

             Buffer buffer = _buffer[index];

             // Do something useful with the buffer now in display order
         }
       }
 }

I would like to improve upon that answer a bit. While the outlined solution works, it requires knowledge of the number of frames needed to produce an output frame. The example uses a buffer size of 2, but in my case I needed a buffer size of 3. To avoid having to specify this in advance one can make use of the fact, that frames (in display order) align exactly in terms of pts/duration. I.e. the end of one frame is exactly the beginning of the next. Thus one can simply accumulate frames until there is no "gap" at the beginning, then pop the first frame, and so on. Also one can take the pts of the first frame (which is always an I-frame) as the initial "head" (as it does not have to be zero...). Here is some code that does this:

#include <CoreVideo/CVImageBuffer.h>

#include <boost/container/flat_set.hpp>

inline bool operator<(const CMTime& left, const CMTime& right)
{
    return CMTimeCompare(left, right) < 0;
}

inline bool operator==(const CMTime& left, const CMTime& right)
{
    return CMTimeCompare(left, right) == 0;
}

inline CMTime operator+(const CMTime& left, const CMTime& right)
{
    return CMTimeAdd(left, right);
}

class reorder_buffer_t
{
public:

    struct entry_t
    {
        CFGuard<CVImageBufferRef> image;
        CMTime pts;
        CMTime duration;
        bool operator<(const entry_t& other) const
        {
            return pts < other.pts;
        }
    };

private:

    typedef boost::container::flat_set<entry_t> buffer_t;

public:

    reorder_buffer_t()
    {
    }

    void push(entry_t entry)
    {
        if (!_head)
            _head = entry.pts;
        _buffer.insert(std::move(entry));
    }

    bool empty() const
    {
        return _buffer.empty();
    }

    bool ready() const
    {
        return !empty() && _buffer.begin()->pts == _head;
    }

    entry_t pop()
    {
        assert(ready());
        auto entry = *_buffer.begin();
        _buffer.erase(_buffer.begin());
        _head = entry.pts + entry.duration;
        return entry;
    }

    void clear()
    {
        _buffer.clear();
        _head = boost::none;
    }

private:

    boost::optional<CMTime> _head;
    buffer_t _buffer;
};