CVMetalTextureCacheCreateTextureFromImage returns -6660 on macOS 10.13

I've come across the same issue, the problem was not asking for Metal compatibility when configuring the AVCaptureVideoDataOutput. I guess the system started to check this in macOS 10.13, possibly to apply some optimization when not requested.

The solution was to add the kCVPixelBufferMetalCompatibilityKey to the videoSettings property of AVCaptureVideoDataOutput.

In Objective-C:

outputCapture.videoSettings = @{
  /* ... */
  (NSString *)kCVPixelBufferMetalCompatibilityKey: @YES
};

In Swift:

outputCapture.videoSettings = [
  /* ... */
  kCVPixelBufferMetalCompatibilityKey as String: true
]

I think this warrants a radar, to ask Apple to at least print a warning message when this occurs. I'll update this if I get to it.

I found a workaround for this, which keeps the 2vuy format in the pixel buffer, but the bad thing is that you make a copy of the pixel buffer data which affects performance. I'm posting this for future reference, or if anyone else finds it useful. Basically we intercept the pixel buffer and then add attributes while copying the data.

NSDictionary *attributes = @{
                             @"IOSurfaceCoreAnimationCompatibility": @YES
                             };
CVPixelBufferRef copy = NULL;

CVPixelBufferCreate(kCFAllocatorDefault,
                    CVPixelBufferGetWidth(pixelBuffer),
                    CVPixelBufferGetHeight(pixelBuffer),
                    CVPixelBufferGetPixelFormatType(pixelBuffer),
                    (__bridge CFDictionaryRef)attributes,
                    &copy);

CVPixelBufferLockBaseAddress(pixelBuffer, kCVPixelBufferLock_ReadOnly);
CVPixelBufferLockBaseAddress(copy, 0);

void *baseAddress = CVPixelBufferGetBaseAddress(pixelBuffer);
void *copyBaseAddress = CVPixelBufferGetBaseAddress(copy);

memcpy(copyBaseAddress, baseAddress, CVPixelBufferGetDataSize(pixelBuffer));

CVPixelBufferUnlockBaseAddress(copy, 0);
CVPixelBufferUnlockBaseAddress(pixelBuffer, kCVPixelBufferLock_ReadOnly);

Another way to get from a CVPixelBufferRef to a Metal texture, you could go via a CIImage and use a CIContext with a Metal device (hopefully minimises getting the CPU involved with copying the pixel buffer);

Make sure your target metal texture is appropriately sized.

CIContext* ciContext = [CIContext contextWithMTLDevice:mtlDevice
    options:[NSDictionary dictionaryWithObjectsAndKeys:@(NO),kCIContextUseSoftwareRenderer,nil]
];

id<MTLCommandBuffer> metalCommandBuffer=[mtlCommandQueue commandBufferWithUnretainedReferences];

CIImage* ciImage = [[CIImage alloc] initWithCVPixelBuffer:cvPixelBuffer];

[ciContext render:ciImage 
    toMTLTexture:metal_texture 
    commandBuffer:mtlCommandBuffer
    bounds:[ciImage extent])
    colorSpace:[ciImage colorSpace]];