Rotating Video with AVMutableVideoCompositionLayerInstruction

Building on what was answered so far. I found a very good way of debugging and finding out what went wrong with your transforms. Using the ramp methods available, you are able to animate the transforms making it easier to see what your transform is doing.

Most of the time I found myself having transforms that appeared to do nothing until I realised that just using preferredTransform property of a video track alone may result in the video feed moving out of the render screen.

AVMutableVideoCompositionLayerInstruction *videoLayerInstruction = [AVMutableVideoCompositionLayerInstruction videoCompositionLayerInstructionWithAssetTrack:videoTrack];

[videoLayerInstruction setTransformRampFromStartTransform:CGAffineTransformIdentity
toEndTransform:videoTrack.preferredTransform 
timeRange:CMTimeRangeMake(projectClipStart, projectClipDuration)];

Eventually, I found that in some cases I needed to apply a translation to bring back the rotated video into the render screen.

CGAffineTransformConcat(videoTrack.preferredTransform, CGAffineTransformMakeTranslation(0, renderSize.height))

Note: Your translation values may be different.

Try this:

AVMutableVideoCompositionLayerInstruction *passThroughLayer = AVMutableVideoCompositionLayerInstruction videoCompositionLayerInstructionWithAssetTrack:videoTrack];
CGAffineTransform rotationTransform = CGAffineTransformMakeRotation(degreesToRadians(90.0));
CGAffineTransform rotateTranslate = CGAffineTransformTranslate(rotateTransform,320,0);
[passThroughLayer setTransform:rotateTranslate atTime:kCMTimeZero];

Essentially the idea is to create a rotation and translation matrix. You rotate it to the proper orientation and then translate it into the view. I did not see any way to specify a center point while I was glancing through the API.

AVAssetTrack *videoAssetTrack= [[videoAsset tracksWithMediaType:AVMediaTypeVideo] lastObject];

AVMutableCompositionTrack *videoCompositionTrack = [composition addMutableTrackWithMediaType:AVMediaTypeVideo preferredTrackID:kCMPersistentTrackID_Invalid];
[videoCompositionTrack insertTimeRange:CMTimeRangeMake(kCMTimeZero, videoAsset.duration) ofTrack:videoAssetTrack atTime:kCMTimeZero error:&error];

videoCompositionTrack.preferredTransform = videoAssetTrack.preferredTransform;