Custom AVVideoCompositing class not working as expected

_{All of the code for this utility is on GitHub}

It turns out that the requiredSourceTrackIDs variable in the custom AVVideoCompositionInstruction class (VideoFilterCompositionInstruction in the question) has to be set to an array containing the track IDs

override var requiredSourceTrackIDs: [NSValue]{
  get{
    return [
      NSNumber(value: Int(self.trackID))
    ]
  }
}

So the final custom composition instruction class is

class VideoFilterCompositionInstruction : AVMutableVideoCompositionInstruction{
    let trackID: CMPersistentTrackID
    let filters: [CIFilter]
    let context: CIContext

    override var passthroughTrackID: CMPersistentTrackID{get{return self.trackID}}
    override var requiredSourceTrackIDs: [NSValue]{get{return [NSNumber(value: Int(self.trackID))]}}
    override var containsTweening: Bool{get{return false}}

    init(trackID: CMPersistentTrackID, filters: [CIFilter], context: CIContext){
        self.trackID = trackID
        self.filters = filters
        self.context = context
    
        super.init()
    
        self.enablePostProcessing = true
    }

    required init?(coder aDecoder: NSCoder){
        fatalError("init(coder:) has not been implemented")
    }
}

All of the code for this utility is also on GitHub

As you've noted, having passthroughTrackID return the track you want to filter isn't the right approach — you need to return the track to be filtered from requiredSourceTrackIDs instead. (And it looks like once you do that, it doesn't matter if you also return it from passthroughTrackID.) To answer the remaining question of why it works this way...

The docs for passthroughTrackID and requiredSourceTrackIDs certainly aren't Apple's clearest writing ever. (File a bug about it and they might improve.) But if you look closely in the description of the former, there's a hint (emphasis added)...

If for the duration of the instruction, the video composition result is one of the source frames, this property returns the corresponding track ID. The compositor won't be run for the duration of the instruction and the proper source frame is used instead.

So, you use passthroughTrackID only when you're making an instruction class that passes a single track through without processing.

If you plan to perform any image processing, even if it's just to a single track with no compositing, specify that track in requiredSourceTrackIDs instead.