Jojodmo Jojodmo - 1 year ago 131
Swift Question

Custom AVVideoCompositing class not working as expected

I'm attempting to apply a CIFilter to an AVAsset, and then save it with the filter applied. The way that I am doing this is by using an

set to an
object with a custom

I am also setting the
of my
object to a custom composition instruction class (conforming to
). This class is passed a track ID, along with a few other unimportant variables.

Unfortunately, I've run into a problem - the
function in my custom video compositor class (conforming to
) is not being called correctly.

When I set the
variable of my custom instruction class to the track ID, the
function in my
is not called.

Yet, when I do not set the
variable of my custom instruction class, the
is called, but not correctly - printing
results in an empty array, and
results in a nil value.

Something interesting that I found was that the
function is always called twice when attempting to export the video with filters. It is either called once before
and once after, or just twice in a row in the case that
is not called.

I've created three classes for exporting the video with filters. Here's the utility class, which basically just includes an
function and calls all of the required code

class VideoFilterExport{

let asset: AVAsset
init(asset: AVAsset){
self.asset = asset

func export(toURL url: NSURL, callback: (url: NSURL?) -> Void){
guard let track: AVAssetTrack = self.asset.tracksWithMediaType(AVMediaTypeVideo).first else{callback(url: nil); return}

let composition = AVMutableComposition()
let compositionTrack = composition.addMutableTrackWithMediaType(AVMediaTypeVideo, preferredTrackID: kCMPersistentTrackID_Invalid)

try compositionTrack.insertTimeRange(track.timeRange, ofTrack: track, atTime: kCMTimeZero)
catch _{callback(url: nil); return}

let videoComposition = AVMutableVideoComposition(propertiesOfAsset: composition)
videoComposition.customVideoCompositorClass = VideoFilterCompositor.self
videoComposition.frameDuration = CMTimeMake(1, 30)
videoComposition.renderSize = compositionTrack.naturalSize

let instruction = VideoFilterCompositionInstruction(trackID: compositionTrack.trackID)
instruction.timeRange = CMTimeRangeMake(kCMTimeZero, self.asset.duration)
videoComposition.instructions = [instruction]

let session: AVAssetExportSession = AVAssetExportSession(asset: composition, presetName: AVAssetExportPresetMediumQuality)!
session.videoComposition = videoComposition
session.outputURL = url
session.outputFileType = AVFileTypeMPEG4

callback(url: url)

Here's the other two classes - I'll put them both into one code block to make this post shorter

// Video Filter Composition Instruction Class - from what I gather,
// AVVideoCompositionInstruction is used only to pass values to
// the AVVideoCompositing class

class VideoFilterCompositionInstruction : AVMutableVideoCompositionInstruction{

let trackID: CMPersistentTrackID
let filters: ImageFilterGroup
let context: CIContext

// When I leave this line as-is, startVideoCompositionRequest: isn't called.
// When commented out, startVideoCompositionRequest(request) is called, but there
// are no valid CVPixelBuffers provided by request.sourceFrameByTrackID(below value)
override var passthroughTrackID: CMPersistentTrackID{get{return self.trackID}}
override var requiredSourceTrackIDs: [NSValue]{get{return []}}
override var containsTweening: Bool{get{return false}}

init(trackID: CMPersistentTrackID, filters: ImageFilterGroup, context: CIContext){
self.trackID = trackID
self.filters = filters
self.context = context


//self.timeRange = timeRange
self.enablePostProcessing = true

required init?(coder aDecoder: NSCoder) {
fatalError("init(coder:) has not been implemented")


// My custom AVVideoCompositing class. This is where the problem lies -
// although I don't know if this is the root of the problem

class VideoFilterCompositor : NSObject, AVVideoCompositing{

var requiredPixelBufferAttributesForRenderContext: [String : AnyObject] = [
kCVPixelBufferPixelFormatTypeKey as String : NSNumber(unsignedInt: kCVPixelFormatType_32BGRA), // The video is in 32 BGRA
kCVPixelBufferOpenGLESCompatibilityKey as String : NSNumber(bool: true),
kCVPixelBufferOpenGLCompatibilityKey as String : NSNumber(bool: true)
var sourcePixelBufferAttributes: [String : AnyObject]? = [
kCVPixelBufferPixelFormatTypeKey as String : NSNumber(unsignedInt: kCVPixelFormatType_32BGRA),
kCVPixelBufferOpenGLESCompatibilityKey as String : NSNumber(bool: true),
kCVPixelBufferOpenGLCompatibilityKey as String : NSNumber(bool: true)

let renderQueue = dispatch_queue_create("co.getblix.videofiltercompositor.renderingqueue", DISPATCH_QUEUE_SERIAL)

override init(){

func startVideoCompositionRequest(request: AVAsynchronousVideoCompositionRequest){
// This code block is never executed when the
// passthroughTrackID variable is in the above class

guard let instruction = request.videoCompositionInstruction as? VideoFilterCompositionInstruction else{
request.finishWithError(NSError(domain: "", code: 760, userInfo: nil))
guard let pixels = request.sourceFrameByTrackID(instruction.passthroughTrackID) else{
// This code block is executed when I comment out the
// passthroughTrackID variable in the above class

request.finishWithError(NSError(domain: "", code: 761, userInfo: nil))
// I have not been able to get the code to reach this point
// This function is either not called, or the guard
// statement above executes

let image = CIImage(CVPixelBuffer: pixels)
let filtered: CIImage = //apply the filter here

let width = CVPixelBufferGetWidth(pixels)
let height = CVPixelBufferGetHeight(pixels)
let format = CVPixelBufferGetPixelFormatType(pixels)

var newBuffer: CVPixelBuffer?
CVPixelBufferCreate(kCFAllocatorDefault, width, height, format, nil, &newBuffer)

if let buffer = newBuffer{
instruction.context.render(filtered, toCVPixelBuffer: buffer)

func renderContextChanged(newRenderContext: AVVideoCompositionRenderContext){
// I don't have any code in this block

// This is interesting - this is called twice,
// Once before startVideoCompositionRequest is called,
// And once after. In the case when startVideoCompositionRequest
// Is not called, this is simply called twice in a row
func cancelAllPendingVideoCompositionRequests(){

I've been looking at Apple's AVCustomEdit sample project a lot for guidance with this, but I can't seem to find in it any reason why this is happening.

How could I get the
function to call correctly, and provide a valid
for each frame?

Answer Source

As you've noted, having passthroughTrackID return the track you want to filter isn't the right approach — you need to return the track to be filtered from requiredSourceTrackIDs instead. (And it looks like once you do that, it doesn't matter if you also return it from passthroughTrackID.) To answer the remaining question of why it works this way...

The docs for passthroughTrackID and requiredSourceTrackIDs certainly aren't Apple's clearest writing ever. (File a bug about it and they might improve.) But if you look closely in the description of the former, there's a hint (emphasis added)...

If for the duration of the instruction, the video composition result is one of the source frames, this property returns the corresponding track ID. The compositor won't be run for the duration of the instruction and the proper source frame is used instead.

So, you use passthroughTrackID only when you're making an instruction class that passes a single track through without processing.

If you plan to perform any image processing, even if it's just to a single track with no compositing, specify that track in requiredSourceTrackIDs instead.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download