Beyond_Infinity Beyond_Infinity - 12 days ago 7
iOS Question

Playing audio while applying video filter in GPUImage

I've googled this question but did not find any solution.

So, my problem is,
I'm applying video filter using

GPUImage
to a video file. At same time, sound of that video is not playing. I know that sound playing is not supported in
GPUImage
while applying filters.

So, how can I achieve that?

Answer

You can achieve sound playing by adding support for AVPlayer like this.

  1. In GPUImageMovie.h,

    @property(readwrite, nonatomic) BOOL playSound;

  2. In GPUImageMovie.m, update @interface GPUImageMovie () like this.

    @interface GPUImageMovie() <AVPlayerItemOutputPullDelegate> 
    {
        // Add this
        BOOL hasAudioTrack;
    
        .........
        ........
    
       // Add all below three lines
       AVPlayer *theAudioPlayer;
       CFAbsoluteTime startActualFrameTime;
       CGFloat currentVideoTime;
    }
    
  3. After that, in -(void)startProcessing method, add following lines.

    - (void)startProcessing
    {
       // Add this
       currentVideoTime = 0.0f;
    
       .....
       .....
    
       // Add this
       if (self.playSound)
       {
           [self setupSound];
       }
    
       GPUImageMovie __block *blockSelf = self;
    
       [inputAsset loadValuesAsynchronouslyForKeys:[NSArray arrayWithObject:@"tracks"] completionHandler: ^{
        runSynchronouslyOnVideoProcessingQueue(^{
    
                .........
                .........
    
                // Add this
                startActualFrameTime = CFAbsoluteTimeGetCurrent() - currentVideoTime;
    
                .......
                .......
            });
       }];
    }
    
  4. Again, In -(AVAssetReader*)createAssetReader update it,

    - (AVAssetReader*)createAssetReader
    {
        .......
        .......  
        BOOL shouldRecordAudioTrack = (([audioTracks count] > 0) &&  (self.audioEncodingTarget != nil));
    
        // Add this
        hasAudioTrack = ([audioTracks count] > 0);
    
        .........
    }
    
  5. In -(void)processAsset,

    -(void)processAsset
    {
    
       .......
       .......
    
       if ([reader startReading] == NO)
       {
          NSLog(@"Error reading from file at URL: %@", self.url);
          return;
       }
    
       // Add this
       if (self.playSound && hasAudioTrack)
       {
          [theAudioPlayer seekToTime:kCMTimeZero];
          [theAudioPlayer play];
       }
    
       .........
       .........
    }
    
  6. In -(void)endProcessing,

    -(void) endProcessing
    {
       .........
       .........
    
       if (self.playerItem && (displayLink != nil))
       {
          [displayLink invalidate]; // remove from all run loops
          displayLink = nil;
       } 
    
       // Add this
       if (theAudioPlayer != nil)
       {
          [theAudioPlayer pause];
          [theAudioPlayer seekToTime:kCMTimeZero];
       }
    
       .........
       .........
    }
    
  7. Add new method -(void) setupSound at last,

    // Add this
    - (void) setupSound
    {
       if (theAudioPlayer != nil)
       {
           [theAudioPlayer pause];
           [theAudioPlayer seekToTime:kCMTimeZero];
           theAudioPlayer = nil;
       }
    
       theAudioPlayer = [[AVPlayer alloc] initWithURL:self.url];
    }
    
  8. At last, when you create your GPUImageMovie object, don't forget to set playSound flag to YES.

    Like this : movieFile.playSound = YES;

    Hope this helps.