WebRTC音视频通话-实现GPUImage视频美颜滤镜效果iOS

WebRTC音视频通话-实现GPUImage视频美颜滤镜效果

在WebRTC音视频通话的GPUImage美颜效果图如下

可以看下
之前搭建ossrs服务，可以查看：https://blog.csdn.net/gloryFlow/article/details/132257196
之前实现iOS端调用ossrs音视频通话，可以查看：https://blog.csdn.net/gloryFlow/article/details/132262724
之前WebRTC音视频通话高分辨率不显示画面问题，可以查看：https://blog.csdn.net/gloryFlow/article/details/132262724
修改SDP中的码率Bitrate，可以查看：https://blog.csdn.net/gloryFlow/article/details/132263021

一、GPUImage是什么？

GPUImage是iOS上一个基于OpenGL进行图像处理的开源框架,内置大量滤镜,架构灵活,可以在其基础上很轻松地实现各种图像处理功能。

GPUImage中包含各种滤镜，这里我不会使用那么多，使用的是GPUImageLookupFilter及GPUImagePicture

GPUImage中有一个专门针对lookup table进行处理的滤镜函数GPUImageLookupFilter，使用这个函数就可以直接对图片进行滤镜添加操作了。代码如下

/**GPUImage中有一个专门针对lookup table进行处理的滤镜函数GPUImageLookupFilter，使用这个函数就可以直接对图片进行滤镜添加操作了。originalImg是你希望添加滤镜的原始图片@param image 原图@return 处理后的图片*/
+ (UIImage *)applyLookupFilter:(UIImage *)image lookUpImage:(UIImage *)lookUpImage {if (lookUpImage == nil) {return image;}UIImage *inputImage = image;UIImage *outputImage = nil;GPUImagePicture *stillImageSource = [[GPUImagePicture alloc] initWithImage:inputImage];//添加滤镜GPUImageLookupFilter *lookUpFilter = [[GPUImageLookupFilter alloc] init];//导入之前保存的NewLookupTable.png文件GPUImagePicture *lookupImg = [[GPUImagePicture alloc] initWithImage:lookUpImage];[lookupImg addTarget:lookUpFilter atTextureLocation:1];[stillImageSource addTarget:lookUpFilter atTextureLocation:0];[lookUpFilter useNextFrameForImageCapture];if([lookupImg processImageWithCompletionHandler:nil] && [stillImageSource processImageWithCompletionHandler:nil]) {outputImage= [lookUpFilter imageFromCurrentFramebuffer];}return outputImage;
}

这个需要lookUpImage，图列表如下

由于暂时没有整理demo的git

这里在使用applyLomofiFilter再试下效果

SDApplyFilter.m中的几个方法

+ (UIImage *)applyBeautyFilter:(UIImage *)image {GPUImageBeautifyFilter *filter = [[GPUImageBeautifyFilter alloc] init];[filter forceProcessingAtSize:image.size];GPUImagePicture *pic = [[GPUImagePicture alloc] initWithImage:image];[pic addTarget:filter];[pic processImage];[filter useNextFrameForImageCapture];return [filter imageFromCurrentFramebuffer];
}/**Amatorka滤镜 Rise滤镜,可以使人像皮肤得到很好的调整@param image image@return 处理后的图片*/
+ (UIImage *)applyAmatorkaFilter:(UIImage *)image
{GPUImageAmatorkaFilter *filter = [[GPUImageAmatorkaFilter alloc] init];[filter forceProcessingAtSize:image.size];GPUImagePicture *pic = [[GPUImagePicture alloc] initWithImage:image];[pic addTarget:filter];[pic processImage];[filter useNextFrameForImageCapture];return [filter imageFromCurrentFramebuffer];
}/**复古型滤镜,感觉像旧上海滩@param image image@return 处理后的图片*/
+ (UIImage *)applySoftEleganceFilter:(UIImage *)image
{GPUImageSoftEleganceFilter *filter = [[GPUImageSoftEleganceFilter alloc] init];[filter forceProcessingAtSize:image.size];GPUImagePicture *pic = [[GPUImagePicture alloc] initWithImage:image];[pic addTarget:filter];[pic processImage];[filter useNextFrameForImageCapture];return [filter imageFromCurrentFramebuffer];
}/**图像黑白化，并有大量噪点@param image 原图@return 处理后的图片*/
+ (UIImage *)applyLocalBinaryPatternFilter:(UIImage *)image
{GPUImageLocalBinaryPatternFilter *filter = [[GPUImageLocalBinaryPatternFilter alloc] init];[filter forceProcessingAtSize:image.size];GPUImagePicture *pic = [[GPUImagePicture alloc] initWithImage:image];[pic addTarget:filter];[pic processImage];[filter useNextFrameForImageCapture];return [filter imageFromCurrentFramebuffer];
}/**单色滤镜@param image 原图@return 处理后的图片*/
+ (UIImage *)applyMonochromeFilter:(UIImage *)image
{GPUImageMonochromeFilter *filter = [[GPUImageMonochromeFilter alloc] init];[filter forceProcessingAtSize:image.size];GPUImagePicture *pic = [[GPUImagePicture alloc] initWithImage:image];[pic addTarget:filter];[pic processImage];[filter useNextFrameForImageCapture];return [filter imageFromCurrentFramebuffer];
}

使用GPUImageSoftEleganceFilter复古型滤镜,感觉像旧上海滩效果图如下

使用GPUImageLocalBinaryPatternFilter图像黑白化效果图如下

使用GPUImageMonochromeFilter 效果图如下

二、WebRTC实现音视频通话中视频滤镜处理

之前实现iOS端调用ossrs音视频通话，可以查看：https://blog.csdn.net/gloryFlow/article/details/132262724
这个已经有完整的代码了，这里暂时做一下调整。

为RTCCameraVideoCapturer的delegate指向代理

- (RTCVideoTrack *)createVideoTrack {RTCVideoSource *videoSource = [self.factory videoSource];self.localVideoSource = videoSource;// 如果是模拟器if (TARGET_IPHONE_SIMULATOR) {if (@available(iOS 10, *)) {self.videoCapturer = [[RTCFileVideoCapturer alloc] initWithDelegate:self];} else {// Fallback on earlier versions}} else{self.videoCapturer = [[RTCCameraVideoCapturer alloc] initWithDelegate:self];}RTCVideoTrack *videoTrack = [self.factory videoTrackWithSource:videoSource trackId:@"video0"];return videoTrack;
}

实现RTCVideoCapturerDelegate的方法didCaptureVideoFrame

#pragma mark - RTCVideoCapturerDelegate处理代理
- (void)capturer:(RTCVideoCapturer *)capturer didCaptureVideoFrame:(RTCVideoFrame *)frame {
//    DebugLog(@"capturer:%@ didCaptureVideoFrame:%@", capturer, frame);// 调用SDWebRTCBufferFliter的滤镜处理RTCVideoFrame *aFilterVideoFrame;if (self.delegate && [self.delegate respondsToSelector:@selector(webRTCClient:didCaptureVideoFrame:)]) {aFilterVideoFrame = [self.delegate webRTCClient:self didCaptureVideoFrame:frame];}//  操作C 需要手动释放  否则内存暴涨
//      CVPixelBufferRelease(_buffer)//    拿到pixelBuffer
//        ((RTCCVPixelBuffer*)frame.buffer).pixelBufferif (!aFilterVideoFrame) {aFilterVideoFrame = frame;}[self.localVideoSource capturer:capturer didCaptureVideoFrame:frame];
}

之后调用SDWebRTCBufferFliter，实现滤镜效果。
实现将((RTCCVPixelBuffer *)frame.buffer).pixelBuffer进行渲染，这里用到了EAGLContext、CIContext

EAGLContext是OpenGL绘制句柄或者上下文，在绘制试图之前，需要指定使用创建的上下文绘制。
CIContext是用来渲染CIImage,将作用在CIImage上的滤镜链应用到原始的图片数据中。我这里需要将UIImage转换成CIImage。

具体代码实现如下

SDWebRTCBufferFliter.h

#import <Foundation/Foundation.h>
#import "WebRTCClient.h"@interface SDWebRTCBufferFliter : NSObject- (RTCVideoFrame *)webRTCClient:(WebRTCClient *)client didCaptureVideoFrame:(RTCVideoFrame *)frame;@end

SDWebRTCBufferFliter.m

#import "SDWebRTCBufferFliter.h"
#import <VideoToolbox/VideoToolbox.h>
#import "SDApplyFilter.h"@interface SDWebRTCBufferFliter ()
// 滤镜
@property (nonatomic, strong) EAGLContext *eaglContext;@property (nonatomic, strong) CIContext *coreImageContext;@property (nonatomic, strong) UIImage *lookUpImage;@end@implementation SDWebRTCBufferFliter- (instancetype)init
{self = [super init];if (self) {self.eaglContext = [[EAGLContext alloc] initWithAPI:kEAGLRenderingAPIOpenGLES3];self.coreImageContext = [CIContext contextWithEAGLContext:self.eaglContext options:nil];self.lookUpImage = [UIImage imageNamed:@"lookup_jiari"];}return self;
}- (RTCVideoFrame *)webRTCClient:(WebRTCClient *)client didCaptureVideoFrame:(RTCVideoFrame *)frame {CVPixelBufferRef pixelBufferRef = ((RTCCVPixelBuffer *)frame.buffer).pixelBuffer;//    CFRetain(pixelBufferRef);if (pixelBufferRef) {CIImage *inputImage = [CIImage imageWithCVPixelBuffer:pixelBufferRef];CGImageRef imgRef = [_coreImageContext createCGImage:inputImage fromRect:[inputImage extent]];UIImage *fromImage = nil;if (!fromImage) {fromImage = [UIImage imageWithCGImage:imgRef];}UIImage *toImage;toImage = [SDApplyFilter applyMonochromeFilter:fromImage];
//
//        if (toImage == nil) {
//            toImage = [SDApplyFilter applyLookupFilter:fromImage lookUpImage:self.lookUpImage];
//        } else {
//            toImage = [SDApplyFilter applyLookupFilter:fromImage lookUpImage:self.lookUpImage];
//        }if (toImage == nil) {toImage = fromImage;}CGImageRef toImgRef = toImage.CGImage;CIImage *ciimage = [CIImage imageWithCGImage:toImgRef];[_coreImageContext render:ciimage toCVPixelBuffer:pixelBufferRef];CGImageRelease(imgRef);//必须释放fromImage = nil;toImage = nil;ciimage = nil;inputImage = nil;}RTCCVPixelBuffer *rtcPixelBuffer =[[RTCCVPixelBuffer alloc] initWithPixelBuffer:pixelBufferRef];RTCVideoFrame *filteredFrame =[[RTCVideoFrame alloc] initWithBuffer:rtcPixelBufferrotation:frame.rotationtimeStampNs:frame.timeStampNs];return filteredFrame;
}@end

至此可以看到在WebRTC音视频通话中GPUImage视频美颜滤镜的具体效果了。

三、小结

WebRTC音视频通话-实现GPUImage视频美颜滤镜效果。主要用到GPUImage处理视频画面CVPixelBufferRef，将处理后的CVPixelBufferRef生成RTCVideoFrame，通过调用localVideoSource中实现的didCaptureVideoFrame方法。内容较多，描述可能不准确，请见谅。

本文地址：https://blog.csdn.net/gloryFlow/article/details/132265842

学习记录，每天不停进步。