Metal学习笔记03 渲染视频原始数据

发表于 | 分类于

上篇笔记

这次来使用Metal渲染视频数据，渲染视频实际上就是不断地把视频数据变成MTLTexture，再使用renderEncoder去渲染显示，本文会利用iOS设备采集RGBA、NV12数据，然后进行渲染显示。

渲染BGRA数据

创建一个layer类型为CAMetalLayer的view

1
2
3
4
+(Class)layerClass
{
    return [CAMetalLayer class];
}

设置layer的属性

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

@property(nonatomic,strong)id<MTLDevice>device;
@property(nonatomic,strong)CAMetalLayer * metalLayer;
...
-(void)prepareLayer{

    self.device = MTLCreateSystemDefaultDevice();
    
    
    self.metalLayer = (CAMetalLayer*)self.layer;
    
    self.metalLayer.pixelFormat = MTLPixelFormatBGRA8Unorm;
    self.metalLayer.framebufferOnly = YES;
    self.metalLayer.drawableSize = self.bounds.size;
    self.metalLayer.device = self.device;
    
}

这里主要是获取了默认的设备（GPU）然后赋值给layer的device属性，然后设置一些渲染格式和尺寸的属性。

准备管道状态

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
@property(nonatomic,strong)id<MTLRenderPipelineState> pipelineState;

...
-(void)preparePipelineState
{
    id<MTLLibrary>library = [self.device newDefaultLibrary];
    id<MTLFunction>vertexFunc = [library newFunctionWithName:@"texture_vertex"];
    id<MTLFunction>fragmentFunc = [library newFunctionWithName:@"texture_fragment"];
    
    MTLRenderPipelineDescriptor*descriptor = [[MTLRenderPipelineDescriptor alloc]init];
    descriptor.vertexFunction = vertexFunc;
    descriptor.fragmentFunction = fragmentFunc;
    descriptor.colorAttachments[0].pixelFormat = MTLPixelFormatBGRA8Unorm;
    
    id<MTLRenderPipelineState> pipelineState = [self.device newRenderPipelineStateWithDescriptor:descriptor error:nil];
    
    self.pipelineState = pipelineState;
    
}


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
#include <metal_stdlib>
using namespace metal;
struct VertexOut
{
    float4 position [[position]];
    float2 textureCoordinate;
};
vertex VertexOut texture_vertex (
    constant float4*vertex_array[[buffer(0)]],
    constant float2*textureCoord_array[[buffer(1)]],
    unsigned int vid[[vertex_id]]){

    VertexOut outputVertices;

    outputVertices.position = vertex_array[vid];
    outputVertices.textureCoordinate = textureCoord_array[vid];

    return outputVertices;
}

fragment float4 texture_fragment(VertexOut fragmentInput [[stage_in]],
                                 texture2d<float> inputTexture [[texture(0)]]) {
    constexpr sampler quadSampler;
    float4 color = inputTexture.sample(quadSampler, fragmentInput.textureCoordinate);

    return color;
}

这里创建了管道状态，设置了两个着色器函数，和上一篇笔记渲染图片纹理的函数一致。

创建命令队列

1
2
3
4
5
6
7
@property(nonatomic,strong)id<MTLCommandQueue>commandQueue;
...
-(void)prepareCommandQueue
{
    id<MTLCommandQueue>commandQueue = [self.device newCommandQueue];
    self.commandQueue = commandQueue;
}

这里创建一个命令队列，用于渲染时取得可用的command给GPU渲染命令。

渲染方法

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
@property(nonatomic,strong)id<MTLTexture>texture;

@property(nonatomic,strong)MTLTextureDescriptor*textureDes;
@property(nonatomic,assign)int  textureHeight;
@property(nonatomic,assign)int  textureWidth;
...
-(void)renderRGBAWith:(uint8_t*)RGBBuffer width:(int)width height:(int)height
{
    if (!self.textureDes || self.textureWidth != width ||self.textureHeight != height) {
        self.textureDes = [MTLTextureDescriptor texture2DDescriptorWithPixelFormat:MTLPixelFormatBGRA8Unorm width:width height:height mipmapped:NO];
    }
	//创建纹理
    self.texture = [self.device newTextureWithDescriptor:self.textureDes];
    MTLRegion region = MTLRegionMake2D(0, 0, width, height);
	//将数据写入纹理
    [self.texture replaceRegion:region mipmapLevel:0 withBytes:RGBBuffer bytesPerRow:width*4];
    
    
    
    id<MTLCommandBuffer>commandBuffer = [self.commandQueue commandBuffer];
    

    id<CAMetalDrawable>drawable = [self.metalLayer nextDrawable];
    
    MTLRenderPassDescriptor*renderPassDes = [[MTLRenderPassDescriptor alloc]init];
    
    renderPassDes.colorAttachments[0].texture = [drawable texture];
    renderPassDes.colorAttachments[0].loadAction = MTLLoadActionClear;
    renderPassDes.colorAttachments[0].clearColor = MTLClearColorMake(1.0, 1.0, 1.0, 1.0);
    renderPassDes.colorAttachments[0].storeAction = MTLStoreActionStore;
    
    id<MTLRenderCommandEncoder>renderEncoder = [commandBuffer renderCommandEncoderWithDescriptor:renderPassDes];
    [renderEncoder setRenderPipelineState:self.pipelineState];
    
    
    float vertexArray[] = {
        -1.0, -1.0,0, 1.0,
        1.0, -1.0, 0, 1.0,
        -1.0,  1.0, 0, 1.0,
        1.0,  1.0, 0, 1.0,
    };
    
    
    //顶点坐标buffer
    id<MTLBuffer>vertexBuffer = [self.device newBufferWithBytes:vertexArray length:sizeof(vertexArray) options:MTLResourceCPUCacheModeDefaultCache];
    
    [renderEncoder setVertexBuffer:vertexBuffer offset:0 atIndex:0];
    
    
    float textureCoord[] = {
        0,1,
        1,1,
        0,0,
        1,0
    };
	//纹理坐标buffer
    id<MTLBuffer>textureCoordBuffer = [self.device newBufferWithBytes:textureCoord length:sizeof(textureCoord) options:MTLResourceCPUCacheModeDefaultCache];
    
    [renderEncoder setVertexBuffer:textureCoordBuffer offset:0 atIndex:1];
    [renderEncoder setFragmentTexture:self.texture atIndex:0];
    
    [renderEncoder drawPrimitives:MTLPrimitiveTypeTriangleStrip vertexStart:0 vertexCount:4];
    
    
    [commandBuffer presentDrawable:drawable];
    
    [renderEncoder endEncoding];
    [commandBuffer commit];
    
    
    
}

这里根据传入的宽高判断是否需要新创建纹理描述，然后根据纹理描述创建纹理，将数据写入。然后创建MTLRenderPassDescriptor，MTLRenderPassDescriptor描述一系列attachments的值，类似GL的FrameBuffer，同时也用来创建MTLRenderCommandEncoder。 然后创建顶点和纹理坐标buffer，然后设置给renderEncoder，然后渲染显示。

将数据传入renderView

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
-(void)captureOutput:(AVCaptureOutput *)output didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection
{
    CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
    
    if(CVPixelBufferLockBaseAddress(imageBuffer, 0) == kCVReturnSuccess)
    {
                UInt8 *rgbBuffer = (UInt8 *)CVPixelBufferGetBaseAddressOfPlane(imageBuffer, 0);
                size_t width = CVPixelBufferGetWidth(imageBuffer);
                size_t height = CVPixelBufferGetHeight(imageBuffer);
        
        [self.renderView renderRGBAWith:rgbBuffer width:width height:height];
        
        
    }
    CVPixelBufferUnlockBaseAddress(imageBuffer, 0);
}

运行后即可看到渲染的视频。

遇到的问题

一开始我在采集之后才将renderView添加到view上，导致崩溃在

1
id<CAMetalDrawable>drawable = [self.metalLayer nextDrawable];

原因是drawableSize为0，无法取出drawable。

渲染NV12

使用两个纹理

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
@property(nonatomic,strong)MTLTextureDescriptor*textureYDes;
@property(nonatomic,strong)MTLTextureDescriptor*textureUVDes;

@property(nonatomic,strong)id<MTLTexture>textureY;
@property(nonatomic,strong)id<MTLTexture>textureUV;

...
-(void)renderNV12With:(uint8_t*)yBuffer uvBuffer:(uint8_t*)uvBuffer width:(int)width height:(int)height
	if (!self.textureYDes || self.textureWidth != width ||self.textureHeight != height) {
			self.textureYDes = [MTLTextureDescriptor texture2DDescriptorWithPixelFormat:MTLPixelFormatR8Unorm width:width height:height mipmapped:NO];
		}
		self.textureY = [self.device newTextureWithDescriptor:self.textureYDes];
		MTLRegion region = MTLRegionMake2D(0, 0, width, height);
		[self.textureY replaceRegion:region mipmapLevel:0 withBytes:yBuffer bytesPerRow:width];

		if (!self.textureUVDes || self.textureWidth != width ||self.textureHeight != height) {
			self.textureUVDes = [MTLTextureDescriptor texture2DDescriptorWithPixelFormat:MTLPixelFormatRG8Unorm width:width/2 height:height/2 mipmapped:NO];
		}

		region = MTLRegionMake2D(0, 0, width/2, height/2);
		self.textureUV = [self.device newTextureWithDescriptor:self.textureUVDes];
		[self.textureUV replaceRegion:region mipmapLevel:0 withBytes:uvBuffer bytesPerRow:width];
		...
}

由于NV12有两个通道，因此需要传入两个纹理。这里使用两个MTLTextureDescriptor创建，y分量的纹理因为只有y值，因此使用MTLPixelFormatR8Unorm格式，表示一个八位的通道。uv分量使用MTLPixelFormatRG8Unorm，有两个八位通道。

修改片段着色器

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
fragment float4 nv12_fragment(VertexOut fragmentInput [[stage_in]],
                              texture2d<float> textureY [[texture(0)]]，
                               texture2d<float> textureUV [[texture(1)]]) {
    constexpr sampler quadSampler;
    
    float y = textureY.sample(quadSampler,fragmentInput.textureCoordinate).r;
    float u = textureUV.sample(quadSampler, fragmentInput.textureCoordinate).r - 0.5;
    
    float v = textureUV.sample(quadSampler, fragmentInput.textureCoordinate).g - 0.5;
    
    float r = y +             1.402 * v;
    float g = y - 0.344 * u - 0.714 * v;
    float b = y + 1.772 * u;
    
    float4 color = float4(r,g,b,1.0);
    
    return color;
}

这里增加一个纹理参数，从textureY中取出y，textureUV分别取出uv，利用公式转换成rgb值。

修改着色器程序

1
2
//    id<MTLFunction>fragmentFunc = [library newFunctionWithName:@"texture_fragment"];
    id<MTLFunction>fragmentFunc = [library newFunctionWithName:@"nv12_fragment"];

给renderEncoder传入两个纹理

1
2
 	[renderEncoder setFragmentTexture:self.textureY atIndex:0];
    [renderEncoder setFragmentTexture:self.textureUV atIndex:1];

调用渲染方法

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
-(void)captureOutput:(AVCaptureOutput *)output didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection
{
    CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
    
    if(CVPixelBufferLockBaseAddress(imageBuffer, 0) == kCVReturnSuccess)
    {
                UInt8 *yBuffer = (UInt8 *)CVPixelBufferGetBaseAddressOfPlane(imageBuffer, 0);
            UInt8 *uvBuffer = (UInt8 *)CVPixelBufferGetBaseAddressOfPlane(imageBuffer, 1);
                size_t width = CVPixelBufferGetWidth(imageBuffer);
                size_t height = CVPixelBufferGetHeight(imageBuffer);
        
        
        size_t linesize_yu = CVPixelBufferGetBytesPerRowOfPlane(imageBuffer, 1);
//        [self.renderView renderRGBAWith:rgbBuffer width:width height:height];
        [self.renderView renderNV12With:yBuffer uvBuffer:uvBuffer width:width height:height];
        
        
    }
    CVPixelBufferUnlockBaseAddress(imageBuffer, 0);
}

运行程序，即可看到采集的画面。

使用CoreVideo相关函数创建纹理

使用CVMetalTextureCacheRef纹理缓存，CVMetalTextureCacheCreateTextureFromImage函数可以直接从CVPixelBufferRef中获取CVMetalTextureRef，然后在从CVMetalTextureRef中获得MTLTexture。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
@property (nonatomic, assign) CVMetalTextureCacheRef textureCache;
...
//创建CVMetalTextureCacheRef
CVMetalTextureCacheCreate(NULL, NULL, self.device, NULL, &_textureCache);

...
		id<MTLTexture> textureY = nil;
        size_t width = CVPixelBufferGetWidthOfPlane(pixelBuffer, 0);
        size_t height = CVPixelBufferGetHeightOfPlane(pixelBuffer, 0);
        MTLPixelFormat pixelFormat = MTLPixelFormatR8Unorm;
	
        CVMetalTextureRef texture = NULL; 
		//将pixelBuffer的0通道写入CVMetalTextureRef
        CVReturn status = CVMetalTextureCacheCreateTextureFromImage(NULL, self.textureCache, pixelBuffer, NULL, pixelFormat, width, height, 0, &texture);
        if(status == kCVReturnSuccess)
        {
            textureY = CVMetalTextureGetTexture(texture); // 转成Metal用的纹理
            CFRelease(texture);
        }

之后就和上面的纹理处理一致了。

总结

感觉使用Metal渲染的视频的代码比OpenGL的实现要容易些，基本就是渲染图片纹理稍作修改即可实现，而且MetalKit还提供了一些直接将AVFoundation数据转换为Metal可以处理的类型的功能，使得开发更加简单。

demo地址