1

I'm wondering if it's possible to achieve a better performance in converting the UIView into CVPixelBuffer.

My app converts a sequence of UIViews first into UIImages and then into CVPixelBuffers as shown below. In the end, I record all these images/frames into an AVAssetWriterInput and save the result as a movie file.

Thank you in advance!

Best, Aibek

func viewToImage(view: UIView) -> CGImage {
  let rect: CGRect = container.frame

  UIGraphicsBeginImageContextWithOptions(rect.size, true, 1)

  let context: CGContext = UIGraphicsGetCurrentContext()!
  view.layer.render(in: context)
  let img = UIGraphicsGetImageFromCurrentImageContext()

  UIGraphicsEndImageContext()

  return img!.cgImage
}

func imageToBuffer(image: CGImage) -> CVPixelBuffer? {
  let frameSize = CGSize(width: image.width, height: image.height)

  var pixelBuffer: CVPixelBuffer?
  let status = CVPixelBufferCreate(kCFAllocatorDefault, Int(frameSize.width), Int(frameSize.height), kCVPixelFormatType_32BGRA, nil, &pixelBuffer)

  if status != kCVReturnSuccess {
    return nil
  }

  CVPixelBufferLockBaseAddress(pixelBuffer!, CVPixelBufferLockFlags(rawValue: 0))
  let data = CVPixelBufferGetBaseAddress(pixelBuffer!)
  let rgbColorSpace = CGColorSpaceCreateDeviceRGB()
  let bitmapInfo = CGBitmapInfo(rawValue: CGBitmapInfo.byteOrder32Little.rawValue | CGImageAlphaInfo.premultipliedFirst.rawValue)
  let context = CGContext(data: data, width: Int(frameSize.width), height: Int(frameSize.height), bitsPerComponent: 8, bytesPerRow: CVPixelBufferGetBytesPerRow(pixelBuffer!), space: rgbColorSpace, bitmapInfo: bitmapInfo.rawValue)

  context?.draw(image, in: CGRect(x: 0, y: 0, width: image.width, height: image.height))

  CVPixelBufferUnlockBaseAddress(pixelBuffer!, CVPixelBufferLockFlags(rawValue: 0))

  return pixelBuffer
}
Improve this question

2 Answers 2

Reset to default
1

Converting the UIViews into MTLTextures and recording them into a video file using the Recorder provided by Mayo didn't increase the performance actually.

However, the recorder is able to write MTLTextures in real-time. That meant for me that I can re-write all the animations using Metal and use the recorder.

Improve this answer
Sign up to request clarification or add additional context in comments.

4 Comments

Long time no see. What do you mean that re-write all the animations?
True, it's been a while! I mean, I have to adopt all the processing stuff with Metal. Currently, I use UIViews to draw animated stuff into the CALayer. This differs from how Metal works. So, I need to re-write all the processing code to create primitives of all my elements and write shaders functions to properly display those primitives.
Good for you anyway that you figure out what you have to do :)
Good luck for all your codes
0

You'd better to see this. https://stackoverflow.com/a/61862728/13680955

In short, this sample converts UIView to MTLTexture in 12ms.

Sure you can use CVPixelBuffer directly, but I used MTLTexture to make video and no issue was on it.

If you are struggling with the performance, too slow or weird to use, try to do this.

With MTLTexture

import AVFoundation
import MetalKit

class VideoRecorder {
    let assetWriter: AVAssetWriter
    let assetWriterVideoInput: AVAssetWriterInput
    let assetWriterInputPixelBufferAdapter: AVAssetWriterInputPixelBufferAdaptor
    var recordingStartTime = TimeInterval(0)
    var recordingElapsedTime = TimeInterval(0)
    let url: URL = {
        let fileName = "exported_video.mp4"
        return FileManager.default.temporaryDirectory.appendingPathComponent(fileName)
    }()

    init(outputSize: CGSize) throws {
        if FileManager.default.fileExists(atPath: url.path) {
            try FileManager.default.removeItem(at: url)
        }
        let fileType: AVFileType = .mov
        assetWriter = try AVAssetWriter(outputURL: url, fileType: fileType)
        let mediaType: AVMediaType = .video
        let outputSettings: [String: Any] = [
            AVVideoCodecKey: AVVideoCodecType.h264,
            AVVideoWidthKey: outputSize.width,
            AVVideoHeightKey: outputSize.height
        ]
        assetWriterVideoInput = AVAssetWriterInput(mediaType: mediaType, outputSettings: outputSettings)
        assetWriterVideoInput.expectsMediaDataInRealTime = false
        let sourcePixelBufferAttributes: [String: Any] = [
            kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_32BGRA,
            kCVPixelBufferWidthKey as String: outputSize.width,
            kCVPixelBufferHeightKey as String: outputSize.height
        ]
        assetWriterInputPixelBufferAdapter = AVAssetWriterInputPixelBufferAdaptor(
            assetWriterInput: assetWriterVideoInput, sourcePixelBufferAttributes: sourcePixelBufferAttributes)
        assetWriter.add(assetWriterVideoInput)
    }

    private static func currentTimestampString() -> String {
        let date = Date()
        let dateFormatter = DateFormatter()
        dateFormatter.dateFormat = "yyyy-MM-dd HH:mm:ss"
        return dateFormatter.string(from: date)
    }

    public func start() {
        print("videoRecorder.start")
        assetWriter.startWriting()
        assetWriter.startSession(atSourceTime: .zero)
        recordingStartTime = CACurrentMediaTime()
    }

    public func cancel() {
        #if DEBUG
        print("videoRecorder.cancel")
        #endif

        assetWriterVideoInput.markAsFinished()
        assetWriter.cancelWriting()
    }

    public func finish(_ callback: @escaping () -> Void) {
        print("videoRecorder.finish")
        assetWriterVideoInput.markAsFinished()
        assetWriter.finishWriting {
            self.recordingElapsedTime = CACurrentMediaTime() - self.recordingStartTime
            print("videoRecorder.finish elapsedTime: \(self.recordingElapsedTime)")
            callback()
        }
    }

    private var pixelBuffer: CVPixelBuffer?

    public func writeFrame(texture: MTLTexture, at presentationTime: CMTime) {
        print("videoRecorder.writeFrame: \(presentationTime)")
        if pixelBuffer == nil {
            guard let pixelBufferPool = assetWriterInputPixelBufferAdapter.pixelBufferPool else {
                print("Pixel buffer asset writer input did not have a pixel buffer pool available;")
                print("cannot retrieve frame")
                return
            }

            var maybePixelBuffer: CVPixelBuffer?
            let status  = CVPixelBufferPoolCreatePixelBuffer(nil, pixelBufferPool, &maybePixelBuffer)
            if status != kCVReturnSuccess {
                print("Could not get pixel buffer from asset writer input; dropping frame...")
                return
            }
            pixelBuffer = maybePixelBuffer
            print("videoRecorder.writeFrame: pixelBuffer was created: \(String(describing: pixelBuffer))")
        }

        guard let pixelBuffer = pixelBuffer else {
            print("videoRecorder.writeFrame: NO pixelBuffer")
            return
        }

        writeFrame(texture: texture, at: presentationTime, with: pixelBuffer)
    }

    private func writeFrame(texture: MTLTexture, at presentationTime: CMTime, with pixelBuffer: CVPixelBuffer) {
        while !assetWriterVideoInput.isReadyForMoreMediaData {
            //
            print("NOT ready for more media data at: \(presentationTime)")
        }
        CVPixelBufferLockBaseAddress(pixelBuffer, [])
        let pixelBufferBytes = CVPixelBufferGetBaseAddress(pixelBuffer)!

        // Use the bytes per row value from the pixel buffer since its stride may be rounded up to be 16-byte aligned
        let bytesPerRow = CVPixelBufferGetBytesPerRow(pixelBuffer)
        let region = MTLRegionMake2D(0, 0, texture.width, texture.height)

        texture.getBytes(pixelBufferBytes, bytesPerRow: bytesPerRow, from: region, mipmapLevel: 0)

        assetWriterInputPixelBufferAdapter.append(pixelBuffer, withPresentationTime: presentationTime)

        CVPixelBufferUnlockBaseAddress(pixelBuffer, [])
    }
}
Improve this answer

5 Comments

Hey @마요mayo! Thanks for replying to this. I like your second suggestion more about recording the MTLTexture into a video file. However, will the converting UIViews to MTLTextures and recording (first suggestion) slow down the recording process? I'm also worried about re-writing the rendering stuff with Metal. Do you think it is worth that? How much performance increase can I expect from Metal? Best, Aibek
In simulator 13pro and real device 13pro, real time playing video, it didn't drop frame. Does your method performs very bad?
Rendering of UIViews with the approach above with video output 1080p @ 60fps and video duration of 2:30 takes around ~20-25 minutes.
oh.. then you can use this method. 1080p, 3:00, iPhone 13pro, it takes 3:00. Sure, I didn't used stackoverflow.com/a/61862728/13680955 this. But, with Metal, it doesn't take that long.
Hi Mayo! Just wanted to share that I've checked your approach. From my perspective, I didn't see any performance increase from converting the UIView to MTLTexture and recording it with your recorder. But I started researching how Metal works in hope that I can re-create all the animations in Metal and use Metal + Recorder to export videos. So far I could make some basic animations and tried to record textures as video frames. Looks like it performs fast enough, considering I write frames in realtime. Thanks for helping me out!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.