Skip to main content

Pixel Formats

What are Pixel Formats?​

A Camera's video pipeline operates in a specific pixel format which specifies how the pixels are laid out in a memory buffer.

If you are simply recording videos (video={true}), the most efficient pixel format will be automatically chosen for you, and buffer compression will be enabled if available.

If you are using Frame Processors, it is important to understand what pixel format you are using.

The most commonly known pixel format is RGB, which lays out pixels in 3 channels (R, G and B), and each channel has a value ranging from 0 to 255 (8-byte), making it a total of 24-bytes per pixel:

RGBRGBRGBRGBRGBRGB

Cameras however don't operate in RGB, they use YUV instead. Instead of storing a color value for each channel, it stores the brightness ("luma") in it's first channel (Y), and the colors ("chroma") in the U and V channels. This is much closer to what a Camera hardware actually sees, as it is essentially a light sensor. Also, it is more memory efficient, since the UV channels are usually half the size of the Y channel:

YYYYUVYYYYUVYYYYUV

In VisionCamera, pixel formats are abstracted under a simple PixelFormat API with two possible values:

  • yuv: The YUV (often 4:2:0, 8-bit per channel) pixel format.
  • rgb: An RGB (often BGRA, 8-bit per channel) pixel format.

Every Camera device can either stream in yuv or rgb, but rgb comes with an additional overhead and sometimes even needs to be converted from yuv first.

function App() {
  const frameProcessor = useFrameProcessor((frame) => {
    'worklet'
    console.log(frame.pixelFormat) // <-- "rgb"
  })

  return (
    <Camera
      style={StyleSheet.absoluteFill}
      pixelFormat="rgb"
      frameProcessor={frameProcessor}
    />
  )
}

As a general tip, try to always use YUV if possible. If you have some specific models (e.g. Face Detectors), try converting them to YUV (4:2:0) instead of trying to run your Camera in RGB, as the conversion beforehand will be worth the effort.

info

A 4k Frame in yuv is roughly 12 MB, while a 4k Frame in rgb is roughly 31 MB.

HDR​

When HDR is enabled, a different pixel format (10-bit instead of 8-bit) will be chosen. Make sure your Frame Processor can handle these formats, or disable HDR. See "Understanding YpCbCr Image Formats" for more information.

Instead of kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange, it uses kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange, same for full-range.

Buffer Compression​

Buffer Compression is automatically enabled if you are not using a Frame Processor. If you are using a Frame Processor, buffer compression will be turned off, as it essentially uses a different format than YUV. See "Understanding YpCbCr Image Formats" for more information.

Instead of kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange, it uses kCVPixelFormatType_Lossy_420YpCbCr8BiPlanarVideoRange, same for full-range.