V4L2 Stateless Codec Support

Embedded Linux Conference Europe,
Microsoft Sponsor Suite / the bar


  1. Alexandre Courbot
  2. Chris Healy
  3. Ezequiel Garcia
  4. Hans Verkuil
  5. Kieran Bingham
  6. Laurent Pinchart
  7. Maxime Ripard
  8. Mauro Carvalho Chehab
  9. Nicolas Dufresne
  10. Niklas Söderlund
  11. Philip Zabell
  12. Sakari Ailus
  13. Tomasz Figa
  14. Victor Jáquez

Nicolas reported an issue with buffer management. V4L2 decouples buffers from buffer layouts, assuming that all buffers that will be used on a queue will have the same layout. This makes importing buffers difficult.

Tomasz proposed using the format passed to VIDIOC_CREATE_BUFS to communicate the buffer format, and keep that format associated with the buffer internally. An ioctl to delete buffers selectively would then be needed.

Could we take a stream parser from an existing project (such as gstreamer), and make it a standalone component, that could then be used by different implementations (gstreamer, libva, ...) ?

Maxime said that VLC recently released a new parser meant to be a library which could be useful. Nicolas believes we need a parser library split out from any other code base, to avoid pulling in other libraries, and being able to maintain it separately from gstreamer, VLC or any other project. One or several people need to step up and maintain a parser, and it might be difficult to find volunteers.

Many existing parsers are meant to be used internally in the project they're part of. Nicolas explained that the ffmpeg parser, for instance, was used by gstreamer, but then got dropped, as it only offered the APIs that ffmpeg needed, but not the APIs needed by gstreamer.

Does ChromeOS have its own in-house parser ? Alexandre believes it does, but isn't sure whether it was initially developed internally.

There's also a language problem: ffmpeg and gstreamer are written in C, the ChomeOS parser in C++, VLC is moving to Rust. What do we pick, how do we ensure interoperability ?

A parser isn't just about extracting a few tables, it's really part of the decoder. Decisions such as error concealment (i.e. what do we do if a frame is missing from the stream ?) are part of the userspace codec component. Applications may have different requirements, so this is possibly best left to applications, that may delegate this to libva.

As a short-term solution, implementing a generic libva on top of the V4L2 stateless codec kernel API would give support for our codecs to all applications that currently use libva, including applications based on ffmpeg and gstreamer as they both have a libva backend (gstreamer uses libva directly, it doesn't go through ffmpeg to do so).

70% of applications pick one stack: ffmpeg. It has a software codec API almost identical to the V4L2 stateful codecs. It would be trivial for applications to switch to V4L2 natively.

Mauro would like us to approach Intel to explain our plans with libva, to avoid later surprises.

AV1 is a complex codec. It's a successor to VP9/VP10 requiring more tables and parameters, which will come through soon.

libva is hosted on freedesktop. Should we host the libva-v4l2-codec backend there, or host it on linuxtv.org ? Hans would prefer linuxtv as it's "closer to our kernel implementation".

libva loads backends in order, and picks the first one that reports it can support the platform. There is also an environment variable that can specify a backend. Ezequiel enquired how to support platforms that would have multiple hardware codecs. libva doesn't seem to support this at the moment. Nicolas reported that there's an Intel SoC that have both an Intel graphics core and a Vega64 graphics core that both have a codec.

Hans said that a platform that expose multiple codecs will likely be used for specialized applications, and requiring those to implement codec support directly is acceptable. Our main focus should be to support the common case.

NVidia is following our  progress and is interested in using the V4L2 stateless API. On the userspace side, vdpau is pretty much dead, they have moved to nvdec. Usage of OMX is phasing out.

On RaspberryPi OMX is going away.

bootlin has developed a debugging tool called v4l2-request-test (https://github.com/bootlin/v4l2-request-test) that has been very useful to debug the codec driver without going through the full userspace stack. This is worth mentioning and integrating.