Video mixing acceleration using gstreamer with VAAPI

jbolduc01

Hello,

We are using a UpSquared board with an Intel Atom E3950. We are also using the system below.
Linux upsquared 5.3.0-42-generic #34~18.04.1-Ubuntu SMP Fri Feb 28 13:42:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

We would like to capture uncompressed video sources and mix them in order to output the result through HDMI and DisplayPort. We intend to use gstreamer in order to capture from UVC devices and would like to use VAAPI in order to do the mixing. Here is the custom pipeline I am using:
gst-launch-1.0 compositor name=comp \
sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1920 sink_0::height=1080 sink_0::zorder=0 \
sink_1::xpos=1920 sink_1::ypos=0 sink_1::width=1920 sink_1::height=1080 sink_1::zorder=0 \
sink_2::xpos=0 sink_2::ypos=1080 sink_2::width=1920 sink_2::height=1080 sink_2::zorder=0 \
sink_3::xpos=1920 sink_3::ypos=1080 sink_3::width=1920 sink_3::height=1080 sink_3::zorder=0 \
! queue ! fpsdisplaysink video-sink="vaapisink fullscreen=true" sync=false text-overlay=false \
v4l2src device=/dev/video0 io-mode=dmabuf ! 'video/x-raw,width=1920,height=1080,framerate=30/1' ! queue ! comp. \
v4l2src device=/dev/video8 io-mode=dmabuf ! 'video/x-raw,width=1920,height=1080,framerate=30/1' ! queue ! comp. \
v4l2src device=/dev/video2 io-mode=dmabuf ! image/jpeg,width=1920,height=1080,framerate=30/1 ! vaapijpegdec ! queue ! comp. \
v4l2src device=/dev/video6 io-mode=dmabuf ! image/jpeg,width=1920,height=1080,framerate=30/1 ! vaapijpegdec ! queue ! comp. -v

The result of this pipeline leads video stutters and I am not able to render video at 30fps. It renders a non stable video.

If I do the same setup but each video is rendered on 4 separate windows, the video is really more stable. It seems to be related to the "compositor" structure that is not hardware accelerated and is managed by CPU.

I also tried videomixer and glvideomixer and these one seems to have problems too.

Is there a way to mix 4 video sources and use VAAPI or any hardware accelerated structure in order to output it to a display?

jbolduc01

After thinking about it, I think all the problems we have is related to X11. Is there a way to use vaapisink without X11? Some other platform like iMX6 or Nvidia offers the possibility to render video as an overlay on the display. Even if we are in console-only mode (where no graphical interface is activated) we have the possibility to render video.
Is it possible with the UpSquared? Even if I have a Yocto image with console-only mode, all gstreamer video renderers are removed from image as well as the vaapi plugins.