tee and videomixer for side-by-side video on Raspberry Pi 3

[dariuskrail]

Hello all,

I am a gstreamer newbie.  I am trying to use the Raspberry Pi 3 to create a side-by-side (SBS) display video playback, where there is a video stream for the left and right side. Below is my pipeline that I have used. I split the decoded h264 stream with a tee and connected them to the sinks of the videomixer to have both streams displayed on the left and right side.

gst-launch-1.0 -v -m --gst-debug=3 \
    videomixer name=mix sink_0::xpos=0 sink_0::ypos=0 sink_1::xpos=760 sink_1::ypos=0 !  videoconvert ! glvideosink sync=false \
    filesrc location=bbb_sunflower_720p_24fps_equal.mp4 ! qtdemux name=demux \
    demux.video_0 ! h264parse ! omxh264dec ! tee name=t \
    t.src_0 ! queue ! mix.sink_0 \
    t.src_1 ! queue ! mix.sink_1 \
    demux.audio_0 ! queue ! aacparse ! faad ! alsasink

While I did get a SBS playback of the video, the FPS is really low at only about ~13FPS. If I set sync=true, it drops further to just ~10FPS, as indicated by fpsdisplaysink. I have tested with a single full HD video at 1920x1080, and the playback with audio was much smoother than a SBS playback of a 720x720 MP4 video with the above pipeline.

Could there be an issue with my pipeline or anything else?

Also to note, I am using a RPi3 with the GStreamer package 1.4.4. I have recompiled the gst-plugins-bad-1.4.4 source with GLES2 and EGL enabled for hardware acceleration for the glimagesink element.

[Arjen Veenhuizen]

I think you have to use the glvideomixer element instead of the videomixer element. The videomixer element requires frames to reside in system memory causing a lot of memcopies from system memory to video memory and vice-versa.

[dariuskrail]

I think you are right about the mem-copies between the system and video memory for videomixer. With the above pipeline, I get ~13fps but the CPU usage is rather high for gst-launch-1.0 at about 50% in as shown in top. Perhaps much of the CPU is used for memory copy operations here.

Then, I replaced the videomixer element with glvideomixer. The frame rate was actually even lower at only about ~6fps. However, the CPU usage was significantly reduced to about 15%. I am not sure why, but could the GPU here be occupied doing the video decoding with the omxh264dec element that it is not able to to process the glvideomixer element quick enough, hence the low fps and the lower CPU usage?

And is there anything else I could try to get a better fps rate?