Combining 2 Video Streams Side by Side, Getting Choppy Results

[Jim Ruxton]

I am trying to combine 2 video streams. One from my laptop camera and one from an external webcam. I'm using the following pipeline and getting very choppy results. Neither my CPU or GPU seem to be working hard. Any ideas how to make this less choppy?

./gst-launch-1.0 v4l2src device=/dev/video4 ! videoscale ! videoconvert ! video/x-raw,format=YUY2, framerate=30/1, width=640, height=480! alpha alpha=1 ! videobox border-alpha=0 left=-640 ! 
videomixer name=Mix ! videoconvert ! autovideosink v4l2src device=/dev/video2 ! videoscale ! videoconvert ! video/x-raw, format=YUY2, framerate=30/1, width=640, height=480! alpha alpha=1 ! videobox border-alpha=0 right=-640 ! Mix.

Thanks,

Jim

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Jim Ruxton]

Just discovered that putting sync=false after autovideosink solved my problem.

Jim

On 2021-02-16 2:50 a.m., Jim Ruxton wrote:

I am trying to combine 2 video streams. One from my laptop camera and one from an external webcam. I'm using the following pipeline and getting very choppy results. Neither my CPU or GPU seem to be working hard. Any ideas how to make this less choppy?

./gst-launch-1.0 v4l2src device=/dev/video4 ! videoscale ! videoconvert ! video/x-raw,format=YUY2, framerate=30/1, width=640, height=480! alpha alpha=1 ! videobox border-alpha=0 left=-640 ! 
videomixer name=Mix ! videoconvert ! autovideosink v4l2src device=/dev/video2 ! videoscale ! videoconvert ! video/x-raw, format=YUY2, framerate=30/1, width=640, height=480! alpha alpha=1 ! videobox border-alpha=0 right=-640 ! Mix.

Thanks,

Jim

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Ian Davidson]

I don't see any queue in your pipeline. A queue causes the pipeline to have a new thread - so all of your pipeline is one thread and is therefore only using one processor.  Experiment with putting a queue after each videoscale.

On 16/02/2021 09:02, Jim Ruxton wrote:

Just discovered that putting sync=false after autovideosink solved my problem.

Jim

On 2021-02-16 2:50 a.m., Jim Ruxton wrote:

I am trying to combine 2 video streams. One from my laptop camera and one from an external webcam. I'm using the following pipeline and getting very choppy results. Neither my CPU or GPU seem to be working hard. Any ideas how to make this less choppy?

./gst-launch-1.0 v4l2src device=/dev/video4 ! videoscale ! videoconvert ! video/x-raw,format=YUY2, framerate=30/1, width=640, height=480! alpha alpha=1 ! videobox border-alpha=0 left=-640 ! 
videomixer name=Mix ! videoconvert ! autovideosink v4l2src device=/dev/video2 ! videoscale ! videoconvert ! video/x-raw, format=YUY2, framerate=30/1, width=640, height=480! alpha alpha=1 ! videobox border-alpha=0 right=-640 ! Mix.

Thanks,

Jim

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Mathieu Duponchelle]

That is a good point, you should also use compositor instead of videomixer,
especially for live streams. In any case, using sync=false on the sinks is not
a desirable solution, rather an admission of defeat :)

Cheers,

On 2/16/21 11:20 AM, Ian Davidson wrote:

I don't see any queue in your pipeline. A queue causes the pipeline to have a new thread - so all of your pipeline is one thread and is therefore only using one processor.  Experiment with putting a queue after each videoscale.

On 16/02/2021 09:02, Jim Ruxton wrote:

Just discovered that putting sync=false after autovideosink solved my problem.

Jim

On 2021-02-16 2:50 a.m., Jim Ruxton wrote:

I am trying to combine 2 video streams. One from my laptop camera and one from an external webcam. I'm using the following pipeline and getting very choppy results. Neither my CPU or GPU seem to be working hard. Any ideas how to make this less choppy?

./gst-launch-1.0 v4l2src device=/dev/video4 ! videoscale ! videoconvert ! video/x-raw,format=YUY2, framerate=30/1, width=640, height=480! alpha alpha=1 ! videobox border-alpha=0 left=-640 ! 
videomixer name=Mix ! videoconvert ! autovideosink v4l2src device=/dev/video2 ! videoscale ! videoconvert ! video/x-raw, format=YUY2, framerate=30/1, width=640, height=480! alpha alpha=1 ! videobox border-alpha=0 right=-640 ! Mix.

Thanks,

Jim

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le mardi 16 février 2021 à 02:50 -0500, Jim Ruxton a écrit :

I am trying to combine 2 video streams. One from my laptop camera and one from an external webcam. I'm using the following pipeline and getting very choppy results. Neither my CPU or GPU seem to be working hard. Any ideas how to make this less choppy?

./gst-launch-1.0 v4l2src device=/dev/video4 ! videoscale ! videoconvert ! video/x-raw,format=YUY2, framerate=30/1, width=640, height=480! alpha alpha=1 ! videobox border-alpha=0 left=-640 ! 
videomixer name=Mix ! videoconvert ! autovideosink v4l2src device=/dev/video2 ! videoscale ! videoconvert ! video/x-raw, format=YUY2, framerate=30/1, width=640, height=480! alpha alpha=1 ! videobox border-alpha=0 right=-640 ! Mix.

First recommendation, move away from videomixer and use compositor (videomixer is just a backward shm on top of compositor). As you have a live pipeline, you should release the composition pressure by configurating a latency on the compositor element.  The latency is in nano-second, one of two frames of latency should be fine in general, but the default is none, and would only worked with perfectly synched sources which has accurate latency (v4l2src does not have accurate latency, it simply claims 1 frame, alway).

Second recommendation, consider adding a queue before you display sink, this will improve the timeout logic inside the compositor, by giving it a bit more freedome (with thread seperation).

Thanks,

Jim

_______________________________________________

gstreamer-devel mailing list

[hidden email]

https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Jim Ruxton]

Thanks so much for the tips. I first put queue in after the videoscales but it didn't seem to help. Using compositor seemed to fix it. I'm going to have to read up on compositor more to understand why that works.

Cheers,

Jim

On 2021-02-16 9:14 a.m., Mathieu Duponchelle wrote:

That is a good point, you should also use compositor instead of videomixer,
especially for live streams. In any case, using sync=false on the sinks is not
a desirable solution, rather an admission of defeat :)

Cheers,

On 2/16/21 11:20 AM, Ian Davidson wrote:

I don't see any queue in your pipeline. A queue causes the pipeline to have a new thread - so all of your pipeline is one thread and is therefore only using one processor.  Experiment with putting a queue after each videoscale.

On 16/02/2021 09:02, Jim Ruxton wrote:

Just discovered that putting sync=false after autovideosink solved my problem.

Jim

On 2021-02-16 2:50 a.m., Jim Ruxton wrote:

I am trying to combine 2 video streams. One from my laptop camera and one from an external webcam. I'm using the following pipeline and getting very choppy results. Neither my CPU or GPU seem to be working hard. Any ideas how to make this less choppy?

./gst-launch-1.0 v4l2src device=/dev/video4 ! videoscale ! videoconvert ! video/x-raw,format=YUY2, framerate=30/1, width=640, height=480! alpha alpha=1 ! videobox border-alpha=0 left=-640 ! 
videomixer name=Mix ! videoconvert ! autovideosink v4l2src device=/dev/video2 ! videoscale ! videoconvert ! video/x-raw, format=YUY2, framerate=30/1, width=640, height=480! alpha alpha=1 ! videobox border-alpha=0 right=-640 ! Mix.

Thanks,

Jim

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le mardi 16 février 2021 à 12:34 -0500, Jim Ruxton a écrit :
> Thanks so much for the tips. I first put queue in after the videoscales but it
> didn't seem to help. Using compositor seemed to fix it. I'm going to have to
> read up on compositor more to understand why that works.

I won't get into details, but compositor is a rewrite of videomixer to solve the
live use cases.

> Cheers,
> Jim
> On 2021-02-16 9:14 a.m., Mathieu Duponchelle wrote:
>  
> >   That is a good point, you should also use compositor instead of
> > videomixer,
> >  especially for live streams. In any case, using sync=false on the sinks is
> > not
> >  a desirable solution, rather an admission of defeat :)
> >  
> >  Cheers,
> >  
> > _______________________________________________
> > gstreamer-devel mailing list
> > [hidden email]
> > https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[mkonstapel]

On 16-02-2021 16:06, Nicolas Dufresne wrote:

Le mardi 16 février 2021 à 02:50 -0500, Jim Ruxton a écrit :

I am trying to combine 2 video streams. One from my laptop camera and one from an external webcam. I'm using the following pipeline and getting very choppy results. Neither my CPU or GPU seem to be working hard. Any ideas how to make this less choppy?

./gst-launch-1.0 v4l2src device=/dev/video4 ! videoscale ! videoconvert ! video/x-raw,format=YUY2, framerate=30/1, width=640, height=480! alpha alpha=1 ! videobox border-alpha=0 left=-640 ! 
videomixer name=Mix ! videoconvert ! autovideosink v4l2src device=/dev/video2 ! videoscale ! videoconvert ! video/x-raw, format=YUY2, framerate=30/1, width=640, height=480! alpha alpha=1 ! videobox border-alpha=0 right=-640 ! Mix.

First recommendation, move away from videomixer and use compositor (videomixer is just a backward shm on top of compositor). As you have a live pipeline, you should release the composition pressure by configurating a latency on the compositor element.  The latency is in nano-second, one of two frames of latency should be fine in general, but the default is none, and would only worked with perfectly synched sources which has accurate latency (v4l2src does not have accurate latency, it simply claims 1 frame, alway).

Second recommendation, consider adding a queue before you display sink, this will improve the timeout logic inside the compositor, by giving it a bit more freedome (with thread seperation).

Interesting! Does any of the above apply to *gl*videomixer as well? Are there any specific considerations for using that in a live pipeline? Should we also give that more latency? I see the glmixerbin superclass has a property for that, also defaulting to zero.

Kind regards,
Michiel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le mardi 02 mars 2021 à 10:29 +0100, Michiel Konstapel a écrit :

On 16-02-2021 16:06, Nicolas Dufresne wrote:

Le mardi 16 février 2021 à 02:50 -0500, Jim Ruxton a écrit :

I am trying to combine 2 video streams. One from my laptop camera and one from an external webcam. I'm using the following pipeline and getting very choppy results. Neither my CPU or GPU seem to be working hard. Any ideas how to make this less choppy?

./gst-launch-1.0 v4l2src device=/dev/video4 ! videoscale ! videoconvert ! video/x-raw,format=YUY2, framerate=30/1, width=640, height=480! alpha alpha=1 ! videobox border-alpha=0 left=-640 ! 
videomixer name=Mix ! videoconvert ! autovideosink v4l2src device=/dev/video2 ! videoscale ! videoconvert ! video/x-raw, format=YUY2, framerate=30/1, width=640, height=480! alpha alpha=1 ! videobox border-alpha=0 right=-640 ! Mix.

First recommendation, move away from videomixer and use compositor (videomixer is just a backward shm on top of compositor). As you have a live pipeline, you should release the composition pressure by configurating a latency on the compositor element.  The latency is in nano-second, one of two frames of latency should be fine in general, but the default is none, and would only worked with perfectly synched sources which has accurate latency (v4l2src does not have accurate latency, it simply claims 1 frame, alway).

Second recommendation, consider adding a queue before you display sink, this will improve the timeout logic inside the compositor, by giving it a bit more freedome (with thread seperation).

Interesting! Does any of the above apply to *gl*videomixer as well? Are there any specific considerations for using that in a live pipeline? Should we also give that more latency? I see the glmixerbin superclass has a property for that, also defaulting to zero.

glvideomixer is based a GstVideoAggregator, hence works fine for livepipeline. The mixer bin is to help taking take of uploading the pixel to your GPU, as GPUs don't or rarely use linear / malloc pixel data.

Kind regards,
Michiel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel