GStreamer-devel

Videomixing two/multiple webcams side by side

Classic

List

Threaded

16 messages Options

scottmil

Videomixing two/multiple webcams side by side

Hello,

I've been playing with gstreamer for two days now. I can successfully display streams from two web cams individually. I can also videomix (side by side) either webcam with a testpattern, but when I attempt to videomix the two webcams it doesn't work. I've tried a lot of different commands but this is where I stand:

*****This works fine with a cam and a testpattern*****
gst-launch v4l2src device=/dev/video1 ! video/x-raw-yuv,width=320,height=240 ! videobox left=-320 ! videomixer name=mix ! ffmpegcolorspace ! xvimagesink videotestsrc pattern=0 ! video/x-raw-yuv,width=320,height=240 ! mix.

*****When I add in a second cam instead I get the following error*****

cs@box:~/Desktop/GStreamer Projects$ gst-launch v4l2src device=/dev/video0 ! video/x-raw-yuv,width=320,height=240 ! videobox left=-320 ! videomixer name=mix ! ffmpegcolorspace ! xvimagesink v4l2src device=/dev/video1 ! video/x-raw-yuv,width=320,height=240 ! mix.

(gst-launch-0.10:8243): GLib-WARNING **: g_set_prgname() called multiple times
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data flow error.
Additional debug info:
gstbasesrc.c(2378): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
streaming task paused, reason not-negotiated (-4)
Execution ended after 108730332 ns.
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...
cs@box:~/Desktop/GStreamer Projects$

I think it may have something to do with not being able to pause a live stream, but I'm not sure how to fix it, or if there is another way around. I absolutely must be using live webcams, not recordings. Anyhelp here is GREATLY appreciated!

Scott

scottmil

Re: Videomixing two/multiple webcams side by side

UPDATE:

I've made some progress. I can get a "wider view" screen and it loads without crashing but only one camera is shown, though both are running. So I assume one is being displayed under the other, but I cant seem to get one on one side and one on the other.

here is the command i'm using:

gst-launch v4l2src device=/dev/video0 ! videobox right=-640 ! video/x-raw-yuv! videomixer name=mix ! ffmpegcolorspace ! xvimagesink sync=false v4l2src device=/dev/video1 ! queue ! video/x-raw-yuv! mix. mix.

Kapil Agrawal

Re: Videomixing two/multiple webcams side by side

In your videobox add border-alpha=0 property , also use queue before videomixer input of both cameras.

for eg.

gst-launch-0.10 v4l2src0 ! video/x-raw-yuv,width=320,height=240 ! videobox left=-320 border-alpha=0 ! queue ! videomixer name=mix ! ffmpegcolorspace ! xvimagesink v4l2src1 ! video/x-raw-yuv,width=320,height=240 ! queue ! mix.

best luck
Kapil

On Sat, May 15, 2010 at 12:37 AM, scottmil <[hidden email]> wrote:

UPDATE:

I've made some progress. I can get a "wider view" screen and it loads
without crashing but only one camera is shown, though both are running. So
I assume one is being displayed under the other, but I cant seem to get one
on one side and one on the other.

here is the command i'm using:

gst-launch v4l2src device=/dev/video0 ! videobox right=-640 !
video/x-raw-yuv! videomixer name=mix ! ffmpegcolorspace ! xvimagesink
sync=false v4l2src device=/dev/video1 ! queue ! video/x-raw-yuv! mix. mix.

--
View this message in context: http://gstreamer-devel.966125.n4.nabble.com/Videomixing-two-multiple-webcams-side-by-side-tp2197804p2217082.html

Sent from the GStreamer-devel mailing list archive at Nabble.com.

------------------------------------------------------------------------------

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

--
twitter handle: @gst_kaps
http://www.linkedin.com/in/kapilagrawal

------------------------------------------------------------------------------

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

scottmil

Re: Videomixing two/multiple webcams side by side

Kapil, Thanks for the reply! Unfortunately this still doesn't display both camera feeds. I'm not getting a green blank area instead of a black one. I assume this is from the border-alpha setting. I also had to add an extra mix. at the end. My command is:

gst-launch-0.10 v4l2src device=/dev/video0 ! video/x-raw-yuv,width=320,height=240 ! videobox left=-320 border-alpha=0 ! queue ! videomixer name=mix ! ffmpegcolorspace ! xvimagesink sync=false v4l2src device=/dev/video1 ! video/x-raw-yuv,width=320,height=240 ! queue ! mix. mix.

Which produces this:

The actual video is from /dev/video0. And I see nothing from /dev/video1. However the LED indicator on the camera turns on so the feed is getting pulled. Just not displayed.

Furthermore, if i define a videobox for /dev/video1 such as:

The stream crashes with the following output:

Setting pipeline to PAUSED ...

Pipeline is live and does not need PREROLL ...

Setting pipeline to PLAYING ...

New clock: GstSystemClock

ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src1: Internal data flow error.

Additional debug info:

gstbasesrc.c(2507): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src1:

streaming task paused, reason not-linked (-1)

Execution ended after 111866092 ns.

Setting pipeline to PAUSED ...

Setting pipeline to READY ...

Setting pipeline to NULL ...

Freeing pipeline ...

scott@scott-ubuntu:~$

Thanks, Scott

Martin Bisson

Re: Videomixing two/multiple webcams side by side

In reply to this post by Kapil Agrawal

This question is a bit unrelated to this thread, and I'm new to
GStreamer, but what does queue actually do? Why is it needed in that case?

Thanks,

Martin

Kapil Agrawal wrote:

> In your videobox add border-alpha=0 property , also use queue before
> videomixer input of both cameras.
>
> for eg.
>
> gst-launch-0.10 v4l2src0 ! video/x-raw-yuv,width=320,height=240 !
> videobox left=-320 border-alpha=0 ! queue ! videomixer name=mix !
> ffmpegcolorspace ! xvimagesink v4l2src1 !
> video/x-raw-yuv,width=320,height=240 ! queue ! mix.
>
> best luck
> Kapil
>
> On Sat, May 15, 2010 at 12:37 AM, scottmil <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>
> UPDATE:
>
> I've made some progress. I can get a "wider view" screen and it loads
> without crashing but only one camera is shown, though both are
> running. So
> I assume one is being displayed under the other, but I cant seem
> to get one
> on one side and one on the other.
>
> here is the command i'm using:
>
> gst-launch v4l2src device=/dev/video0 ! videobox right=-640 !
> video/x-raw-yuv! videomixer name=mix ! ffmpegcolorspace ! xvimagesink
> sync=false v4l2src device=/dev/video1 ! queue ! video/x-raw-yuv!
> mix. mix.
>
> --
> View this message in context:
> http://gstreamer-devel.966125.n4.nabble.com/Videomixing-two-multiple-webcams-side-by-side-tp2197804p2217082.html
> Sent from the GStreamer-devel mailing list archive at Nabble.com.
>
> ------------------------------------------------------------------------------
>
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> <mailto:[hidden email]>
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>
>
>
>
> --
> twitter handle: @gst_kaps
> http://www.linkedin.com/in/kapilagrawal
> ------------------------------------------------------------------------
>
> ------------------------------------------------------------------------------
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>

------------------------------------------------------------------------------

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

Kapil Agrawal

Re: Videomixing two/multiple webcams side by side

In reply to this post by scottmil

Scott,

Dont use sync=false , also I dont think there is any need of adding extra mix. , its of no use.

If this doesnt work please run the pipeline with GST_DEBUG=*:2 and attach logs for better understanding of issue.

best
Kapil

On Sat, May 22, 2010 at 2:44 AM, scottmil <[hidden email]> wrote:

Kapil,

Thanks for the reply! Unfortunately this still doesn't display both camera
feeds. I'm not getting a green blank area instead of a black one. I assume
this is from the border-alpha setting. I also had to add an extra mix. at
the end. My command is:

gst-launch-0.10 v4l2src device=/dev/video0 !

video/x-raw-yuv,width=320,height=240 ! videobox left=-320 border-alpha=0 !

queue ! videomixer name=mix ! ffmpegcolorspace ! xvimagesink sync=false
v4l2src device=/dev/video1 ! video/x-raw-yuv,width=320,height=240 ! queue !
mix. mix.

Which produces this:

http://n4.nabble.com/file/n2226729/Screenshot.png

The actual video is from /dev/video0. And I see nothing from /dev/video1.
However the LED indicator on the camera turns on so the feed is getting
pulled. Just not displayed.

Scott
--
View this message in context: http://gstreamer-devel.966125.n4.nabble.com/Videomixing-two-multiple-webcams-side-by-side-tp2197804p2226729.html

Sent from the GStreamer-devel mailing list archive at Nabble.com.

------------------------------------------------------------------------------

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

Stefan Sauer

Re: Videomixing two/multiple webcams side by side

In reply to this post by scottmil

Am 14.05.2010 22:07, schrieb scottmil:

>
> UPDATE:
>
> I've made some progress. I can get a "wider view" screen and it loads
> without crashing but only one camera is shown, though both are running. So
> I assume one is being displayed under the other, but I cant seem to get one
> on one side and one on the other.
>
> here is the command i'm using:
>
> gst-launch v4l2src device=/dev/video0 ! videobox right=-640 !
> video/x-raw-yuv! videomixer name=mix ! ffmpegcolorspace ! xvimagesink
> sync=false v4l2src device=/dev/video1 ! queue ! video/x-raw-yuv! mix. mix.
>

Just set the position of the 2nd video when using videomixer:
... ! videomixer name=mix sink_00::xpos=0 sink_01::xpos=640 ! ...

check the pad names of videomixer from running gst-launch -v

Stefan

------------------------------------------------------------------------------

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

Stefan Sauer

Re: Videomixing two/multiple webcams side by side

In reply to this post by Martin Bisson

Am 22.05.2010 01:00, schrieb Martin Bisson:
> This question is a bit unrelated to this thread, and I'm new to
> GStreamer, but what does queue actually do? Why is it needed in that case?

I creates a thread barrier to decouple the capture threads from the mixing thread.

Stefan

>
> Thanks,
>
> Martin
>
>
> Kapil Agrawal wrote:
>> In your videobox add border-alpha=0 property , also use queue before
>> videomixer input of both cameras.
>>
>> for eg.
>>
>> gst-launch-0.10 v4l2src0 ! video/x-raw-yuv,width=320,height=240 !
>> videobox left=-320 border-alpha=0 ! queue ! videomixer name=mix !
>> ffmpegcolorspace ! xvimagesink v4l2src1 !
>> video/x-raw-yuv,width=320,height=240 ! queue ! mix.
>>
>> best luck
>> Kapil
>>
>> On Sat, May 15, 2010 at 12:37 AM, scottmil <[hidden email]
>> <mailto:[hidden email]>> wrote:
>>
>>
>> UPDATE:
>>
>> I've made some progress. I can get a "wider view" screen and it loads
>> without crashing but only one camera is shown, though both are
>> running. So
>> I assume one is being displayed under the other, but I cant seem
>> to get one
>> on one side and one on the other.
>>
>> here is the command i'm using:
>>
>> gst-launch v4l2src device=/dev/video0 ! videobox right=-640 !
>> video/x-raw-yuv! videomixer name=mix ! ffmpegcolorspace ! xvimagesink
>> sync=false v4l2src device=/dev/video1 ! queue ! video/x-raw-yuv!
>> mix. mix.
>>
>> --
>> View this message in context:
>> http://gstreamer-devel.966125.n4.nabble.com/Videomixing-two-multiple-webcams-side-by-side-tp2197804p2217082.html
>> Sent from the GStreamer-devel mailing list archive at Nabble.com.
>>
>> ------------------------------------------------------------------------------
>>
>> _______________________________________________
>> gstreamer-devel mailing list
>> [hidden email]
>> <mailto:[hidden email]>
>> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>>
>>
>>
>>
>> --
>> twitter handle: @gst_kaps
>> http://www.linkedin.com/in/kapilagrawal
>> ------------------------------------------------------------------------
>>
>> ------------------------------------------------------------------------------
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> gstreamer-devel mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>>
>
>
> ------------------------------------------------------------------------------
>
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

scottmil

Re: Videomixing two/multiple webcams side by side

Well for some reason when I used to have only one "mix." it would crash. So, I went ahead and gave it a try with only one and walla it worked. I'm not sure why it did this time and not before, must have been something else I had changed.

Working command is:

gst-launch-0.10 v4l2src device=/dev/video0 ! video/x-raw-yuv,width=320,height=240 ! videobox left=-320 border-alpha=0 ! queue ! videomixer name=mix ! ffmpegcolorspace ! xvimagesink v4l2src device=/dev/video1 ! video/x-raw-yuv,width=320,height=240 ! videobox left=0 ! queue ! mix.

Now I need to figure out if there is a way to synchronize the streams. I've tried sync=false and sync=true parameters with xbimagesink. Anyone have any ideas. I also tried fiddling with setting x_pos in the videomixer instead of the videobox, but that caused crashes too.

P.S. You guys absolutely rock!

Scott

scottmil

Re: Videomixing two/multiple webcams side by side

Made even more progress. I was using two cameras of different makes and seeing the delay. I was also receiving warnings about dropped buffers. So, I switched to two identical cameras. Now I'm seeing nearly no delay. I also tried using ximagesink instead of xvimagesink, but then I would see a delay build up over time. The future of this project will be to eventually have 5-6 cameras running simultaneously, so I will need to figure out how to mix 3-4 etc cameras, and then output this as a stream that can be picked up somewhere else. Hopefully this won't drastically increase the delay. Thoughts? Scott

Stefan Sauer

Re: Videomixing two/multiple webcams side by side

In reply to this post by scottmil

On 24.05.2010 19:45, scottmil wrote:

> Well for some reason when I used to have only one "mix." it would crash. So,
> I went ahead and gave it a try with only one and walla it worked. I'm not
> sure why it did this time and not before, must have been something else I
> had changed.
>
> Working command is:
>
> gst-launch-0.10 v4l2src device=/dev/video0 !
> video/x-raw-yuv,width=320,height=240 ! videobox left=-320 border-alpha=0 !
> queue ! videomixer name=mix ! ffmpegcolorspace ! xvimagesink v4l2src
> device=/dev/video1 ! video/x-raw-yuv,width=320,height=240 ! videobox left=0
> ! queue ! mix.
>
> Now I need to figure out if there is a way to synchronize the streams. I've
> tried sync=false and sync=true parameters with xbimagesink. Anyone have any
> ideas. I also tried fiddling with setting x_pos in the videomixer instead
> of the videobox, but that caused crashes too.
>
> P.S. You guys absolutely rock!
>
> Scott
>

If you get crashers, please run under gdb (preferably with the latest
version from git or a release as new as possible) and get a backtrace
from the crash. Then file a bug and attach the backtraces.

Stefan

------------------------------------------------------------------------------

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

Luciana Fujii Pontello

Re: Videomixing two/multiple webcams side by side

In reply to this post by scottmil

On Mon, 2010-05-24 at 11:14 -0700, scottmil wrote:
> Made even more progress. I was using two cameras of different makes and
> seeing the delay. I was also receiving warnings about dropped buffers. So,
> I switched to two identical cameras. Now I'm seeing nearly NO delay.

>From what I observed, I think videomixer doesn't work well with videos
with different framerates and different colorspaces. Try adding a
videorate and a colorspace on one (or both) streams before the
videomixer to make your different cameras work. If that doesn't work,
then try also a capsfilter after the videorate and colorspace right
before the videomixer.

I have a PictureInPicture class in python that adds all those elements
before the videomixer if you want to take a look. It gets kind of big,
but that's the only way I was able to mix any two different streams.

Regards,

Luciana Fujii

------------------------------------------------------------------------------

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

scottmil

Re: Videomixing two/multiple webcams side by side

I would LOVE to see it. Is it posted somewhere it can be downloaded?

Scott

Luciana Fujii Pontello

Re: Videomixing two/multiple webcams side by side

On Tue, 2010-05-25 at 13:01 -0700, scottmil wrote:
> I would LOVE to see it. Is it posted somewhere it can be downloaded?
>
> Scott

I only received your e-mail today, so I don't know if this will still be
useful.

The module I use to mix 2 pipelines in a picture-in-picture is in here:
http://git.holoscopio.com/?p=sltv.git;a=blob;f=sltv/videomixer.py;hb=HEAD. It does others crazy things (I use it as if it was the input-selector also), but it should not be a problem if you don't link more stuff to it. Actually, I think if you go back in time you can get a simpler version like this: http://git.holoscopio.com/?p=sltv.git;a=blob;f=sltv/videomixer.py;h=91e8d0bdd59d1a0c1a3718cbc29ebe5018f16b33;hb=2cabc48c982650dd0d3f69499374fa2e46a11e31

I have a small test case in http://holoscopio.com/~fujii/testpip.py that
is a good example and you can check the project repository to see how we
use it for real.

Regards,

Luciana Fujii
Holoscópio Tecnologia - http://holoscopio.com

------------------------------------------------------------------------------
ThinkGeek and WIRED's GeekDad team up for the Ultimate
GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the
lucky parental unit. See the prize list and enter to win:
http://p.sf.net/sfu/thinkgeek-promo
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

Michael Joachimiak

Re: Videomixing two/multiple webcams side by side

Scott,

USB cameras are a little bit bad since they don't provide a constant framerate.
It depends on the integration time (longer integration times -- lower framerate) .
And integration time depends on changing light conditions.
You can consider dropping all frames but latter in the stream to get them synchronize.
If framerates will be different faster camera will be dropping and you will get constant framerate.
I had the same problem before.
In your chain function you can check the size of the buffer and if it has more than one frame you can take a pointer to the latest frame.
Hmm.. But I guess you don't have your own plugin? Do you?

This strikes another question. Is it possible to drop frames like I mentioned using videomixer?

2010/6/7 Luciana Fujii Pontello <[hidden email]>

On Tue, 2010-05-25 at 13:01 -0700, scottmil wrote:
> I would LOVE to see it. Is it posted somewhere it can be downloaded?
>
> Scott

I only received your e-mail today, so I don't know if this will still be
useful.

The module I use to mix 2 pipelines in a picture-in-picture is in here:
http://git.holoscopio.com/?p=sltv.git;a=blob;f=sltv/videomixer.py;hb=HEAD. It does others crazy things (I use it as if it was the input-selector also), but it should not be a problem if you don't link more stuff to it. Actually, I think if you go back in time you can get a simpler version like this: http://git.holoscopio.com/?p=sltv.git;a=blob;f=sltv/videomixer.py;h=91e8d0bdd59d1a0c1a3718cbc29ebe5018f16b33;hb=2cabc48c982650dd0d3f69499374fa2e46a11e31

I have a small test case in http://holoscopio.com/~fujii/testpip.py that
is a good example and you can check the project repository to see how we
use it for real.

Regards,

Luciana Fujii
Holoscópio Tecnologia - http://holoscopio.com

------------------------------------------------------------------------------
ThinkGeek and WIRED's GeekDad team up for the Ultimate
GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the
lucky parental unit. See the prize list and enter to win:
http://p.sf.net/sfu/thinkgeek-promo
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

--
Your Sincerely
Michael Joachimiak

------------------------------------------------------------------------------
ThinkGeek and WIRED's GeekDad team up for the Ultimate
GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the
lucky parental unit. See the prize list and enter to win:
http://p.sf.net/sfu/thinkgeek-promo
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

Michael Joachimiak

Re: Videomixing two/multiple webcams side by side

Ok. Now I see it should work with videorate element.
Does it work for you?

2010/6/9 Michael Joachimiak <[hidden email]>

Scott,

USB cameras are a little bit bad since they don't provide a constant framerate.
It depends on the integration time (longer integration times -- lower framerate) .
And integration time depends on changing light conditions.
You can consider dropping all frames but latter in the stream to get them synchronize.
If framerates will be different faster camera will be dropping and you will get constant framerate.
I had the same problem before.
In your chain function you can check the size of the buffer and if it has more than one frame you can take a pointer to the latest frame.
Hmm.. But I guess you don't have your own plugin? Do you?

This strikes another question. Is it possible to drop frames like I mentioned using videomixer?

2010/6/7 Luciana Fujii Pontello <[hidden email]>

On Tue, 2010-05-25 at 13:01 -0700, scottmil wrote:
> I would LOVE to see it. Is it posted somewhere it can be downloaded?
>
> Scott

I only received your e-mail today, so I don't know if this will still be
useful.

The module I use to mix 2 pipelines in a picture-in-picture is in here:
http://git.holoscopio.com/?p=sltv.git;a=blob;f=sltv/videomixer.py;hb=HEAD. It does others crazy things (I use it as if it was the input-selector also), but it should not be a problem if you don't link more stuff to it. Actually, I think if you go back in time you can get a simpler version like this: http://git.holoscopio.com/?p=sltv.git;a=blob;f=sltv/videomixer.py;h=91e8d0bdd59d1a0c1a3718cbc29ebe5018f16b33;hb=2cabc48c982650dd0d3f69499374fa2e46a11e31

I have a small test case in http://holoscopio.com/~fujii/testpip.py that
is a good example and you can check the project repository to see how we
use it for real.

Regards,

Luciana Fujii
Holoscópio Tecnologia - http://holoscopio.com

------------------------------------------------------------------------------
ThinkGeek and WIRED's GeekDad team up for the Ultimate
GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the
lucky parental unit. See the prize list and enter to win:
http://p.sf.net/sfu/thinkgeek-promo
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

--
Your Sincerely
Michael Joachimiak