GStreamer-devel

Mixing starting/stopping/disconnecting sources: multiqueue + freeze?

Classic

List

Threaded

6 messages Options

Sjoerd Op 't Land

Mixing starting/stopping/disconnecting sources: multiqueue + freeze?

Hello all,

For a live video mixer/switcher I would like to perform mixing/
switching between multiple asynchronous and starting/stopping/
(dis)connecting video sources. In hardware, this is normally done by a
TBC (Time Base Corrector). What would be the way to go in GStreamer?

This is the solution I thought up thus far: every source is a pipeline
that can be started and stopped at will. Then, a multiqueue to
simultaneously output a frame from each source:

source1(pipeline) -> | | -> | |
| multiqueue | | videomixer | -> sink
source2(pipeline) -> | | -> | |

However, it seems from experiments that multiqueue only starts when it
has received one buffer from every source. Should I insert something
like the 'freeze' element to feed the multiqueue with a continuous
stream? (Not the current 'freeze' element, because it freezes always,
and not only a stopped source. Furthermore, it should increase or
remove timestamps.)

source1(pipeline) -> 'freeze' -> | | -> | |
| multiqueue | | videomixer | ->
sink
source2(pipeline) -> 'freeze' -> | | -> | |

Does such an element exist? The existence of the GST_BUFFER_FLAG_GAP
makes me think that it must exist, but I cannot find it. Is pull-based
scheduling possible for such a solution?

Thanks in advance for any hints or suggestions,
Sjoerd

------------------------------------------------------------------------------
This SF.Net email is sponsored by the Verizon Developer Community
Take advantage of Verizon's best-in-class app development support
A streamlined, 14 day to market process makes app distribution fast and easy
Join now and get one step closer to millions of Verizon customers
http://p.sf.net/sfu/verizon-dev2dev
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

Frans van Berckel

Re: Mixing starting/stopping/disconnecting sources: multiqueue + freeze?

Not knowing it in the details but did you checkout the GStreamer
Non-Linear plugins?

http://gstreamer.freedesktop.org/modules/gnonlin.html

It contains a set of plug-ins for GStreamer to ease the creation of
multimedia editors, or any other application where a timeline-oriented
use of GStreamer makes sense.

http://wiki.pitivi.org/wiki/GNonLin_documentation

Thanks,

Frans van Berckel

On Sat, 2010-01-02 at 13:05 +0100, Sjoerd Op 't Land wrote:

> Hello all,
>
> For a live video mixer/switcher I would like to perform mixing/
> switching between multiple asynchronous and starting/stopping/
> (dis)connecting video sources. In hardware, this is normally done by a
> TBC (Time Base Corrector). What would be the way to go in GStreamer?
>
> This is the solution I thought up thus far: every source is a pipeline
> that can be started and stopped at will. Then, a multiqueue to
> simultaneously output a frame from each source:
>
> source1(pipeline) -> | | -> | |
> | multiqueue | | videomixer | -> sink
> source2(pipeline) -> | | -> | |
>
> However, it seems from experiments that multiqueue only starts when it
> has received one buffer from every source. Should I insert something
> like the 'freeze' element to feed the multiqueue with a continuous
> stream? (Not the current 'freeze' element, because it freezes always,
> and not only a stopped source. Furthermore, it should increase or
> remove timestamps.)
>
> source1(pipeline) -> 'freeze' -> | | -> | |
> | multiqueue | | videomixer | ->
> sink
> source2(pipeline) -> 'freeze' -> | | -> | |
>
> Does such an element exist? The existence of the GST_BUFFER_FLAG_GAP
> makes me think that it must exist, but I cannot find it. Is pull-based
> scheduling possible for such a solution?
>
> Thanks in advance for any hints or suggestions,
> Sjoerd

------------------------------------------------------------------------------
This SF.Net email is sponsored by the Verizon Developer Community
Take advantage of Verizon's best-in-class app development support
A streamlined, 14 day to market process makes app distribution fast and easy
Join now and get one step closer to millions of Verizon customers
http://p.sf.net/sfu/verizon-dev2dev
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

Sjoerd Op 't Land

Re: Mixing starting/stopping/disconnecting sources: multiqueue + freeze?

Op 2 jan 2010, om 13:42 heeft Frans van Berckel het volgende geschreven:

> Not knowing it in the details but did you checkout the GStreamer
> Non-Linear plugins?
>
> http://gstreamer.freedesktop.org/modules/gnonlin.html
>
> It contains a set of plug-ins for GStreamer to ease the creation of
> multimedia editors, or any other application where a timeline-oriented
> use of GStreamer makes sense.
>
> http://wiki.pitivi.org/wiki/GNonLin_documentation

Looks interesting, both GNonLine and PiTiVi. Perhaps this will become
useful in a later stage, to 'playback' live cuts and to enable post-
production. For the moment, we are looking for live seamless (low-
latency) cutting and fading between sources. If I understand the
GNonLin documentation correctly, this is not possible with GNonLin
specifically. Correct?

Thanks for your help,
Sjoerd

>
> Thanks,
>
>
> Frans van Berckel
>
> On Sat, 2010-01-02 at 13:05 +0100, Sjoerd Op 't Land wrote:
>> Hello all,
>>
>> For a live video mixer/switcher I would like to perform mixing/
>> switching between multiple asynchronous and starting/stopping/
>> (dis)connecting video sources. In hardware, this is normally done
>> by a
>> TBC (Time Base Corrector). What would be the way to go in GStreamer?
>>
>> This is the solution I thought up thus far: every source is a
>> pipeline
>> that can be started and stopped at will. Then, a multiqueue to
>> simultaneously output a frame from each source:
>>
>> source1(pipeline) -> | | -> | |
>> | multiqueue | | videomixer | -> sink
>> source2(pipeline) -> | | -> | |
>>
>> However, it seems from experiments that multiqueue only starts when
>> it
>> has received one buffer from every source. Should I insert something
>> like the 'freeze' element to feed the multiqueue with a continuous
>> stream? (Not the current 'freeze' element, because it freezes always,
>> and not only a stopped source. Furthermore, it should increase or
>> remove timestamps.)
>>
>> source1(pipeline) -> 'freeze' -> | | -> | |
>> | multiqueue | | videomixer | ->
>> sink
>> source2(pipeline) -> 'freeze' -> | | -> | |
>>
>> Does such an element exist? The existence of the GST_BUFFER_FLAG_GAP
>> makes me think that it must exist, but I cannot find it. Is pull-
>> based
>> scheduling possible for such a solution?
>>
>> Thanks in advance for any hints or suggestions,
>> Sjoerd
>
>
> ------------------------------------------------------------------------------
> This SF.Net email is sponsored by the Verizon Developer Community
> Take advantage of Verizon's best-in-class app development support
> A streamlined, 14 day to market process makes app distribution fast
> and easy
> Join now and get one step closer to millions of Verizon customers
> http://p.sf.net/sfu/verizon-dev2dev
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>

Florent THIERY-2

Re: Mixing starting/stopping/disconnecting sources: multiqueue + freeze?

Hi

> For the moment, we are looking for live seamless (low-
> latency) cutting and fading between sources. If I understand the
> GNonLin documentation correctly, this is not possible with GNonLin
> specifically. Correct?

low level gnonlin elements will let you do file-based compositing
(i.e. defining portions of media to feed to the pipeline, such as e.g.
second 2 to 16 from a video file) with the gstreamer approach, but it
is not live performance oriented.

For live compositing/switching, you have two options as far as i know:
* the lagacy videomixer [1], which might let you overlay multiple AYUV streams
* gst-plugins-gl's glmosaic plugin [2], which is an example of
opengl-based compositing in gstreamer. It can be used as a starting
point to create a video mixer

There used to be a "switcher" element, but i don't know why it disappeared.

[1] http://gstreamer.freedesktop.org/data/doc/gstreamer/0.10.1/gst-plugins-good-plugins/html/gst-plugins-good-plugins-videomixer.html
[2] http://cgit.freedesktop.org/gstreamer/gst-plugins-gl/tree/gst/gl/gstglmosaic.c

Florent

------------------------------------------------------------------------------
This SF.Net email is sponsored by the Verizon Developer Community
Take advantage of Verizon's best-in-class app development support
A streamlined, 14 day to market process makes app distribution fast and easy
Join now and get one step closer to millions of Verizon customers
http://p.sf.net/sfu/verizon-dev2dev
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

Sjoerd Op 't Land

Re: Mixing starting/stopping/disconnecting sources: multiqueue + freeze?

Thanks Florent for your suggestion, I didn't know the glmosaic plugin
and it might prove useful in a later stage. However, this is not my
current problem.

My current problem is how to mix (e.g. using videomixer) between
multiple, asynchronous, starting and stopping streams. For example, I
want to be able to connect an extra source (e.g. an RTP stream, a
movie file, a text generator) during a live performance, and then
dissolve to this source. Later on I want to switch away from this
source, and then disconnect the source. Meanwhile, the live output
should still continue.

If I understand correctly, multiqueue synchronises multiple sources to
one clock (correct?). If that's correct, part of the problem is
solved. However, it seems that multiqueue stays STOPPED as long as it
hasn't received at least one buffer from all of its inputs. Therefore,
I thought, maybe I could write or use an element that repeats the same
buffer until the real source produces a (new) buffer.

Is this the way to go?

Thanks in advance,
Sjoerd

Op 2 jan 2010, om 18:19 heeft Florent het volgende geschreven:

> Hi
>
>> For the moment, we are looking for live seamless (low-
>> latency) cutting and fading between sources. If I understand the
>> GNonLin documentation correctly, this is not possible with GNonLin
>> specifically. Correct?
>
> low level gnonlin elements will let you do file-based compositing
> (i.e. defining portions of media to feed to the pipeline, such as e.g.
> second 2 to 16 from a video file) with the gstreamer approach, but it
> is not live performance oriented.
>
> For live compositing/switching, you have two options as far as i know:
> * the lagacy videomixer [1], which might let you overlay multiple
> AYUV streams
> * gst-plugins-gl's glmosaic plugin [2], which is an example of
> opengl-based compositing in gstreamer. It can be used as a starting
> point to create a video mixer
>
> There used to be a "switcher" element, but i don't know why it
> disappeared.
>
> [1] http://gstreamer.freedesktop.org/data/doc/gstreamer/0.10.1/gst-plugins-good-plugins/html/gst-plugins-good-plugins-videomixer.html
> [2] http://cgit.freedesktop.org/gstreamer/gst-plugins-gl/tree/gst/gl/gstglmosaic.c
>
> Florent
>
> ------------------------------------------------------------------------------
> This SF.Net email is sponsored by the Verizon Developer Community
> Take advantage of Verizon's best-in-class app development support
> A streamlined, 14 day to market process makes app distribution fast
> and easy
> Join now and get one step closer to millions of Verizon customers
> http://p.sf.net/sfu/verizon-dev2dev
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>

Stefan Sauer

Re: Mixing starting/stopping/disconnecting sources: multiqueue + freeze?

In reply to this post by Florent THIERY-2

Am 02.01.2010 19:19, schrieb Florent:

There are {input,output}-selectur elements

Stefan

>
> [1] http://gstreamer.freedesktop.org/data/doc/gstreamer/0.10.1/gst-plugins-good-plugins/html/gst-plugins-good-plugins-videomixer.html
> [2] http://cgit.freedesktop.org/gstreamer/gst-plugins-gl/tree/gst/gl/gstglmosaic.c
>
> Florent
>
> ------------------------------------------------------------------------------
> This SF.Net email is sponsored by the Verizon Developer Community
> Take advantage of Verizon's best-in-class app development support
> A streamlined, 14 day to market process makes app distribution fast and easy
> Join now and get one step closer to millions of Verizon customers
> http://p.sf.net/sfu/verizon-dev2dev
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel