Mixing lossy live sources

[Benjamin Le Bozec]

Hello,

Can you advise me on the following point.

I'm mixing two RTMP live sources with the compositor element : 

rtmpsrc (do-imestamp=true) ==>

                                                       compositor ==> rtmpsink (sync=true)

rtmpsrc (do-imestamp=true)  ==>

But the sources are coming from cellphones, through a lossy network (the cellular network), and packets are lost from time to time. Consequently, the compositor elements tends to freeze, waiting for both streams to deliver packets. 

Is there a way to ensure a continuous streaming, that is for the compositor to continue to deliver images even if some (or all) sources do not deliver images ?

Thank you,

Ben

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Baby Octopus]

Compositor exactly works the way you have described it to work. The issue
that you are reporting was with older videomixer element. What exact issue
do you see in compositor? Can you share the pipeline and debug logs?

~BO



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Tim Müller]

On Thu, 2018-05-17 at 12:28 +0200, Benjamin Le Bozec wrote:

Hi Benjamin,

> I'm mixing two RTMP live sources with the compositor element :
> rtmpsrc (do-imestamp=true) ==>
>                                                        compositor ==>
> rtmpsink (sync=true)
> rtmpsrc (do-imestamp=true)  ==>
>
> But the sources are coming from cellphones, through a lossy network
> (the cellular network), and packets are lost from time to time.
> Consequently, the compositor elements tends to freeze, waiting for
> both streams to deliver packets.
>
> Is there a way to ensure a continuous streaming, that is for the
> compositor to continue to deliver images even if some (or all)
> sources do not deliver images ?

compositor *should* be able to handle this scenario well. However, it
will do so only if the source elements have identified themselves to be
'live' sources (like rtspsrc/udpsrc). I don't think rtmpsrc does that.

You *might* be able to hack around it by using gst_base_src_set_live()
and setting the "latency" property on compositor to some reasonable
value; or add a pad probe and intercept the latency query from
compositor to the sources.

Not sure if rtmpsrc will gracefully handle this case though (drop
packets to 'catch up').

Cheers
 -Tim

--
Tim Müller, Centricular Ltd - http://www.centricular.com
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le jeudi 17 mai 2018 à 12:52 +0100, Tim Müller a écrit :

> On Thu, 2018-05-17 at 12:28 +0200, Benjamin Le Bozec wrote:
>
> Hi Benjamin,
>
> > I'm mixing two RTMP live sources with the compositor element :
> > rtmpsrc (do-imestamp=true) ==>
> >                                                        compositor
> > ==>
> > rtmpsink (sync=true)
> > rtmpsrc (do-imestamp=true)  ==>
> >
> > But the sources are coming from cellphones, through a lossy network
> > (the cellular network), and packets are lost from time to time.
> > Consequently, the compositor elements tends to freeze, waiting for
> > both streams to deliver packets.
> >
> > Is there a way to ensure a continuous streaming, that is for the
> > compositor to continue to deliver images even if some (or all)
> > sources do not deliver images ?
>
> compositor *should* be able to handle this scenario well. However, it
> will do so only if the source elements have identified themselves to
> be
> 'live' sources (like rtspsrc/udpsrc). I don't think rtmpsrc does
> that.
>
> You *might* be able to hack around it by using
> gst_base_src_set_live()
> and setting the "latency" property on compositor to some reasonable
> value; or add a pad probe and intercept the latency query from
> compositor to the sources.

I believe you can turn this into a lice source using the following
construction, let me know if that works for you.

  rmptsrc do-timestamp=1 ! identity sync=1 ! ...

You can place the identity elsewhere if needed.

>
> Not sure if rtmpsrc will gracefully handle this case though (drop
> packets to 'catch up').
>
> Cheers
>  -Tim
>
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[daniel-2]

W dniu 2018-05-17 17:21, Nicolas Dufresne napisał(a):

> Le jeudi 17 mai 2018 à 12:52 +0100, Tim Müller a écrit :
>> On Thu, 2018-05-17 at 12:28 +0200, Benjamin Le Bozec wrote:
>>
>> Hi Benjamin,
>>
>> > I'm mixing two RTMP live sources with the compositor element :
>> > rtmpsrc (do-imestamp=true) ==>
>> >                                                        compositor
>> > ==>
>> > rtmpsink (sync=true)
>> > rtmpsrc (do-imestamp=true)  ==>
>> >
>> > But the sources are coming from cellphones, through a lossy network
>> > (the cellular network), and packets are lost from time to time.
>> > Consequently, the compositor elements tends to freeze, waiting for
>> > both streams to deliver packets.
>> >
>> > Is there a way to ensure a continuous streaming, that is for the
>> > compositor to continue to deliver images even if some (or all)
>> > sources do not deliver images ?
>>
>> compositor *should* be able to handle this scenario well. However, it
>> will do so only if the source elements have identified themselves to
>> be
>> 'live' sources (like rtspsrc/udpsrc). I don't think rtmpsrc does
>> that.
>>
>> You *might* be able to hack around it by using
>> gst_base_src_set_live()
>> and setting the "latency" property on compositor to some reasonable
>> value; or add a pad probe and intercept the latency query from
>> compositor to the sources.
>
> I believe you can turn this into a lice source using the following
> construction, let me know if that works for you.
>
>   rmptsrc do-timestamp=1 ! identity sync=1 ! ...
>
> You can place the identity elsewhere if needed.
>
>>
>> Not sure if rtmpsrc will gracefully handle this case though (drop
>> packets to 'catch up').
>>
>> Cheers
>>  -Tim
>>
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

You can also try using audiorate element to insert extra samples into
stream where necessary.

Regards,
Daniel
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Benjamin Le Bozec]

Ok guys, thanks for the heads-up !

I managed to have something working using the "identity sync=1" trick, and adding a latency to the compositor. Here is the pipeline :

gst-launch-1.0 -e rtmpsrc location="rtmp://..." ! identity sync=1 ! 

queue max-size-time=2000000000 ! decodebin ! videorate ! 'video/x-raw,format=(string)I420,framerate=30/1, width=1280, height=720' ! mix.sink_0 \

rtmpsrc location="rtmp://..." ! identity sync=1 ! \

queue max-size-time=2000000000 ! decodebin ! videoscale ! videorate ! 'video/x-raw,format=(string)I420,framerate=30/1, width=320, height=240' ! mix.sink_1 \

compositor name=mix latency=1000000000 ! \

videoconvert ! x264enc threads=0 bitrate=2000 tune=zerolatency key-int-max=60 speed-preset=veryfast ! \

flvmux streamable=true name=mux ! rtmpsink sync=1 location="rtmp://... live=1"

When one of the inputs starts to lag, the compositor continues to produce images. Amazing :)

The only thing i'm not confortable with is the arbitrary latency set to 1sec.

I also managed to mix the audio, but i spare you the pipeline ...

Thank you,

Ben

2018-05-17 17:21 GMT+02:00 Nicolas Dufresne <[hidden email]>:

Le jeudi 17 mai 2018 à 12:52 +0100, Tim Müller a écrit :
> On Thu, 2018-05-17 at 12:28 +0200, Benjamin Le Bozec wrote:
>
> Hi Benjamin,
>
> > I'm mixing two RTMP live sources with the compositor element :
> > rtmpsrc (do-imestamp=true) ==>
> >                                                        compositor
> > ==>
> > rtmpsink (sync=true)
> > rtmpsrc (do-imestamp=true)  ==>
> >
> > But the sources are coming from cellphones, through a lossy network
> > (the cellular network), and packets are lost from time to time.
> > Consequently, the compositor elements tends to freeze, waiting for
> > both streams to deliver packets.
> >
> > Is there a way to ensure a continuous streaming, that is for the
> > compositor to continue to deliver images even if some (or all)
> > sources do not deliver images ?
>
> compositor *should* be able to handle this scenario well. However, it
> will do so only if the source elements have identified themselves to
> be
> 'live' sources (like rtspsrc/udpsrc). I don't think rtmpsrc does
> that.
>
> You *might* be able to hack around it by using
> gst_base_src_set_live()
> and setting the "latency" property on compositor to some reasonable
> value; or add a pad probe and intercept the latency query from
> compositor to the sources.

I believe you can turn this into a lice source using the following
construction, let me know if that works for you.

  rmptsrc do-timestamp=1 ! identity sync=1 ! ...

You can place the identity elsewhere if needed.

>
> Not sure if rtmpsrc will gracefully handle this case though (drop
> packets to 'catch up').
>
> Cheers
>  -Tim
>

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

--

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel