Fast switching between H264 streams with input-selector

[omer.tal]

Hey guys,

I'm using input-selector to toggle between several RTSP streams.
Whenever I toggle between two streams, I have a small duration where there's
no valid I-frame to use for decoding so I get garbage.

I thought that I could use a queue before the input-selector to store the
last second of each stream, so when I will toggle between them, the decoder
will decode (as fast as possible) the last second (which will include an
I-frame), and will then get me my current frame decoded properly.

I'm not sure what parameters I should use to achieve this behavior.

The pipeline I have:

-----------	---------
| rtspsrc | --> | queue | --
-----------	---------  |    ------------------     ----------------     -------------     -------------     ----------------     --------------     -----------
                            --> | input-selector | --> | rtph264depay | --> | h264parse | --> | avdec_h264| --> | videoconvert | --> | videoscale | --> | appsink |
-----------     ---------  |    ------------------     ----------------     -------------     -------------     ----------------     --------------     -----------
| rtspsrc | --> | queue | --
-----------     ---------

Should I block every source pad that is inactive in the input-selector? That seems like an awkward way to go...

Thank you for your time and help :)


--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[e-mail]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Marc Leeman]

This is something that I have been using for some time now (both for
streaming and for decoding). What you can do is the following:

1. You start from a steady state

uridecodebin uri=rtsp://..../1  ! input-selector !  videoconvert ! autovideosink

2. When you want to switch, you add a new uridecodebin to your
pipeline, connect the pad-added, and set the uri to the new RTSP uri

uridecodebin uri=rtsp://..../1 ! input-selector !  videoconvert ! autovideosink
uridecodebin uri=rtsp://..../2

3. When you get a new pad; attach it to the input selector; and swap
the active-pad
4 clean up the previous uridecodebin

You end up with
uridecodebin uri=rtsp://..../2 caps="video/x-h264" ! input-selector !
decodebin ! autovideosink

This will make certain that you only switch when there is a full
decoded frame and you should have no artifacts. The previous stream
will keep on playing until you do. If the 2nd camera is off-line; you
will keep on seeing the first one, unless you use a timeout.

On Tue, 12 Jan 2021 at 11:15, omer.tal <[hidden email]> wrote:

>
> Hey guys,
>
> I'm using input-selector to toggle between several RTSP streams.
> Whenever I toggle between two streams, I have a small duration where there's
> no valid I-frame to use for decoding so I get garbage.
>
> I thought that I could use a queue before the input-selector to store the
> last second of each stream, so when I will toggle between them, the decoder
> will decode (as fast as possible) the last second (which will include an
> I-frame), and will then get me my current frame decoded properly.
>
> I'm not sure what parameters I should use to achieve this behavior.
>
> The pipeline I have:
>
>
>
> Thank you for your time and help :)
>
>
>
> --
> Sent from: http://gstreamer-devel.966125.n4.nabble.com/
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

--
g. Marc
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[omer.tal]

Hi and thank you for your replay.

I couldn't however understand how does it guarantee that I get a full frame?
What if the pad was added during a p-frame? So the first frame I get is a
p-frame.

Anyway, I thought about something like that, but I do like to keep both
streams connected at all time to reduce the time it takes to perform the
connection.

I think that since data was flowing through the pipeline, I can always store
the last I-frame in a queue and just decode from there...
I'm just not sure if the way to go is to use a PadProbe to block the
inactive pads.



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Marc Leeman]

with uridecodebin you pass 2 things (of relevance here).

1. pass the uri (in this case rtsp)
2. pass the caps you want uridecodebin to produce (defaults to decoded
media, you might want to specify video/x-raw to avoid e.g. audio).

The decoder is in uridecodebin, so you get decoded frames.

I suspect that in your setup, you re-use the decoder by setting the
decoder *after* the input-selector. In the example, de decoder is in
front of the input-selector (hidden in uridecodebin).

This makes sure that you are switching in the decoded domain, not in
the compressed domain. At this point, frames have no temporal
dependencies (that produces the macro blocks in your case).

If you want to switch in the compressed domain and re-use thedecoder,
this is also possible; but in your use-case a bit more complex. You
will need to add an extra step on the produced pad of uridecodebin and
check if the buffers that are produced, are 'non-delta frames'. There
is a flag; *and* you will need to make certain, somehow, that you only
switch after getting the SPS/PPS in the stream (or out of band).

On Tue, 12 Jan 2021 at 14:30, omer.tal <[hidden email]> wrote:

>
> Hi and thank you for your replay.
>
> I couldn't however understand how does it guarantee that I get a full frame?
> What if the pad was added during a p-frame? So the first frame I get is a
> p-frame.
>
> Anyway, I thought about something like that, but I do like to keep both
> streams connected at all time to reduce the time it takes to perform the
> connection.
>
> I think that since data was flowing through the pipeline, I can always store
> the last I-frame in a queue and just decode from there...
> I'm just not sure if the way to go is to use a PadProbe to block the
> inactive pads.
>
>
>
> --
> Sent from: http://gstreamer-devel.966125.n4.nabble.com/
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

--
g. Marc
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel