RTSP latency issues, effect of sync=false

[Guillermo Rodriguez Garcia-2]

Hi all,

I am using GStreamer to fetch live video from IP cameras (RTSP /
H.264). My application does not require audio.

In some cases I am seeing stuttering video with many dropped frames,
and I see the typical "There may be a timestamping problem, or this
computer is too slow" messages on GStreamer's debug output.

Nonetheless I don't think this is a performance issue in the receiver:
Sometimes this happens for a low-bandwidth video stream (640x480, 15
fps, < 1 Mbps), and the problems will go away if I actually _increase_
the image resolution, frame rate, or bit rate in the camera itself.

I have checked the following:
- Adding queue elements between other elements in the pipeline doesn't
seem to help
- If I increase the latency parameter in the rtspsrc element (by
default I am setting this to 200ms), this fixes the problem (but also
introduces undesirable delays)
- If I add sync=false to the sink, this also fixes the problem

The last option seems to always fix the problem without introducing
any adverse side-effects (that I am aware of). My question is: Is
there any reason not to use this? Anything I should bear in mind? Note
that all streams are live, at any given time I am only using one
source and one sink (not mixing anything), and I am not using audio at
all.

Thank you,

Guillermo Rodriguez Garcia
[hidden email]
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le mercredi 22 juillet 2020 à 13:21 +0200, Guillermo Rodriguez Garcia a
écrit :

> Hi all,
>
> I am using GStreamer to fetch live video from IP cameras (RTSP /
> H.264). My application does not require audio.
>
> In some cases I am seeing stuttering video with many dropped frames,
> and I see the typical "There may be a timestamping problem, or this
> computer is too slow" messages on GStreamer's debug output.
>
> Nonetheless I don't think this is a performance issue in the receiver:
> Sometimes this happens for a low-bandwidth video stream (640x480, 15
> fps, < 1 Mbps), and the problems will go away if I actually _increase_
> the image resolution, frame rate, or bit rate in the camera itself.

That seems similar to the effect of burst in the network, like when
using wifi.

>
> I have checked the following:
> - Adding queue elements between other elements in the pipeline doesn't
> seem to help
> - If I increase the latency parameter in the rtspsrc element (by
> default I am setting this to 200ms), this fixes the problem (but also
> introduces undesirable delays)
> - If I add sync=false to the sink, this also fixes the problem
>
> The last option seems to always fix the problem without introducing
> any adverse side-effects (that I am aware of). My question is: Is
> there any reason not to use this? Anything I should bear in mind? Note
> that all streams are live, at any given time I am only using one
> source and one sink (not mixing anything), and I am not using audio at
> all.

With sync=false, the playback rate will be unstable during these
bursts. On some wifi, I could see clear freeze and speedup effect from
that. It will then get worst when things get congested.

What happens is that the latency is a deadline, when the frames get
near that deadline, GStreamer try to play the "cachtup" game. But
catching up on live stream isn't something reliable or doable. In fact,
I've had better result with qos=0 for this case.

What advance apps can do is monitor CPU usage and buffer drops. If the
CPU usage is low, but you see a lot of buffer drops, you could
increasing the latency (this setting can be changed live). This would
effectively let you receiver adapt to the network conditions.

It's of course up to you, for some cases, smooth playback isn't
strictly required.

>
> Thank you,
>
> Guillermo Rodriguez Garcia
> [hidden email]
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Guillermo Rodriguez Garcia-2]

Hello,

El mié., 22 jul. 2020 a las 22:03, Nicolas Dufresne
(<[hidden email]>) escribió:

>
> Le mercredi 22 juillet 2020 à 13:21 +0200, Guillermo Rodriguez Garcia a
> écrit :
> > Hi all,
> >
> > I am using GStreamer to fetch live video from IP cameras (RTSP /
> > H.264). My application does not require audio.
> >
> > In some cases I am seeing stuttering video with many dropped frames,
> > and I see the typical "There may be a timestamping problem, or this
> > computer is too slow" messages on GStreamer's debug output.
> >
> > Nonetheless I don't think this is a performance issue in the receiver:
> > Sometimes this happens for a low-bandwidth video stream (640x480, 15
> > fps, < 1 Mbps), and the problems will go away if I actually _increase_
> > the image resolution, frame rate, or bit rate in the camera itself.
>
> That seems similar to the effect of burst in the network, like when
> using wifi.

I see. Is there any way to verify this hypothesis?

Also.. if this (network issues) was the cause, why would the problem
only happen for a given configuration, and then go away if I e.g.
increase the image resolution (-> higher bandwidth requirements) ?

>
> >
> > I have checked the following:
> > - Adding queue elements between other elements in the pipeline doesn't
> > seem to help
> > - If I increase the latency parameter in the rtspsrc element (by
> > default I am setting this to 200ms), this fixes the problem (but also
> > introduces undesirable delays)
> > - If I add sync=false to the sink, this also fixes the problem
> >
> > The last option seems to always fix the problem without introducing
> > any adverse side-effects (that I am aware of). My question is: Is
> > there any reason not to use this? Anything I should bear in mind? Note
> > that all streams are live, at any given time I am only using one
> > source and one sink (not mixing anything), and I am not using audio at
> > all.
>
> With sync=false, the playback rate will be unstable during these
> bursts. On some wifi, I could see clear freeze and speedup effect from
> that. It will then get worst when things get congested.

I understand sync=false means "play incoming frames as soon as they
arrive, ignoring the timestamp" (i.e. "don't discard frames that
arrive too late"). Is this correct?

I then assume that the speedup is caused by the sink rendering a bunch
of frames that "arrived together"?

Is there any way to tell Gstreamer to "play incoming frames as soon as
they arrive, i.e. don't discard frames that arrive too late, _but_ if
there are multiple frames waiting, discard the older ones" ?

>
> What happens is that the latency is a deadline, when the frames get
> near that deadline, GStreamer try to play the "cachtup" game. But
> catching up on live stream isn't something reliable or doable. In fact,
> I've had better result with qos=0 for this case.

Just tried this (qos=0), but it doesn't seem to help.

>
> What advance apps can do is monitor CPU usage and buffer drops. If the
> CPU usage is low, but you see a lot of buffer drops, you could
> increasing the latency (this setting can be changed live). This would
> effectively let you receiver adapt to the network conditions.

Understood, thank you.

>
> It's of course up to you, for some cases, smooth playback isn't
> strictly required.

Thanks,

Guillermo Rodriguez Garcia
[hidden email]
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le jeudi 23 juillet 2020 à 19:17 +0200, Guillermo Rodriguez Garcia a
écrit :

> Hello,
>
> El mié., 22 jul. 2020 a las 22:03, Nicolas Dufresne
> (<[hidden email]>) escribió:
> > Le mercredi 22 juillet 2020 à 13:21 +0200, Guillermo Rodriguez Garcia a
> > écrit :
> > > Hi all,
> > >
> > > I am using GStreamer to fetch live video from IP cameras (RTSP /
> > > H.264). My application does not require audio.
> > >
> > > In some cases I am seeing stuttering video with many dropped frames,
> > > and I see the typical "There may be a timestamping problem, or this
> > > computer is too slow" messages on GStreamer's debug output.
> > >
> > > Nonetheless I don't think this is a performance issue in the receiver:
> > > Sometimes this happens for a low-bandwidth video stream (640x480, 15
> > > fps, < 1 Mbps), and the problems will go away if I actually _increase_
> > > the image resolution, frame rate, or bit rate in the camera itself.
> >
> > That seems similar to the effect of burst in the network, like when
> > using wifi.
>
> I see. Is there any way to verify this hypothesis?
>
> Also.. if this (network issues) was the cause, why would the problem
> only happen for a given configuration, and then go away if I e.g.
> increase the image resolution (-> higher bandwidth requirements) ?

When you increase the resolution, you also increase the data rate. So
for the wifi case, which optimize throughput by bursting data, it means
the burst buffer will be more often full, so it will be sending data
more often, reducing the latency.

>
> > > I have checked the following:
> > > - Adding queue elements between other elements in the pipeline doesn't
> > > seem to help
> > > - If I increase the latency parameter in the rtspsrc element (by
> > > default I am setting this to 200ms), this fixes the problem (but also
> > > introduces undesirable delays)
> > > - If I add sync=false to the sink, this also fixes the problem
> > >
> > > The last option seems to always fix the problem without introducing
> > > any adverse side-effects (that I am aware of). My question is: Is
> > > there any reason not to use this? Anything I should bear in mind? Note
> > > that all streams are live, at any given time I am only using one
> > > source and one sink (not mixing anything), and I am not using audio at
> > > all.
> >
> > With sync=false, the playback rate will be unstable during these
> > bursts. On some wifi, I could see clear freeze and speedup effect from
> > that. It will then get worst when things get congested.
>
> I understand sync=false means "play incoming frames as soon as they
> arrive, ignoring the timestamp" (i.e. "don't discard frames that
> arrive too late"). Is this correct?

Correct.

>
> I then assume that the speedup is caused by the sink rendering a bunch
> of frames that "arrived together"?

Together or closer by then what they were initially produced. Also
correct.

>
> Is there any way to tell Gstreamer to "play incoming frames as soon as
> they arrive, i.e. don't discard frames that arrive too late, _but_ if
> there are multiple frames waiting, discard the older ones" ?

The sink element have no queues, so it does know if there is some late
frame already there. So I believe the answer is no.

>
> > What happens is that the latency is a deadline, when the frames get
> > near that deadline, GStreamer try to play the "cachtup" game. But
> > catching up on live stream isn't something reliable or doable. In fact,
> > I've had better result with qos=0 for this case.
>
> Just tried this (qos=0), but it doesn't seem to help.
>
> > What advance apps can do is monitor CPU usage and buffer drops. If the
> > CPU usage is low, but you see a lot of buffer drops, you could
> > increasing the latency (this setting can be changed live). This would
> > effectively let you receiver adapt to the network conditions.
>
> Understood, thank you.
>
> > It's of course up to you, for some cases, smooth playback isn't
> > strictly required.
>
> Thanks,
>
> Guillermo Rodriguez Garcia
> [hidden email]
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Guillermo Rodriguez Garcia-2]

El jue., 23 jul. 2020 a las 22:57, Nicolas Dufresne
(<[hidden email]>) escribió:

>
> Le jeudi 23 juillet 2020 à 19:17 +0200, Guillermo Rodriguez Garcia a
> écrit :
> > Hello,
> >
> > El mié., 22 jul. 2020 a las 22:03, Nicolas Dufresne
> > (<[hidden email]>) escribió:
> > > Le mercredi 22 juillet 2020 à 13:21 +0200, Guillermo Rodriguez Garcia a
> > > écrit :
> > > > Hi all,
> > > >
> > > > I am using GStreamer to fetch live video from IP cameras (RTSP /
> > > > H.264). My application does not require audio.
> > > >
> > > > In some cases I am seeing stuttering video with many dropped frames,
> > > > and I see the typical "There may be a timestamping problem, or this
> > > > computer is too slow" messages on GStreamer's debug output.
> > > >
> > > > Nonetheless I don't think this is a performance issue in the receiver:
> > > > Sometimes this happens for a low-bandwidth video stream (640x480, 15
> > > > fps, < 1 Mbps), and the problems will go away if I actually _increase_
> > > > the image resolution, frame rate, or bit rate in the camera itself.
> > >
> > > That seems similar to the effect of burst in the network, like when
> > > using wifi.
> >
> > I see. Is there any way to verify this hypothesis?
> >
> > Also.. if this (network issues) was the cause, why would the problem
> > only happen for a given configuration, and then go away if I e.g.
> > increase the image resolution (-> higher bandwidth requirements) ?
>
> When you increase the resolution, you also increase the data rate. So
> for the wifi case, which optimize throughput by bursting data, it means
> the burst buffer will be more often full, so it will be sending data
> more often, reducing the latency.

OK, makes sense (even though I am not using Wifi -- all devices are on
the same wired LAN)

> > > > I have checked the following:
> > > > - Adding queue elements between other elements in the pipeline doesn't
> > > > seem to help
> > > > - If I increase the latency parameter in the rtspsrc element (by
> > > > default I am setting this to 200ms), this fixes the problem (but also
> > > > introduces undesirable delays)
> > > > - If I add sync=false to the sink, this also fixes the problem
> > > >
> > > > The last option seems to always fix the problem without introducing
> > > > any adverse side-effects (that I am aware of). My question is: Is
> > > > there any reason not to use this? Anything I should bear in mind? Note
> > > > that all streams are live, at any given time I am only using one
> > > > source and one sink (not mixing anything), and I am not using audio at
> > > > all.
> > >
> > > With sync=false, the playback rate will be unstable during these
> > > bursts. On some wifi, I could see clear freeze and speedup effect from
> > > that. It will then get worst when things get congested.
> >
> > I understand sync=false means "play incoming frames as soon as they
> > arrive, ignoring the timestamp" (i.e. "don't discard frames that
> > arrive too late"). Is this correct?
>
> Correct.
>
> >
> > I then assume that the speedup is caused by the sink rendering a bunch
> > of frames that "arrived together"?
>
> Together or closer by then what they were initially produced. Also
> correct.
>
> >
> > Is there any way to tell Gstreamer to "play incoming frames as soon as
> > they arrive, i.e. don't discard frames that arrive too late, _but_ if
> > there are multiple frames waiting, discard the older ones" ?
>
> The sink element have no queues, so it does know if there is some late
> frame already there. So I believe the answer is no.

Uhm.. but then for the same reason, I would expect the "speedup
effect" to be rather limited.
I mean, since the sink has no queues (and in fact I am not using
queues anywhere else in the pipeline) there shouldn't be that too many
"late" frames to play -- only the bunch of frames that were sent "more
or less together" due to the network bursts you mentioned.

Anyway my problem seems to be that when the frames arrive, the sink
discards them because of the timestamp (i.e. this is not a performance
problem, but a latency problem). Using sync=false helps because the
sink will not discard frames, but is subject to the speedup problems
you describe.

Is there any way to configure the sink so that it does not drop late
frames (or at least to be a bit more lenient) without using sync=false
?

Thank you for the help and the feedback.

BR,

Guillermo

>
> >
> > > What happens is that the latency is a deadline, when the frames get
> > > near that deadline, GStreamer try to play the "cachtup" game. But
> > > catching up on live stream isn't something reliable or doable. In fact,
> > > I've had better result with qos=0 for this case.
> >
> > Just tried this (qos=0), but it doesn't seem to help.
> >
> > > What advance apps can do is monitor CPU usage and buffer drops. If the
> > > CPU usage is low, but you see a lot of buffer drops, you could
> > > increasing the latency (this setting can be changed live). This would
> > > effectively let you receiver adapt to the network conditions.
> >
> > Understood, thank you.
> >
> > > It's of course up to you, for some cases, smooth playback isn't
> > > strictly required.
> >
> > Thanks,
> >
> > Guillermo Rodriguez Garcia
> > [hidden email]
> > _______________________________________________
> > gstreamer-devel mailing list
> > [hidden email]
> > https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

--
Guillermo Rodriguez Garcia
[hidden email]
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le vendredi 24 juillet 2020 à 14:33 +0200, Guillermo Rodriguez Garcia a
écrit :

> > The sink element have no queues, so it does know if there is some late
> > frame already there. So I believe the answer is no.
>
> Uhm.. but then for the same reason, I would expect the "speedup
> effect" to be rather limited.
> I mean, since the sink has no queues (and in fact I am not using
> queues anywhere else in the pipeline) there shouldn't be that too many
> "late" frames to play -- only the bunch of frames that were sent "more
> or less together" due to the network bursts you mentioned.
>
> Anyway my problem seems to be that when the frames arrive, the sink
> discards them because of the timestamp (i.e. this is not a performance
> problem, but a latency problem). Using sync=false helps because the
> sink will not discard frames, but is subject to the speedup problems
> you describe.
>
> Is there any way to configure the sink so that it does not drop late
> frames (or at least to be a bit more lenient) without using sync=false
> ?

Sinks have a max-lateness property, if you set it to -1, then it will
behave like udpsink, which is it will send the data on sync, or late
and won't drop. If you PC is too slow, it will never catch up.

Nicolas

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Guillermo Rodriguez Garcia-2]

El vie., 24 jul. 2020 a las 16:04, Nicolas Dufresne
(<[hidden email]>) escribió:

>
> Le vendredi 24 juillet 2020 à 14:33 +0200, Guillermo Rodriguez Garcia a
> écrit :
> > > The sink element have no queues, so it does know if there is some late
> > > frame already there. So I believe the answer is no.
> >
> > Uhm.. but then for the same reason, I would expect the "speedup
> > effect" to be rather limited.
> > I mean, since the sink has no queues (and in fact I am not using
> > queues anywhere else in the pipeline) there shouldn't be that too many
> > "late" frames to play -- only the bunch of frames that were sent "more
> > or less together" due to the network bursts you mentioned.
> >
> > Anyway my problem seems to be that when the frames arrive, the sink
> > discards them because of the timestamp (i.e. this is not a performance
> > problem, but a latency problem). Using sync=false helps because the
> > sink will not discard frames, but is subject to the speedup problems
> > you describe.
> >
> > Is there any way to configure the sink so that it does not drop late
> > frames (or at least to be a bit more lenient) without using sync=false
> > ?
>
> Sinks have a max-lateness property, if you set it to -1, then it will
> behave like udpsink, which is it will send the data on sync, or late
> and won't drop. If you PC is too slow, it will never catch up.

I have tried to set max-lateness to -1 and it doesn't help; I still
see stuttering video.
So I guess this means my hypothesis was wrong -- the sink is not
discarding frames because they are late.

I am a bit lost now.
If neither max-lateness=-1 nor qos=0 help, but sync=false does, then
what might be the problem?

Guillermo
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel