Gstreamer error: clock problem (Azure VM)

[Joao Miguel Ferreira]

Hello list,

I am facing a problem using gstreamer on Azure VMs. I have no previous experience with Gstreamer, and maybe that is the real problem. But I need to get this up&running and I'm finding these issues specifically in Azure VMs.

We are using a python based wrapper for gstreamer which also plays as REST API. It's here: https://github.com/bbc/brave. It's a fine software and it helps us a lot.

So... my problem: we are overlaying a PNG file onto an RTMP stream and then pushing the result to an RTMP server (nginx for tests, twitch TV too). It works well on my laptop at 720p and 1080p. The setup runs on Docker, both on Debian OS and also on ubuntu VMs.

But it never works well on Azure VMs. I have tried several VM families (even with GPU) and I always get some issue. The most frequent one is:

GStreamer error: clock problem

The logs show additional information concerning latency:

 WARNING: [32m[   output1] [0m GStreamer warning from mux: gst-core-error-quark: GStreamer error: clock problem. (13)
 WARNING: [32m[   output1] [0m GStreamer warning debug: gstaggregator.c(1731): gst_aggregator_query_latency_unlocked (): /GstPipeline:pipeline2/GstFlvMux:mux:
Impossible to configure latency: max 0:00:02.000000000 < min 0:00:02.100000000. Add queues or other buffering elements.
 WARNING: [32m[   output1] [0m GStreamer warning message: GStreamer error: clock problem.

This does not happen at lower resolutions like 640x360

Could this be related to the fact that these are VMs ? Because on my laptop (Lenovo thinkpad) it works fine.

I would appreciate any suggestions you might have or sharing anyone's experience with this kind of deploy.

Thank you

João

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le vendredi 05 avril 2019 à 12:37 +0100, Joao Miguel Ferreira a écrit :

> Hello list,
>
> I am facing a problem using gstreamer on Azure VMs. I have no previous experience with Gstreamer, and maybe that is the real problem. But I need to get this up&running and I'm finding these issues specifically in Azure VMs.
>
> We are using a python based wrapper for gstreamer which also plays as REST API. It's here: https://github.com/bbc/brave. It's a fine software and it helps us a lot.
>
> So... my problem: we are overlaying a PNG file onto an RTMP stream and then pushing the result to an RTMP server (nginx for tests, twitch TV too). It works well on my laptop at 720p and 1080p. The setup runs on Docker, both on Debian OS and also on ubuntu VMs.
>
> But it never works well on Azure VMs. I have tried several VM families (even with GPU) and I always get some issue. The most frequent one is:
>
> GStreamer error: clock problem
>
> The logs show additional information concerning latency:
>
>  WARNING: [32m[   output1] [0m GStreamer warning from mux: gst-core-error-quark: GStreamer error: clock problem. (13)
>  WARNING: [32m[   output1] [0m GStreamer warning debug: gstaggregator.c(1731): gst_aggregator_query_latency_unlocked (): /GstPipeline:pipeline2/GstFlvMux:mux:
> Impossible to configure latency: max 0:00:02.000000000 < min 0:00:02.100000000. Add queues or other buffering elements.
>  WARNING: [32m[   output1] [0m GStreamer warning message: GStreamer error: clock problem.

This warning indicates a severe defect in your live pipeline. It means
that the sum of requested latency of each plugins is bigger then the
latency available at your muxer (aggregator). In this message, it says
that the maximum latency your pipeline can accumulate is 2s, but the
minimum latency (the amount of requested latency) is 2.1s. This
indicates that stream synchronization isn't possible. Typically that
happens when two branches have latency that is too different. E.g.:

   videotestsrc is-live=1 ! x264enc ! mux.
   audiotestsrc is-live-1 ! avenc_aac ! mux.

In this case, x264enc, which has pretty high quality default
configuration will requires around 1s (or more) of latency, while
avenc_aac will likely need less then 0.1s. So the audio will reach the
muxer earlier then video, which means audio need to be buffered. The
maximum latency is also an amount of buffering capability.

By default, flvmux is configured for non-live procession, hence have 0
latency. So it is expected the upstream branches have large enough
queues to buffer. In life scenarios, adding queues isn't very nice,
because generic 'queue' are not really latency aware. Fortunatly,
flvmux (GstAggregator) have a queuing mechanism. The size if configured
though the 'latency' property in nanosecond. In special cases, like
dynamic pipeline, flvmux might not be able to query the appropriate
latency. As an example, the latency of an up-coming branch might be
bigger then any of the initial branches. We have added "min-upstream-
latency" in order to accommodate this use case.
>
> This does not happen at lower resolutions like 640x360
>
> Could this be related to the fact that these are VMs ? Because on my laptop (Lenovo thinkpad) it works fine.

Laptop usually have a system tick running at higher frequency (or on
demand tick / tick less) which reduce latency and makes your desktop
more reponsive. While servers usually have slower tick to improves
throughput, but often increase scheduling latency. This type of latency
(along with processing latency) is quite variable, so GStreamer cannot
perfectly accommodate them.

>
> I would appreciate any suggestions you might have or sharing anyone's experience with this kind of deploy.
>
> Thank you
> João
>
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Joao Miguel Ferreira]

Hello Nicolas,

thank you for your explanations. I'm still working on this subject. There is a lot I need to understand.

If I come to any conclusions I will post the to the list.

thx

Joao

On Fri, Apr 5, 2019 at 3:16 PM Nicolas Dufresne <[hidden email]> wrote:

Le vendredi 05 avril 2019 à 12:37 +0100, Joao Miguel Ferreira a écrit :
> Hello list,
>
> I am facing a problem using gstreamer on Azure VMs. I have no previous experience with Gstreamer, and maybe that is the real problem. But I need to get this up&running and I'm finding these issues specifically in Azure VMs.
>
> We are using a python based wrapper for gstreamer which also plays as REST API. It's here: https://github.com/bbc/brave. It's a fine software and it helps us a lot.
>
> So... my problem: we are overlaying a PNG file onto an RTMP stream and then pushing the result to an RTMP server (nginx for tests, twitch TV too). It works well on my laptop at 720p and 1080p. The setup runs on Docker, both on Debian OS and also on ubuntu VMs.
>
> But it never works well on Azure VMs. I have tried several VM families (even with GPU) and I always get some issue. The most frequent one is:
>
> GStreamer error: clock problem
>
> The logs show additional information concerning latency:
>
>  WARNING: [32m[   output1] [0m GStreamer warning from mux: gst-core-error-quark: GStreamer error: clock problem. (13)
>  WARNING: [32m[   output1] [0m GStreamer warning debug: gstaggregator.c(1731): gst_aggregator_query_latency_unlocked (): /GstPipeline:pipeline2/GstFlvMux:mux:
> Impossible to configure latency: max 0:00:02.000000000 < min 0:00:02.100000000. Add queues or other buffering elements.
>  WARNING: [32m[   output1] [0m GStreamer warning message: GStreamer error: clock problem.

This warning indicates a severe defect in your live pipeline. It means
that the sum of requested latency of each plugins is bigger then the
latency available at your muxer (aggregator). In this message, it says
that the maximum latency your pipeline can accumulate is 2s, but the
minimum latency (the amount of requested latency) is 2.1s. This
indicates that stream synchronization isn't possible. Typically that
happens when two branches have latency that is too different. E.g.:

   videotestsrc is-live=1 ! x264enc ! mux.
   audiotestsrc is-live-1 ! avenc_aac ! mux.

In this case, x264enc, which has pretty high quality default
configuration will requires around 1s (or more) of latency, while
avenc_aac will likely need less then 0.1s. So the audio will reach the
muxer earlier then video, which means audio need to be buffered. The
maximum latency is also an amount of buffering capability.

By default, flvmux is configured for non-live procession, hence have 0
latency. So it is expected the upstream branches have large enough
queues to buffer. In life scenarios, adding queues isn't very nice,
because generic 'queue' are not really latency aware. Fortunatly,
flvmux (GstAggregator) have a queuing mechanism. The size if configured
though the 'latency' property in nanosecond. In special cases, like
dynamic pipeline, flvmux might not be able to query the appropriate
latency. As an example, the latency of an up-coming branch might be
bigger then any of the initial branches. We have added "min-upstream-
latency" in order to accommodate this use case.
>
> This does not happen at lower resolutions like 640x360
>
> Could this be related to the fact that these are VMs ? Because on my laptop (Lenovo thinkpad) it works fine.

Laptop usually have a system tick running at higher frequency (or on
demand tick / tick less) which reduce latency and makes your desktop
more reponsive. While servers usually have slower tick to improves
throughput, but often increase scheduling latency. This type of latency
(along with processing latency) is quite variable, so GStreamer cannot
perfectly accommodate them.

>
> I would appreciate any suggestions you might have or sharing anyone's experience with this kind of deploy.
>
> Thank you
> João
>
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Joao Miguel Ferreira]

Hello list and Nicolas,

After some dedication to this matter I was able to find a solution which consisted in reducing the "key-int-max" value (from 60 to 30) on the "x264enc" segment of my setup (roughly: "flvmux // rtmpsink // x264enc").

The python application that I am using is setting that value to 60. Reducing to 30 eliminates the problem. I believe that this change does not have any undesirable impacts, but as I said before I am not familiar with gstreamer. I could be doing something that causes other issues. Maybe the value 60 was used to obtain some other effect and I could be hurting it.

I would appreciate to know your thoughts on this topic :)

Thank you

João

On Fri, Apr 5, 2019 at 6:55 PM Joao Miguel Ferreira <[hidden email]> wrote:

Hello Nicolas,

thank you for your explanations. I'm still working on this subject. There is a lot I need to understand.

If I come to any conclusions I will post the to the list.

thx

Joao

On Fri, Apr 5, 2019 at 3:16 PM Nicolas Dufresne <[hidden email]> wrote:

Le vendredi 05 avril 2019 à 12:37 +0100, Joao Miguel Ferreira a écrit :
> Hello list,
>
> I am facing a problem using gstreamer on Azure VMs. I have no previous experience with Gstreamer, and maybe that is the real problem. But I need to get this up&running and I'm finding these issues specifically in Azure VMs.
>
> We are using a python based wrapper for gstreamer which also plays as REST API. It's here: https://github.com/bbc/brave. It's a fine software and it helps us a lot.
>
> So... my problem: we are overlaying a PNG file onto an RTMP stream and then pushing the result to an RTMP server (nginx for tests, twitch TV too). It works well on my laptop at 720p and 1080p. The setup runs on Docker, both on Debian OS and also on ubuntu VMs.
>
> But it never works well on Azure VMs. I have tried several VM families (even with GPU) and I always get some issue. The most frequent one is:
>
> GStreamer error: clock problem
>
> The logs show additional information concerning latency:
>
>  WARNING: [32m[   output1] [0m GStreamer warning from mux: gst-core-error-quark: GStreamer error: clock problem. (13)
>  WARNING: [32m[   output1] [0m GStreamer warning debug: gstaggregator.c(1731): gst_aggregator_query_latency_unlocked (): /GstPipeline:pipeline2/GstFlvMux:mux:
> Impossible to configure latency: max 0:00:02.000000000 < min 0:00:02.100000000. Add queues or other buffering elements.
>  WARNING: [32m[   output1] [0m GStreamer warning message: GStreamer error: clock problem.

This warning indicates a severe defect in your live pipeline. It means
that the sum of requested latency of each plugins is bigger then the
latency available at your muxer (aggregator). In this message, it says
that the maximum latency your pipeline can accumulate is 2s, but the
minimum latency (the amount of requested latency) is 2.1s. This
indicates that stream synchronization isn't possible. Typically that
happens when two branches have latency that is too different. E.g.:

   videotestsrc is-live=1 ! x264enc ! mux.
   audiotestsrc is-live-1 ! avenc_aac ! mux.

In this case, x264enc, which has pretty high quality default
configuration will requires around 1s (or more) of latency, while
avenc_aac will likely need less then 0.1s. So the audio will reach the
muxer earlier then video, which means audio need to be buffered. The
maximum latency is also an amount of buffering capability.

By default, flvmux is configured for non-live procession, hence have 0
latency. So it is expected the upstream branches have large enough
queues to buffer. In life scenarios, adding queues isn't very nice,
because generic 'queue' are not really latency aware. Fortunatly,
flvmux (GstAggregator) have a queuing mechanism. The size if configured
though the 'latency' property in nanosecond. In special cases, like
dynamic pipeline, flvmux might not be able to query the appropriate
latency. As an example, the latency of an up-coming branch might be
bigger then any of the initial branches. We have added "min-upstream-
latency" in order to accommodate this use case.
>
> This does not happen at lower resolutions like 640x360
>
> Could this be related to the fact that these are VMs ? Because on my laptop (Lenovo thinkpad) it works fine.

Laptop usually have a system tick running at higher frequency (or on
demand tick / tick less) which reduce latency and makes your desktop
more reponsive. While servers usually have slower tick to improves
throughput, but often increase scheduling latency. This type of latency
(along with processing latency) is quite variable, so GStreamer cannot
perfectly accommodate them.

>
> I would appreciate any suggestions you might have or sharing anyone's experience with this kind of deploy.
>
> Thank you
> João
>
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel