Clock skewing algorithm

[Tvrtko Ursulin]

Hi Wim, all,

I was just talking with Jan Schmidt on #gstreamer about a curious clock
skewing observation you might sched some light at.

Essentially I am debugging AV sync issues when streaming video from v4l2src
and audio from alsasrc. To narrow it down I went on to test purely audio
pipeline in the form of "alsasrc ! audio/x-raw-int,rate=48000 ! queue min-
threshold-time=2000000000 ! alsasink".

What happens occasionally is that audio gets choppy and then stops completely.
I have captured logs of one such incident, pulled out some information from
clock skewing and plotted it with gnuplot for which I am attaching two plots.

On the full plot you can see that after the second large skew spike average
skew calculation seems to go wrong and starts oscillating between positive and
negative values. On the zoomed in view you can see that there is no negative
skew values, while average keeps oscillating. However on the first occurence of
such large spike it seems to keep the average correct.

Basically I have two question here. First is whether wrong average skew is
expected to have negative effects on audio, including stopping completely, and
then secondly whether it looks like a bug in average skew calculation which
should be investigated deeper?

Regards,

Tvrtko

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Wim Taymans]

On 10/11/2011 03:15 PM, Tvrtko Ursulin wrote:

>
> Hi Wim, all,
>
> I was just talking with Jan Schmidt on #gstreamer about a curious clock
> skewing observation you might sched some light at.
>
> Essentially I am debugging AV sync issues when streaming video from v4l2src
> and audio from alsasrc. To narrow it down I went on to test purely audio
> pipeline in the form of "alsasrc ! audio/x-raw-int,rate=48000 ! queue min-
> threshold-time=2000000000 ! alsasink".

This pipeline is rather impossible to run, the queue will always overrun
and block the source from producing more samples. You can see this with
gst-launch with:

gstbaseaudiosrc.c(838): gst_base_audio_src_create ():
/GstPipeline:pipeline0/GstAlsaSrc:alsasrc0:
Dropped 37920 samples. This is most likely because downstream can't keep
up and is consuming samples too slowly.

The reason is that the default max size of the queue is 1 second and you
ask it to buffer 2 seconds at least. You need to increase the
max-size-time to something bigger than 2 seconds. We should make queue
error on these invalid configurations..

It does not seem to cause and wrong skews here, though. Things just get
choppy while the source drops samples and the sink consumes the
occasional bursts of samples.

>
> What happens occasionally is that audio gets choppy and then stops completely.
> I have captured logs of one such incident, pulled out some information from
> clock skewing and plotted it with gnuplot for which I am attaching two plots.
>
> On the full plot you can see that after the second large skew spike average
> skew calculation seems to go wrong and starts oscillating between positive and
> negative values. On the zoomed in view you can see that there is no negative
> skew values, while average keeps oscillating. However on the first occurence of
> such large spike it seems to keep the average correct.
>
> Basically I have two question here. First is whether wrong average skew is
> expected to have negative effects on audio, including stopping completely, and
> then secondly whether it looks like a bug in average skew calculation which
> should be investigated deeper?

A wrong average skew could indeed mess up the timing.

Wim
>
> Regards,
>
> Tvrtko

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Tvrtko Ursulin]

On Tuesday 11 Oct 2011 17:10:05 Wim Taymans wrote:

> On 10/11/2011 03:15 PM, Tvrtko Ursulin wrote:
> > Hi Wim, all,
> >
> > I was just talking with Jan Schmidt on #gstreamer about a curious clock
> > skewing observation you might sched some light at.
> >
> > Essentially I am debugging AV sync issues when streaming video from
> > v4l2src and audio from alsasrc. To narrow it down I went on to test
> > purely audio pipeline in the form of "alsasrc !
> > audio/x-raw-int,rate=48000 ! queue min- threshold-time=2000000000 !
> > alsasink".
>
> This pipeline is rather impossible to run, the queue will always overrun
> and block the source from producing more samples. You can see this with
> gst-launch with:
>
> gstbaseaudiosrc.c(838): gst_base_audio_src_create ():
> /GstPipeline:pipeline0/GstAlsaSrc:alsasrc0:
> Dropped 37920 samples. This is most likely because downstream can't keep
> up and is consuming samples too slowly.
>
> The reason is that the default max size of the queue is 1 second and you
> ask it to buffer 2 seconds at least. You need to increase the
> max-size-time to something bigger than 2 seconds. We should make queue
> error on these invalid configurations..

Sorry I just accidentally omitted the part (when writing this email) where I  
set max-size-time to 4 seconds and buffer and size limits to zero. So in fact
the pipeline with which I tested did not have this error.
 
> It does not seem to cause and wrong skews here, though. Things just get
> choppy while the source drops samples and the sink consumes the
> occasional bursts of samples.

> > What happens occasionally is that audio gets choppy and then stops
> > completely. I have captured logs of one such incident, pulled out some
> > information from clock skewing and plotted it with gnuplot for which I
> > am attaching two plots.
> >
> > On the full plot you can see that after the second large skew spike
> > average skew calculation seems to go wrong and starts oscillating
> > between positive and negative values. On the zoomed in view you can see
> > that there is no negative skew values, while average keeps oscillating.
> > However on the first occurence of such large spike it seems to keep the
> > average correct.
> >
> > Basically I have two question here. First is whether wrong average skew
> > is expected to have negative effects on audio, including stopping
> > completely, and then secondly whether it looks like a bug in average
> > skew calculation which should be investigated deeper?
>
> A wrong average skew could indeed mess up the timing.

Let's look at the part where it is going bad then:

<sample_num> <time> <skew> <avg_skew>
28504 0:05:12.396161786 20999999 18596044
28505 0:05:12.775732497 218666667 24848250

    sink->priv->avg_skew = (31 * sink->priv->avg_skew + skew) / 32;

(31 * 24848250 + 218666667 ) / 32 = 30905075

  mdrift = sink->priv->drift_tolerance * 1000; = 40000000;
  mdrift2 = mdrift / 2; = 20000000

  if (sink->priv->avg_skew > mdrift2) {
...
    sink->priv->avg_skew -= mdrift; = -9094925

avg_skew goes negative here, although strangely it does not match the next
real data point. I assumed drift tolerance is a default 40ms, but maybe it can
get auto-adjusted at runtime somehow? Or I just made an oversight in following
the code and doing the arithmetic...

Anyway, avg_skew from now on oscillates:

28506 0:05:13.150555343 360000000 -3428257
28507 0:05:13.154222121 353500000 7725751
28508 0:05:13.154989086 353500000 18531196
28509 0:05:13.155440401 354333333 29025012
28510 0:05:13.194284943 310833333 -918477
28511 0:05:13.194986604 310833333 8823767

Even when this skew spike passes - samples 28600 onward.

Thanks,

Tvrtko
_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Felipe Contreras]

On Wed, Oct 12, 2011 at 11:26 AM, Tvrtko Ursulin
<[hidden email]> wrote:
> Anyway, avg_skew from now on oscillates:

Have you tried these patches?
https://bugzilla.gnome.org/show_bug.cgi?id=640859

--
Felipe Contreras
_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Tvrtko Ursulin]

On Sunday 06 Nov 2011 05:22:45 Felipe Contreras wrote:
> On Wed, Oct 12, 2011 at 11:26 AM, Tvrtko Ursulin
>
> <[hidden email]> wrote:
> > Anyway, avg_skew from now on oscillates:
> Have you tried these patches?
> https://bugzilla.gnome.org/show_bug.cgi?id=640859

Unfortunately I did not find your report when our problem was actual. I only
glanced over it now but it does seem like there is something there.

We have solved our use case via some workarounds in other layers so
unfortunately I have no time to further play with this at the moment.

Tvrtko
_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel