pitch element breaks lip sync in Chrome

[Juan Navarro]

Hi,

It seems that the 'pitch' filter (in the soundtouch plugin,
gst-plugins-bad) introduces some kind of wrong timestamp, clock skew,
bad sequence, a combination of those, or maybe something different (but
probably related).

I'm seeing an accumulative delay in Chrome between video and audio in a
WebRTC call (_not_ using the new GStreamer's WebRTC element, yet) where
the source is sending a video+audio stream, and the audio is filtered
with the pitch element. Chrome is unable to perform the lip sync
successfully, and for some reason deduces that somehow the audio is
lagging behind the video (which is actually not), so it delays the video
indefinitely until the delay gets to Chrome's maximum, 10 seconds.

The net effect of this issue is practically the same as what happened in
this Chrome bug:
https://bugs.chromium.org/p/webrtc/issues/detail?id=5456 (just check the
screenshots)
with 'googCurrentDelayMs' and 'goodMinPlayoutDelayMs' growing linearly.
At that time it happened to be Chrome wrongly using the webcam's
timestamp, which had a different clock rate than the system's timestamp.

But, in this case I don't think it's a Chrome bug; I have verified that
this is caused by the GStreamer's 'pitch' filter, by sourcing this
simple test pipeline to my custom WebRTC source element:

... -> (raw audio)
   -> audioconvert -> audioresample -> pitch ->
   -> audioconvert -> audioresample -> WebRTC

(Probably most, if not all of those audioconvert/audioresample elements
are not needed, I added them just to fall on the safe side)

This generates the mentioned delay in the video presentation handled by
Chrome.

However nothing of this happens if the 'pitch' element is removed and
any other is used, e.g. an 'scaletempo' element:

... -> (raw audio)
   -> audioconvert -> audioresample -> scaletempo ->
   -> audioconvert -> audioresample -> WebRTC

This produces a normal lip sync result in Chrome. Delay (latency) stays
at around 100, 150 ms.

I've been reading google's WebRTC code, wanting to know exactly what is
the name of the value that is to blame:
https://cs.chromium.org/chromium/src/third_party/webrtc/video/stream_synchronization.cc

but finding out what is the correct function chain is difficult, and I'm
still not sure of exactly *what* is making Chrome confused and wrongly
assuming that the audio is behind the video, when it's not.

I have cherry-picked and applied all commits that touched the file
'./ext/soundtouch/gstpitch.cc' into a custom built version of
gst-plugins-bad, but the issue persists so it's not a matter of trying
the latest code (after a lot of time without changes, the pitch filter
received some patches in June so I wanted to test if those helped...)

Only idea I have is that the pitch element is missing some sequence
number handling, or something about the pipeline's clock rate... but I'm
out of ideas.


Please help :)
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Sorry for top posting, but this seems like a good bug report to me. Can
you paste this on a bug in bugs.gnome.org please ?

Le jeudi 06 septembre 2018 à 19:11 +0200, Juan Navarro a écrit :

> Hi,
>
> It seems that the 'pitch' filter (in the soundtouch plugin,
> gst-plugins-bad) introduces some kind of wrong timestamp, clock skew,
> bad sequence, a combination of those, or maybe something different (but
> probably related).
>
> I'm seeing an accumulative delay in Chrome between video and audio in a
> WebRTC call (_not_ using the new GStreamer's WebRTC element, yet) where
> the source is sending a video+audio stream, and the audio is filtered
> with the pitch element. Chrome is unable to perform the lip sync
> successfully, and for some reason deduces that somehow the audio is
> lagging behind the video (which is actually not), so it delays the video
> indefinitely until the delay gets to Chrome's maximum, 10 seconds.
>
> The net effect of this issue is practically the same as what happened in
> this Chrome bug:
> https://bugs.chromium.org/p/webrtc/issues/detail?id=5456 (just check the
> screenshots)
> with 'googCurrentDelayMs' and 'goodMinPlayoutDelayMs' growing linearly.
> At that time it happened to be Chrome wrongly using the webcam's
> timestamp, which had a different clock rate than the system's timestamp.
>
> But, in this case I don't think it's a Chrome bug; I have verified that
> this is caused by the GStreamer's 'pitch' filter, by sourcing this
> simple test pipeline to my custom WebRTC source element:
>
> ... -> (raw audio)
>    -> audioconvert -> audioresample -> pitch ->
>    -> audioconvert -> audioresample -> WebRTC
>
> (Probably most, if not all of those audioconvert/audioresample elements
> are not needed, I added them just to fall on the safe side)
>
> This generates the mentioned delay in the video presentation handled by
> Chrome.
>
> However nothing of this happens if the 'pitch' element is removed and
> any other is used, e.g. an 'scaletempo' element:
>
> ... -> (raw audio)
>    -> audioconvert -> audioresample -> scaletempo ->
>    -> audioconvert -> audioresample -> WebRTC
>
> This produces a normal lip sync result in Chrome. Delay (latency) stays
> at around 100, 150 ms.
>
> I've been reading google's WebRTC code, wanting to know exactly what is
> the name of the value that is to blame:
> https://cs.chromium.org/chromium/src/third_party/webrtc/video/stream_synchronization.cc
>
> but finding out what is the correct function chain is difficult, and I'm
> still not sure of exactly *what* is making Chrome confused and wrongly
> assuming that the audio is behind the video, when it's not.
>
> I have cherry-picked and applied all commits that touched the file
> './ext/soundtouch/gstpitch.cc' into a custom built version of
> gst-plugins-bad, but the issue persists so it's not a matter of trying
> the latest code (after a lot of time without changes, the pitch filter
> received some patches in June so I wanted to test if those helped...)
>
> Only idea I have is that the pitch element is missing some sequence
> number handling, or something about the pipeline's clock rate... but I'm
> out of ideas.
>
>
> Please help :)
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Juan Navarro]

I'll do; normally I prefer to first ask in the mail list instead of
assuming it's a bug. Most times it's not.
Currently I'm gathering as much info as possible before reporting, for
that I also posted in discuss-webrtc:
https://groups.google.com/d/topic/discuss-webrtc/k9so0_12pqM/discussion

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le jeu. 6 sept. 2018 15:29, Juan Navarro <[hidden email]> a écrit :

I'll do; normally I prefer to first ask in the mail list instead of
assuming it's a bug. Most times it's not.
Currently I'm gathering as much info as possible before reporting, for
that I also posted in discuss-webrtc:
https://groups.google.com/d/topic/discuss-webrtc/k9so0_12pqM/discussion

No worries, it all makes sense. To further investigate, I would enable some debug in GST. GST_DEBUG="*SCHED*:5" and then look at that timestamp before after the pitch element, and also look at the buffer size, it raw audio at same rate same format, so it should not add or remove anything.

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel