Audio/video sync problems

[Daniel Drake-5]

Hi,

I'm diagnosing a bug in OLPC's Record activity. We use gstreamer to
capture audio and video from the microphone and webcam, allowing the
laptop to be used like a camcorder.

The problem is that the A/V sync on the output file is really bad.

I'm wondering if anyone can suggest a place to start investigating
this, or anything else to try. Happy to jump into the code, but would
love to have some suggestion of where to start.

This is brand new hardware (new "1.5" version of the laptop) so
unfortunately we don't have a prior baseline to be working towards.

Environment:
Fedora 11
gstreamer-0.10.25
gstreamer-plugins-base-0.10.25
gstreamer-plugins-good-0.10.16
No PulseAudio
alsa-lib-1.0.23
Linux 2.6.31
XO-1.5 laptop (VIA 1ghz CPU, HDA sound with Conexant codec)

I've narrowed down the problem to a simple pygst test program. The
program encodes the webcam input on-the-fly to theora, while recording
audio to a .wav file.
After recording for a few seconds, it runs a 2nd pipeline which
transcodes the .wav into vorbis, and muxes it with the .ogg theora
file, resulting in a vorbis+theora .ogg output file

(the reason for doing it in 2 passes is that the system is not capable
of recording smooth video while encoding to both theora and vorbis
simultaneously)

Test program:
http://dev.laptop.org/~dsd/20100428/record.py

Intermediate theora:
http://dev.laptop.org/~dsd/20100428/video.ogg

Intermediate .wav:
http://dev.laptop.org/~dsd/20100428/audio.wav

Resultant file with obvious bad sync:
http://dev.laptop.org/~dsd/20100428/output.ogg

Logs from running with GST_DEBUG=2
http://dev.laptop.org/~dsd/20100428/log2

Logs from running with GST_DEBUG=4
http://dev.laptop.org/~dsd/20100428/log4.gz

Any tips appreciated

cheers
Daniel

------------------------------------------------------------------------------
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

[Stefan Sauer]

Am 28.04.2010 22:13, schrieb Daniel Drake:

> Hi,
>
> I'm diagnosing a bug in OLPC's Record activity. We use gstreamer to
> capture audio and video from the microphone and webcam, allowing the
> laptop to be used like a camcorder.
>
> The problem is that the A/V sync on the output file is really bad.
>
> I'm wondering if anyone can suggest a place to start investigating
> this, or anything else to try. Happy to jump into the code, but would
> love to have some suggestion of where to start.
>
> This is brand new hardware (new "1.5" version of the laptop) so
> unfortunately we don't have a prior baseline to be working towards.
>
> Environment:
> Fedora 11
> gstreamer-0.10.25
> gstreamer-plugins-base-0.10.25
> gstreamer-plugins-good-0.10.16
> No PulseAudio
> alsa-lib-1.0.23
> Linux 2.6.31
> XO-1.5 laptop (VIA 1ghz CPU, HDA sound with Conexant codec)
>
> I've narrowed down the problem to a simple pygst test program. The
> program encodes the webcam input on-the-fly to theora, while recording
> audio to a .wav file.
> After recording for a few seconds, it runs a 2nd pipeline which
> transcodes the .wav into vorbis, and muxes it with the .ogg theora
> file, resulting in a vorbis+theora .ogg output file
>
> (the reason for doing it in 2 passes is that the system is not capable
> of recording smooth video while encoding to both theora and vorbis
> simultaneously)

I was recently suggesting someone working on OLPC to record in one pass, but use
speex instead of vorbis and he reported that this makes it work fine.

You could try to replace wavenc/wavparse with aiffmux/aiffparse to check if its
wav plugin related.

Stefan

>
> Test program:
> http://dev.laptop.org/~dsd/20100428/record.py
>
> Intermediate theora:
> http://dev.laptop.org/~dsd/20100428/video.ogg
>
> Intermediate .wav:
> http://dev.laptop.org/~dsd/20100428/audio.wav
>
> Resultant file with obvious bad sync:
> http://dev.laptop.org/~dsd/20100428/output.ogg
>
> Logs from running with GST_DEBUG=2
> http://dev.laptop.org/~dsd/20100428/log2
>
> Logs from running with GST_DEBUG=4
> http://dev.laptop.org/~dsd/20100428/log4.gz
>
> Any tips appreciated
>
> cheers
> Daniel
>
> ------------------------------------------------------------------------------
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

------------------------------------------------------------------------------
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

[Arnout Vandecappelle]

 To facilitate debugging, you can insert identities in the audio and video
pipelines, and compare the timestamps reported by them (when using gst-
launch -v or displaying the messages in your messagecb).  That could help to
identify where the wrong timestamps are created.

 Regards,
 Arnout

On Wednesday 28 April 2010 21:13:07, Daniel Drake wrote:

> Hi,
>
> I'm diagnosing a bug in OLPC's Record activity. We use gstreamer to
> capture audio and video from the microphone and webcam, allowing the
> laptop to be used like a camcorder.
>
> The problem is that the A/V sync on the output file is really bad.
>
> I'm wondering if anyone can suggest a place to start investigating
> this, or anything else to try. Happy to jump into the code, but would
> love to have some suggestion of where to start.
>
> This is brand new hardware (new "1.5" version of the laptop) so
> unfortunately we don't have a prior baseline to be working towards.
>
> Environment:
> Fedora 11
> gstreamer-0.10.25
> gstreamer-plugins-base-0.10.25
> gstreamer-plugins-good-0.10.16
> No PulseAudio
> alsa-lib-1.0.23
> Linux 2.6.31
> XO-1.5 laptop (VIA 1ghz CPU, HDA sound with Conexant codec)
>
> I've narrowed down the problem to a simple pygst test program. The
> program encodes the webcam input on-the-fly to theora, while recording
> audio to a .wav file.
> After recording for a few seconds, it runs a 2nd pipeline which
> transcodes the .wav into vorbis, and muxes it with the .ogg theora
> file, resulting in a vorbis+theora .ogg output file
>
> (the reason for doing it in 2 passes is that the system is not capable
> of recording smooth video while encoding to both theora and vorbis
> simultaneously)
>
> Test program:
> http://dev.laptop.org/~dsd/20100428/record.py
>
> Intermediate theora:
> http://dev.laptop.org/~dsd/20100428/video.ogg
>
> Intermediate .wav:
> http://dev.laptop.org/~dsd/20100428/audio.wav
>
> Resultant file with obvious bad sync:
> http://dev.laptop.org/~dsd/20100428/output.ogg
>
> Logs from running with GST_DEBUG=2
> http://dev.laptop.org/~dsd/20100428/log2
>
> Logs from running with GST_DEBUG=4
> http://dev.laptop.org/~dsd/20100428/log4.gz
>
> Any tips appreciated
>
> cheers
> Daniel
>
> -------------------------------------------------------------------------
> ----- _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>

--
Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 [phone]
Essensium/Mind                                     http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  31BB CF53 8660 6F88 345D  54CC A836 5879 20D7 CF43

------------------------------------------------------------------------------
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

[Daniel Drake-5]

On 28 April 2010 16:13, Daniel Drake <[hidden email]> wrote:
> I'm diagnosing a bug in OLPC's Record activity. We use gstreamer to
> capture audio and video from the microphone and webcam, allowing the
> laptop to be used like a camcorder.
>
> The problem is that the A/V sync on the output file is really bad.

Thanks for the suggestions!

I didn't get around to trying AIFF, but I did try speex + theora
on-the-fly, writing to an oggmux. The resultant file was unplayable.
Possible bug there.

Uncompressed AVI on-the-fly recording worked quite well, as did
on-the-fly recording to vorbis + theora using 2 very large queue
buffers.

But in the end, I solved the problem and obtained the best sync simply
by sticking with the 2-pass technique, but adding a queue buffer in
the audio recording path, after alsasrc. It seems like the queue is
able to smooth out the overruns/underruns that occur at the start of
recording, which was throwing off the sync for the whole recording
session.

I think this indicates a bug somewhere in gstreamer or alsa -- if the
audio recording can't keep up for whatever reason, I feel like it
should resync later on, but that's not working well.

Due to time pressures I think this is the "solution" we'll go with for now.

cheers,
Daniel

------------------------------------------------------------------------------
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

[Tim-Philipp Müller-2]

On Thu, 2010-04-29 at 12:14 -0300, Daniel Drake wrote:

> But in the end, I solved the problem and obtained the best sync simply
> by sticking with the 2-pass technique, but adding a queue buffer in
> the audio recording path, after alsasrc. It seems like the queue is
> able to smooth out the overruns/underruns that occur at the start of
> recording, which was throwing off the sync for the whole recording
> session.
>
> I think this indicates a bug somewhere in gstreamer or alsa -- if the
> audio recording can't keep up for whatever reason, I feel like it
> should resync later on, but that's not working well.

A queue between capture source and encoder is not a bad idea in any
case, but an audiorate element before the audio encoder might also help
smooth over any gaps in the input.

 Cheers
  -Tim


------------------------------------------------------------------------------
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel