Automatic time alignment by signal analysis

[Benjamin Schwartz]

I often find myself wanting to edit multiple audio streams that were
recorded by separate devices.  For example, I might have a high quality
audio recorder, and a separate consumer videocamera with a low-quality
microphone.  The two devices, being separate, have different start times
and slightly different clock rates.

I would like to align these two tracks automatically based on analysis of
the audio signals' contents.  Does anyone know of a tool that does this?
If not, I am considering writing such a program, as I am interested in the
signal processing problem.

I would like to make this tool as friendly to gstreamer as possible, so
that it can be used in gst-world applications like PiTiVi.  However, I do
not know how to structure the program.  The program must perform analysis
on the entire duration of two different audio streams (potentially
multiple hours or longer) before determining the required offset and
stretch factor.  These parameters then need to be applied, possibly by
editing a gnonlin timeline.

How should I write such a program ... or have you already written it?

--Ben


_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sean McNamara-4]

Hi,

On Wed, Apr 20, 2011 at 12:50 PM, Benjamin M. Schwartz
<[hidden email]> wrote:

> I often find myself wanting to edit multiple audio streams that were
> recorded by separate devices.  For example, I might have a high quality
> audio recorder, and a separate consumer videocamera with a low-quality
> microphone.  The two devices, being separate, have different start times
> and slightly different clock rates.
>
> I would like to align these two tracks automatically based on analysis of
> the audio signals' contents.  Does anyone know of a tool that does this?
> If not, I am considering writing such a program, as I am interested in the
> signal processing problem.
>
> I would like to make this tool as friendly to gstreamer as possible, so
> that it can be used in gst-world applications like PiTiVi.  However, I do
> not know how to structure the program.  The program must perform analysis
> on the entire duration of two different audio streams (potentially
> multiple hours or longer) before determining the required offset and
> stretch factor.  These parameters then need to be applied, possibly by
> editing a gnonlin timeline.
>
> How should I write such a program ... or have you already written it?
>
> --Ben

It may not be what you're looking for (no integration with Gstreamer),
but PulseAudio has code in "module-combine" to:

1. Combine two separate sinks into one virtual sink. Example: two sound cards.
2. Dynamically calculate the timing deviation, on the fly, and adapt
to keep the two sinks in sync by resampling one of them
3. Do software mixing on the two streams to combine them into one
stream that then becomes the output stream of the virtual sink.

Sounds like you may not want 1. or 3. but just the functionality in 2.
Well, PulseAudio is open source, so if you're just looking for some
algorithmic inspiration on how to do this, check out the source of PA:
http://git.0pointer.de/?p=pulseaudio.git;a=blob;f=src/modules/module-combine.c;h=ab93c05efebf77236c2204a6765f21be6ee92f92;hb=HEAD

Have a look at the adjust_rates() functionality in particular, but
also the contexts in which that function is called.

HTH,

Sean

>
>
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>
>
_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Benjamin Schwartz]

On 04/20/2011 01:08 PM, Sean McNamara wrote:
> It may not be what you're looking for (no integration with Gstreamer),
> but PulseAudio has code in "module-combine" to:
>
> 1. Combine two separate sinks into one virtual sink. Example: two sound cards.
> 2. Dynamically calculate the timing deviation, on the fly, and adapt
> to keep the two sinks in sync by resampling one of them
> 3. Do software mixing on the two streams to combine them into one
> stream that then becomes the output stream of the virtual sink.

That's very interesting, but also very different from my application.
PulseAudio is determining clock skew by counting the samples processed by
each device and watching that one of them processes faster than the other.

I am trying to combine two independent recordings after recording is
complete, and the two recordings may have started and stopped at
substantially different times.  To determine the correct alignment, the
program must compare the contents of the two signals.

--Ben


_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Bouillot]

could be of interest:
http://www.eecs.qmul.ac.uk/~simond/match/index.html

Nicolas

On Wed, 2011-04-20 at 13:52 -0400, Benjamin M. Schwartz wrote:

> On 04/20/2011 01:08 PM, Sean McNamara wrote:
> > It may not be what you're looking for (no integration with Gstreamer),
> > but PulseAudio has code in "module-combine" to:
> >
> > 1. Combine two separate sinks into one virtual sink. Example: two sound cards.
> > 2. Dynamically calculate the timing deviation, on the fly, and adapt
> > to keep the two sinks in sync by resampling one of them
> > 3. Do software mixing on the two streams to combine them into one
> > stream that then becomes the output stream of the virtual sink.
>
> That's very interesting, but also very different from my application.
> PulseAudio is determining clock skew by counting the samples processed by
> each device and watching that one of them processes faster than the other.
>
> I am trying to combine two independent recordings after recording is
> complete, and the two recordings may have started and stopped at
> substantially different times.  To determine the correct alignment, the
> program must compare the contents of the two signals.
>
> --Ben
>
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel