Sound stopping (pipeline sync issue?)

[Tvrtko Ursulin]

Hi all,

I have a test pipeline where sound works very unreliable unless sync property
on alsasink is set to false and I am wondering about why, what is happening
under the hood and so on.

Pipeline looks like this:

gst-launch-0.10 v4l2src device=/dev/video0 ! video/x-raw-
yuv,width=1920,height=1080,framerate=50/1 ! queue ! deinterlace ! queue !
xvimagesink alsasrc device=hw:2,0 provide-clock=false ! audio/x-raw-
int,rate=48000 ! queue ! alsasink

What is happening is that sound stops less than a second after pipeline is
launched and stays off. Video works fine.

If I add "sync=false" to alsasink then it all works fine. "provide-clock" on
the alsasrc has no effect there, I added it assuming video clock must be a
better choice? How

My main question would be is "sync=false" on the audio output sync in any way
dangerous or not recommended?

How does GStreamer synchronise video and audio with this kind of pipeline and
how does "sync=false" on audio sink affect that?

Many thanks,

Tvrtko
_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Marco Ballesio]

Hi,

some notes below.

On Thu, Jun 2, 2011 at 2:45 PM, Tvrtko Ursulin
<[hidden email]> wrote:

>
> Hi all,
>
> I have a test pipeline where sound works very unreliable unless sync property
> on alsasink is set to false and I am wondering about why, what is happening
> under the hood and so on.
>
> Pipeline looks like this:
>
> gst-launch-0.10 v4l2src device=/dev/video0 ! video/x-raw-
> yuv,width=1920,height=1080,framerate=50/1 ! queue ! deinterlace ! queue !
> xvimagesink alsasrc device=hw:2,0 provide-clock=false ! audio/x-raw-
> int,rate=48000 ! queue ! alsasink
>

here the video processing is performed at quite high frame rate and
resolution. Even though no actual processing is performed on the
frames, memory copies are performed here and there, so I'd check the
CPU load to begin with.

> What is happening is that sound stops less than a second after pipeline is
> launched and stays off. Video works fine.
>
> If I add "sync=false" to alsasink then it all works fine. "provide-clock" on
> the alsasrc has no effect there, I added it assuming video clock must be a
> better choice?

In your case, as the pipeline is live and no sources would be
providing a clock, the pipeline would use the system one:

http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer/html/GstPipeline.html

(v4l2src doesn't provide a clock).

It's usually a bad choice, as it could lead to (short) audio dropouts
when compensating skews. As audio is (almost) a continuous signal, a
drop there is more annoying than in the case of video.

>
> My main question would be is "sync=false" on the audio output sync in any way
> dangerous or not recommended?

it prevents the audio sink from dropping frames when they arrive too
late, either because of audio skew or for delayed scheduling in the
audio pipeline (or the cumulative effect of both). The intermediate
queue risks to delay audio further because of more context switches (a
queue "splits" the pipe in two threads), but here opinions may differ.

You might try and play with the alsa buffer size by increasing both
"buffer-time" and "latency-time". Please note that this would increase
the overall latency in your pipeline and, being it a live stream, it
could not be what you really want.

>
> How does GStreamer synchronise video and audio with this kind of pipeline and
> how does "sync=false" on audio sink affect that?
>

"grep"ping (or "find"ing) on the gstreamer core sources would have
given you a quicker answer ;) . btw here you go:

http://cgit.freedesktop.org/gstreamer/gstreamer/tree/docs/design/part-synchronisation.txt

Regards

> Many thanks,
>
> Tvrtko
>
_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Tvrtko Ursulin]

Hi Marco,

Thank you for replying.

On Sunday 05 Jun 2011 07:45:24 Marco Ballesio wrote:

> On Thu, Jun 2, 2011 at 2:45 PM, Tvrtko Ursulin
> <[hidden email]> wrote:
> > Hi all,
> >
> > I have a test pipeline where sound works very unreliable unless sync
> > property on alsasink is set to false and I am wondering about why, what
> > is happening under the hood and so on.
> >
> > Pipeline looks like this:
> >
> > gst-launch-0.10 v4l2src device=/dev/video0 ! video/x-raw-
> > yuv,width=1920,height=1080,framerate=50/1 ! queue ! deinterlace ! queue !
> > xvimagesink alsasrc device=hw:2,0 provide-clock=false ! audio/x-raw-
> > int,rate=48000 ! queue ! alsasink
>
> here the video processing is performed at quite high frame rate and
> resolution. Even though no actual processing is performed on the
> frames, memory copies are performed here and there, so I'd check the
> CPU load to begin with.

This seems to be fine, gst-launch load is 10-15% of a single core.
 

> > What is happening is that sound stops less than a second after pipeline
> > is launched and stays off. Video works fine.
> >
> > If I add "sync=false" to alsasink then it all works fine. "provide-clock"
> > on the alsasrc has no effect there, I added it assuming video clock must
> > be a better choice?
>
> In your case, as the pipeline is live and no sources would be
> providing a clock, the pipeline would use the system one:
>
> http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer/html/Gst
> Pipeline.html
>
> (v4l2src doesn't provide a clock).
>
> It's usually a bad choice, as it could lead to (short) audio dropouts
> when compensating skews. As audio is (almost) a continuous signal, a
> drop there is more annoying than in the case of video.

Right, so you suggest leaving provide-clock to true for alsasrc on the
rationale that audio clock is more reliable than the system clock? Or some
other reason?

> > My main question would be is "sync=false" on the audio output sync in any
> > way dangerous or not recommended?
>
> it prevents the audio sink from dropping frames when they arrive too
> late, either because of audio skew or for delayed scheduling in the
> audio pipeline (or the cumulative effect of both). The intermediate
> queue risks to delay audio further because of more context switches (a
> queue "splits" the pipe in two threads), but here opinions may differ.

sync=false prevents dropping frame or sync=true?

As I said with sync=true I get no sound at all after the initial sub second
period. This is even when alsasrc and alsasink are on the same device.
 
> You might try and play with the alsa buffer size by increasing both
> "buffer-time" and "latency-time". Please note that this would increase
> the overall latency in your pipeline and, being it a live stream, it
> could not be what you really want.

Neither increasing or decreasing these by order of magnitude has any
observable effect.

> > How does GStreamer synchronise video and audio with this kind of pipeline
> > and how does "sync=false" on audio sink affect that?
>
> "grep"ping (or "find"ing) on the gstreamer core sources would have
> given you a quicker answer ;) . btw here you go:
>
> http://cgit.freedesktop.org/gstreamer/gstreamer/tree/docs/design/part-synch
> ronisation.txt

Well it is a bit complicated but in essence it says that things should
synchronize, right? :)

Even with "disjoint" streams in a pipeline (not two streams coming for a
demuxer which would then output correct timestamps for each)?

Many thanks,

Tvrtko
_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel