Audio/video synchronization with pre-buffered video

[Anton Protopopov]

Hello everyone,

I have a pipeline with live H264 video source and live AAC audio
source muxed together using mpegtsmux. One problem with this pipeline
is that the video encoder hardware can buffer some number of frames,
so when I am creating and setting a pipeline to PLAYING state, there
could be already, say, 1-2 seconds of video buffered. Audio is not
buffered in hardware. So when the pipeline is set to the PLAYING
state, video source produces a set of frames with almost the same time
stamps and several first seconds of video are corrupted (they contain
old video frames with some audio frames). Is there a way to
synchronize such sources without dropping video frames?

Thanks,
Anton
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Mikl]

Hello, Anton

I don`t believe gstreamer have solution fro this. It can not know about specific features of your hardware.

As a solution, can you feed pipeline with "silence" audio during cleaning of the video buffer?

Mikl

---

From: gstreamer-devel <[hidden email]> on behalf of Anton Protopopov <[hidden email]>
Sent: Wednesday, February 15, 2017 6:03:02 PM
To: [hidden email]
Subject: Audio/video synchronization with pre-buffered video

 

Hello everyone,

I have a pipeline with live H264 video source and live AAC audio
source muxed together using mpegtsmux. One problem with this pipeline
is that the video encoder hardware can buffer some number of frames,
so when I am creating and setting a pipeline to PLAYING state, there
could be already, say, 1-2 seconds of video buffered. Audio is not
buffered in hardware. So when the pipeline is set to the PLAYING
state, video source produces a set of frames with almost the same time
stamps and several first seconds of video are corrupted (they contain
old video frames with some audio frames). Is there a way to
synchronize such sources without dropping video frames?

Thanks,
Anton
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On Wed, 2017-02-15 at 12:03 -0500, Anton Protopopov wrote:

> Hello everyone,
>
> I have a pipeline with live H264 video source and live AAC audio
> source muxed together using mpegtsmux. One problem with this pipeline
> is that the video encoder hardware can buffer some number of frames,
> so when I am creating and setting a pipeline to PLAYING state, there
> could be already, say, 1-2 seconds of video buffered. Audio is not
> buffered in hardware. So when the pipeline is set to the PLAYING
> state, video source produces a set of frames with almost the same
> time
> stamps and several first seconds of video are corrupted (they contain
> old video frames with some audio frames). Is there a way to
> synchronize such sources without dropping video frames?

That sounds like you're handling timestamps wrong. What should happen
is that buffers that belong together have the same timestamps, and if
you have 1-2s of video before the first audio, then the audio will just
start with a timestamps 1-2s later than the first video timestamp.

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Anton Protopopov-2]

2017-03-17 6:06 GMT-04:00 Sebastian Dröge <[hidden email]>:

> On Wed, 2017-02-15 at 12:03 -0500, Anton Protopopov wrote:
>> Hello everyone,
>>
>> I have a pipeline with live H264 video source and live AAC audio
>> source muxed together using mpegtsmux. One problem with this pipeline
>> is that the video encoder hardware can buffer some number of frames,
>> so when I am creating and setting a pipeline to PLAYING state, there
>> could be already, say, 1-2 seconds of video buffered. Audio is not
>> buffered in hardware. So when the pipeline is set to the PLAYING
>> state, video source produces a set of frames with almost the same
>> time
>> stamps and several first seconds of video are corrupted (they contain
>> old video frames with some audio frames). Is there a way to
>> synchronize such sources without dropping video frames?
>
> That sounds like you're handling timestamps wrong. What should happen
> is that buffers that belong together have the same timestamps, and if
> you have 1-2s of video before the first audio, then the audio will just
> start with a timestamps 1-2s later than the first video timestamp.

Thanks for the answer Sebastian. Currently, I solved the problem by
just dropping the buffered video (it turned out that we loose at most
1-2 frames in our situation). However, you are definitely right that I
am handling timestamps wrong. Can't you tell me how can I adjust
timestamps generated by pulseaudio? If, say, my video-source produces
buffers with timestamps starting from zero, then how can I tell
pulseaudio to generate timestamps starting from some offset X?

Anton

>
> --
> Sebastian Dröge, Centricular Ltd · http://www.centricular.com
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel