Audio video muxing issue

[pzymen]

Hi, I'm having some problem with combining the audio and video I get from two network streams. In the end I want it to be output as a HLS-stream but in this attempt I'm just outputing the mpegts to a file for testing. The resulting video plays fine in Windows Media Player but not in VLC. The VLC log prints a bunch of messagage saying something like "core warning: picture is too late to be displayed (missing 80 ms)". If I remove all the audio stuff the video plays fine and it also works in in HLS. I have tried diffrent combinations with or without do-timestamps but nothing seems to work very good.

This is how the pipeline looks:
https://www.dropbox.com/s/rlidka62z2le7mr/pipeline.png?dl=0

This is the a sample clip produced by the above pipeline. The source is an IP-camera (1280x720p H264 10fps / 8000Hz s16le mono)
https://www.dropbox.com/s/5kibfvaotddufws/test.ts?dl=0

This is the code used to build the pipeline and push data to it. The pushVideo/pushAudio functions are called as soon as i get data from the source. The timestamps are unix timestamp from the source that i offset to make the first keyframe start at 0. That keyframe is also the first data that gets pushed to pipeline.
https://www.dropbox.com/s/szm981gj3iiqwcc/VideoStreamer.cpp?dl=0

Would be grateful for any tips on how to make this work.

[Ashish Kumar Mishra]

Hi ,

Not exactly , but i has faced similar problem while capturing VIDEO + AUDIO as avi file.

This happens due to audio path , the below mentione two tweaks helped me :-
a) Add an QUEUE element in audio path.
b) Configuring x264enc tune=zerolatency

http://gstreamer-devel.966125.n4.nabble.com/gstreamer-1-0-v4l2src-webcam-alsasrc-has-audio-sample-drop-problem-td4675768.html#a4675808

Hope it gives you some pointers.

[pzymen]

Hello Ashish, thanks for your reply.

a) I have tried adding queus to both the video and the audio path but it did not solve the problem.

b) As I'm not the one encoding the video I can't set any parameters for it. It is done on the IP-Camera. I get the audio and video streams trough a Peer-To-Peer API.

[Ashish Kumar Mishra]

The only other test i could think is :-
1) Remove the queue from video path ,but retaining queue on audio path.
2) Increase the audio buffer size


   

[Tim Müller]

On Mon, 2016-04-04 at 23:26 -0700, pzymen wrote:

Hi,

> This is how the pipeline looks:
> https://www.dropbox.com/s/rlidka62z2le7mr/pipeline.png?dl=0
> <https://www.dropbox.com/s/rlidka62z2le7mr/pipeline.png?dl=0>  
>
> (snip)
> This is the code used to build the pipeline and push data to it. The
> pushVideo/pushAudio functions are called as soon as i get data from
> the source. The timestamps are unix timestamp from the source that i
> offset to make the first keyframe start at 0. That keyframe is also
> the first data that gets pushed to pipeline.
> https://www.dropbox.com/s/szm981gj3iiqwcc/VideoStreamer.cpp?dl=0
> <https://www.dropbox.com/s/szm981gj3iiqwcc/VideoStreamer.cpp?dl=0>  
>
> Would be grateful for any tips on how to make this work.

My first guess would be that there's something wrong with the buffers
and/or timestamps you're pushing into the appsrc elements.

Couple of things to check:

 - are you configuring the appsrc elements to operate in format=time
(so the segment event sent is a GST_FORMAT_TIME segment)

 - are you offsetting the buffers to start with timestamp 0 ?

 - the audioparse element should not be needed and might mess up
timestamping if you don't feed it a perfect stream. You should set the
right audio caps on appsrc and push audio buffers with the right
timestamps that are a multiple of the audio sample size.

 - does it work if you don't set timestamps but instead use do-
timestamp=true is-live=true ?

 - does it work fine if you just pipe both audio/video to an
audiosink/videosink ?

Cheers
 -Tim
--
Tim Müller, Centricular Ltd - http://www.centricular.com


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[pzymen]

I have found the solution and figured I would share it here for future reference.

I noticed that the HLS stream with video only wouldn't play on apple devices and after some time figured out that the problem was that the h264 data was missing AU NAL Headers. By manualy prefixing each data frame with an AU-delimiter before pushing to appsrc I was able to get the stream working on apple devices. And after this the audio just started working.

// Data to add before every frame
static const char au_delim[] = {
        0x00, 0x00, 0x00, 0x01,  /* nal prefix */
        0x09,  /* nal_unit_type = access unit delimiter */
        (char)0xf0   /* allow any slice type */
};

[Sebastian Dröge-3]

On Fr, 2016-04-29 at 00:15 -0700, pzymen wrote:
> I have found the solution and figured I would share it here for future
> reference.
>
> I noticed that the HLS stream with video only wouldn't play on apple devices
> and after some time figured out that the problem was that the h264 data was
> missing AU NAL Headers. By manualy prefixing each data frame with an
> AU-delimiter before pushing to appsrc I was able to get the stream working
> on apple devices. And after this the audio just started working.

This sounds like
  https://bugzilla.gnome.org/show_bug.cgi?id=736213

Some changes are needed to the parser here so it can do exactly what
you did in your code now :)

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel