Video+Audio together stutter (but not separately)

[fadnavis]

Hello all,

I have been stuck on this issue for past few days and would appreciate if someone can guide me in the right direction.
I am building a pipeline to stream video (with audio) over UDP from my Raspberry Pi to Android.

When I build a gstreamer pipeline for video-only streaming, it works fine. The video renders smoothly and perfectly even on my old Galaxy Nexus without any glitches or stuttering. Similarly, when I build an audio-only pipeline to stream from the Pi to Android, everything works fine again. No glitches or stuttering in the audio.

However, when I create a pipeline to play both together - this is where the problem starts. When both are streamed, the video and audio both stutter a lot. Here is the pipeline I have built -

SENDER:
gst-launch-1.0 filesrc location=/path/to/file.mp4 ! qtdemux name=demux demux.audio_0 ! queue ! rtpmp4apay ! udpsink host=<Target IP> port=5001 demux.video_0 ! queue !  decodebin ! jpegenc ! rtpjpegpay ! udpsink host=<Target IP> port=5000

RECEIVER:
udpsrc port=5000 caps="application/x-rtp" ! rtpjpegdepay ! decodebin ! videoscale ! autovideosink  udpsrc port=5001 caps="application/x-rtp,media=(string)audio,clock-rate=(int)44100,encoding-name=(string)MP4A-LATM,cpresent=(string)0,config=(string)40002420,payload=(int)96,ssrc=(uint)1152421420,timestamp-offset=(uint)3767852666,seqnum-offset=(uint)2211" ! rtpmp4adepay ! decodebin ! queue ! audioconvert ! autoaudiosink

I have also tried adding "sync=false" at the end of both sinks on the receiver but the behavior is the same. I did some research around this and guessed there could be three reasons :

a) Bad timestamps
b) Android CPU not powerful enough to render both audio and video together (remember - individual pipelines work fine. Also, same behavior on a powerful OnePlus one device with 3GB RAM)
c) Data is being read from the buffer faster than it is being fed into it - this is called underflow right? (or could it be overflow? I doubt though)

I'd appreciate if someone can help me understand if it is one of these three issues or could it be something else? Even if the issue is amongst the above, I have no idea how to solve it.

Thanks in advance.

Cheers,
Fadnavis

[Nicolas Dufresne-4]

Le dimanche 27 mars 2016 à 11:17 -0700, fadnavis a écrit :

> RECEIVER:
> udpsrc port=5000 caps="application/x-rtp" ! rtpjpegdepay ! decodebin
> !
> videoscale ! autovideosink  udpsrc port=5001
> caps="application/x-rtp,media=(string)audio,clock-
> rate=(int)44100,encoding-name=(string)MP4A-
> LATM,cpresent=(string)0,config=(string)40002420,payload=(int)96,ssrc=
> (uint)1152421420,timestamp-offset=(uint)3767852666,seqnum-
> offset=(uint)2211"
> ! rtpmp4adepay ! decodebin ! queue ! audioconvert ! autoaudiosink 
>
> I have also tried adding "sync=false" at the end of both sinks on the
> receiver but the behavior is the same. I did some research around
> this and
> guessed there could be three reasons :
>
> a) Bad timestamps

On the receiver side, you should be using rtpjitterbuffer elements.
This will provide both buffering and timestamp adjustment.

> b) Android CPU not powerful enough to render both audio and video
> together
> (remember - individual pipelines work fine. Also, same behavior on a
> powerful OnePlus one device with 3GB RAM)

You can test that. Just mux the stream (e.g. using mp4mux), copy that
file to your phone and play it with various audio/video player.

> c) Data is being read from the buffer faster than it is being fed
> into it -
> this is called underflow right? (or could it be overflow? I doubt
> though)

This is possible, that's also what a jitter buffer is responsible for.
A way to simplify your work here and more toward a more stable
solution, would be to use gst-rtsp-server library. And then on the
client, you could simply use playbin with rtps:// URI.

>
> I'd appreciate if someone can help me understand if it is one of
> these three
> issues or could it be something else? Even if the issue is amongst
> the
> above, I have no idea how to solve it. 
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[fadnavis]

Hi Nicolas,

Thanks for this. I'll be trying out the library this weekend and update the thread with the results.

Thanks again.