Synchronizing separate audio and video streams based on rtp timestamp

[Wedge, Ryan J. (JSC-CD4)[SGT, INC]]

We’ve encountered a non-standard synchronization issue in a new system. A platform running GStreamer will receive two multicast streams, one for audio and another for video. The audio stream will only be present while there is actual intelligible audio. A concern is that the audio may be encoded with much less delay than the video and received by our platform unsynchronized, such that the audio will be out of sync with the video when output to speakers/monitor.  

 

I’m hoping someone with more knowledge of GStreamer’s limitations and capabilities can provide input on whether this is a trivial problem that can be solved using an existing GStreamer 1.x deployment. A proxy-like application prior to GStreamer’s udpsrc could be implemented to perform this synchronization upstream of GStreamer entirely.   

 

This proxy-like application may do the following:

Receive video packets, extract current RTP timestamp. Receive initial audio packet, extract RTP timestamp. Compare Audio RTP timestamp to current Video RTP timestamp. If Video Timestamp >= Audio Timestamp: send audio packet immediately. Else, buffer audio packets with spacing information until Video RTP timestamp >= Audio RTP timestamp. Then, empty the buffered audio at the rate it was received. Assumes video is always delayed more than audio.

 

 

Ryan Wedge

Software Engineer, MCCS Communications Subsystem

 

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Tim Müller]

On Mon, 2019-08-26 at 18:36 +0000, Wedge, Ryan J. (JSC-CD4)[SGT, INC]
wrote:

Hi Ryan,

> We’ve encountered a non-standard synchronization issue in a new
> system. A platform running GStreamer will receive two multicast
> streams, one for audio and another for video. The audio stream will
> only be present while there is actual intelligible audio. A concern
> is that the audio may be encoded with much less delay than the video
> and received by our platform unsynchronized, such that the audio will
> be out of sync with the video when output to speakers/monitor.  
>  
> I’m hoping someone with more knowledge of GStreamer’s limitations and
> capabilities can provide input on whether this is a trivial problem
> that can be solved using an existing GStreamer 1.x deployment.

This is a fairly standard problem in the context of RTP streaming, and
there should be a standard solution that fits your streaming scenario.

Usually the sender would also send Sender Report (SR) RTCP packets to
receivers, and these packets will map the RTP timestamps for each
stream to a common NTP timebase. GStreamer's rtpbin will automatically
take care of synchronising the streams in a GStreamer context in that
case.

If you don't have RTCP packets, there are multiple RTP header
extensions that will carry a common reference time and that can then be
used for the purpose of synchronising streams.

Or the sender can make an SDP available with a mapping.

It all depends a bit on the details, but it's a standard problem and
you should be able to use a standard solution.

Do you have control over the sender as well? Is it GStreamer?

If it's an option I would recommend using RTSP to setup the streaming
(the actual data can then still be sent via UDP multicast).

Cheers
 Tim

--
Tim Müller, Centricular Ltd - http://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Wedge, Ryan J. (JSC-CD4)[SGT, INC]]

Thanks so much for your response Tim.

> Usually the sender would also send Sender Report (SR) RTCP packets to
> receivers, and these packets will map the RTP timestamps for each
> stream to a common NTP timebase. GStreamer's rtpbin will automatically
> take care of synchronising the streams in a GStreamer context in that
> case.
Having the source send RTCP SR packets may not be possible, so we are looking for a solution independent of RTCP. Does rtpbin rely on SR packets for synchronization? I did not see a fallback method mentioned in the plugin docs.  

> If you don't have RTCP packets, there are multiple RTP header
> extensions that will carry a common reference time and that can then be
> used for the purpose of synchronising streams.
We will investigate what is open to change on the sender. The extension flag is not currently set on the video RTP headers. Browsing through some of the RTP packets in wireshark shows the timestamp field actually jumping around quite dramatically. Sometimes single units, sometimes 10k units, others 50k units within milliseconds of wallclock time. I assume this is related to i-frames and p-frames and the 90KHz clock.

> Or the sender can make an SDP available with a mapping.
I researched the possible parameters within an SDP and didn't see a field that appeared to describe a value used to sync the streams. Can you please elaborate on how the SDP can help with synchronization?

> Do you have control over the sender as well? Is it GStreamer?
We do not have control over either the audio or video sender and neither originate from the same physical device. I am not aware of whether the senders are using GStreamer. The two sending devices will have their system clocks synchronized.

You have absolutely shown there exist several standard solutions to this type of multi-stream synchronization issue, but from what is available for us to change from the receiver only side, it's not looking like we have many standard options open to us. I'll continue researching rtpbin, RTP extensions, RTSP, and RTCP packets and try to get a better understanding of where we lie in the solution space without requesting sender changes.  

Thanks,
Ryan
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel