Synchronization of multiple senders using rtpBin

[andrew2412]

Dear GST-devs,

I am using gstreamer pipelines to send packets with encoded audio and to receive those packets, decode and play/save the audio on the receiver. My case differs from other similar applications, because I have many senders and only one receiver. I'd like to have streams from all senders synchronised, so I use RTPbin with property rtpjitterbuffer-mode set to "4" - synced receiver and sender clocks. System clocks on all devices are synchronized outside the gstreamer just before streaming (no clock update during the streaming).

Problem: I have observed that streams from different senders are shifted in time by at least several microseconds. Exact number of ms changes with time of recording - at the beginning the difference is very low, and after a couple of minutes I see that the difference gets bigger. Also, when turning the streaming app on and off again, the initial difference seems to be random.  My test signal is metronom, so I can compare streams in an easier way.

Can you please give me some advice on how to minimise the time difference between streams from multiple senders?
Second thing, I have a concern that the RTPJitterBuffer in RTPbin element does not use NTP time from RTCP SR packets  - is that so?

Best regards,
Andrzej Ruminski

[Sebastian Dröge-3]

On Mi, 2015-11-04 at 07:41 -0800, andrew2412 wrote:

> Dear GST-devs,
>
> I am using gstreamer pipelines to send packets with encoded audio and
> to receive those packets, decode and play/save the audio on the
> receiver. My case differs from other similar applications, because I
> have many senders and only one receiver. I'd like to have streams
> from all senders synchronised, so I use RTPbin with property
> rtpjitterbuffer-mode set to "4" - synced receiver and sender clocks.
> System clocks on all devices are synchronized outside the gstreamer
> just before streaming (no clock update during the streaming).
>
> Problem: I have observed that streams from different senders are
> shifted in time by at least several microseconds. Exact number of ms
> changes with time of recording - at the beginning the difference is
> very low, and after a couple of minutes I see that the difference
> gets bigger. Also, when turning the streaming app on and off again,
> the initial difference seems to be random.  My test signal is
> metronom, so I can compare streams in an easier way.
>
> Can you please give me some advice on how to minimise the time
> difference between streams from multiple senders?
> Second thing, I have a concern that the RTPJitterBuffer in RTPbin
> element does not use NTP time from RTCP SR packets  - is that so?

I was talking about this at the GStreamer conference this year, maybe
you'd like to start with watching that :)
https://gstconf.ubicast.tv/videos/synchronised-multi-room-media-playback-and-distributed-live-media-processing-and-mixing-with-gstreamer/

In any case, basically what you need to do is:
1) use the same (!) clock as pipeline clock on all senders and the
receiver, e.g. an NTP or PTP clock
2) use ntp-time-source=3 (clock-time) on the rtpbin on the senders and
the receiver. Then the SR NTP times are based on the pipeline clock
time
3) use rtcp-sync-send-time=false on the rtpbin on the senders. Then
all senders will put times based on the capture times into the SR,
not the send time. You care about when the frame was captured, not
when it was sent
4) use buffer-mode=4 (synced) on the rtpbin on the receiver
5) use ntp-sync=true on the rtpbin on the receiver, then it will sync
things based on the NTP time of the SRs

You need GIT 1.6.0 for all this except for 3), for that one you need
GIT master unfortunately.

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com


_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Marzena Malczewska]

Dear Sebastian,
first of all I'd like to thank you for such precise description what should be set on the RTPbin. The 1st point is realized in this way: on the receiver device runs ptpd-2.3.1 as a master clock, but there's no gst_clock_wait_for_sync() in the receiver pipeline. It turned out that when both - the ptpd and the pipeline - run on the same device, the pipeline never stops waiting.  On each sender we have added:

    GstClock* ptpClock = gst_ptp_clock_new ("PTP-clock", 1);
    gst_pipeline_use_clock( GST_PIPELINE(pipelineAS), ptpClock );
    gst_clock_wait_for_sync( ptpClock, GST_CLOCK_TIME_NONE );

and it seems to be working. Previously the time difference between streams changing during recording was mentioned. Now it is rather stable, but the initial difference between streams is still present. What I have observed is 30 ms more or less (both on the ethernet as well as wifi connection). We would like to minimize it as much as possible. Do you know how? Or maybe something in the implementation of sender and/or receiver is still missing?

Second issue:
from time to time (seems to occur randomly) I get no stream on the receiver side while senders (in this case three of them, two channels per each) are still streaming, please look at the attachment. In logs on the receiver I get warnings:

1:36:11.377741278  6136 0x7fdf5c0028f0 WARN           audiobasesink gstaudiobasesink.c:1509:gst_audio_base_sink_skew_slaving:<Charlie> correct clock skew -0:00:00.500320897 < - 0:00:00.500000000
1:36:11.377741289  6136 0x7fdf580032d0 WARN           audiobasesink gstaudiobasesink.c:1509:gst_audio_base_sink_skew_slaving:<Alpha> correct clock skew -0:00:00.500262138 < - 0:00:00.500000000
1:36:11.377767860  6136 0x7fdf5c0028a0 WARN           audiobasesink gstaudiobasesink.c:1509:gst_audio_base_sink_skew_slaving:<Bravo> correct clock skew -0:00:00.500299680 < - 0:00:00.500000000
1:36:12.380328082  6136 0x7fdf580032d0 WARN           audiobasesink gstaudiobasesink.c:1804:gst_audio_base_sink_get_alignment:<Alpha> Unexpected discontinuity in audio timestamps of -0:00:01.000000000, resyncing
1:36:12.380328435  6136 0x7fdf5c0028a0 WARN           audiobasesink gstaudiobasesink.c:1804:gst_audio_base_sink_get_alignment:<Bravo> Unexpected discontinuity in audio timestamps of -0:00:01.000000000, resyncing
1:36:12.380385420  6136 0x7fdf5c0028f0 WARN           audiobasesink gstaudiobasesink.c:1804:gst_audio_base_sink_get_alignment:<Charlie> Unexpected discontinuity in audio timestamps of -0:00:01.000000000, resyncing

I suppose that this is the reason of blank spaces in the recording, but what can I do to have continuous streams? And why it takes so long (e.g. 2 min, 7 min) to have the streams back?

Best Regards,
Marzena Malczewska

P.S. I am writing on behalf of Andrzej Ruminski, who started this issue, as we are working on the same project.

W dniu 04.11.2015 o 19:18, Sebastian Dröge pisze:

On Mi, 2015-11-04 at 07:41 -0800, andrew2412 wrote:

Dear GST-devs,

I am using gstreamer pipelines to send packets with encoded audio and
to receive those packets, decode and play/save the audio on the
receiver. My case differs from other similar applications, because I
have many senders and only one receiver. I'd like to have streams
from all senders synchronised, so I use RTPbin with property
rtpjitterbuffer-mode set to "4" - synced receiver and sender clocks.
System clocks on all devices are synchronized outside the gstreamer
just before streaming (no clock update during the streaming).

Problem: I have observed that streams from different senders are
shifted in time by at least several microseconds. Exact number of ms
changes with time of recording - at the beginning the difference is
very low, and after a couple of minutes I see that the difference
gets bigger. Also, when turning the streaming app on and off again,
the initial difference seems to be random.  My test signal is
metronom, so I can compare streams in an easier way.

Can you please give me some advice on how to minimise the time
difference between streams from multiple senders?
Second thing, I have a concern that the RTPJitterBuffer in RTPbin
element does not use NTP time from RTCP SR packets  - is that so?

I was talking about this at the GStreamer conference this year, maybe
you'd like to start with watching that :)
https://gstconf.ubicast.tv/videos/synchronised-multi-room-media-playback-and-distributed-live-media-processing-and-mixing-with-gstreamer/

In any case, basically what you need to do is:
1) use the same (!) clock as pipeline clock on all senders and the
receiver, e.g. an NTP or PTP clock
2) use ntp-time-source=3 (clock-time) on the rtpbin on the senders and
the receiver. Then the SR NTP times are based on the pipeline clock
time
3) use rtcp-sync-send-time=false on the rtpbin on the senders. Then
all senders will put times based on the capture times into the SR,
not the send time. You care about when the frame was captured, not
when it was sent
4) use buffer-mode=4 (synced) on the rtpbin on the receiver
5) use ntp-sync=true on the rtpbin on the receiver, then it will sync
things based on the NTP time of the SRs

You need GIT 1.6.0 for all this except for 3), for that one you need
GIT master unfortunately.

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[danny.smith]

Hi,

I am using a similar scenario where we have synchronized senders
transmitting audio from a microphone to one receiver.
Have followed the above description and we are using net-time-provider and
net-client-clock to provide the synchronization.

We have the net-time-provider on one unit and the senders/receivers
net-client-clock synchronize against this provider.
This works well if the unit which runs the time-provider has been started
before the sender units, i.e. the time-provider units uptime is larger than
the sender units. If it works, the function call to gst_clock_id_wait() in
gst_base_sink_wait_clock() in gstbasesink.c returns GST_CLOCK_EARLY (this is
through the sender udpsink). If it does not work, we hang in the same
function call. Have tried setting sync=false which should disable this check
but the setting does not seem to be respected.

Using Gstreamer 1.10.4. Have we missed something? Have tried experimenting
with a lot of settings but have failed to solve this issue.

Thankful for any insights!

Regards,
Danny



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[danny.smith]

setting sync=false actually works, was an error on my part. Should I set
sync=false on my senders and if so why?

Regards,
Danny



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel