Dynamically add sources to a pipeline sending RTP packets

[gst-noob-1.0]

I am trying to build a MCU that mixes streams from multiple webrtcbin elements and sends that single mixed stream over RTP (not webRTC). A user joins the room, a webrtc bin elements gets added to the pipeline, that webrtcbin audio is decoded and sent to the mixer that outputs one stream that is encoded and sent over RTP via udp.

The problem I am facing is with the timing. For example, the server starts up and the pipeline plays, 45 seconds later a user joins the room, a webrtcbin is dynamically added to the pipeline and audio starts flowing. The issue is the mixed audio going out RTP via UDP is delayed by 45 seconds (since the user joined the room 45 seconds after the pipeline started). I am very new to GStreamer so I am not too sure how to fix this or really what is occurring. I assume it has something to do with the clock. Most of what I have found regarding this suggests setting sync=false on all sinks which is what I have done. The audio is definitely being sent as soon as it arrives, but I believe the timestamps are what are doing me in.

These are live streams so sync does not matter and I have set sync=False on the udpsink. Using wireshark I can see the packets are arriving on time and in order, but the clockdrift  is equal to the time that passed from when the pipeline started and when the webrtcbin got added and started sending to the udp sink.

 

[gst-noob-1.0]

I have also tried setting the timestamp-offset property on my payloader to 0 and it does not seem to have an effect. When I am dynamically adding these elements, I do not stop the pipeline from playing. I use blocking probes on whatever needs them and connect/reconnect things as needed. Everything works great except for this timing thing

I should also add that I am using rtpbin linked to a udp sink to send the audio.

So you’ve got something like:

Webrtcbin->depay->decode->audiomixer->encode->pay->rtpbin->udpsink

And multiple webrtcbins are linked to the mixer. Using an adder instead of the mixer doesn’t seem to have this issue, but I’d rather use the mixer since it seems more fleshed out.

[gst-noob-1.0]

Alright I figured out what was missing. I needed to set the “start-time-selection" on the mixer (an inherited property from Aggregator) to GST_AGGREGATOR_START_TIME_SELECTION_FIRST (1). This resolved the playback delay issues. Instead of the time being relative to the server start, it is now relative to the first packet received by the mixer.