webrtcbin pad-added callback is not invoked

[Michael S]

20240130_post_attachment.logHello,

Thanks for taking the time to read this. In short, "pad-added" is not being invoked and autovideosink is not appearing.

Environment: WSL2 running Ubuntu 23.04, gstreamer 1.22.1 and Python 3.11.4.

In a Python program I'm working on, my goal is to use the webrtcbin element as a receive only bin and source that once connected and receiving an RTP stream, is sent the stream to a further sink.

I have been modeling my program after this webrtc_sendrecv.py demo.  Thanks so much to the auther.

There is a difference in what I'm doing to theirs. Theirs is receiving and sending and mine is receiving only and I'm building my pipeline as follows:

            Gst.init(None)
            
            # inspired by: https://gitlab.freedesktop.org/gstreamer/gst-examples/-/blob/master/webrtc/sendrecv/gst/webrtc_sendrecv.py            

            self.pipe = Gst.Pipeline.new()
            self.wsrc = Gst.ElementFactory.make('webrtcbin', 'wsrc')
            self.pipe.add(self.wsrc)
            
            self.log.debug(f"setting wsrc.turn-servers and transport policy")
            res = self.wsrc.set_property("ice-transport-policy", GstWebRTC.WebRTCICETransportPolicy.RELAY)
            self.wsrc.set_property("turn-server", iceserver.getSecureTcpTurnServer(self.iceServers))
            self.wsrc.emit("add-turn-server", iceserver.getUdpTurnServer(self.iceServers))
            self.wsrc.emit("add-turn-server", iceserver.getUnsecureTcpTurnServer(self.iceServers))
            
            self.wsrc.set_property("message-forward", True)

            self.log.debug(f"setting callbacks")
            res = self.wsrc.connect("pad-added", self.on_incoming_stream, None)
            res = self.wsrc.connect("on-negotiation-needed", self.wsrc_on_negotiation_needed)
            res = self.wsrc.connect("on-ice-candidate", self.wsrc_on_ice_candidate)
            res = self.wsrc.connect("on-data-channel", self.wsrc_on_data_channel)
            res = self.wsrc.connect("on-new-transceiver", self.wsrc_on_new_transceiver)
            res = self.wsrc.connect("pad-removed", self.wsrc_on_pad_removed)
            res = self.wsrc.connect("no-more-pads", self.wsrc_on_no_more_pads)
            res = self.wsrc.connect("notify::ice-gathering-state", self.wsrc_notify_on_ice_gathering_state)
            res = self.wsrc.connect("request-aux-sender", self.wsrc_notify_on_request_aux_sender)
            
            self.addTransceivers()
            
            # Set up the pipeline state
            self.log.debug("setting gst.pipeline state to PLAYING")
            self.pipe.set_state(Gst.State.PLAYING)

    def addTransceivers(self):
        self.log.debug(f"adding video transceiver")
        profileLevelId = "42e01f"
        video_caps = Gst.caps_from_string(f"application/x-rtp,media=video,encoding-name=H264,payload=96,clock-rate=90000,packetization-mode=(string)1,profile-level-id=(string){profileLevelId}")
        videoTransceiver = self.wsrc.emit("add-transceiver", GstWebRTC.WebRTCRTPTransceiverDirection.RECVONLY, video_caps)            
        self.mediaData["video"].transceiver = videoTransceiver
        return

The pipeline is running in its own process, with no interference from other threads or asynchronous IO. My signaling is handled using an IPC mechanism that I built. The create-offer, set-local-description and set-remote-description, and the ice-candidate exchange appears to be going very well. The GST log and my log (attached with code) shows the stream (from a Kurento media server) is being received, but the the "pad-added" callback is not being invoked. I have tried many things:

1. Fiddled with the number of arguments in my callback function.
2. Located the callback as private method, global function, a locally scoped function with in the function that builds the pipeline.
3. Tried cobbling together many (for days) different variations of pipeline descriptors, with and without caps.
4. Omitting the addTransceivers() call.
5. Instead of setting the pipeline state to PLAYING, set it to READY or PAUSED and set it to READY in the on-new-receiver callbacks or notify::ice-gathering-state state change to COMPLETE.
6. Fiddling with a variety of different webrtcbin properties to learn what the impact is on the behavior.
7. Tried handing my signaling IPC events in another thread.
8. Tried interleaving IPC message lookups with calls to self.pipe.get_bus().timed_pop_filtered(0.5 * Gst.SECOND, ...)
9. I added all the callbacks to learn if there are sequences of events that play into my issue.  Very few get invoked, just the on-negotiation-needed, on-ice-candidate, on-new-transceiver and notify::ice-gathering-state are being called.

A couple of observations that may illuminate (I use "may" with a log of hope but I also don't want to lead the witness):

1. When adding a transceiver manually, it gets an mlineindex of -1 (or unsigned int max of 4294967295) as seen here:

<wsrc> created new transceiver <webrtctransceiver0> with direction recvonly (3), mline 4294967295, kind video (2)
(3.7 seconds later...)
<transportstream0> Adding mapping for rtp session 0 media_idx 4294967295 direction sendonly ssrc 1018892234

   I tried very hard to set the mlineindex of the transceiver to 0 in many different ways and I receive an error saying it cannot be set after the transceiver has been constructed. Even though documentation says it's is a writable property.

Thanks so much for reading and hopefully some insights from you. I've honestly been at this for a couple of weeks, started with Windows based python, then Ubuntu 20.04 and gstreamer 1.16.3 which was crashing. I had a model of the application that was using asynchio for signaling and another socket for receiving input to drive the stream and went to a multi-process model. This last issue has had me stumped for several days.

The code, GST_DEBUG and my log output attached.

Thanks,

Mike (Staff Developer at X2OMEDIA)

[Michael S]

Following up on this.

In the end it was logic error in another area of my application where I needed to instruct kurento to start the stream. Once doing so and the stream started, webrtcbin correctly called the "pad-added" callback and I was able to link with the given pad.

Thanks to anyone who may have read this.