WebRTCBin not receiving caps signal

[Dominik Röttsches]

Hi,

I am working on a python app that turns an rtsp stream from a networked camera into a stream that you can access through the browser via WebRTC.

For that, I am using a dynamic pipeline, the rtp audio and video streams go to a fakesink through a tee element. For each incoming websocket connection, I build an WebRTC connection: I am adding two queues, one for audio, one for video to the tee elements and connect them to a new WebRTC element. Then I exchange the SDP offer and response via websockets.

Now, in some situations, after quick reloads of the web page that brokers the connection, the newest added WebRTCBin element does not send the “on-negotiation-neeed” signal. I am assuming this is because it did not receive caps on the newly requested pad, thus the gst_webrtcbin_sink_event method did not get triggered and the WebRTCBin is not checking whether a negotiation is needed. 

A PDF of the pipeline is here: http://roettsch.es/debug_gst_webrtcbin.pdf and an image of the relevant portions is attached.

The WebRTCBin element sendrecv-2-772371 has blocked pads, as it hasn’t done any negotiation and unblocked its pads, but also the CAPS on the links to the queue elements only show “ANY”.

I have a few questions to further investigate this:

What could be the reason the caps event is not received by the GstWebRTCBin?

When is the caps event usually sent downstream? Is it sent only once and could it be that the GstWebRTCBin somehow missed it through some race condition?

When connecting queue and webrtcbin to the tee element, do I need to follow a certain order? Do I need to insert probes in order not to miss the caps event?

Thanks for your help,

Dominik

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Pascal Jacquemart]

Hi Dominik,

I read your email a while ago and for some reason decided to implement
roughly the same thing. But I am having some trouble...
Sorry I could not help in the first place with your
'on-negociation-needed' message missing. It seems clearly a bug in the
webrtcbin as you described.

So in my case, I want to implement a one to one RTSP to WebRTC
transcoding. Seems rather simple but I never got it to work.
It seems that the RTSP connection takes few seconds to initialise and
the pipeline goes into many PLAY / PAUSE transitions before stabilising.
From this I understand that it is not possible to start with webrtcbin
in a static pipeline:

rtspsrc location=rtsp://whatever ! rtph264depay ! rtph264pay !
queue ! application/x-rtp,media=video,encoding-name=H264,payload=97 !
webrtcbin name=sendrecv bundle-policy=max-bundle

Then I tried your strategy to redirect the rtsp stream to fakesink and
only attach the webrtcbin later. In this case, I see the SDP exchange
but the other end never receive the video.

Have you seen this same problem before?

Thanks,             Pascal

On Sun, 10 Mar 2019 11:25:23 +0200
Dominik Röttsches <[hidden email]> wrote:

> Hi,
>
> I am working on a python app that turns an rtsp stream from a
> networked camera into a stream that you can access through the
> browser via WebRTC.
>
> For that, I am using a dynamic pipeline, the rtp audio and video
> streams go to a fakesink through a tee element. For each incoming
> websocket connection, I build an WebRTC connection: I am adding two
> queues, one for audio, one for video to the tee elements and connect
> them to a new WebRTC element. Then I exchange the SDP offer and
> response via websockets.
>
> Now, in some situations, after quick reloads of the web page that
> brokers the connection, the newest added WebRTCBin element does not
> send the “on-negotiation-neeed” signal. I am assuming this is because
> it did not receive caps on the newly requested pad, thus the
> gst_webrtcbin_sink_event method did not get triggered and the
> WebRTCBin is not checking whether a negotiation is needed.
>
> A PDF of the pipeline is here:
> http://roettsch.es/debug_gst_webrtcbin.pdf and an image of the
> relevant portions is attached.
>
> The WebRTCBin element sendrecv-2-772371 has blocked pads, as it
> hasn’t done any negotiation and unblocked its pads, but also the CAPS
> on the links to the queue elements only show “ANY”.
>
> I have a few questions to further investigate this:
>
> What could be the reason the caps event is not received by the
> GstWebRTCBin? When is the caps event usually sent downstream? Is it
> sent only once and could it be that the GstWebRTCBin somehow missed
> it through some race condition? When connecting queue and webrtcbin
> to the tee element, do I need to follow a certain order? Do I need to
> insert probes in order not to miss the caps event?
>
> Thanks for your help,
>
> Dominik
>

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[mkonstapel]

Hi Dominik et al,

I think I am running into the exact same problem - quickly reloading the
"player" page (thus adding webrtcbins) eventually, and apparently randomly,
locks up the pipeline. Did you ever find a solution?

Kind regards,
Michiel Konstapel




--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[mkonstapel]

Apologies, that lost a lot of context since it references an old email
to the list. It concerns this message by Dominik:
http://gstreamer-devel.966125.n4.nabble.com/WebRTCBin-not-receiving-caps-signal-tt4689946.html

Quoting:

> I am working on a python app that turns an rtsp stream from a
> networked camera into a stream that you can access through the browser
> via WebRTC.
>
> For that, I am using a dynamic pipeline, the rtp audio and video
> streams go to a fakesink through a tee element. For each incoming
> websocket connection, I build an WebRTC connection: I am adding two
> queues, one for audio, one for video to the tee elements and connect
> them to a new WebRTC element. Then I exchange the SDP offer and
> response via websockets.
>
> Now, in some situations, after quick reloads of the web page that
> brokers the connection, the newest added WebRTCBin element does not
> send the “on-negotiation-neeed” signal. I am assuming this is because
> it did not receive caps on the newly requested pad, thus the
> gst_webrtcbin_sink_event method did not get triggered and the
> WebRTCBin is not checking whether a negotiation is needed.
>
> A PDF of the pipeline is here:
> http://roettsch.es/debug_gst_webrtcbin.pdf and an image of the
> relevant portions is attached.
>
> The WebRTCBin element sendrecv-2-772371 has blocked pads, as it hasn’t
> done any negotiation and unblocked its pads, but also the CAPS on the
> links to the queue elements only show “ANY”.
>
> I have a few questions to further investigate this:
>
> What could be the reason the caps event is not received by the
> GstWebRTCBin?
> When is the caps event usually sent downstream? Is it sent only once
> and could it be that the GstWebRTCBin somehow missed it through some
> race condition?
> When connecting queue and webrtcbin to the tee element, do I need to
> follow a certain order? Do I need to insert probes in order not to
> miss the caps event?

On 14-01-2021 12:00, mkonstapel wrote:
> Hi Dominik et al,
>
> I think I am running into the exact same problem - quickly reloading the
> "player" page (thus adding webrtcbins) eventually, and apparently randomly,
> locks up the pipeline. Did you ever find a solution?
>
> Kind regards,
> Michiel Konstapel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[mkonstapel]

After several days of banging my head on the wall, I've gone and split off the webrtc part into a separate pipeline, connected to the main one through udpsink/udpsrc. However, even then adding webrtcbins will randomly lock up. I've narrowed it down to a pretty small testcase in the attached python file.

In my environment (1.18.3 built from source) I had to run

export GI_TYPELIB_PATH=/usr/local/lib/x86_64-linux-gnu/girepository-1.0

or it fails with:

ValueError: Namespace GstWebRTC not available

Run it once as a producer pipeline:

python3 testcase.py producer

And then run the consumer:

python3 testcase.py consumer

This will just add a whole bunch of webrtcbins to the running pipeline.

Occasionally, this will successfully add 100 webrtcbins to the pipeline, but usually, it'll lock up:

gstwebrtcbin.c:319:gst_webrtc_bin_pad_new:<'':sink_0> new visible pad with direction sink
gstwebrtcbin.c:469:_find_transceiver_for_mline:<webrtcbin97> Found transceiver (NULL) for mlineindex 0
gstwebrtcbin.c:5823:gst_webrtc_bin_request_new_pad:<webrtcbin97> Created new transceiver <webrtctransceiver97> for mline 0
gstwebrtcbin.c:5737:gst_webrtc_bin_change_state: changing state: NULL => READY
gstwebrtcbin.c:1354:_check_if_negotiation_is_needed:<webrtcbin97> checking if negotiation is needed
gstwebrtcbin.c:1360:_check_if_negotiation_is_needed:<webrtcbin97> no negotiation possible until caps have been received on all sink pads
gstwebrtcbin.c:5737:gst_webrtc_bin_change_state: changing state: READY => PAUSED
gstwebrtcbin.c:5737:gst_webrtc_bin_change_state: changing state: PAUSED => PLAYING

And then it just freezes.

Sometimes it's the 98th, sometimes it's the 10th. I've ran it with maximum GST_DEBUG, added print statements to gstwebrtcbin.c, and I can't find anything that sets apart the successful and failing attempts. Adding sleeps seems to help, but it's not foolproof and I have no idea why it would help.

Matthew or other experts, any ideas? Is there an issue tracker I should add this to? Is there something wrong with this approach?

I did find a workaround: if I pause the whole consumer pipeline, add the webrtcbin, and then set the pipeline playing, that appears to prevent the race condition. Now that it's in a separate pipeline, that's a viable option, but it still doesn't sit right with me that it just randomly... stops.

Kind regards,
Michiel

On 14-01-2021 12:04, Michiel Konstapel wrote:

Apologies, that lost a lot of context since it references an old email to the list. It concerns this message by Dominik: http://gstreamer-devel.966125.n4.nabble.com/WebRTCBin-not-receiving-caps-signal-tt4689946.html

Quoting:

I am working on a python app that turns an rtsp stream from a networked camera into a stream that you can access through the browser via WebRTC.

For that, I am using a dynamic pipeline, the rtp audio and video streams go to a fakesink through a tee element. For each incoming websocket connection, I build an WebRTC connection: I am adding two queues, one for audio, one for video to the tee elements and connect them to a new WebRTC element. Then I exchange the SDP offer and response via websockets.

Now, in some situations, after quick reloads of the web page that brokers the connection, the newest added WebRTCBin element does not send the “on-negotiation-neeed” signal. I am assuming this is because it did not receive caps on the newly requested pad, thus the gst_webrtcbin_sink_event method did not get triggered and the WebRTCBin is not checking whether a negotiation is needed.

A PDF of the pipeline is here: http://roettsch.es/debug_gst_webrtcbin.pdf and an image of the relevant portions is attached.

The WebRTCBin element sendrecv-2-772371 has blocked pads, as it hasn’t done any negotiation and unblocked its pads, but also the CAPS on the links to the queue elements only show “ANY”.

I have a few questions to further investigate this:

What could be the reason the caps event is not received by the GstWebRTCBin?
When is the caps event usually sent downstream? Is it sent only once and could it be that the GstWebRTCBin somehow missed it through some race condition?
When connecting queue and webrtcbin to the tee element, do I need to follow a certain order? Do I need to insert probes in order not to miss the caps event?

On 14-01-2021 12:00, mkonstapel wrote:

Hi Dominik et al,

I think I am running into the exact same problem - quickly reloading the
"player" page (thus adding webrtcbins) eventually, and apparently randomly,
locks up the pipeline. Did you ever find a solution?

Kind regards,
Michiel Konstapel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[mkonstapel]

Oh, the producer pipeline in the example was not quite correct: it should include key frames, a profile specification and config-interval for the h264 parser and payloader:

... nvh264enc gop-size=50 ! video/x-h264, profile=baseline ! h264parse config-interval=-1 ! rtph264pay config-interval=-1 ...

Updated version is attached. But even then, it still breaks.

On 16-03-2021 20:51, Michiel Konstapel wrote:

After several days of banging my head on the wall, I've gone and split off the webrtc part into a separate pipeline, connected to the main one through udpsink/udpsrc. However, even then adding webrtcbins will randomly lock up. I've narrowed it down to a pretty small testcase in the attached python file.

In my environment (1.18.3 built from source) I had to run

export GI_TYPELIB_PATH=/usr/local/lib/x86_64-linux-gnu/girepository-1.0

or it fails with:

ValueError: Namespace GstWebRTC not available

Run it once as a producer pipeline:

python3 testcase.py producer

And then run the consumer:

python3 testcase.py consumer

This will just add a whole bunch of webrtcbins to the running pipeline.

Occasionally, this will successfully add 100 webrtcbins to the pipeline, but usually, it'll lock up:

gstwebrtcbin.c:319:gst_webrtc_bin_pad_new:<'':sink_0> new visible pad with direction sink
gstwebrtcbin.c:469:_find_transceiver_for_mline:<webrtcbin97> Found transceiver (NULL) for mlineindex 0
gstwebrtcbin.c:5823:gst_webrtc_bin_request_new_pad:<webrtcbin97> Created new transceiver <webrtctransceiver97> for mline 0
gstwebrtcbin.c:5737:gst_webrtc_bin_change_state: changing state: NULL => READY
gstwebrtcbin.c:1354:_check_if_negotiation_is_needed:<webrtcbin97> checking if negotiation is needed
gstwebrtcbin.c:1360:_check_if_negotiation_is_needed:<webrtcbin97> no negotiation possible until caps have been received on all sink pads
gstwebrtcbin.c:5737:gst_webrtc_bin_change_state: changing state: READY => PAUSED
gstwebrtcbin.c:5737:gst_webrtc_bin_change_state: changing state: PAUSED => PLAYING

And then it just freezes.

Sometimes it's the 98th, sometimes it's the 10th. I've ran it with maximum GST_DEBUG, added print statements to gstwebrtcbin.c, and I can't find anything that sets apart the successful and failing attempts. Adding sleeps seems to help, but it's not foolproof and I have no idea why it would help.

Matthew or other experts, any ideas? Is there an issue tracker I should add this to? Is there something wrong with this approach?

I did find a workaround: if I pause the whole consumer pipeline, add the webrtcbin, and then set the pipeline playing, that appears to prevent the race condition. Now that it's in a separate pipeline, that's a viable option, but it still doesn't sit right with me that it just randomly... stops.

Kind regards,
Michiel

On 14-01-2021 12:04, Michiel Konstapel wrote:

Apologies, that lost a lot of context since it references an old email to the list. It concerns this message by Dominik: http://gstreamer-devel.966125.n4.nabble.com/WebRTCBin-not-receiving-caps-signal-tt4689946.html

Quoting:

I am working on a python app that turns an rtsp stream from a networked camera into a stream that you can access through the browser via WebRTC.

For that, I am using a dynamic pipeline, the rtp audio and video streams go to a fakesink through a tee element. For each incoming websocket connection, I build an WebRTC connection: I am adding two queues, one for audio, one for video to the tee elements and connect them to a new WebRTC element. Then I exchange the SDP offer and response via websockets.

Now, in some situations, after quick reloads of the web page that brokers the connection, the newest added WebRTCBin element does not send the “on-negotiation-neeed” signal. I am assuming this is because it did not receive caps on the newly requested pad, thus the gst_webrtcbin_sink_event method did not get triggered and the WebRTCBin is not checking whether a negotiation is needed.

A PDF of the pipeline is here: http://roettsch.es/debug_gst_webrtcbin.pdf and an image of the relevant portions is attached.

The WebRTCBin element sendrecv-2-772371 has blocked pads, as it hasn’t done any negotiation and unblocked its pads, but also the CAPS on the links to the queue elements only show “ANY”.

I have a few questions to further investigate this:

What could be the reason the caps event is not received by the GstWebRTCBin?
When is the caps event usually sent downstream? Is it sent only once and could it be that the GstWebRTCBin somehow missed it through some race condition?
When connecting queue and webrtcbin to the tee element, do I need to follow a certain order? Do I need to insert probes in order not to miss the caps event?

On 14-01-2021 12:00, mkonstapel wrote:

Hi Dominik et al,

I think I am running into the exact same problem - quickly reloading the
"player" page (thus adding webrtcbins) eventually, and apparently randomly,
locks up the pipeline. Did you ever find a solution?

Kind regards,
Michiel Konstapel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel