Occasionally no audio between webrtcbin and chrome

[Trey Hutcheson]

Gstreamer 1.16.2; arch linux.

Ok, this one is really weird. I've got a media server (written in rust, if that matters) 

that juggles multiple gstreamer pipelines, with webrtc being the central use case. Occasionally when a browser based peer (google chrome) joins, it receives no audio.

The pipeline is pretty basic; an audiotestsrc emits a test tone, encoding as opus, to webrtcbin. The browser side should simply pipe the audio to the local speaker. About 50% of the time, after signalling and ice connectivity has been established, there is no audio. In the cases when there is no audio, I've looked at chrome://webrtc-internals, and the audio track for the browser peer is not receiving any samples; no audio level, audio energy, etc. 

So I've tested a variety of scenarios. If I restart the browser between each test, there is no change. If I restart my media server between each iteration, then I receive audio 100% of the time. I added an identity element after `rtpopuspay` to dump the data to the console, and the data is definitely there. I monitored udp traffic with wireshark; the number and size of packets to the browser is roughly the same when there is audio and when there is not. 

So then I turned up gstreamer logging to see if it gave me any insight, and viola! I now have audio 100% of the time. When I set GST_DEBUG to 4 or higher, I get audio on every test iteration. Anything below 3, and the browser emits audio only occasionally. 

Does anyone have insight on what might be going on here?

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Olivier Crête-3]

Hi,

To debug this, I would check a couple things:

1. If you look at the SDP, are all the payload types unique across all media?

2. Using Wireshark, can you use audio RTP packets (you should be able to identify them by their payload type).

3. If GStreamer is actually not sending anything, can you check with GST_DEBUG=*SCHED*:9 how far down the audio packets go? Ie where are they dropped? Or maybe some thread is blocked?

Olivier

On Mon, 2020-07-20 at 15:15 -0500, Trey Hutcheson wrote:

Gstreamer 1.16.2; arch linux.

Ok, this one is really weird. I've got a media server (written in rust, if that matters) 

that juggles multiple gstreamer pipelines, with webrtc being the central use case. Occasionally when a browser based peer (google chrome) joins, it receives no audio.

The pipeline is pretty basic; an audiotestsrc emits a test tone, encoding as opus, to webrtcbin. The browser side should simply pipe the audio to the local speaker. About 50% of the time, after signalling and ice connectivity has been established, there is no audio. In the cases when there is no audio, I've looked at chrome://webrtc-internals, and the audio track for the browser peer is not receiving any samples; no audio level, audio energy, etc. 

So I've tested a variety of scenarios. If I restart the browser between each test, there is no change. If I restart my media server between each iteration, then I receive audio 100% of the time. I added an identity element after `rtpopuspay` to dump the data to the console, and the data is definitely there. I monitored udp traffic with wireshark; the number and size of packets to the browser is roughly the same when there is audio and when there is not. 

So then I turned up gstreamer logging to see if it gave me any insight, and viola! I now have audio 100% of the time. When I set GST_DEBUG to 4 or higher, I get audio on every test iteration. Anything below 3, and the browser emits audio only occasionally. 

Does anyone have insight on what might be going on here?

_______________________________________________

gstreamer-devel mailing list

[hidden email]

https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

-- 
Olivier Crête
[hidden email]

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Trey Hutcheson]

Fortunately I am in control of the signaling layer as well, and I can confirm that the sdp is correct and media lines do have unique payload types. In my workflow, the browser peer drives the connection by providing the offer; the media server answers. So I make sure that I set the pt property on rtpopuspay to the payload type specified in the browser's offer (and rtpvp8pay, when video is available). 

I did apply the rpt view in wireshark and can confirm that rtp packets with correct payload types are being sent even in the case where the browser is not emitting audio. 

At this point I'm out of ideas. By turning up the logging, I get audio (out of the browser). Something about setting the logging (minimum of 4) has some kind of internal timing effect. I've observed this behavior within my development environment (vs code) as well as with the actual binaries (both release and debug builds). 

On Mon, Jul 20, 2020 at 4:02 PM Olivier Crête <[hidden email]> wrote:

Hi,

To debug this, I would check a couple things:

1. If you look at the SDP, are all the payload types unique across all media?

2. Using Wireshark, can you use audio RTP packets (you should be able to identify them by their payload type).

3. If GStreamer is actually not sending anything, can you check with GST_DEBUG=*SCHED*:9 how far down the audio packets go? Ie where are they dropped? Or maybe some thread is blocked?

Olivier

On Mon, 2020-07-20 at 15:15 -0500, Trey Hutcheson wrote:

Gstreamer 1.16.2; arch linux.

Ok, this one is really weird. I've got a media server (written in rust, if that matters) 

that juggles multiple gstreamer pipelines, with webrtc being the central use case. Occasionally when a browser based peer (google chrome) joins, it receives no audio.

The pipeline is pretty basic; an audiotestsrc emits a test tone, encoding as opus, to webrtcbin. The browser side should simply pipe the audio to the local speaker. About 50% of the time, after signalling and ice connectivity has been established, there is no audio. In the cases when there is no audio, I've looked at chrome://webrtc-internals, and the audio track for the browser peer is not receiving any samples; no audio level, audio energy, etc. 

So I've tested a variety of scenarios. If I restart the browser between each test, there is no change. If I restart my media server between each iteration, then I receive audio 100% of the time. I added an identity element after `rtpopuspay` to dump the data to the console, and the data is definitely there. I monitored udp traffic with wireshark; the number and size of packets to the browser is roughly the same when there is audio and when there is not. 

So then I turned up gstreamer logging to see if it gave me any insight, and viola! I now have audio 100% of the time. When I set GST_DEBUG to 4 or higher, I get audio on every test iteration. Anything below 3, and the browser emits audio only occasionally. 

Does anyone have insight on what might be going on here?

_______________________________________________

gstreamer-devel mailing list

[hidden email]

https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

-- 
Olivier Crête
[hidden email]

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Toshick]

Hello Trey.

According to your description, I can assume, that during some webrtc configuration, you're not relying on Gstreamer's callbacks. And, when you have more resources, something happens quicker than it should. So, please, inspect your code and try to put as much initialization on gst-bus callbacks as you can.

Maybe it helps.

Best regards,

Anton.

On Tue, Jul 21, 2020 at 4:40 PM Trey Hutcheson <[hidden email]> wrote:

Fortunately I am in control of the signaling layer as well, and I can confirm that the sdp is correct and media lines do have unique payload types. In my workflow, the browser peer drives the connection by providing the offer; the media server answers. So I make sure that I set the pt property on rtpopuspay to the payload type specified in the browser's offer (and rtpvp8pay, when video is available). 

I did apply the rpt view in wireshark and can confirm that rtp packets with correct payload types are being sent even in the case where the browser is not emitting audio. 

At this point I'm out of ideas. By turning up the logging, I get audio (out of the browser). Something about setting the logging (minimum of 4) has some kind of internal timing effect. I've observed this behavior within my development environment (vs code) as well as with the actual binaries (both release and debug builds). 

On Mon, Jul 20, 2020 at 4:02 PM Olivier Crête <[hidden email]> wrote:

Hi,

To debug this, I would check a couple things:

1. If you look at the SDP, are all the payload types unique across all media?

2. Using Wireshark, can you use audio RTP packets (you should be able to identify them by their payload type).

3. If GStreamer is actually not sending anything, can you check with GST_DEBUG=*SCHED*:9 how far down the audio packets go? Ie where are they dropped? Or maybe some thread is blocked?

Olivier

On Mon, 2020-07-20 at 15:15 -0500, Trey Hutcheson wrote:

Gstreamer 1.16.2; arch linux.

Ok, this one is really weird. I've got a media server (written in rust, if that matters) 

that juggles multiple gstreamer pipelines, with webrtc being the central use case. Occasionally when a browser based peer (google chrome) joins, it receives no audio.

The pipeline is pretty basic; an audiotestsrc emits a test tone, encoding as opus, to webrtcbin. The browser side should simply pipe the audio to the local speaker. About 50% of the time, after signalling and ice connectivity has been established, there is no audio. In the cases when there is no audio, I've looked at chrome://webrtc-internals, and the audio track for the browser peer is not receiving any samples; no audio level, audio energy, etc. 

So I've tested a variety of scenarios. If I restart the browser between each test, there is no change. If I restart my media server between each iteration, then I receive audio 100% of the time. I added an identity element after `rtpopuspay` to dump the data to the console, and the data is definitely there. I monitored udp traffic with wireshark; the number and size of packets to the browser is roughly the same when there is audio and when there is not. 

So then I turned up gstreamer logging to see if it gave me any insight, and viola! I now have audio 100% of the time. When I set GST_DEBUG to 4 or higher, I get audio on every test iteration. Anything below 3, and the browser emits audio only occasionally. 

Does anyone have insight on what might be going on here?

_______________________________________________

gstreamer-devel mailing list

[hidden email]

https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

-- 
Olivier Crête
[hidden email]

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Trey Hutcheson]

I'm sorry Anton, but I don't really know what initialization I should do in response to messages received on the bus. As part of the signalling path, this code dynamically constructs bins and add themes to the pipeline, performing the sdp negotiation with webrtcbin. That includes setting up the audio source and encoding into opus.

The code then sets the remote description, then generates the answer and returns it along the signalling path, and immediately gets out of the way. 

On Tue, Jul 21, 2020 at 9:14 AM Anton Pryima <[hidden email]> wrote:

Hello Trey.

According to your description, I can assume, that during some webrtc configuration, you're not relying on Gstreamer's callbacks. And, when you have more resources, something happens quicker than it should. So, please, inspect your code and try to put as much initialization on gst-bus callbacks as you can.

Maybe it helps.

Best regards,

Anton.

On Tue, Jul 21, 2020 at 4:40 PM Trey Hutcheson <[hidden email]> wrote:

Fortunately I am in control of the signaling layer as well, and I can confirm that the sdp is correct and media lines do have unique payload types. In my workflow, the browser peer drives the connection by providing the offer; the media server answers. So I make sure that I set the pt property on rtpopuspay to the payload type specified in the browser's offer (and rtpvp8pay, when video is available). 

I did apply the rpt view in wireshark and can confirm that rtp packets with correct payload types are being sent even in the case where the browser is not emitting audio. 

At this point I'm out of ideas. By turning up the logging, I get audio (out of the browser). Something about setting the logging (minimum of 4) has some kind of internal timing effect. I've observed this behavior within my development environment (vs code) as well as with the actual binaries (both release and debug builds). 

On Mon, Jul 20, 2020 at 4:02 PM Olivier Crête <[hidden email]> wrote:

Hi,

To debug this, I would check a couple things:

1. If you look at the SDP, are all the payload types unique across all media?

2. Using Wireshark, can you use audio RTP packets (you should be able to identify them by their payload type).

3. If GStreamer is actually not sending anything, can you check with GST_DEBUG=*SCHED*:9 how far down the audio packets go? Ie where are they dropped? Or maybe some thread is blocked?

Olivier

On Mon, 2020-07-20 at 15:15 -0500, Trey Hutcheson wrote:

Gstreamer 1.16.2; arch linux.

Ok, this one is really weird. I've got a media server (written in rust, if that matters) 

that juggles multiple gstreamer pipelines, with webrtc being the central use case. Occasionally when a browser based peer (google chrome) joins, it receives no audio.

The pipeline is pretty basic; an audiotestsrc emits a test tone, encoding as opus, to webrtcbin. The browser side should simply pipe the audio to the local speaker. About 50% of the time, after signalling and ice connectivity has been established, there is no audio. In the cases when there is no audio, I've looked at chrome://webrtc-internals, and the audio track for the browser peer is not receiving any samples; no audio level, audio energy, etc. 

So I've tested a variety of scenarios. If I restart the browser between each test, there is no change. If I restart my media server between each iteration, then I receive audio 100% of the time. I added an identity element after `rtpopuspay` to dump the data to the console, and the data is definitely there. I monitored udp traffic with wireshark; the number and size of packets to the browser is roughly the same when there is audio and when there is not. 

So then I turned up gstreamer logging to see if it gave me any insight, and viola! I now have audio 100% of the time. When I set GST_DEBUG to 4 or higher, I get audio on every test iteration. Anything below 3, and the browser emits audio only occasionally. 

Does anyone have insight on what might be going on here?

_______________________________________________

gstreamer-devel mailing list

[hidden email]

https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

-- 
Olivier Crête
[hidden email]

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel