Webrtcbin, chrome, and h264

[Trey Hutcheson]

I can't get Chrome to understand or render h264 content sent by another chrome peer. Honestly I have no idea where the problem is. Here's the scenario:

* chrome (87) produces an offer with h264 video, sendonly, and connects to one instance of webrtcbin

* A second remote peer connects to a second instance of webrtcbin. In this case, webrtcbin produces the offer. I connect video content received from webrtcbin1 to webrtcbin2, by repayloading (webrtcbin1 > rtph264depay > rtph264pay > webrtcbin2). Webrtcbin produces an offer with h264 video. The second chrome peer  responds with an answer, accepting the offer.

* The second chrome peer fires the ontrack event for the new video stream, but it doesn't actually understand the video. I've attached the stream to a video element, but it doesn't render. Furthermore, chrome://webrtc-internals reports "unknown" for the codec and the decoderImplementation. 

Now, to narrow things down, on the server side I have tried taking the stream received from webrtcbin1 and rendering it onscreen (rtp624depay ! avdec_h264 ! videoconvert ! videoscale ! autovideosink) and it renders just fine, proving that at the least the rtp is decoded properly and the h264 decode process produces good video. Thus - the chrome provided video must be correct.

I've also tried inserting an h264parse element between the depay and payload elements, and that had no effect. 

I have *also* tried depayloading, decoding, re-encoding, and re-payloading, and that works just fine. Remote side understands the stream. 

For the record, I'm using the following code to make chrome provide h264:

const video_transceiver = peer.addTransceiver('video', {"direction": "sendonly"})
const capabilities = RTCRtpSender.getCapabilities('video')
const noVP8 = capabilities.codecs
    .filter(codec => codec.mimeType.toLocaleLowerCase() != "video/vp8")
    .filter(codec => codec.mimeType.toLocaleLowerCase() != "video/vp9");
video_transceiver.setCodecPreferences(noVP8);

Does anyone have any insight?

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sean DuBois]

Hey,

Lots of things can go wrong here, but these are the big ones I have run
into!

* Make sure you have a SPS/PPS with every IDR. GStreamer doesn't do that
  by default. `config-interval=-1` on `h264parse` will fix that.

* Make sure your PayloadTypes/SSRCes are correct. I don't believe this
  is the issue since Chrome sees the incoming media. But I see this bite
  a lot of people when the implementation doesn't handle this for them.

* Make sure your browser supports the H264 profile. I don't believe this
  is an issue since you are doing Chrome <-> Chrome. One thing to note
  that Chrome doesn't support H264 on all platforms (Chromium on Debian
  has no H264, and Android builds with hardware support)

* General packet loss/ordering woes? Repacketizing could be exacerbating
  issues. You could be transforming an out of order/lossy stream to an
  in order one.

* Lots of other possible issues :) I would try saving the RTP stream to
  disk and using a tool to analyze it further. Just so you can know all
  the attributes you are working with.

On Wed, Dec 09, 2020 at 01:52:29PM -0600, Trey Hutcheson wrote:

> I can't get Chrome to understand or render h264 content sent by another
> chrome peer. Honestly I have no idea where the problem is. Here's the
> scenario:
> * chrome (87) produces an offer with h264 video, sendonly, and connects to
> one instance of webrtcbin
> * A second remote peer connects to a second instance of webrtcbin. In this
> case, webrtcbin produces the offer. I connect video content received from
> webrtcbin1 to webrtcbin2, by repayloading (webrtcbin1 > rtph264depay >
> rtph264pay > webrtcbin2). Webrtcbin produces an offer with h264 video. The
> second chrome peer  responds with an answer, accepting the offer.
> * The second chrome peer fires the ontrack event for the new video stream,
> but it doesn't actually understand the video. I've attached the stream to a
> video element, but it doesn't render. Furthermore,
> chrome://webrtc-internals reports "unknown" for the codec and the
> decoderImplementation.
>
> Now, to narrow things down, on the server side I have tried taking the
> stream received from webrtcbin1 and rendering it onscreen (rtp624depay
> ! avdec_h264 ! videoconvert ! videoscale ! autovideosink) and it renders
> just fine, proving that at the least the rtp is decoded properly and the
> h264 decode process produces good video. Thus - the chrome provided video
> must be correct.
>
> I've also tried inserting an h264parse element between the depay and
> payload elements, and that had no effect.
>
> I have *also* tried depayloading, decoding, re-encoding, and re-payloading,
> and that works just fine. Remote side understands the stream.
>
> For the record, I'm using the following code to make chrome provide h264:
>
> const video_transceiver = peer.addTransceiver('video', {"direction":
> "sendonly"})
> const capabilities = RTCRtpSender.getCapabilities('video')
> const noVP8 = capabilities.codecs
>     .filter(codec => codec.mimeType.toLocaleLowerCase() != "video/vp8")
>     .filter(codec => codec.mimeType.toLocaleLowerCase() != "video/vp9");
> video_transceiver.setCodecPreferences(noVP8);
>
> Does anyone have any insight?

> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Trey Hutcheson]

Wow that was it. h264parse's config interval! Thank you so much. Fortunately I still had that branch and that one property change got me going. 

You are a lifesaver. And I'm a total h264 newb. Again, thanks. 

On Wed, Dec 9, 2020 at 3:46 PM Sean DuBois <[hidden email]> wrote:

Hey,

Lots of things can go wrong here, but these are the big ones I have run
into!

* Make sure you have a SPS/PPS with every IDR. GStreamer doesn't do that
  by default. `config-interval=-1` on `h264parse` will fix that.

* Make sure your PayloadTypes/SSRCes are correct. I don't believe this
  is the issue since Chrome sees the incoming media. But I see this bite
  a lot of people when the implementation doesn't handle this for them.

* Make sure your browser supports the H264 profile. I don't believe this
  is an issue since you are doing Chrome <-> Chrome. One thing to note
  that Chrome doesn't support H264 on all platforms (Chromium on Debian
  has no H264, and Android builds with hardware support)

* General packet loss/ordering woes? Repacketizing could be exacerbating
  issues. You could be transforming an out of order/lossy stream to an
  in order one.

* Lots of other possible issues :) I would try saving the RTP stream to
  disk and using a tool to analyze it further. Just so you can know all
  the attributes you are working with.

On Wed, Dec 09, 2020 at 01:52:29PM -0600, Trey Hutcheson wrote:
> I can't get Chrome to understand or render h264 content sent by another
> chrome peer. Honestly I have no idea where the problem is. Here's the
> scenario:
> * chrome (87) produces an offer with h264 video, sendonly, and connects to
> one instance of webrtcbin
> * A second remote peer connects to a second instance of webrtcbin. In this
> case, webrtcbin produces the offer. I connect video content received from
> webrtcbin1 to webrtcbin2, by repayloading (webrtcbin1 > rtph264depay >
> rtph264pay > webrtcbin2). Webrtcbin produces an offer with h264 video. The
> second chrome peer  responds with an answer, accepting the offer.
> * The second chrome peer fires the ontrack event for the new video stream,
> but it doesn't actually understand the video. I've attached the stream to a
> video element, but it doesn't render. Furthermore,
> chrome://webrtc-internals reports "unknown" for the codec and the
> decoderImplementation.
>
> Now, to narrow things down, on the server side I have tried taking the
> stream received from webrtcbin1 and rendering it onscreen (rtp624depay
> ! avdec_h264 ! videoconvert ! videoscale ! autovideosink) and it renders
> just fine, proving that at the least the rtp is decoded properly and the
> h264 decode process produces good video. Thus - the chrome provided video
> must be correct.
>
> I've also tried inserting an h264parse element between the depay and
> payload elements, and that had no effect.
>
> I have *also* tried depayloading, decoding, re-encoding, and re-payloading,
> and that works just fine. Remote side understands the stream.
>
> For the record, I'm using the following code to make chrome provide h264:
>
> const video_transceiver = peer.addTransceiver('video', {"direction":
> "sendonly"})
> const capabilities = RTCRtpSender.getCapabilities('video')
> const noVP8 = capabilities.codecs
>     .filter(codec => codec.mimeType.toLocaleLowerCase() != "video/vp8")
>     .filter(codec => codec.mimeType.toLocaleLowerCase() != "video/vp9");
> video_transceiver.setCodecPreferences(noVP8);
>
> Does anyone have any insight?

> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel