RTP Forward to GStreamer has corrupted frames

[Nicolas]

Hi all,

We have started playing with GStreamer recently, and it works very nicely !

Our use case is video only. We stream in webRTC from browsers to Janus
Gateway, which itself forwards an RTP stream to our backend (using Janus
Video Room plugin). Janus is also recording the video in a file.

Browser ---------> Janus -----------> Gstreamer
             WebRTC                RTP

In our backend, the RTP stream is decoded with Gstreamer, to be analyzed
with computer vision algorithms.
The whole pipeline works nicely, but we observed a weird behavior. We
compared the frames decoded by GStreamer, and the video recorded by Janus.
We realized that frames read by GStreamer are sometimes corrupted, while the
video file never have issues. This corruption starts low, and amplifies
until frames get totally unreadable. After a few seconds, the stream comes
back to normal. (Attached some examples)

The Gstreamer pipeline we use, that provided the best results so far is:
```udpsrc port=17004 ! application/x-rtp,clock-rate=90000,payload=96 ! queue
!  rtph264depay ! h264parse ! avdec_h264 ! videoconvert ! app```

We've tried using a rtpjitterbuffer, which did not solve the issue.


I'd be glad to hear suggestions on where to look at next, and even experts
that could help pipoint the issue !
Best,
nicolas

<http://gstreamer-devel.966125.n4.nabble.com/file/t378556/9d800b06-f62f-4a29-983c-c34490f56287-0-1537968887013-NO_ALGO.png>

<http://gstreamer-devel.966125.n4.nabble.com/file/t378556/9d800b06-f62f-4a29-983c-c34490f56287-0-1537968887320-NO_ALGO.png>

<http://gstreamer-devel.966125.n4.nabble.com/file/t378556/9d800b06-f62f-4a29-983c-c34490f56287-0-1537968889714-NO_ALGO.png>

<http://gstreamer-devel.966125.n4.nabble.com/file/t378556/9d800b06-f62f-4a29-983c-c34490f56287-0-1537968897195-NO_ALGO.png>



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Matteo Valdina]

Hi,

Could you try to add a queue after the decoder? 

My suspect it is a slow down in the pipeline that could lead to some data be discarded by the decoder.

Best

Matteo

On Wed, Sep 26, 2018, 10:44 Nicolas <[hidden email]> wrote:

Hi all,

We have started playing with GStreamer recently, and it works very nicely !

Our use case is video only. We stream in webRTC from browsers to Janus
Gateway, which itself forwards an RTP stream to our backend (using Janus
Video Room plugin). Janus is also recording the video in a file.

Browser ---------> Janus -----------> Gstreamer
             WebRTC                RTP

In our backend, the RTP stream is decoded with Gstreamer, to be analyzed
with computer vision algorithms.
The whole pipeline works nicely, but we observed a weird behavior. We
compared the frames decoded by GStreamer, and the video recorded by Janus.
We realized that frames read by GStreamer are sometimes corrupted, while the
video file never have issues. This corruption starts low, and amplifies
until frames get totally unreadable. After a few seconds, the stream comes
back to normal. (Attached some examples)

The Gstreamer pipeline we use, that provided the best results so far is:
```udpsrc port=17004 ! application/x-rtp,clock-rate=90000,payload=96 ! queue
!  rtph264depay ! h264parse ! avdec_h264 ! videoconvert ! app```

We've tried using a rtpjitterbuffer, which did not solve the issue.


I'd be glad to hear suggestions on where to look at next, and even experts
that could help pipoint the issue !
Best,
nicolas

<http://gstreamer-devel.966125.n4.nabble.com/file/t378556/9d800b06-f62f-4a29-983c-c34490f56287-0-1537968887013-NO_ALGO.png>

<http://gstreamer-devel.966125.n4.nabble.com/file/t378556/9d800b06-f62f-4a29-983c-c34490f56287-0-1537968887320-NO_ALGO.png>

<http://gstreamer-devel.966125.n4.nabble.com/file/t378556/9d800b06-f62f-4a29-983c-c34490f56287-0-1537968889714-NO_ALGO.png>

<http://gstreamer-devel.966125.n4.nabble.com/file/t378556/9d800b06-f62f-4a29-983c-c34490f56287-0-1537968897195-NO_ALGO.png>



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Arjen Veenhuizen]

Nicolas wrote
> The Gstreamer pipeline we use, that provided the best results so far is:
> ```udpsrc port=17004 ! application/x-rtp,clock-rate=90000,payload=96 !
> queue
> !  rtph264depay ! h264parse ! avdec_h264 ! videoconvert ! app```

You must use the rtpjitterbuffer to accommodate for packets that were
received out of order. Depending on network latency, you might need to tune
its latency property (200 msec being the default). Furthermore, as Matteo
Valdina suggested as welll, add a queue, e.g.:





--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas]

Hi Matteo, Arjen and all,

We've made a series of tests on the GStreamer pipeline, as well as running
GStreamer in the same machine as Janus.

We ended up with a setup that seems to yield a good quality for our use
case:
 - adding a rtpjitterbuffer as suggested (without playing with latency for
now)
 - adding threads to the vp8dec element (threads=4)
 - running GStreamer natively instead of within Python

This uncovered another issue... we're streaming 3 successive sequences. In
between each sequence, the publisher stops the stream, then relaunch a new
one, forwarded in a similar way to GStreamer. GStreamer is unchanged.

This works fine with all setups we tested *except* when we use a
Jitterbuffer. When a Jitterbuffer is used, nothing is received for the
second scene.

Do you have any idea where the issue might be coming from?

Thanks,
nicolas



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Matteo Valdina]

I suggest enabling the log. My first suspect, it is something related to the SSRC/TS  of the new stream. You can give a try with the rtpbin that contains the jitter.

On Wed, Oct 3, 2018 at 7:29 AM Nicolas <[hidden email]> wrote:

Hi Matteo, Arjen and all,

We've made a series of tests on the GStreamer pipeline, as well as running
GStreamer in the same machine as Janus.

We ended up with a setup that seems to yield a good quality for our use
case:
 - adding a rtpjitterbuffer as suggested (without playing with latency for
now)
 - adding threads to the vp8dec element (threads=4)
 - running GStreamer natively instead of within Python

This uncovered another issue... we're streaming 3 successive sequences. In
between each sequence, the publisher stops the stream, then relaunch a new
one, forwarded in a similar way to GStreamer. GStreamer is unchanged.

This works fine with all setups we tested *except* when we use a
Jitterbuffer. When a Jitterbuffer is used, nothing is received for the
second scene.

Do you have any idea where the issue might be coming from?

Thanks,
nicolas



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

--

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel