Time distortion on WebRTC video streams

[silently]

Hello,

I am encountering an issue with a server side processing of WebRTC video
streams: an added/variable latency appears and increases over time (but
I don't notice any obvious "slow motion" effect).

The pipeline is defined as:
appsrc format=time is-live=true do-timestamp=true name=src !
application/x-rtp, encoding-name=VP8-DRAFT-IETF-01 ! queue ! rtpvp8depay
! decodebin ! videoconvert ! identity ! videoconvert ! vp8enc
error-resilient=partitions keyframe-max-dist=10 auto-alt-ref=true
cpu-used=5 deadline=1 ! rtpvp8pay ! appsink name=sink

(the intent is to replace identity by other effects)

The processing is done in parallel on two video streams:
peer A webcam from browser -> webrtc app -> GStreamer pipeline -> webrtc
app -> peer B screen in browser
peer B webcam from browser -> webrtc app -> GStreamer pipeline -> webrtc
app -> peer A screen in browser

The webrtc app is made with the pion/webrtc library (for
signaling/connecting/routing of streams...), and an additional
processing is done (with GStreamer) on audio streams, without any time
distortion.

I have two ideas:

* either the entire processing (vp8 decoding/encoding to start with) is
expensive enough to "break" real-time and introduce time distortions. I
wonder if in this case there is a way to adapt the processing (for
instance adjusting the framerate) and try to maintain a near constant
latency
* there is a problem with how RTP packets are "time labeled" (with pts
for instance) and synced

I would greatly appreciate any help, including how to debug/specify more
precisely the encountered behavior and where it occurs.

Many thanks,
Guillaume
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Olivier Crête-3]

Hi,

One thing you can do is enable qos=true on both the sink (to generate the events). An then enable it also on the encoder to drop frames before encoding to catch up.

Olivier

On Fri, 2021-04-16 at 09:43 +0200, Guillaume Denis wrote:

Hello,

I am encountering an issue with a server side processing of WebRTC video

streams: an added/variable latency appears and increases over time (but

I don't notice any obvious "slow motion" effect).

The pipeline is defined as:

appsrc format=time is-live=true do-timestamp=true name=src !

application/x-rtp, encoding-name=VP8-DRAFT-IETF-01 ! queue ! rtpvp8depay

! decodebin ! videoconvert ! identity ! videoconvert ! vp8enc

error-resilient=partitions keyframe-max-dist=10 auto-alt-ref=true

cpu-used=5 deadline=1 ! rtpvp8pay ! appsink name=sink

(the intent is to replace identity by other effects)

The processing is done in parallel on two video streams:

peer A webcam from browser -> webrtc app -> GStreamer pipeline -> webrtc

app -> peer B screen in browser

peer B webcam from browser -> webrtc app -> GStreamer pipeline -> webrtc

app -> peer A screen in browser

The webrtc app is made with the pion/webrtc library (for

signaling/connecting/routing of streams...), and an additional

processing is done (with GStreamer) on audio streams, without any time

distortion.

I have two ideas:

* either the entire processing (vp8 decoding/encoding to start with) is

expensive enough to "break" real-time and introduce time distortions. I

wonder if in this case there is a way to adapt the processing (for

instance adjusting the framerate) and try to maintain a near constant

latency

* there is a problem with how RTP packets are "time labeled" (with pts

for instance) and synced

I would greatly appreciate any help, including how to debug/specify more

precisely the encountered behavior and where it occurs.

Many thanks,

Guillaume

_______________________________________________

gstreamer-devel mailing list

[hidden email]

https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

-- 

Olivier Crête

[hidden email]

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel