Video drop issue from rtpjitterbuffer and xvimagesink

[Krishnamoorthy]

I am using the below gstreamer filter path to decode and display the h.264 video. It is streaming application.

gst-launch-1.0 rtspsrc location=rtsp://<IP Address>:554/MediaINput/h264 latency=50 protocols=3 ! rtpjitterbuffer latency=200 drop-on-latency=false ! rtph264depay ! avdec_h264 ! xvimagesink

This filter path is not working for the below use case. I found continuous drop in video frames.

1) When the packets arrived with delay. And the delay is not uniform between packets.
I found the rtpjitterbuffer continuously dropping the video packets.

---------------------
I checked with enabling the gstreamer logs and found continuous packet drops
0:00:11.071297582 17238 0xb2e0eac0 WARN         rtpjitterbuffer gstrtpjitterbuffer.c:2160:gst_rtp_jitter_buffer_chain:<rtpjitterbuffer1> Packet #20413 too late as #20416 was already popped, dropping
0:00:12.068382413 17238 0xb2e0eac0 WARN         rtpjitterbuffer gstrtpjitterbuffer.c:2160:gst_rtp_jitter_buffer_chain:<rtpjitterbuffer1> Packet #20526 too late as #20530 was already popped, dropping
0:00:12.544864153 17238 0xb2e0eac0 WARN         rtpjitterbuffer gstrtpjitterbuffer.c:2160:gst_rtp_jitter_buffer_chain:<rtpjitterbuffer1> Packet #20582 too late as #20584 was already popped, dropping
---------------------
But this is OK. since the packets are delayed. It is expected by gstreamer to drop it.



But, after around 20 seconds, the video packets are reaching without any delay.
But, still the gstreamer is dropping the video frames.

I found this time, the video frames are dropped by xvimagesink filter with the below reason.
---------------------
WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.
Additional debug info:
gstbasesink.c(2791): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:
There may be a timestamping problem, or this computer is too slow.
---------------------


And, the main concerns is
1) I understand the packet drop from  rtpjitterbuffer when packets are delayed.
2) But, after around 20 sec, the packets are not delayed. But, still gstreamer is dropping the frames.

Could any one please suggest on how to address this condition.


I tried the below option.
1) sync=false in xvimagesink
With this the my application is able to handle the use case. But, end with out of sync between multiple video feed.

2) Increased the Latency in jitterbuffer from 200ms to 1000ms
It also resolve the problem. But it introduce the delay.

Is there any way to resolve without delay or out of sync issue.

[Nicolas Dufresne-5]

Le mercredi 31 mai 2017 à 03:59 -0700, Krishnamoorthy a écrit :
> I am using the below gstreamer filter path to decode and display the h.264
> video. It is streaming application.
>
> gst-launch-1.0 rtspsrc location=rtsp://<IP Address>:554/MediaINput/h264
> latency=50 protocols=3 ! rtpjitterbuffer latency=200 drop-on-latency=false !
> rtph264depay ! avdec_h264 ! xvimagesink

Any rational in adding a second rtpjitterbuffer ? there is already one
inside rtspsrc. For the dropping, it's possible 50ms on the rtspsrc
jitterbuffer is too low.

Nicolas
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Krishnamoorthy]

Hi Nicolas,

Thanks for your feedback.

There is no specific reason for rtpjitterbuffer. I have removed rtpjitterbuffer and increased the latency in the jitterbuffer. The decoder application works fine when there is no issue in the network.

But, my network environment is -
1) Wireless network
2) There is possibility of random delay (delay is not uniform between packets) between video packets.

To simulate the random delay, I have used the below command in linux PC (Another PC in between the video application and wireless router.)
sudo tc qdisc add dev eth1 root netem delay 250ms 750ms 10%


With this "non uniform" delay, the decoder pipeline dropping the frames continuously (introduce the delay for around 30 sec and then stop introduce the delay).

I found the below gstreamer pipeline is able to handle the "non uniform delay".
gst-launch-1.0 rtspsrc location=rtsp://<IP Address>/MediaINput/h264 protocols=3 ! queue max-size-time=100000000 ! rtph264depay ! avdec_h264 ! xvimagesink sync=false

List of changes compared to my earlier filter path:
1) Removed the rtpjitterbuffer
2) sync=false in xvimagesink
3) rtspsrc with default latency (latency=2000 ms)
4) Introduce queue  with max-size-time=100000000

This pipeline is able to manage the non-uniform delay in network packets. But, i found the below side effects.

1) Observed 1 sec delay in the displayed video frames over long run (1 hr).
2) Video frames are out of sync between the video feeds (I have 3 video decoders running simultaneously). Nearly around 500ms difference found between the video feeds.

Could any help me to review the above decoder pipeline and feedback on how to address the issue in delay, out of sync between multiple video feed issue.

[Arjen Veenhuizen]

Krishnamoorthy wrote

1) Observed 1 sec delay in the displayed video frames over long run (1 hr).

This is perhaps the internal jitterbuffer slowly filling up.

Krishnamoorthy wrote

2) Video frames are out of sync between the video feeds (I have 3 video decoders running simultaneously). Nearly around 500ms difference found between the video feeds.

That is obvious since you set sync=false. The pipeline will ignore the timestamps of the video frames and it will play them as they arrive ignoring all timing information (setting sync=false is almost never a solution).

[Krishnamoorthy]

Hi Arjen,

Thanks for your feedback.

Could you please clarify on
1) How to control the internal jitterbuffer from slowly filling up
2) xvimagesink filter is dropping the frame due to timestamping problem. And it looks that the reason for drop in xvimagesink filter is due to rtpjitterbuffer timestamp calculation when "non uniform" delay in network.

Is there anyway to handle the "non uniform" delay in video packets due to network condition in gstreamer decoder.

I am assuming that the out of sync between multiple video streams can be controlled by introducing queue. With that assumption, i tried the below pipeline.

gst-launch-1.0 rtspsrc location=rtsp://<IP Address>/MediaINput/h264 protocols=3 ! queue max-size-time=100000000 ! rtph264depay ! avdec_h264 ! xvimagesink sync=false

I am observing out of sync issue between multiple video feeds and the time difference is around 500 ms. But, I am nor sure whether the time difference will keep increase during long run or not.

Could you please suggest for any other way to handle the "non uniform" delay in video packets.

My original decoder filter path is as below:
rtspsrc location=rtsp://<IP Address>:554/MediaINput/h264 latency=50 protocols=3 ! rtpjitterbuffer latency=200 drop-on-latency=false ! rtph264depay ! avdec_h264 ! xvimagesink