Possible bug in rtpjpegpay and/or rtpjpegdepay.

[Stirling Westrup]

It seems that one or the other of the two elements has trouble correctly dealing with fragmented jpeg images. As the size of the images increase, one starts getting lots and lots of warnings and dropped frames from rtpjepgdepay:

0:00:14.129355327 19338      0x1b11370 WARN            rtpjpegdepay gstrtpjpegdepay.c:760:gst_rtp_jpeg_depay_process:<rtpjpegdepay0> discarding data packets received when we have no header

Here are our test pipelines. Please let us know if this is an actual bug, or if we are doing something wrong. We've found that with MTU=1400 and QUAL=75, we start dropping long (multi-second) sequences of frames. If we set MTU to 64000, we have almost no dropped frames, and if we set QUAL=5 we have no trouble with dropped frames (although the image is terrible).

SERVER:
#!/bin/bash

PATH="/home/sti/Videos/BBB"
FILE="bbb_sunflower_1080p_30fps_normal.mp4"
MTU=1400
QUAL=75
PATHNAME="$PATH/$FILE"

/usr/bin/gst-launch-1.0                             \
  rtpbin ntp-time-source=1 ntp-sync=true buffer-mode=synced name=sendbin \
  filesrc location="$PATHNAME" ! qtdemux ! h264parse ! avdec_h264     \
    ! videoconvert ! queue ! jpegenc quality=$QUAL ! rtpjpegpay mtu=$MTU \
    ! sendbin.send_rtp_sink_0                         \
  sendbin.send_rtp_src_0                         \
    ! udpsink port=25000 host="127.0.0.1"                 \
  sendbin.send_rtcp_src_0                         \
    ! udpsink port=25001 host="127.0.0.1" sync=false async=false      \
  udpsrc port=25005 ! sendbin.recv_rtcp_sink_0

--8<---CUT HERE--->8--

CLIENT:

#!/bin/bash

/usr/bin/gst-launch-1.0   \
  rtpbin name=rtpbin                            \
  udpsrc port=25000                             \
    ! application/x-rtp,media=video,payload=26,encoding-name=JPEG    \
    ,clock-rate=90000 ! rtpbin.recv_rtp_sink_0                \
  rtpbin. ! rtpjpegdepay ! queue ! jpegparse ! jpegdec ! videoconvert     \
    ! ximagesink display=:0                        \
  udpsrc port=25001 ! rtpbin.recv_rtcp_sink_0                \
  rtpbin.send_rtcp_src_0                        \
    ! udpsink port=25005 host="127.0.0.1" sync=false async=false



--

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Hello,

Le jeudi 13 juillet 2017 à 13:25 -0400, Stirling Westrup a écrit :
> It seems that one or the other of the two elements has trouble
> correctly dealing with fragmented jpeg images. As the size of the
> images increase, one starts getting lots and lots of warnings and
> dropped frames from rtpjepgdepay:
>
> 0:00:14.129355327 19338      0x1b11370 WARN            rtpjpegdepay
> gstrtpjpegdepay.c:760:gst_rtp_jpeg_depay_process:<rtpjpegdepay0>
> discarding data packets received when we have no header

I was testing your scripts, and notice that packets are dropped by the
receiving socket. I could workaround the issue locally.

1. By increasing kernel UDP memory: sysctl -w net.core.rmem_max=26214400
2. Increase client socket buffer size in client: buffer-size=$((512*1024))

Note that it works with smaller rmem_max value. It also seems to work
with receiver buffer-size of 256 kb. I'm not sure how one calculate the
requirement for these.

regards,
Nicolas

>  
> Here are our test pipelines. Please let us know if this is an actual
> bug, or if we are doing something wrong. We've found that with
> MTU=1400 and QUAL=75, we start dropping long (multi-second) sequences
> of frames. If we set MTU to 64000, we have almost no dropped frames,
> and if we set QUAL=5 we have no trouble with dropped frames (although
> the image is terrible).
>
> SERVER:
> #!/bin/bash
>
> PATH="/home/sti/Videos/BBB"
> FILE="bbb_sunflower_1080p_30fps_normal.mp4"
> MTU=1400
> QUAL=75
> PATHNAME="$PATH/$FILE"
>
> /usr/bin/gst-launch-1.0                             \
>   rtpbin ntp-time-source=1 ntp-sync=true buffer-mode=synced
> name=sendbin \
>   filesrc location="$PATHNAME" ! qtdemux ! h264parse ! avdec_h264  
>  \
>     ! videoconvert ! queue ! jpegenc quality=$QUAL ! rtpjpegpay
> mtu=$MTU \
>     ! sendbin.send_rtp_sink_0                         \
>   sendbin.send_rtp_src_0                         \
>     ! udpsink port=25000 host="127.0.0.1"                 \
>   sendbin.send_rtcp_src_0                         \
>     ! udpsink port=25001 host="127.0.0.1" sync=false async=false    
>  \
>   udpsrc port=25005 ! sendbin.recv_rtcp_sink_0
>
> --8<---CUT HERE--->8--
> CLIENT:
>
> #!/bin/bash
>
> /usr/bin/gst-launch-1.0   \
>   rtpbin name=rtpbin                            \
>   udpsrc port=25000                             \
>     ! application/x-rtp,media=video,payload=26,encoding-name=JPEG  
> \
>     ,clock-rate=90000 ! rtpbin.recv_rtp_sink_0                \
>   rtpbin. ! rtpjpegdepay ! queue ! jpegparse ! jpegdec ! videoconvert
>     \
>     ! ximagesink display=:0                        \
>   udpsrc port=25001 ! rtpbin.recv_rtcp_sink_0                \
>   rtpbin.send_rtcp_src_0                        \
>     ! udpsink port=25005 host="127.0.0.1" sync=false async=false
>
>
>
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[rrazavi]

Thank you for your answer, looks like udpsink also has buffer-size option and buy setting correct value in both udpsink and udpsrc it works fine.

[Stirling Westrup]

Thanks for the workaround. We've tested it and it works in our environment, but it leaves us wondering at its necessity. Will we have to do this in production, or is there some other fix that could be applied to jpegpay/depay?

On Thu, Jul 13, 2017 at 3:35 PM, Nicolas Dufresne <[hidden email]> wrote:

Hello,

Le jeudi 13 juillet 2017 à 13:25 -0400, Stirling Westrup a écrit :
> It seems that one or the other of the two elements has trouble
> correctly dealing with fragmented jpeg images. As the size of the
> images increase, one starts getting lots and lots of warnings and
> dropped frames from rtpjepgdepay:
>
> 0:00:14.129355327 19338      0x1b11370 WARN            rtpjpegdepay
> gstrtpjpegdepay.c:760:gst_rtp_jpeg_depay_process:<rtpjpegdepay0>
> discarding data packets received when we have no header

I was testing your scripts, and notice that packets are dropped by the
receiving socket. I could workaround the issue locally.

1. By increasing kernel UDP memory: sysctl -w net.core.rmem_max=26214400
2. Increase client socket buffer size in client: buffer-size=$((512*1024))

Note that it works with smaller rmem_max value. It also seems to work
with receiver buffer-size of 256 kb. I'm not sure how one calculate the
requirement for these.

regards,
Nicolas

>
> Here are our test pipelines. Please let us know if this is an actual
> bug, or if we are doing something wrong. We've found that with
> MTU=1400 and QUAL=75, we start dropping long (multi-second) sequences
> of frames. If we set MTU to 64000, we have almost no dropped frames,
> and if we set QUAL=5 we have no trouble with dropped frames (although
> the image is terrible).
>
> SERVER:
> #!/bin/bash
>
> PATH="/home/sti/Videos/BBB"
> FILE="bbb_sunflower_1080p_30fps_normal.mp4"
> MTU=1400
> QUAL=75
> PATHNAME="$PATH/$FILE"
>
> /usr/bin/gst-launch-1.0                             \
>   rtpbin ntp-time-source=1 ntp-sync=true buffer-mode=synced
> name=sendbin \
>   filesrc location="$PATHNAME" ! qtdemux ! h264parse ! avdec_h264
>  \
>     ! videoconvert ! queue ! jpegenc quality=$QUAL ! rtpjpegpay
> mtu=$MTU \
>     ! sendbin.send_rtp_sink_0                         \
>   sendbin.send_rtp_src_0                         \
>     ! udpsink port=25000 host="127.0.0.1"                 \
>   sendbin.send_rtcp_src_0                         \
>     ! udpsink port=25001 host="127.0.0.1" sync=false async=false
>  \
>   udpsrc port=25005 ! sendbin.recv_rtcp_sink_0
>
> --8<---CUT HERE--->8--
> CLIENT:
>
> #!/bin/bash
>
> /usr/bin/gst-launch-1.0   \
>   rtpbin name=rtpbin                            \
>   udpsrc port=25000                             \
>     ! application/x-rtp,media=video,payload=26,encoding-name=JPEG
> \
>     ,clock-rate=90000 ! rtpbin.recv_rtp_sink_0                \
>   rtpbin. ! rtpjpegdepay ! queue ! jpegparse ! jpegdec ! videoconvert
>     \
>     ! ximagesink display=:0                        \
>   udpsrc port=25001 ! rtpbin.recv_rtcp_sink_0                \
>   rtpbin.send_rtcp_src_0                        \
>     ! udpsink port=25005 host="127.0.0.1" sync=false async=false
>
>
>

> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

--

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le jeudi 13 juillet 2017 à 16:04 -0400, Stirling Westrup a écrit :
> Thanks for the workaround. We've tested it and it works in our
> environment, but it leaves us wondering at its necessity. Will we
> have to do this in production, or is there some other fix that could
> be applied to jpegpay/depay?

rtpjpegpay/depay are not implicated in this issue. Setting the buffer-
size for your use case on udpsrc/sink is something we normally do for
high rate streaming. The kernel default (4K) is pretty small. For the
kernel side, it's unclear to me why it is so small, I would need to
research a bit more the subject.

I know that in high throughput servers, it pretty common to tweak the
linux kernel. Those settings are usually placed in /etc/sysctl.d/. As
these are split files, you can ship a config as part of your
application (dropbox does that to increase the number of FD you can
watch with inotify).

Nicolas
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le jeudi 13 juillet 2017 à 12:46 -0700, rrazavi a écrit :
> Thank you for your answer, looks like udpsink also has buffer-size
> option and
> buy setting correct value in both udpsink and udpsrc it works fine.
>

I'm quite happy when a reply helps multiple users. Would you mind
sharing how you kernel is configured in redgar to rmem/wmem ? For me I
had the following:

$ sysctl net.core.rmem_default net.core.rmem_max net.core.wmem_default net.core.wmem_max
net.core.rmem_default = 212992
net.core.rmem_max = 212992
net.core.wmem_default = 212992
net.core.wmem_max = 212992

And 208kb was not sufficient for this test to work. Oracle requires
256, but still, that isn't sufficient. I manage to get it running at
300kb. But the speed of the CPU and the general can have an impact I
suppose. I don't know yet why the _max are set that low and same to the
default. It does not give the application much control really.

regards,
Nicolas
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[rrazavi]

It's giving me exact same results:
sysctl net.core.rmem_default net.core.rmem_max net.core.wmem_default net.core.wmem_max
net.core.rmem_default = 212992
net.core.rmem_max = 212992
net.core.wmem_default = 212992
net.core.wmem_max = 212992

[rrazavi]

I'm trying to playback rtsp with jpegpay in an Android device. Client pipeline is something simple like this :
udpsrc buffer-size=26214400 port=5000 caps=application/x-rtp,media=video,clock-rate=90000,encoding-name=JPEG,payload=96 ! queue ! rtpjitterbuffer ! queue ! rtpjpegdepay ! queue ! jpegdec ! queue  ! glupload ! queue  ! glcolorconvert ! queue ! glimagesinkelement

I've already increased kernel udp maximum memory size in device and get this from Android device command line:
sysctl net.core.rmem_default net.core.rmem_max net.core.wmem_default net.core.wmem_max
net.core.rmem_default = 212992
net.core.rmem_max = 26214400
net.core.wmem_default = 212992
net.core.wmem_max = 212992

But I'm still getting the same error as before :
gstrtpjpegdepay.c:760:gst_rtp_jpeg_depay_process:<rtpjpegdepay0> discarding data packets received when we have no header

[Nicolas Dufresne-5]

Le jeudi 27 juillet 2017 à 14:20 -0700, rrazavi a écrit :

> I'm trying to playback rtsp with jpegpay in an Android device. Client
> pipeline is something simple like this :
> udpsrc buffer-size=26214400 port=5000
> caps=application/x-rtp,media=video,clock-rate=90000,encoding-
> name=JPEG,payload=96
> ! queue ! rtpjitterbuffer ! queue ! rtpjpegdepay ! queue ! jpegdec !
> queue
> ! glupload ! queue  ! glcolorconvert ! queue ! glimagesinkelement
>
> I've already increased kernel udp maximum memory size in device and
> get this
> from Android device command line:
> sysctl net.core.rmem_default net.core.rmem_max net.core.wmem_default
> net.core.wmem_max
> net.core.rmem_default = 212992
> net.core.rmem_max = 26214400
> net.core.wmem_default = 212992
> net.core.wmem_max = 212992
>
> But I'm still getting the same error as before :
> gstrtpjpegdepay.c:760:gst_rtp_jpeg_depay_process:<rtpjpegdepay0>
> discarding
> data packets received when we have no header

Maybe this time, the receiver is just too slow. If your reduce the
framerate, is it better ? What is the size of the jpeg images ? (peek
size is important)

Nicolas
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[rrazavi]

Thank you for your response.
About the performance I'm pretty sure that I have a decent speed on my system.
a pipeline like this plays smooth and device CPU usage is around 50%
tcpserversrc host=192.168.120.27 port=7000 ! queue ! gdpdepay ! queue ! jpegdec ! queue ! glupload ! queue ! glcolorconvert ! queue ! glimagesinkelement

We have costume version of jpegdec by Stirling that is multi threaded and once it's completely debuged will commit it to upstream.

There is sender that makes sure jpegs sizes are not getting bigger than .3MB but still we're having this problem.
What you suggest ?