Gstreamer h264 udp stream is to laggy

[kingalione]

Hello, I'm very new in gstreamer and have a problem:

I'm using gstreamer to stream a h264 encoded udp stream to an iPad. It works but there is one big problem. The stream is way to laggy. I mean if I do something in front of the camera, I see it on the iPad after 3-6 Seconds which is way to much.


this is how my pipeline looks:

sudo gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw, width=3072,height=2048,framerate=15/1 ! videoconvert ! queue ! videoscale ! video/x-raw, width=600,height=450,framerate=15/1 ! x264enc tune=zerolatency speed-preset=ultrafast bitrate=500 ! queue ! rtph264pay config-interval=2 pt=96 ! udpsink host=192.168.178.141 port=5000


The camera is plugged-in to a intel nuc celeron and if I run the pipeline I get about 80% processor usage on both cores.


I use this camera to stream:

sudo v4l2-ctl --all

Driver Info (not using libv4l2):
        Driver name   : uvcvideo
        Card type     : DFK 33UX178
        Bus info      : usb-0000:00:14.0-1
        Driver version: 3.16.7
        Capabilities  : 0x84000001
                Video Capture
                Streaming
                Device Capabilities
        Device Caps   : 0x04000001
                Video Capture
                Streaming
Priority: 2
Video input : 0 (Camera 1: ok)
Format Video Capture:
        Width/Height  : 3072/2048
        Pixel Format  : 'YUYV'
        Field         : None
        Bytes per Line: 6144
        Size Image    : 12582912
        Colorspace    : Unknown (00000000)
Crop Capability Video Capture:
        Bounds      : Left 0, Top 0, Width 3072, Height 2048
        Default     : Left 0, Top 0, Width 3072, Height 2048
        Pixel Aspect: 1/1
Streaming Parameters Video Capture:
        Capabilities     : timeperframe
        Frames per second: 15.000 (15/1)
        Read buffers     : 0
                     brightness (int)    : min=0 max=4095 step=1 default=200 value=333
                     saturation (int)    : min=0 max=255 step=1 default=64 value=240
                            hue (int)    : min=-180 max=180 step=1 default=0 value=0
   white_balance_component_auto (bool)   : default=1 value=0
    white_balance_red_component (int)    : min=0 max=255 step=1 default=64 value=112
   white_balance_blue_component (int)    : min=0 max=255 step=1 default=64 value=136
                          gamma (int)    : min=1 max=500 step=1 default=100 value=100
                           gain (int)    : min=0 max=480 step=1 default=0 value=0
               exposure_time_us (int)    : min=20 max=60000000 step=1 default=333 value=50000
                   auto_shutter (bool)   : default=1 value=0
        auto_exposure_reference (int)    : min=0 max=255 step=1 default=128 value=128
                      gain_auto (bool)   : default=1 value=0
                   trigger_mode (bool)   : default=0 value=0
               software_trigger (button) : flags=write-only
                  trigger_delay (int)    : min=0 max=1000000 step=1 default=0 value=31
                  strobe_enable (bool)   : default=0 value=0
                strobe_polarity (bool)   : default=0 value=0
                strobe_exposure (bool)   : default=1 value=1
                strobe_duration (int)    : min=0 max=32767 step=1 default=100 value=100
                   strobe_delay (int)    : min=0 max=32767 step=1 default=0 value=0
                          gpout (bool)   : default=0 value=0
                           gpin (bool)   : default=0 value=0
                   roi_offset_x (int)    : min=0 max=2816 step=4 default=0 value=0
                   roi_offset_y (int)    : min=0 max=2044 step=4 default=0 value=0
                roi_auto_center (bool)   : default=1 value=1
               trigger_polarity (menu)   : min=0 max=1 default=0 value=1
            trigger_burst_count (int)    : min=1 max=1000 step=1 default=1 value=1
       trigger_debounce_time_us (int)    : min=0 max=1000000 step=1 default=0 value=0
           trigger_mask_time_us (int)    : min=0 max=1000000 step=1 default=0 value=0
  trigger_noise_supression_time (int)    : min=0 max=1000000 step=1 default=0 value=0
     auto_functions_roi_control (bool)   : default=1 value=1
        auto_functions_roi_left (int)    : min=0 max=3056 step=1 default=0 value=768
         auto_functions_roi_top (int)    : min=0 max=2032 step=1 default=0 value=512
       auto_functions_roi_width (int)    : min=16 max=3072 step=1 default=0 value=1536
      auto_functions_roi_height (int)    : min=16 max=2048 step=1 default=0 value=1024
              whitebalance_mode (menu)   : min=0 max=1 default=0 value=0
      white_balance_auto_preset (menu)   : min=0 max=4 default=0 value=0
  white_balance_green_component (int)    : min=0 max=255 step=1 default=64 value=64
whitebalance_temperature_preset (menu)   : min=0 max=8 default=6500 value=4
      white_balance_temperature (int)    : min=2500 max=10000 step=100 default=6500 value=4000
                flip_horizontal (bool)   : default=0 value=0
 exposure_auto_upper_limit_auto (bool)   : default=1 value=0
      exposure_auto_upper_limit (int)    : min=20 max=1000000 step=1 default=333 value=50000



So what is the problem here to get a smooth live stream? I tried to play a bit with the framerates up and down but got no significant difference on the lagging time.

[Nicolas Dufresne-4]

Le jeudi 03 novembre 2016 à 07:11 -0700, kingalione a écrit :
>
> So what is the problem here to get a smooth live stream? I tried to play a
> bit with the framerates up and down but got no significant difference on the
> lagging time.

You may try to set a simplier H264 profile, but default it is high.
This is done by adding a caps filter after the encoder:

  ... ! x264enc tune=zerolatency ... ! video/x-h264,profile=constrained-baseline ! ...

If that is not enough, you should try and enable an HW encoder on your platform.

Nicolas
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Peter Maersk-Moller-2]

Hi.

What are you using to play the stream on the iPad? Software/setup ?

Regards

Peter

On Thu, Nov 3, 2016 at 3:11 PM, kingalione <[hidden email]> wrote:

Hello, I'm very new in gstreamer and have a problem:

I'm using gstreamer to stream a h264 encoded udp stream to an iPad. It works
but there is one big problem. The stream is way to laggy. I mean if I do
something in front of the camera, I see it on the iPad after 3-6 Seconds
which is way to much.


this is how my pipeline looks:

sudo gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,
width=3072,height=2048,framerate=15/1 ! videoconvert ! queue ! videoscale !
video/x-raw, width=600,height=450,framerate=15/1 ! x264enc tune=zerolatency
speed-preset=ultrafast bitrate=500 ! queue ! rtph264pay config-interval=2
pt=96 ! udpsink host=192.168.178.141 port=5000


The camera is plugged-in to a intel nuc celeron and if I run the pipeline I
get about 80% processor usage on both cores.


I use this camera to stream:

sudo v4l2-ctl --all

Driver Info (not using libv4l2):
        Driver name   : uvcvideo
        Card type     : DFK 33UX178
        Bus info      : usb-0000:00:14.0-1
        Driver version: 3.16.7
        Capabilities  : 0x84000001
                Video Capture
                Streaming
                Device Capabilities
        Device Caps   : 0x04000001
                Video Capture
                Streaming
Priority: 2
Video input : 0 (Camera 1: ok)
Format Video Capture:
        Width/Height  : 3072/2048
        Pixel Format  : 'YUYV'
        Field         : None
        Bytes per Line: 6144
        Size Image    : 12582912
        Colorspace    : Unknown (00000000)
Crop Capability Video Capture:
        Bounds      : Left 0, Top 0, Width 3072, Height 2048
        Default     : Left 0, Top 0, Width 3072, Height 2048
        Pixel Aspect: 1/1
Streaming Parameters Video Capture:
        Capabilities     : timeperframe
        Frames per second: 15.000 (15/1)
        Read buffers     : 0
                     brightness (int)    : min=0 max=4095 step=1 default=200
value=333
                     saturation (int)    : min=0 max=255 step=1 default=64
value=240
                            hue (int)    : min=-180 max=180 step=1 default=0
value=0
   white_balance_component_auto (bool)   : default=1 value=0
    white_balance_red_component (int)    : min=0 max=255 step=1 default=64
value=112
   white_balance_blue_component (int)    : min=0 max=255 step=1 default=64
value=136
                          gamma (int)    : min=1 max=500 step=1 default=100
value=100
                           gain (int)    : min=0 max=480 step=1 default=0
value=0
               exposure_time_us (int)    : min=20 max=60000000 step=1
default=333 value=50000
                   auto_shutter (bool)   : default=1 value=0
        auto_exposure_reference (int)    : min=0 max=255 step=1 default=128
value=128
                      gain_auto (bool)   : default=1 value=0
                   trigger_mode (bool)   : default=0 value=0
               software_trigger (button) : flags=write-only
                  trigger_delay (int)    : min=0 max=1000000 step=1
default=0 value=31
                  strobe_enable (bool)   : default=0 value=0
                strobe_polarity (bool)   : default=0 value=0
                strobe_exposure (bool)   : default=1 value=1
                strobe_duration (int)    : min=0 max=32767 step=1
default=100 value=100
                   strobe_delay (int)    : min=0 max=32767 step=1 default=0
value=0
                          gpout (bool)   : default=0 value=0
                           gpin (bool)   : default=0 value=0
                   roi_offset_x (int)    : min=0 max=2816 step=4 default=0
value=0
                   roi_offset_y (int)    : min=0 max=2044 step=4 default=0
value=0
                roi_auto_center (bool)   : default=1 value=1
               trigger_polarity (menu)   : min=0 max=1 default=0 value=1
            trigger_burst_count (int)    : min=1 max=1000 step=1 default=1
value=1
       trigger_debounce_time_us (int)    : min=0 max=1000000 step=1
default=0 value=0
           trigger_mask_time_us (int)    : min=0 max=1000000 step=1
default=0 value=0
  trigger_noise_supression_time (int)    : min=0 max=1000000 step=1
default=0 value=0
     auto_functions_roi_control (bool)   : default=1 value=1
        auto_functions_roi_left (int)    : min=0 max=3056 step=1 default=0
value=768
         auto_functions_roi_top (int)    : min=0 max=2032 step=1 default=0
value=512
       auto_functions_roi_width (int)    : min=16 max=3072 step=1 default=0
value=1536
      auto_functions_roi_height (int)    : min=16 max=2048 step=1 default=0
value=1024
              whitebalance_mode (menu)   : min=0 max=1 default=0 value=0
      white_balance_auto_preset (menu)   : min=0 max=4 default=0 value=0
  white_balance_green_component (int)    : min=0 max=255 step=1 default=64
value=64
whitebalance_temperature_preset (menu)   : min=0 max=8 default=6500 value=4
      white_balance_temperature (int)    : min=2500 max=10000 step=100
default=6500 value=4000
                flip_horizontal (bool)   : default=0 value=0
 exposure_auto_upper_limit_auto (bool)   : default=1 value=0
      exposure_auto_upper_limit (int)    : min=20 max=1000000 step=1
default=333 value=50000



So what is the problem here to get a smooth live stream? I tried to play a
bit with the framerates up and down but got no significant difference on the
lagging time.



--
View this message in context: http://gstreamer-devel.966125.n4.nabble.com/Gstreamer-h264-udp-stream-is-to-laggy-tp4680421.html
Sent from the GStreamer-devel mailing list archive at Nabble.com.
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[kingalione]

Hi Nicolas,

thanks for your help. I tried your solution and it didn't helped much. I still got a lag 3-5 seconds long. How can I enable an HW encoder for my platform?

I'm running gstreamer on a Intel Nuc celeron with linux ubuntu 14.04

[kingalione]

I'm using a own decoder. I don't think that the problem is the iPad side, because before we choose this camera we used another camera with lower resolution and the lag time there was about 0.2-0.8 seconds which is acceptable.

[Nicolas Dufresne-4]

Le vendredi 04 novembre 2016 à 02:41 -0700, kingalione a écrit :

> Hi Nicolas,
>
> thanks for your help. I tried your solution and it didn't helped
> much. I
> still got a lag 3-5 seconds long. How can I enable an HW encoder for
> my
> platform?
>
> I'm running gstreamer on a Intel Nuc celeron with linux ubuntu 14.04
> 

If it's supported by this HW, the encoder will be provided by
gstreamer-vaapi. Ubuntu 14.04 built-in gstreamer is too old. You likely
want to try with something really really recent, but check if this NUC
have such encoding HW, they don't all have it.

Note, you are likely chasing the wrong target. I just notice that you
capture at 2K from the camera, and SW convert + scale to 600x450.
That's ways too much for the NUC CPU. Try scaling in GL, you'll pay the
cost of upload/download, but it's more likely to work.

If you have a really recent version of the linux kernel and GStreamer,
you can make the upload zero-copy using dmabuf, something like:

  v4l2src io-mode=dmabuf ! glfilterbin filter="glcolorscale" ! x264enc ...

Replace x264enc with vaapih264enc if you have it for better perf.

regards,
Nicolas
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[kingalione]

Hello Nicolas and thanks you for your help.

What do you mean with:

Try scaling in GL, you'll pay the
cost of upload/download, but it's more likely to work.  ?


How whould the pipeline than look?


And if I update x264enc with vaapih264enc I get this error:
WARNING: erroneous pipeline: no element "vaapih264enc"

[Nicolas Dufresne-4]

Le vendredi 04 novembre 2016 à 07:54 -0700, kingalione a écrit :
> Hello Nicolas and thanks you for your help.
>
> What do you mean with:
>
> Try scaling in GL, you'll pay the 
> cost of upload/download, but it's more likely to work.  ?

Sending/receiving data from/to the GPU has a cost, but it's likely
lower then doing video scaling in software like you are doing now.

>
>
> How whould the pipeline than look?
>
>
> And if I update x264enc with vaapih264enc I get this error:
> WARNING: erroneous pipeline: no element "vaapih264enc"

I'm not sure what is the name of the vaapi element, I left you the
exercise:
 a) Install required plugins for you target linux distribution
 b) Check if the target HW support is available
 c) Find the right name and parameters using gst-inspect-1.0

regards,
Nicolas
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel