Lower latency suggestions using opencv and gstreamer

[WisdomPill]

I'm trying to stream manipulated images from opencv and I came across a latency problem. I'm new to gstreamer and opencv and I was wondering if someone with more experience can tell me if I can improve the pipeline from a latency perspective.
I think the pipeline is the problem here since there's no heavy processing in opencv, but I'm not sure about that.
I'm using opencv-3.2 with python3 on a raspberry pi 3.

My code is the following.

import cv2

cap = cv2.VideoCapture(0)

framerate = 10.0

out = cv2.VideoWriter('appsrc ! videoconvert ! '
                      'x264enc bitrate=500 speed-preset=ultrafast quantizer=20 tune=zerolatency ! '
                      'rtph264pay config-interval=1 pt=96 ! '
                      'tcpserversink host=192.168.1.27 port=5000 ',
                      0, framerate, (640, 480))

while cap.isOpened():
    ret, frame = cap.read()
    if ret:
        frame = cv2.flip(frame, 0)

        # write the flipped frame
        out.write(frame)

        if cv2.waitKey(1) & 0xFF == ord('q'):
            break
    else:
        break

# Release everything if job is finished
cap.release()
out.release()

[Michael MacIntosh]

You could have a color conversion happening, which could add some
overhead, you could try adding a queue between your videoconvert and
your encoder to split them into other threads, or try outputting in a
format so you don't need the videoconvert.

What kind of latency are you seeing?

Also if you just want to flip frames there is a videoflip element:

https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-good/html/gst-plugins-good-plugins-videoflip.html

Cheers,
Michael.


On 8/11/2017 10:25 AM, WisdomPill wrote:

> I'm trying to stream manipulated images from opencv and I came across a
> latency problem. I'm new to gstreamer and opencv and I was wondering if
> someone with more experience can tell me if I can improve the pipeline from
> a latency perspective.
> I think the pipeline is the problem here since there's no heavy processing
> in opencv, but I'm not sure about that.
> I'm using opencv-3.2 with python3 on a raspberry pi 3.
>
> My code is the following.
>
> import cv2
>
> cap = cv2.VideoCapture(0)
>
> framerate = 10.0
>
> out = cv2.VideoWriter('appsrc ! videoconvert ! '
>                        'x264enc bitrate=500 speed-preset=ultrafast
> quantizer=20 tune=zerolatency ! '
>                        'rtph264pay config-interval=1 pt=96 ! '
>                        'tcpserversink host=192.168.1.27 port=5000 ',
>                        0, framerate, (640, 480))
>
> while cap.isOpened():
>      ret, frame = cap.read()
>      if ret:
>          frame = cv2.flip(frame, 0)
>
>          # write the flipped frame
>          out.write(frame)
>
>          if cv2.waitKey(1) & 0xFF == ord('q'):
>              break
>      else:
>          break
>
> # Release everything if job is finished
> cap.release()
> out.release()
>
>
>
>
> --
> View this message in context: http://gstreamer-devel.966125.n4.nabble.com/Lower-latency-suggestions-using-opencv-and-gstreamer-tp4684168.html
> Sent from the GStreamer-devel mailing list archive at Nabble.com.
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>
>

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[WisdomPill]

About flipping the frame, it was only a test to check that opencv was working, just ignore that.
However I tried with this pipeline

out = cv2.VideoWriter('appsrc ! video/x-raw,format=RGB,width=640,height=480,framerate=10/1 ! videoconvert ! '
                      'x264enc bitrate=256 noise-reduction=10000 pass=qual speed-preset=ultrafast quantizer=20 tune=zerolatency ! '
                      'rtph264pay config-interval=1 pt=96 ! gdppay !'
                      'tcpserversink host=192.168.1.27 port=5000 sync=false',
                      0, framerate, (640, 480))

but I cant get it to work, it gives me an error that I don't know how to debug.
Is it the rigth way to change color format?
Can you give me an example of what you were suggesting?

Thank you for answering.

[Baby Octopus]

Try the same pipeline in gst-launch to see if there is something wrong in the pipeline. You could replace appsrc with videotestsrc and audiotestsrc just to see if the rest of the pipeline is fine

[WisdomPill]

I tried the following:

gst-launch-1.0 -v videotestsrc ! video/x-raw,format=RGB,width=640,height=480,framerate=10/1 ! videoconvert ! x264enc bitrate=256 noise-reduction=10000 pass=qual speed-preset=ultrafast quantizer=20 tune=zerolatency ! rtph264pay config-interval=1 pt=96 ! gdppay ! tcpserversink host=192.168.1.27 port=5000 sync=false

and it worked like a charm, I think the problem is with appsrc, I tried different pipelines and everytime that I put the capsfilter it gives me the following error:

GStreamer Plugin: Embedded video playback halted; module appsrc0 reported: Internal data flow error.

The strange thing about appsrc is also that if I don't put videoconvert in the pipeline the latter error comes up again.

[WisdomPill]

I found the solution, the key was to use speed-preset=ultrafast in x264enc.
I don't know why it works because from the terminal even choosing the v4l2src it works seamlessly but it seems that from opencv it gives better results.

If someone else has better solutions or an explanation for this behaviour is welcome to reply