Gstremer vaapi pipeline to change color space

[Andressio]

Hello everybody. I'm trying to build a pipeline to decode an h264 stream
coming from a rtsp camera.
As the video is 4k i'm trying to use vaapi plugins.

If I launch this pipeline
/gst-launch-1.0 rtspsrc
location="rtsp://root:root@192.168.0.91:554/axis-media/media.amp?videocodec=h264&resolution=3840x2160"
protocols=GST_RTSP_LOWER_TRANS_TCP ! rtph264depay ! queue ! vaapih264dec !
vaapisink/

i can see the video and the cpu usage is around 20%.
But if I use that pipeline in Opencv the usage becomes 80%. The piepline in
opencv ends like this:
/... ! rtph264depay ! queue ! vaapih264dec ! videoconvert ! appsink/

If I understand correcty it is necessary the change of memory space from
VAsurface to standard memory that opencv can mange, and also a change of
color space from (probably) something like NV12 to BGR which opencv normally
use.
So I tried to perform the color space conversion directly in hw in order to
reduce the cpu usage but unfortunatly I did not succed.

I tried to use the vaapipostproc and vaapidecodebin plugins like this
1) /...rtph264depay ! queue ! vaapih264dec ! videopostproc format=bgr !
videoconvert ! appsikn/
2) /...rtph264depay ! queue ! vaapih264dec ! videopostproc !
video/x-raw,format=(string)BGR ! videoconvert ! appsikn/
3) /...rtph264depay ! queue ! vaapidecodebin ! videoconvert !
video/x-raw,format=(string)BGR ! videoconvert ! appsikn/

The last one launched from terminal with this pipeline
/... ! rtph264depay ! queue ! vaapih264dec ! vaapipostproc !
video/x-raw,format=(string)BGR ! autovideosink/
gave this error: /could not link vaapipostproc0 to autovideosink0,
vaapipostproc0 can't handle caps video/x-raw, format=(string)BGR/, but the
GStreamer docs states that one of the output formats can be the one I want.

Do you now if ther is a way to perform the change in color space that
directly in hw? What is the proper way? Can this be ported to opencv?

Many thanks




--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Victor Jaquez]

The possible color conversions are restricted by the driver/hardware. If the
vaapi postproc available by your driver cannot convert to BGR, you'll have to
rely on software conversion, or perhaps use an opengl shader (glcoloconvertion),
but you still could lose the zero-copy.

With i915 driver in kabylake, vaapipostproc reports that can convert to NV12,
I420, YV12, YUY2, UYVY, RGBA, RGBx, BGRA, BGRx and P010_10LE

Perhaps OpenCV can process BGRA (adding the alpha channel)

vmjl

On Tue, 09 Oct 2018 at 05:27, Andressio wrote:

> Hello everybody. I'm trying to build a pipeline to decode an h264 stream
> coming from a rtsp camera.
> As the video is 4k i'm trying to use vaapi plugins.
>
> If I launch this pipeline
> /gst-launch-1.0 rtspsrc
> location="rtsp://root:root@192.168.0.91:554/axis-media/media.amp?videocodec=h264&resolution=3840x2160"
> protocols=GST_RTSP_LOWER_TRANS_TCP ! rtph264depay ! queue ! vaapih264dec !
> vaapisink/
>
> i can see the video and the cpu usage is around 20%.
> But if I use that pipeline in Opencv the usage becomes 80%. The piepline in
> opencv ends like this:
> /... ! rtph264depay ! queue ! vaapih264dec ! videoconvert ! appsink/
>
> If I understand correcty it is necessary the change of memory space from
> VAsurface to standard memory that opencv can mange, and also a change of
> color space from (probably) something like NV12 to BGR which opencv normally
> use.
> So I tried to perform the color space conversion directly in hw in order to
> reduce the cpu usage but unfortunatly I did not succed.
>
> I tried to use the vaapipostproc and vaapidecodebin plugins like this
> 1) /...rtph264depay ! queue ! vaapih264dec ! videopostproc format=bgr !
> videoconvert ! appsikn/
> 2) /...rtph264depay ! queue ! vaapih264dec ! videopostproc !
> video/x-raw,format=(string)BGR ! videoconvert ! appsikn/
> 3) /...rtph264depay ! queue ! vaapidecodebin ! videoconvert !
> video/x-raw,format=(string)BGR ! videoconvert ! appsikn/
>
> The last one launched from terminal with this pipeline
> /... ! rtph264depay ! queue ! vaapih264dec ! vaapipostproc !
> video/x-raw,format=(string)BGR ! autovideosink/
> gave this error: /could not link vaapipostproc0 to autovideosink0,
> vaapipostproc0 can't handle caps video/x-raw, format=(string)BGR/, but the
> GStreamer docs states that one of the output formats can be the one I want.
>
> Do you now if ther is a way to perform the change in color space that
> directly in hw? What is the proper way? Can this be ported to opencv?
>
> Many thanks
>
>
>
>
> --
> Sent from: http://gstreamer-devel.966125.n4.nabble.com/
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Andressio]

Thank you Victor for your reply.

It works because the pipeline started grabbing frames. This is currently the
end of the pipeline
/.. ! rtph264depay ! vaapih264dec ! vaapipostproc ! video/x-raw,format=BGRx
! videoconvert ! appsink/

Still the cpu is very close to 80%. By exploring the opencv code of the
gstreamer backend seems that opencv accepts at most 3 channels. This means a
copy is necessary. Will explore further.

Could you please tell me how did you get that infomation regarding i915
dirver? Is there a way to force to use it? Because I've installed /vainfo/,
/intel-vaapi-dirver/ and /gstreamer/ via apt and if I launch the command
/vainfo/, i965 driver are opened.



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Victor Jaquez]

On Tue, 09 Oct 2018 at 10:35, Andressio wrote:

> Thank you Victor for your reply.
>
> It works because the pipeline started grabbing frames. This is currently the
> end of the pipeline
> /.. ! rtph264depay ! vaapih264dec ! vaapipostproc ! video/x-raw,format=BGRx
> ! videoconvert ! appsink/
>
> Still the cpu is very close to 80%. By exploring the opencv code of the
> gstreamer backend seems that opencv accepts at most 3 channels. This means a
> copy is necessary. Will explore further.
>
> Could you please tell me how did you get that infomation regarding i915
> dirver? Is there a way to force to use it? Because I've installed /vainfo/,
> /intel-vaapi-dirver/ and /gstreamer/ via apt and if I launch the command
> /vainfo/, i965 driver are opened.

If you run the pipeline with debug of vaapipostproc:5 you'll see the list for
formats that the driver can convert to, given the current stream.

vmjl

>
>
>
> --
> Sent from: http://gstreamer-devel.966125.n4.nabble.com/
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel