omxh264enc: incorrectly advertised sink capabilities?
Locked
omxh264enc: incorrectly advertised sink capabilities?
 
|
Hey there,
I'm trying to optimize a gstreamer pipeline running on a Raspberry Pi 3b+. According to gst-shark, a lot of proctime is spend in a videoconvert I added to get the pipeline working at all. The simiplified version looks like this: ``` gst-launch-1.0 gltestsrc ! glcolorconvert ! "video/x- raw(memory:GLMemory),format=RGB16" ! queue ! gldownload ! omxh264enc control- rate=1 target-bitrate=1145000 ! video/x-h264,profile=baseline ! h264parse ! avimux ! filesink location=test.avi ``` I.e. I have some part of the pipeline that lives in OpenGL land and thus operates on RGBA frames. Then I wanted to convert that via glcolorconvert to a format that can be understood by omxh264enc before downloading, but this does not work without a videoconvert before the `omxh264enc`, despite the fact that omxh264enc advertises the following: ``` Factory Details: Rank primary + 1 (257) Long-name OpenMAX H.264 Video Encoder Klass Codec/Encoder/Video/Hardware Description Encode H.264 video streams Author Sebastian Dröge <[hidden email]> Plugin Details: Name omx Description GStreamer OpenMAX Plug-ins Filename /usr/lib/gstreamer-1.0/libgstomx.so Version 1.16.1 License LGPL Source module gst-omx Source release date 2019-09-23 Binary package GStreamer OpenMAX Plug-ins source release Origin URL Unknown package origin ... Pad Templates: SINK template: 'sink' Availability: Always Capabilities: video/x-raw format: { (string)NV16_10LE32, (string)NV12_10LE32, (string)NV16, (string)YUY2, (string)YVYU, (string)UYVY, (string)N V12, (string)I420, (string)RGB16, (string)BGR16, (string)ABGR, (string)ARGB, (string)BGR, (string)GRAY8 } width: [ 1, 2147483647 ] height: [ 1, 2147483647 ] framerate: [ 0/1, 2147483647/1 ] ``` The error I get, for all RGB-like formats (BGR16, ABGR, ABGR, ARGB, BGR) is: ``` 0:00:00.216186510 1356 0x1454e60 WARN basesrc gstbasesrc.c: 3072:gst_base_src_loop:<gltestsrc0> error: Internal data stream error. 0:00:00.216280364 1356 0x1454e60 WARN basesrc gstbasesrc.c: 3072:gst_base_src_loop:<gltestsrc0> error: streaming stopped, reason not- negotiated (-4) ERROR: from element /GstPipeline:pipeline0/GstGLTestSrc:gltestsrc0: Internal data stream error. ``` Looking at the source for gst-omx I wonder why these formats are included in GST_OMX_VIDEO_SUPPORTED_FORMATS? They do not seem to be supported internally at all by omxh264enc, if I'm not mistaken? Are those maybe formats supported by omxh264dec, i.e. the decoder, but not by the encoder I'm trying to use? Generally, does someone know if the hardware encoder would in theory support being fed with such frames? Is this complicated to implement for someone with zero experience in writing gst plugins and OMX? or would it be somewhat straight forward? Could someone maybe give me some hints as to what one should do roughly - then I would try to implement this and upstream that code. My hunch is that this would greatly improve the performance of my specific pipeline, as I could get rid of the on-CPU videoconvert and leverage the glcolorconvert instead. Cheers -- Milian Wolff | [hidden email] | Senior Software Engineer KDAB (Deutschland) GmbH, a KDAB Group company Tel: +49-30-521325470 KDAB - The Qt, C++ and OpenGL Experts _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
smime.p7s (5K)
Re: omxh264enc: incorrectly advertised sink capabilities?
|
|
On Montag, 6. April 2020 11:44:17 CEST Milian Wolff wrote:
> Hey there, > > I'm trying to optimize a gstreamer pipeline running on a Raspberry Pi 3b+. > According to gst-shark, a lot of proctime is spend in a videoconvert I added > to get the pipeline working at all. > > The simiplified version looks like this: > > ``` > gst-launch-1.0 gltestsrc ! glcolorconvert ! "video/x- > raw(memory:GLMemory),format=RGB16" ! queue ! gldownload ! omxh264enc > control- rate=1 target-bitrate=1145000 ! video/x-h264,profile=baseline ! > h264parse ! avimux ! filesink location=test.avi > ``` > > I.e. I have some part of the pipeline that lives in OpenGL land and thus > operates on RGBA frames. Then I wanted to convert that via glcolorconvert to > a format that can be understood by omxh264enc before downloading, but this > does not work without a videoconvert before the `omxh264enc`, despite the > fact that omxh264enc advertises the following: > > ``` > Factory Details: > Rank primary + 1 (257) > Long-name OpenMAX H.264 Video Encoder > Klass Codec/Encoder/Video/Hardware > Description Encode H.264 video streams > Author Sebastian Dröge > <[hidden email]> > > Plugin Details: > Name omx > Description GStreamer OpenMAX Plug-ins > Filename /usr/lib/gstreamer-1.0/libgstomx.so > Version 1.16.1 > License LGPL > Source module gst-omx > Source release date 2019-09-23 > Binary package GStreamer OpenMAX Plug-ins source release > Origin URL Unknown package origin > ... > > Pad Templates: > SINK template: 'sink' > Availability: Always > Capabilities: > video/x-raw > format: { (string)NV16_10LE32, (string)NV12_10LE32, > (string)NV16, (string)YUY2, (string)YVYU, (string)UYVY, (string)N > V12, (string)I420, (string)RGB16, (string)BGR16, (string)ABGR, (string)ARGB, > (string)BGR, (string)GRAY8 } > width: [ 1, 2147483647 ] > height: [ 1, 2147483647 ] > framerate: [ 0/1, 2147483647/1 ] > ``` > > The error I get, for all RGB-like formats (BGR16, ABGR, ABGR, ARGB, BGR) is: > > ``` > 0:00:00.216186510 1356 0x1454e60 WARN basesrc > gstbasesrc.c: 3072:gst_base_src_loop:<gltestsrc0> error: Internal data > stream error. 0:00:00.216280364 1356 0x1454e60 WARN > basesrc gstbasesrc.c: 3072:gst_base_src_loop:<gltestsrc0> error: streaming > stopped, reason not- negotiated (-4) > ERROR: from element /GstPipeline:pipeline0/GstGLTestSrc:gltestsrc0: Internal > data stream error. > ``` > > Looking at the source for gst-omx I wonder why these formats are included in > GST_OMX_VIDEO_SUPPORTED_FORMATS? They do not seem to be supported > internally at all by omxh264enc, if I'm not mistaken? Are those maybe > formats supported by omxh264dec, i.e. the decoder, but not by the encoder > I'm trying to use? > > Generally, does someone know if the hardware encoder would in theory support > being fed with such frames? Is this complicated to implement for someone > with zero experience in writing gst plugins and OMX? or would it be > somewhat straight forward? Could someone maybe give me some hints as to > what one should do roughly - then I would try to implement this and > upstream that code. > > My hunch is that this would greatly improve the performance of my specific > pipeline, as I could get rid of the on-CPU videoconvert and leverage the > glcolorconvert instead. 1.16.1 improves the performance significantly, but the video's colors are all wrong. I.e. red turns orange, green turns violet and blue turns yellow... So clearly there's more to be done but I have no clue what would be needed to get there... Does anyone have a suggestion what to try? ``` diff --git a/omx/gstomxvideoenc.c b/omx/gstomxvideoenc.c index dfc03c5..c1a5b71 100644 --- a/omx/gstomxvideoenc.c +++ b/omx/gstomxvideoenc.c @@ -1881,6 +1881,11 @@ gst_omx_video_enc_configure_input_buffer (GstOMXVideoEnc * self, ((port_def.format.video.nFrameHeight + 1) / 2)); break; + case OMX_COLOR_Format32bitBGRA8888: + case OMX_COLOR_Format32bitARGB8888: + port_def.nBufferSize = 4 * (port_def.format.video.nStride * port_def.format.video.nFrameHeight); + break; + default: GST_ERROR_OBJECT (self, "Unsupported port format %x", port_def.format.video.eColorFormat); @@ -3211,8 +3216,8 @@ filter_supported_formats (GList * negotiation_map) case GST_VIDEO_FORMAT_NV12_10LE32: case GST_VIDEO_FORMAT_NV16: case GST_VIDEO_FORMAT_NV16_10LE32: - //case GST_VIDEO_FORMAT_ABGR: - //case GST_VIDEO_FORMAT_ARGB: + case GST_VIDEO_FORMAT_ABGR: + case GST_VIDEO_FORMAT_ARGB: cur = g_list_next (cur); continue; default: ``` Additionally, I would really appreciate if someone could explain the following to me: When I try to feed `omxh264enc` after applying the above patch with `ARGB` input I get: ``` 0:00:02.013120260 2588 0x1fc3e60 DEBUG GST_CAPS gstutils.c: 3111:gst_pad_peer_query_caps:<omxh264enc-omxh264enc0:src> peer query returned video/x-h264, width=(int)[ 16, 4096 ], height=(int)[ 16, 4096 ], framerate=(fraction)[ 0/1, 2147483647/1 ], profile=(string)baseline, stream- format=(string)byte-stream, alignment=(string)au; video/x-h264, width=(int)[ 16, 4096 ], height=(int)[ 16, 4096 ], framerate=(fraction)[ 0/1, 2147483647/1 ], profile=(string)baseline 0:00:02.013263854 2588 0x1fc3e60 LOG default gstvideoutilsprivate.c: 128:__gst_video_element_proxy_getcaps:<omxh264enc-omxh264enc0> template caps video/x-raw, format=(string)I420; video/x-raw, format=(string)NV12; video/x- raw, format=(string)ABGR 0:00:02.013389583 2588 0x1fc3e60 LOG default gstvideoutilsprivate.c: 129:__gst_video_element_proxy_getcaps:<omxh264enc-omxh264enc0> allowed caps video/x-h264, width=(int)[ 16, 4096 ], height=(int)[ 16, 4096 ], framerate=(fraction)[ 0/1, 2147483647/1 ], profile=(string)baseline, stream- format=(string)byte-stream, alignment=(string)au; video/x-h264, width=(int)[ 16, 4096 ], height=(int)[ 16, 4096 ], framerate=(fraction)[ 0/1, 2147483647/1 ], profile=(string)baseline 0:00:02.013797604 2588 0x1fc3e60 LOG default gstvideoutilsprivate.c: 138:__gst_video_element_proxy_getcaps:<omxh264enc-omxh264enc0> intersecting with video/x-raw(memory:GLMemory), format=(string)ARGB, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ], texture-target=(string){ 2D, rectangle, external-oes }; video/ x-raw(memory:DMABuf), format=(string)ARGB, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ]; video/x-raw, format=(string)ARGB, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ] 0:00:02.013934218 2588 0x1fc3e60 LOG default gstvideoutilsprivate.c: 147:__gst_video_element_proxy_getcaps:<omxh264enc-omxh264enc0> proxy caps EMPTY 0:00:02.014005156 2588 0x1fc3e60 LOG omxvideoenc gstomxvideoenc.c:3258:gst_omx_video_enc_getcaps:<omxh264enc-omxh264enc0> Supported caps EMPTY ``` But for `ABGR` it works like the following - why? Shouldn't the code handle both now? ``` 0:00:02.026753801 2733 0xf85c60 DEBUG GST_CAPS gstutils.c: 3111:gst_pad_peer_query_caps:<omxh264enc-omxh264enc0:src> peer query returned video/x-h264, width=(int)[ 16, 4096 ], height=(int)[ 16, 4096 ], framerate=(fraction)[ 0/1, 2147483647/1 ], profile=(string)baseline, stream- format=(string)byte-stream, alignment=(string)au; video/x-h264, width=(int)[ 16, 4096 ], height=(int)[ 16, 4096 ], framerate=(fraction)[ 0/1, 2147483647/1 ], profile=(string)baseline 0:00:02.026899114 2733 0xf85c60 LOG default gstvideoutilsprivate.c: 128:__gst_video_element_proxy_getcaps:<omxh264enc-omxh264enc0> template caps video/x-raw, format=(string)I420; video/x-raw, format=(string)NV12; video/x- raw, format=(string)ABGR 0:00:02.027026353 2733 0xf85c60 LOG default gstvideoutilsprivate.c: 129:__gst_video_element_proxy_getcaps:<omxh264enc-omxh264enc0> allowed caps video/x-h264, width=(int)[ 16, 4096 ], height=(int)[ 16, 4096 ], framerate=(fraction)[ 0/1, 2147483647/1 ], profile=(string)baseline, stream- format=(string)byte-stream, alignment=(string)au; video/x-h264, width=(int)[ 16, 4096 ], height=(int)[ 16, 4096 ], framerate=(fraction)[ 0/1, 2147483647/1 ], profile=(string)baseline 0:00:02.027436614 2733 0xf85c60 LOG default gstvideoutilsprivate.c: 138:__gst_video_element_proxy_getcaps:<omxh264enc-omxh264enc0> intersecting with video/x-raw(memory:GLMemory), format=(string)ABGR, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ], texture-target=(string){ 2D, rectangle, external-oes }; video/ x-raw(memory:DMABuf), format=(string)ABGR, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ]; video/x-raw, format=(string)ABGR, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ] 0:00:02.027611770 2733 0xf85c60 LOG default gstvideoutilsprivate.c: 147:__gst_video_element_proxy_getcaps:<omxh264enc-omxh264enc0> proxy caps video/x-raw, width=(int)[ 16, 4096 ], height=(int)[ 16, 4096 ], framerate=(fraction)[ 0/1, 2147483647/1 ], format=(string)ABGR 0:00:02.027717499 2733 0xf85c60 LOG omxvideoenc gstomxvideoenc.c:3258:gst_omx_video_enc_getcaps:<omxh264enc-omxh264enc0> Supported caps video/x-raw, width=(int)[ 16, 4096 ], height=(int)[ 16, 4096 ], framerate=(fraction)[ 0/1, 2147483647/1 ], format=(string)ABGR ``` -- Milian Wolff | [hidden email] | Senior Software Engineer KDAB (Deutschland) GmbH, a KDAB Group company Tel: +49-30-521325470 KDAB - The Qt, C++ and OpenGL Experts _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
| Free forum by Nabble | Edit this page |