BGRA for omxh264enc and omxh264dec

[Peter Maersk-Moller-2]

Hi Sebastian.

Seems you wrote most of the omx modules.

Is it possible to extend omxh264enc and omxh264dec to accept respectively provide BGRA in addition to the RGBA format they already support?

Such an addition can make a big difference on the RasPi platform in case BGRA is needed due to the high penalty one has to pay when converting from RGBA to BGRA. As far as I can see, the hardware encoder/decoder on the RasPi supports BGRA in addition to the selected RGBA format.

I420 may also be useful.

Best regards

Peter Maersk-Moller

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On Mo, 2016-02-22 at 16:30 +0100, Peter Maersk-Moller wrote:

> Hi Sebastian.
>
> Seems you wrote most of the omx modules.
>
> Is it possible to extend omxh264enc and omxh264dec to accept
> respectively provide BGRA in addition to the RGBA format they already
> support?
>
> Such an addition can make a big difference on the RasPi platform in
> case BGRA is needed due to the high penalty one has to pay when
> converting from RGBA to BGRA. As far as I can see, the hardware
> encoder/decoder on the RasPi supports BGRA in addition to the
> selected RGBA format.
>
> I420 may also be useful.

I420 is already supported, no?

For the other formats, adding support for them would definitely be a
good idea. Can you file a bug for this, ideally with a patch?

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Peter Maersk-Moller-2]

Hi Sebastian.

See replies further down.

On Tue, Feb 23, 2016 at 10:14 AM, Sebastian Dröge <[hidden email]> wrote:

On Mo, 2016-02-22 at 16:30 +0100, Peter Maersk-Moller wrote:
> Hi Sebastian.
> Seems you wrote most of the omx modules.
> Is it possible to extend omxh264enc and omxh264dec to accept
> respectively provide BGRA in addition to the RGBA format they already
> support?
> Such an addition can make a big difference on the RasPi platform in
> case BGRA is needed due to the high penalty one has to pay when
> converting from RGBA to BGRA. As far as I can see, the hardware
> encoder/decoder on the RasPi supports BGRA in addition to the
> selected RGBA format.
> I420 may also be useful.

I420 is already supported, no?

My mistake. So far it seems I420 is the only one that works.

Testing the following pipeline:

gst-launch-1.0 -v videotestsrc is-live=1 ! 'video/x-raw,format='$format ! omxh264enc ! fakesink

with these formats :I420 YV12 YUY2 UYVY AYUV RGBx BGRx xRGB xBGR RGBA BGRA ARGB ABGR RGB BGR Y41B Y42B YVYU Y444 v210 v216 NV12 NV21 NV16 NV61 NV24 GRAY8 GRAY16_BE GRAY16_LE v308 RGB16 BGR16 RGB15 BGR15 UYVP A420 RGB8P YUV9 YVU9 IYU1 ARGB64 AYUV64 r210 I420_10LE I420_10BE I422_10LE I422_10BE Y444_10LE Y444_10BE GBR GBR_10LE GBR_10BE NV12_64Z32 A420_10LE A420_10BE A422_10LE A422_10BE A444_10LE A444_10BE

Then only format that works is I420.

Shouldn't 'gst-inspect-1.0 omxh264enc' report that the only format it takes is I420?

The format RGB16 with the above pipeline reports :

ERROR:gstomxvideoenc.c:1072:gst_omx_video_enc_set_format: code should not be reached

while all the other formats reports:

ERROR: from element /GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0: Internal data flow error.

When using this pipeline:

gst-launch-1.0 -v videotestsrc is-live=1 ! 'video/x-raw,format='$format ! videoconvert ! omxh264enc ! fakesink

the following formats works (but gets converted to I420 by videoconvert):

I420, YV12, NV12, NV21, A420, YUV9, YVU9, I420_10LE, I420_10BE, NV12_64Z32, A420_10LE, A420_10BE,

while the rest does not producing this error:

ERROR:gstomxvideoenc.c:1072:gst_omx_video_enc_set_format: code should not be reached

 

For the other formats, adding support for them would definitely be a
good idea. Can you file a bug for this, ideally with a patch?

I can. Is the above description enough for the bug report?

For the patch, can you point to the code where you detect hardware capabilities before reporting it for the sink pad of omxh264enc ?

Is the code in omxh264enc prepared for reporting multiple sink capabilities, or just one. Not sure I know how to extend the code to report multiple capabilities. When done, I can try to look into how to make the hardware encoder accept other formats than I420 as it should be able to. Maybe it needs some of these *broadcom tunnels* for conversion that some of the encoding examples for BCM2835/2836 shows.

Best regards

Peter Maersk-Moller

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On Di, 2016-02-23 at 13:43 +0100, Peter Maersk-Moller wrote:

> 
> > I420 is already supported, no?
> My mistake. So far it seems I420 is the only one that works.
> Testing the following pipeline:
>
> gst-launch-1.0 -v videotestsrc is-live=1 ! 'video/x-
> raw,format='$format ! omxh264enc ! fakesink
>
> with these formats :I420 YV12 YUY2 UYVY AYUV RGBx BGRx xRGB xBGR RGBA
> BGRA ARGB ABGR RGB BGR Y41B Y42B YVYU Y444 v210 v216 NV12 NV21 NV16
> NV61 NV24 GRAY8 GRAY16_BE GRAY16_LE v308 RGB16 BGR16 RGB15 BGR15 UYVP
> A420 RGB8P YUV9 YVU9 IYU1 ARGB64 AYUV64 r210 I420_10LE I420_10BE
> I422_10LE I422_10BE Y444_10LE Y444_10BE GBR GBR_10LE GBR_10BE
> NV12_64Z32 A420_10LE A420_10BE A422_10LE A422_10BE A444_10LE
> A444_10BE
>
> Then only format that works is I420.
>
> Shouldn't 'gst-inspect-1.0 omxh264enc' report that the only format it
> takes is I420?

As said in the previous mail, it support I420 *and* NV12 currently.
It's just that on the RPi the encoder only supports I420 and nothing
else.

So what it should do is to report I420 and NV12 in general, and in the
RPi specific gstomx.conf you would override the pad template to only
list I420.

> > For the other formats, adding support for them would definitely be
> > a
> > good idea. Can you file a bug for this, ideally with a patch?
> I can. Is the above description enough for the bug report?
>
> For the patch, can you point to the code where you detect hardware
> capabilities before reporting it for the sink pad of omxh264enc ?
> Is the code in omxh264enc prepared for reporting multiple sink
> capabilities, or just one. Not sure I know how to extend the code to
> report multiple capabilities. When done, I can try to look into how
> to make the hardware encoder accept other formats than I420 as it
> should be able to. Maybe it needs some of these *broadcom tunnels*
> for conversion that some of the encoding examples for BCM2835/2836
> shows.

There is no hardware capability probing API for this (or it's not
implemented anywhere in a meaningful way). The code I showed you is
where the format specific handling is, the other changes would be for
the pad templates which would just be hardcoded... in code with a
default and in the gstomx.conf for the RPi.

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Peter Maersk-Moller-2]

Hi Sebastian.

Looking into the to the problem of converting and/or resize I420 or NV12 output from the omxh264dec (and others), there seem to be ways to both convert to other raw formats as well as perhaps resize the output all using the GPU.

The OpenMAX Il-component video_splitter (input port 250, output ports 251-254 - only one is needed) can take input as I420 and output as OMX_COLOR_Format32bitARGB8888 or OMX_COLOR_Format32bitBGRA8888 (one of which I believe is called a byte stream of R, G, B and A bytes (RGBA)) and possibly OMX_COLOR_Format24bitRGB888 or OMX_COLOR_Format24bitBGR888 (one of which is believed to be RGB). I'm not sure it can do both RGB(A) and BGR(A).

Anyway, it should be possible to connect the encoder's port 131 to the video_splitter's port 250, set the splitter port 250 format to I420, perhaps provide buffers for this, set the splitter port 251 to RGBA or RGB and feed it data buffers from downstream (perhaps shm) for zero-copying. An of course the video_splitter component has to be started and prepared for state chage etc.

For video resizing (and conversion) the OpenMAX image component resize can take I420 input on port 60 AND scale AND convert to image formats such as RGBA and RGB (I am 95% sure) and output these to output port 61 on data buffers from downstreams (perhaps shm).

All this of course applies to the other omxXXXdec modules.

The same should be possible for the input for omxh264enc and the other encoders in a way so the encoders can take I420/RGB/RGBA at any size and the decoders can output I420/RGB/RGBA at any size.

The "any size" may be to take it to far. There are some restrictions for either the encoder and the decoder. For one of them, the width and height must both be even numbers and for the other, both must be a multipla of 16. I just don't know if you can link OpenMAX video modules to OpenMAX image modules. At least some timing may be set and controlled out-of-band when going over an image module .... perhaps.

I tried looking to the GStreamer code, but I can't really see where to add the functionality and I am still struggling with the very poorly documented OpenMAX examples and docs.

I hope this is worth looking at. Especailly converting from I420 to RGBA for is nearly impossible on the Pi-platform for 720p and quite impossible for 1080p with the current gstreamer version, conversion alone take a lot of CPU not really present.

Does it help to open a ticket for an improvement?

Best regards

Peter Maersk-Moller

On Tue, Feb 23, 2016 at 3:18 PM, Sebastian Dröge <[hidden email]> wrote:

On Di, 2016-02-23 at 13:43 +0100, Peter Maersk-Moller wrote:
> 
> > I420 is already supported, no?
> My mistake. So far it seems I420 is the only one that works.
> Testing the following pipeline:
>
> gst-launch-1.0 -v videotestsrc is-live=1 ! 'video/x-
> raw,format='$format ! omxh264enc ! fakesink
>
> with these formats :I420 YV12 YUY2 UYVY AYUV RGBx BGRx xRGB xBGR RGBA
> BGRA ARGB ABGR RGB BGR Y41B Y42B YVYU Y444 v210 v216 NV12 NV21 NV16
> NV61 NV24 GRAY8 GRAY16_BE GRAY16_LE v308 RGB16 BGR16 RGB15 BGR15 UYVP
> A420 RGB8P YUV9 YVU9 IYU1 ARGB64 AYUV64 r210 I420_10LE I420_10BE
> I422_10LE I422_10BE Y444_10LE Y444_10BE GBR GBR_10LE GBR_10BE
> NV12_64Z32 A420_10LE A420_10BE A422_10LE A422_10BE A444_10LE
> A444_10BE
>
> Then only format that works is I420.
>
> Shouldn't 'gst-inspect-1.0 omxh264enc' report that the only format it
> takes is I420?

As said in the previous mail, it support I420 *and* NV12 currently.
It's just that on the RPi the encoder only supports I420 and nothing
else.

So what it should do is to report I420 and NV12 in general, and in the
RPi specific gstomx.conf you would override the pad template to only
list I420.

> > For the other formats, adding support for them would definitely be
> > a
> > good idea. Can you file a bug for this, ideally with a patch?
> I can. Is the above description enough for the bug report?
>
> For the patch, can you point to the code where you detect hardware
> capabilities before reporting it for the sink pad of omxh264enc ?
> Is the code in omxh264enc prepared for reporting multiple sink
> capabilities, or just one. Not sure I know how to extend the code to
> report multiple capabilities. When done, I can try to look into how
> to make the hardware encoder accept other formats than I420 as it
> should be able to. Maybe it needs some of these *broadcom tunnels*
> for conversion that some of the encoding examples for BCM2835/2836
> shows.

There is no hardware capability probing API for this (or it's not
implemented anywhere in a meaningful way). The code I showed you is
where the format specific handling is, the other changes would be for
the pad templates which would just be hardcoded... in code with a
default and in the gstomx.conf for the RPi.

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On Mo, 2016-03-14 at 05:39 +0100, Peter Maersk-Moller wrote:

> Hi Sebastian.
>
> Looking into the to the problem of converting and/or resize I420 or
> NV12 output from the omxh264dec (and others), there seem to be ways
> to both convert to other raw formats as well as perhaps resize the
> output all using the GPU.
>
> The OpenMAX Il-component video_splitter (input port 250, output ports
> 251-254 - only one is needed) can take input as I420 and output as
> OMX_COLOR_Format32bitARGB8888 or OMX_COLOR_Format32bitBGRA8888 (one
> of which I believe is called a byte stream of R, G, B and A bytes
> (RGBA)) and possibly OMX_COLOR_Format24bitRGB888 or
> OMX_COLOR_Format24bitBGR888 (one of which is believed to be RGB). I'm
> not sure it can do both RGB(A) and BGR(A).
>
> Anyway, it should be possible to connect the encoder's port 131 to
> the video_splitter's port 250, set the splitter port 250 format to
> I420, perhaps provide buffers for this, set the splitter port 251 to
> RGBA or RGB and feed it data buffers from downstream (perhaps shm)
> for zero-copying. An of course the video_splitter component has to be
> started and prepared for state chage etc.
>
> For video resizing (and conversion) the OpenMAX image component
> resize can take I420 input on port 60 AND scale AND convert to image
> formats such as RGBA and RGB (I am 95% sure) and output these to
> output port 61 on data buffers from downstreams (perhaps shm).
>
> All this of course applies to the other omxXXXdec modules.
>
> The same should be possible for the input for omxh264enc and the
> other encoders in a way so the encoders can take I420/RGB/RGBA at any
> size and the decoders can output I420/RGB/RGBA at any size.
>
> The "any size" may be to take it to far. There are some restrictions
> for either the encoder and the decoder. For one of them, the width
> and height must both be even numbers and for the other, both must be
> a multipla of 16. I just don't know if you can link OpenMAX video
> modules to OpenMAX image modules. At least some timing may be set and
> controlled out-of-band when going over an image module .... perhaps.
>
> I tried looking to the GStreamer code, but I can't really see where
> to add the functionality and I am still struggling with the very
> poorly documented OpenMAX examples and docs.
>
> I hope this is worth looking at. Especailly converting from I420 to
> RGBA for is nearly impossible on the Pi-platform for 720p and quite
> impossible for 1080p with the current gstreamer version, conversion
> alone take a lot of CPU not really present.
>
> Does it help to open a ticket for an improvement?

Here's a ticket with an (outdated) patch that integrates the resize
component via tunneling into the decoder:
https://bugzilla.gnome.org/show_bug.cgi?id=722686

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Peter Maersk-Moller-2]

Hi Sebastian.

That is exactly what is needed. So it can be done in HW.

Any remote chance this gets into a release in future not too distant future before practical realization of teleportation, hot fusion and singularity?

If not, what is the latest GStreamer version this works with and are there some instructions somehwree on how merge this with some version on gst-omx ?

I am preparing a release of Snowmix that will make simple and advanced 2D and 3D scriptable video mixing possible and practical on the Raspberry Pi 2 platform and I have all the other components necessary working pretty well and efficiently except it just can't fly unless I can find a way to convert a video stream to a RGBA frame stream without a crippling load on the CPU. The omxcolorscale, either as an independent module or buit-in within the omxXXXXdec module, is all that is needed. Same for omxXXXXenc would be nice too, but converting from RGBA to I420 in software does not have same perfomance penalty as the other way around with current implementation for Raspberry Pi.

The release of Raspberry PI 3 with 30-100% more juice depending on what is measured, just makes it all more realistic to do serious video mixing on these low cost platforms.

I'm trying to read through the code, but would have no idea on how to add/merge it to/with current gst-omx sources. Not that I can't add code, just that I don't know all the internals og Gstreamer.

Best regards

Peter Maersk-Moller

On Mon, Mar 14, 2016 at 9:06 AM, Sebastian Dröge <[hidden email]> wrote:

On Mo, 2016-03-14 at 05:39 +0100, Peter Maersk-Moller wrote:
> Hi Sebastian.
>
> Looking into the to the problem of converting and/or resize I420 or
> NV12 output from the omxh264dec (and others), there seem to be ways
> to both convert to other raw formats as well as perhaps resize the
> output all using the GPU.
>
> The OpenMAX Il-component video_splitter (input port 250, output ports
> 251-254 - only one is needed) can take input as I420 and output as
> OMX_COLOR_Format32bitARGB8888 or OMX_COLOR_Format32bitBGRA8888 (one
> of which I believe is called a byte stream of R, G, B and A bytes
> (RGBA)) and possibly OMX_COLOR_Format24bitRGB888 or
> OMX_COLOR_Format24bitBGR888 (one of which is believed to be RGB). I'm
> not sure it can do both RGB(A) and BGR(A).
>
> Anyway, it should be possible to connect the encoder's port 131 to
> the video_splitter's port 250, set the splitter port 250 format to
> I420, perhaps provide buffers for this, set the splitter port 251 to
> RGBA or RGB and feed it data buffers from downstream (perhaps shm)
> for zero-copying. An of course the video_splitter component has to be
> started and prepared for state chage etc.
>
> For video resizing (and conversion) the OpenMAX image component
> resize can take I420 input on port 60 AND scale AND convert to image
> formats such as RGBA and RGB (I am 95% sure) and output these to
> output port 61 on data buffers from downstreams (perhaps shm).
>
> All this of course applies to the other omxXXXdec modules.
>
> The same should be possible for the input for omxh264enc and the
> other encoders in a way so the encoders can take I420/RGB/RGBA at any
> size and the decoders can output I420/RGB/RGBA at any size.
>
> The "any size" may be to take it to far. There are some restrictions
> for either the encoder and the decoder. For one of them, the width
> and height must both be even numbers and for the other, both must be
> a multipla of 16. I just don't know if you can link OpenMAX video
> modules to OpenMAX image modules. At least some timing may be set and
> controlled out-of-band when going over an image module .... perhaps.
>
> I tried looking to the GStreamer code, but I can't really see where
> to add the functionality and I am still struggling with the very
> poorly documented OpenMAX examples and docs.
>
> I hope this is worth looking at. Especailly converting from I420 to
> RGBA for is nearly impossible on the Pi-platform for 720p and quite
> impossible for 1080p with the current gstreamer version, conversion
> alone take a lot of CPU not really present.
>
> Does it help to open a ticket for an improvement?

Here's a ticket with an (outdated) patch that integrates the resize
component via tunneling into the decoder:
https://bugzilla.gnome.org/show_bug.cgi?id=722686

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On Mo, 2016-03-14 at 13:18 +0100, Peter Maersk-Moller wrote:
> Hi Sebastian.
>
> That is exactly what is needed. So it can be done in HW.
>
> Any remote chance this gets into a release in future not too distant
> future before practical realization of teleportation, hot fusion and
> singularity?

Read the comments in the bug report. Unless someone updates the patch
to latest GIT master and ideally also cleans it up a bit, it's not
going to be merged.

> If not, what is the latest GStreamer version this works with and are
> there some instructions somehwree on how merge this with some version
> on gst-omx ?

It probably applies to the version that was current when the patch was
written :)

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-4]

Le lundi 14 mars 2016 à 13:18 +0100, Peter Maersk-Moller a écrit :
> If not, what is the latest GStreamer version this works with and are
> there some instructions somehwree on how merge this with some version
> on gst-omx ?

There was another proposition, which I found more promising.

https://bugzilla.gnome.org/show_bug.cgi?id=726325

It proposes to figure-out a way, using allocation query, to be able to
tunnel between two GstElement. This way, we could integrated all the
specialized HW into their own GstElement, using proper baseclass, and
gain in performance by avoiding the GPU/CPU roundtrip the is so
expensive on RPi. The resize element hack is from the RPi Foundation,
and is supposedly integrated into Raspbian distribution already. This
is how they managed to get "decent" video playback in Epiphany/WebKit
without enabling any GL Texture acceleration.

cheers,
Nicolas
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel