Limits of the Gst-omx OpenMAX IL H.264 encoder on the Raspberry Pi 3?

[Matthew Thyer]

Dear GStreamer developers,

 

I'm encoding H.264 video for the GStreamer RTSP server using the hardware accelerated OpenMAX IL encoder (omxh264enc).

My video source is currently the Raspberry Pi V2 camera module (ref: https://www.raspberrypi.org/products/camera-module-v2/) but I’d like to support the V1 module as well (a 5 MP OmniVision OV5647 sensor as opposed to the 8 MP Sony IMX219 sensor of the V2 module).

 

I've built GStreamer 1.14.1 from source and correctly targeted the Raspberry Pi when building the gst-omx module and things are working but I now need to quantify the limits of the application system.

 

A post (https://www.raspberrypi.org/forums/viewtopic.php?t=6852&start=200#p427960) on the Raspberry Pi forums (in thread "gstreamer openmax") has shown me a GStreamer pipeline that works (with the capability filters I need to specify camera resolution, framerate, encoding profile etc) and my code is working with the following pipeline with gpu_mem=128 (in /boot/config.txt):

"v4l2src ! capsfilter caps=video/x-raw,width=640,height=480,framerate=30/1 ! omxh264enc target-bitrate=1000000 control-rate=variable ! capsfilter caps=video/x-h264,profile=high ! rtph264pay name=pay0 pt=96"

 

I'm now wanting to understand the following issues:

• The available resolutions for my video source
• The available frame rates for my video source
• How to intelligently choose the target-bitrate parameter of the OpenMax H.264 encoder
• The meaning of the control-rate parameter of the OpenMax H.264 encoder
• The H.264 encoding profiles that are supported by the OpenMax H.264 encoder and in what scenarios I would want to use each of them

Additionally, I need to understand whether there are particular hardware platform configurations required to support different configurations (e.g. gpu_mem in /boot/config.txt).

 

Ideally, I'd like to be able to programmatically assess whether a given configuration is viable.

It would also be great if I could query the camera for its capabilities if this is possible.

 

Does anyone have any pointers to assist with my problem?

 

---

Matthew Thyer

[phone]

4.38 Ingkarni Wardli

Adelaide University

 

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[James Cameron]

On Mon, Jul 02, 2018 at 08:37:56AM +0000, Matthew Thyer wrote:
> [...]
> I'm now wanting to understand the following issues:
>
>   ● The available resolutions for my video source

G'day Matthew,

For the ov5647, a good information source is the sensor datasheet,
which shows the registers can be programmed for almost any image
dimensions up to the resolution of the sensor array, 2592x1944.

https://cdn.sparkfun.com/datasheets/Dev/RaspberryPi/ov5647_full.pdf

This may be bandwidth constrained by the cable, the bus, the bus
controller or the host processor.

Another source of information is the kernel driver, both in source
code and exposed interfaces.  It is common for a kernel driver to only
implement popular resolutions, despite what the sensor can do.

>   ● The available frame rates for my video source

For the ov5647 sensor itself, there's a list of maximum frame rates by
resolution; 2592x1944@15, 1080p@30, 960p@45, 720p@60, 640x480@90, and
320x240@120.

However, for a dark scene the automatic exposure control (when
configured) may force an integration time that is beyond the time
between frames at your preferred rate, causing the frame to be
delayed.

If later parts of your pipeline cannot cope with variable frame rate,
add the videorate element.

>   ● How to intelligently choose the target-bitrate parameter of the OpenMax
>     H.264 encoder

I've no comment on the encoder; I just wanted to dump my brain for you
on the OmniVision sensor IP.  ;-)

>   ● The meaning of the control-rate parameter of the OpenMax H.264 encoder
>   ● The H.264 encoding profiles that are supported by the OpenMax H.264 encoder
>     and in what scenarios I would want to use each of them
>
> Additionally, I need to understand whether there are particular hardware
> platform configurations required to support different configurations (e.g.
> gpu_mem in /boot/config.txt).
>
> Ideally, I'd like to be able to programmatically assess whether a given
> configuration is viable.
>
> It would also be great if I could query the camera for its capabilities if this
> is possible.

You could read the camera registers, I guess.  I've not tried it on
this sensor.  But reading the v4l2 layer interfaces would be handy to
assess the driver implementation.

> Does anyone have any pointers to assist with my problem?
>
> ---
>
> Matthew Thyer
>
> [phone]
>
> 4.38 Ingkarni Wardli
>
> Adelaide University
>

--
James Cameron
http://quozl.netrek.org/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Tim Müller]

On Mon, 2018-07-02 at 08:37 +0000, Matthew Thyer wrote:

Hi Matthew,

Just in case you weren't aware of it, there's also an rpicamsrc which
outputs H.264 from the rpi camera directly:

 https://github.com/thaytan/gst-rpicamsrc

Cheers
-Tim

--
Tim Müller, Centricular Ltd - http://www.centricular.com

Join us at the GStreamer Conference! 25-26 October 2018 in Edinburgh,
Scotland
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Matthew Thyer]

I am aware of gst-rpicamsrc however I'd prefer to avoid bringing in another component that is not maintained by the GStreamer team.
The command "v4l2-ctl --list-formats" seems to imply that the driver used for v4l2 for the Raspberry Pi camera module V2 can do H.264 encoding itself as 'H264' (compressed) is listed as one of the pixel formats.
I think I need to look more closely at the element " v4l2src" to see what control I have over selection of the pixel format.
Inspecting the element shows that there's support for the capability "video/x-h264" but I'm not sure as to whether I can get as much control over the H.264 encoding as I can with omxh264enc.

-----Original Message-----
From: gstreamer-devel <[hidden email]> On Behalf Of Tim Müller
Sent: Tuesday, 3 July 2018 8:35 AM
To: [hidden email]
Subject: Re: Limits of the Gst-omx OpenMAX IL H.264 encoder on the Raspberry Pi 3?

On Mon, 2018-07-02 at 08:37 +0000, Matthew Thyer wrote:

Hi Matthew,

Just in case you weren't aware of it, there's also an rpicamsrc which outputs H.264 from the rpi camera directly:

 https://github.com/thaytan/gst-rpicamsrc

Cheers
-Tim

--
Tim Müller, Centricular Ltd - http://www.centricular.com

Join us at the GStreamer Conference! 25-26 October 2018 in Edinburgh, Scotland _______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel