Readback OpenGL texture from omxh264dec

[Rafael Savignon]

Hi all,

I'm developing a Qt5 application to playback hardware decoded h.264 files. But on raspberry pi 3 its struggling to play media greater than 720p. After some investigation of QGstVideoBuffer class I realized that calls to gst_video_frame_map function passing bigger frames were consuming much cpu time ~50ms, which was resulting in the postponing of next frames delivery. So I had to find out a more performatic way to read the video frame content. After some search I found out that a special buffer mechanism could enable applications to map GPU memory directly to addressable process memory, avoiding unnecessary copy (VCSM). Then I took another look at QtMultimedia source code and saw that a OpenGL texture handle was being packaged in  with QGstVideoBuffer, as shown below.

guint textureId = gst_gl_memory_get_texture_id(glmem);

videoBuffer = new QGstVideoBuffer(buffer, m_videoInfo, m_format.handleType(), textureId); 

So I thought that things were starting to get clear. Thus I decided to use vcsm to map the texture and speed things up .

First, on my QAbstractVideoSurface::start implemented method I initialized vcsm and created a EGLimageKHR.

int w = Util::nextPOT(size.width());

int h = Util::nextPOT(size.height());

const EGLint attrib[] = { EGL_IMAGE_PRESERVED_KHR, EGL_TRUE, EGL_NONE, EGL_NONE };

vcsm_info.width = w;

vcsm_info.height = h;

vcsm_init();

eglFbImage = eglCreateImageKHR(eglGetCurrentDisplay(), EGL_NO_CONTEXT, EGL_IMAGE_BRCM_VCSM, &vcsm_info, attrib); 

Second, on my QAbstractVideoSurface::present  implementation method I tried to grab the passed Texture content, but it didn't work. Despite this, all OpenGL calls succeeded. I can access the mapped buffer, but it doesn't contain anything meaningful.

QOpenGLFunctions* f = ctx->functions(); 

GLuint framebuffer; 

GLuint depthRenderbuffer; 

GLint prevFbo;

GLenum status = GL_FRAMEBUFFER_COMPLETE;

GLuint texture = static_cast<GLuint>( currentFrame.handle().toInt() );

int texWidth = Util::nextPOT(currentFrame.width());

int texHeight = Util::nextPOT(currentFrame.height());

  

GLCHK(f->glGetIntegerv( GL_FRAMEBUFFER_BINDING, &prevFbo )); 

GLCHK(f->glGenFramebuffers(1, &framebuffer));

GLCHK(f->glBindFramebuffer(GL_FRAMEBUFFER, framebuffer));

GLCHK(glActiveTexture(GL_TEXTURE0));

GLCHK(glBindTexture(GL_TEXTURE_2D, texture));

GLCHK(glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST));

GLCHK(glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST)); 

GLCHK(glEGLImageTargetTexture2DOES(GL_TEXTURE_2D, eglFbImage)); 

GLCHK(f->glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, texture, 0)); 

GLCHK(glBindTexture(GL_TEXTURE_2D, 0)); 

status = glCheckFramebufferStatus(GL_FRAMEBUFFER);

// check fbo status 

GLCHK(f->glFinish()); 

uint8_t *vcsmBuffer; 

VCSM_CACHE_TYPE_T cacheType; 

vcsmBuffer = (uint8_t*)vcsm_lock_cache(vcsm_info.vcsm_handle, VCSM_CACHE_TYPE_HOST, &cacheType);

// print buffer

vcsm_unlock_ptr(vcsmBuffer); 

Is the Texture Id correctly filled by omxh264dec or glupload ?

Can i bind the texture id to a different fbo ?

Why gst_video_frame_map correctly map the video frame content and vcsm do not despite being addressable?

If someone could give a tip on this I would be thankful.

regards.

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Matthew Waters]

The OpenGL texture out of omxh264dec on the RPi is wrapped in a GstGLMemoryEGL which already wraps the EGLImage produced by the OMX decoder.  I'm not entirely sure what VCSM is or how it actually relates to all this but I assume that the following pipeline works for you? 'omxh264dec ! glimagesinkelement'  If so, that is already using the EGLImage/OpenGL texture produced by omxh264dec and rendering using OpenGL with glimagesink.  It is very hard in general for the RPi to do anything but only decode and display a 1080p@30 video and even then it can just barely do it.

On 21/6/20 10:22 pm, Rafael Savignon wrote:

Hi all,

I'm developing a Qt5 application to playback hardware decoded h.264 files. But on raspberry pi 3 its struggling to play media greater than 720p. After some investigation of QGstVideoBuffer class I realized that calls to gst_video_frame_map function passing bigger frames were consuming much cpu time ~50ms, which was resulting in the postponing of next frames delivery. So I had to find out a more performatic way to read the video frame content. After some search I found out that a special buffer mechanism could enable applications to map GPU memory directly to addressable process memory, avoiding unnecessary copy (VCSM). Then I took another look at QtMultimedia source code and saw that a OpenGL texture handle was being packaged in  with QGstVideoBuffer, as shown below.

guint textureId = gst_gl_memory_get_texture_id(glmem);

videoBuffer = new QGstVideoBuffer(buffer, m_videoInfo, m_format.handleType(), textureId); 

So I thought that things were starting to get clear. Thus I decided to use vcsm to map the texture and speed things up .

First, on my QAbstractVideoSurface::start implemented method I initialized vcsm and created a EGLimageKHR.

int w = Util::nextPOT(size.width());

int h = Util::nextPOT(size.height());

const EGLint attrib[] = { EGL_IMAGE_PRESERVED_KHR, EGL_TRUE, EGL_NONE, EGL_NONE };

vcsm_info.width = w;

vcsm_info.height = h;

vcsm_init();

eglFbImage = eglCreateImageKHR(eglGetCurrentDisplay(), EGL_NO_CONTEXT, EGL_IMAGE_BRCM_VCSM, &vcsm_info, attrib); 

Second, on my QAbstractVideoSurface::present  implementation method I tried to grab the passed Texture content, but it didn't work. Despite this, all OpenGL calls succeeded. I can access the mapped buffer, but it doesn't contain anything meaningful.

QOpenGLFunctions* f = ctx->functions(); 

GLuint framebuffer; 

GLuint depthRenderbuffer; 

GLint prevFbo;

GLenum status = GL_FRAMEBUFFER_COMPLETE;

GLuint texture = static_cast<GLuint>( currentFrame.handle().toInt() );

int texWidth = Util::nextPOT(currentFrame.width());

int texHeight = Util::nextPOT(currentFrame.height());

  

GLCHK(f->glGetIntegerv( GL_FRAMEBUFFER_BINDING, &prevFbo )); 

GLCHK(f->glGenFramebuffers(1, &framebuffer));

GLCHK(f->glBindFramebuffer(GL_FRAMEBUFFER, framebuffer));

GLCHK(glActiveTexture(GL_TEXTURE0));

GLCHK(glBindTexture(GL_TEXTURE_2D, texture));

GLCHK(glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST));

GLCHK(glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST)); 

GLCHK(glEGLImageTargetTexture2DOES(GL_TEXTURE_2D, eglFbImage)); 

GLCHK(f->glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, texture, 0)); 

GLCHK(glBindTexture(GL_TEXTURE_2D, 0)); 

status = glCheckFramebufferStatus(GL_FRAMEBUFFER);

// check fbo status 

GLCHK(f->glFinish()); 

uint8_t *vcsmBuffer; 

VCSM_CACHE_TYPE_T cacheType; 

vcsmBuffer = (uint8_t*)vcsm_lock_cache(vcsm_info.vcsm_handle, VCSM_CACHE_TYPE_HOST, &cacheType);

// print buffer

vcsm_unlock_ptr(vcsmBuffer); 

Is the Texture Id correctly filled by omxh264dec or glupload ?

Can i bind the texture id to a different fbo ?

Why gst_video_frame_map correctly map the video frame content and vcsm do not despite being addressable?

If someone could give a tip on this I would be thankful.

regards.

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Rafael Savignon]

Thanks Matthew,

to confirm it i did a little patch to check if the passed GstMemory was indeed a GstGLMemoryEGL, but apparently it is not, the below assert failed right after  the qt video sink received a new frame.

GstMemory *mem = gst_buffer_peek_memory (buffer, 0);

g_assert (gst_is_gl_memory_egl (mem));

As you suggested, I tried to execute the pipeline 'omxh264dec ! glimagesinkelement', but it ends up giving me an error when calling glGetString(GL_VERSION).

$ gst-launch-1.0 filesrc location=/home/root/test.mp4 ! omxh264dec ! glimagesinkelement

  glcontext gstglcontext_egl.c:464:gst_gl_context_egl_choose_config:<glcontextegl0> chosen EGLConfig                                                                                                  
  glcontext gstglcontext_egl.c:174:gst_gl_context_egl_dump_config:<glcontextegl0> dumping EGLConfig 0x4 with id 0x4 and native visual id 0x8428 of type 0x3038                                        
  glcontext gstglcontext_egl.c:207:gst_gl_context_egl_dump_config:<glcontextegl0> Conformant for OpenGL ES|OpenGL ES 2.x|OpenVG                                                                      
  glcontext gstglcontext_egl.c:240:gst_gl_context_egl_dump_config:<glcontextegl0> Renderable for OpenGL ES|OpenGL ES 2.x|OpenVG                                                                      
  glcontext gstglcontext_egl.c:273:gst_gl_context_egl_dump_config:<glcontextegl0> Surface for window|pbuffer|multisample-resolve-box|swap-behaviour-preserved|vg-alpha-format-pre|vg-colorspace-linear
  glcontext gstglcontext_egl.c:313:gst_gl_context_egl_dump_config:<glcontextegl0> [R, G, B, A] = [8, 8, 8, 0]                                                                                        
  glcontext gstglcontext_egl.c:335:gst_gl_context_egl_dump_config:<glcontextegl0> [D, S] = [24, 0]                                                                                                    
  glcontext gstglcontext_egl.c:346:gst_gl_context_egl_dump_config:<glcontextegl0> Swap interval range is [0, 2147483647]                                                                              
  glcontext gstglcontext_egl.c:360:gst_gl_context_egl_dump_config:<glcontextegl0> PBuffer maximum dimensions are [2048, 2048]. Max pixels are 4194304                                                
  glcontext gstglcontext_egl.c:373:gst_gl_context_egl_dump_config:<glcontextegl0> Multisample buffers: 0 and Samples per pixel: 0                                                                    
  glcontext gstglcontext_egl.c:501:_create_context_with_flags:<glcontextegl0> attempting to create OpenGL ES context version 2.0 flags 0 profile 0                                                    
  glcontext gstglcontext_egl.c:746:gst_gl_context_egl_create_context: gl context created: 1                                                                                                          
  glwindow gstglwindow_dispmanx_egl.c:226:window_resize: resizing invisible window from 0x0 to 16x16                                                                                                
  videosink gstvideosink.c:132:gst_video_sink_center_rect: source is 16x16 dest is 640x480, result is 16x16 with x,y 312x232                                                                        
  glcontext gstglcontext_egl.c:794:gst_gl_context_egl_create_context: Creating EGLSurface from window_handle 0x5fd100                                                                                
  glcontext gstglcontext_egl.c:828:gst_gl_context_egl_create_context: surface created                                                                                                                
  glcontext gstglcontext.c:1247:gst_gl_context_create_thread:<glcontextegl0> created context                                                                                                          
  glcontext gstglcontext.c:750:gst_gl_context_activate:<glcontextegl0> activate:1                                                                                                                    
  glcontext gstglcontext.c:1263:gst_gl_context_create_thread:<glcontextegl0> available GL APIs: gles2                                                                                                
  glcontext gstglcontext.c:1282:gst_gl_context_create_thread:<glcontextegl0> Filling info                                                                                                            
  glcontext gstglcontext.c:1042:gst_gl_context_create:<glcontextegl0> gl thread created                                                                                                              
  glimagesink gstglimagesink.c:1010:_ensure_gl_setup:<glimagesink0> error: glGetString not defined or returned invalid value

After some searching I find out that this problem could be related with the wrong GL lib loading. So I certified that gst-launch-1.0 was loading the correct glGetString version.

My system listed two libraries with the symbol:

$ find . -name \*.so -exec bash -c "nm --defined-only -D {} 2>/dev/null | grep glGetString && echo {}" \;
00008810 T glGetString
./libbrcmGLESv2.so
0000880c T glGetString

./libGLESv2.so

 

According to rpi doc, the corrected one is the brcm. Thus, I ran gdb to check which one is being loaded right before  glGetString(GL_VERSION) was called. gst_gl_display_create_context was chosen to the the point of verification.

 

$ gdb gst-launch-1.0

Thread 1 "gst-launch-1.0" hit Breakpoint 2, 0x7693fa22 in gst_gl_display_create_context () from /usr/lib/libgstgl-1.0.so.0
(gdb) info sharedlibrary
From        To          Syms Read   Shared Object Library
0x76fd5b80  0x76fe96b0  Yes (*)     /lib/ld-linux-armhf.so.3
0x76f0d9d8  0x76f94684  Yes (*)     /usr/lib/libgstreamer-1.0.so.0
0x76e1d7e8  0x76e7850c  Yes (*)     /usr/lib/libglib-2.0.so.0
0x76dc2c40  0x76de9810  Yes (*)     /usr/lib/libgobject-2.0.so.0
0x76d99540  0x76da4b94  Yes (*)     /lib/libpthread.so.0
0x76cbba40  0x76d61ac4  Yes (*)     /lib/libc.so.6
0x76c8dbb0  0x76c8e6dc  Yes (*)     /usr/lib/libgmodule-2.0.so.0
0x76c32148  0x76c59c60  Yes (*)     /lib/libm.so.6
0x76c16b38  0x76c17518  Yes (*)     /lib/libdl.so.2
0x76bb8f48  0x76befd8c  Yes (*)     /usr/lib/libpcre.so.1
0x76ba3360  0x76ba6680  Yes (*)     /usr/lib/libffi.so.7
0x76b85118  0x76b8ff10  Yes (*)     /lib/libgcc_s.so.1
0x76b28520  0x76b59774  Yes (*)     /usr/lib/gstreamer-1.0/libgstcoreelements.so
0x76ac55e8  0x76b012a4  Yes (*)     /usr/lib/libgstbase-1.0.so.0
0x76a77e08  0x76a9ce4c  Yes (*)     /usr/lib/gstreamer-1.0/libgstomx.so
0x769fedd0  0x76a4623c  Yes (*)     /usr/lib/libgstvideo-1.0.so.0
0x76998058  0x769ce23c  Yes (*)     /usr/lib/libgstaudio-1.0.so.0
0x7697ad60  0x7697b910  Yes (*)     /usr/lib/libgstallocators-1.0.so.0
0x769355a0  0x76957ae8  Yes (*)     /usr/lib/libgstgl-1.0.so.0
0x768ccfe8  0x7690893c  Yes (*)     /usr/lib/liborc-0.4.so.0
0x76892618  0x768a9e14  Yes (*)     /usr/lib/libgsttag-1.0.so.0
0x76878a0c  0x76879ed4  Yes (*)     /usr/lib/libbcm_host.so
0x7685114c  0x768642dc  Yes (*)     /usr/lib/libbrcmEGL.so
0x76829928  0x76831dc4  Yes (*)     /usr/lib/libvchostif.so
0x76806dc8  0x76812f2c  Yes (*)     /usr/lib/libbrcmGLESv2.so
0x767e3bc8  0x767ecc88  Yes (*)     /lib/libz.so.1
0x767cd704  0x767cfb30  Yes (*)     /usr/lib/libvchiq_arm.so
0x767b6c4c  0x767b9e28  Yes (*)     /usr/lib/libvcos.so
0x7679fae0  0x767a1f30  Yes (*)     /lib/librt.so.1
0x76764228  0x7677d648  Yes (*)     /usr/lib/gstreamer-1.0/libgstopengl.so
0x76742240  0x76748fb4  Yes (*)     /usr/lib/libgstcontroller-1.0.so.0
0x767135b0  0x7672a9f0  Yes (*)     /usr/lib/libpng16.so.16
0x766d89e0  0x766fb670  Yes (*)     /usr/lib/libjpeg.so.62

It's picking the right lib (libbrcmGLESv2.so) so glGetString is returning a invalid value.

Finally, to validate i coded a simple sample to call glGetString, which surprisingly returned : OpenGL ES 2.0

What could be wrong with glimagesinkelement ?

regards.  

On Sun, Jun 21, 2020 at 11:39 AM Matthew Waters <[hidden email]> wrote:

The OpenGL texture out of omxh264dec on the RPi is wrapped in a GstGLMemoryEGL which already wraps the EGLImage produced by the OMX decoder.  I'm not entirely sure what VCSM is or how it actually relates to all this but I assume that the following pipeline works for you? 'omxh264dec ! glimagesinkelement'  If so, that is already using the EGLImage/OpenGL texture produced by omxh264dec and rendering using OpenGL with glimagesink.  It is very hard in general for the RPi to do anything but only decode and display a 1080p@30 video and even then it can just barely do it.

On 21/6/20 10:22 pm, Rafael Savignon wrote:

Hi all,

I'm developing a Qt5 application to playback hardware decoded h.264 files. But on raspberry pi 3 its struggling to play media greater than 720p. After some investigation of QGstVideoBuffer class I realized that calls to gst_video_frame_map function passing bigger frames were consuming much cpu time ~50ms, which was resulting in the postponing of next frames delivery. So I had to find out a more performatic way to read the video frame content. After some search I found out that a special buffer mechanism could enable applications to map GPU memory directly to addressable process memory, avoiding unnecessary copy (VCSM). Then I took another look at QtMultimedia source code and saw that a OpenGL texture handle was being packaged in  with QGstVideoBuffer, as shown below.

guint textureId = gst_gl_memory_get_texture_id(glmem);

videoBuffer = new QGstVideoBuffer(buffer, m_videoInfo, m_format.handleType(), textureId); 

So I thought that things were starting to get clear. Thus I decided to use vcsm to map the texture and speed things up .

First, on my QAbstractVideoSurface::start implemented method I initialized vcsm and created a EGLimageKHR.

int w = Util::nextPOT(size.width());

int h = Util::nextPOT(size.height());

const EGLint attrib[] = { EGL_IMAGE_PRESERVED_KHR, EGL_TRUE, EGL_NONE, EGL_NONE };

vcsm_info.width = w;

vcsm_info.height = h;

vcsm_init();

eglFbImage = eglCreateImageKHR(eglGetCurrentDisplay(), EGL_NO_CONTEXT, EGL_IMAGE_BRCM_VCSM, &vcsm_info, attrib); 

Second, on my QAbstractVideoSurface::present  implementation method I tried to grab the passed Texture content, but it didn't work. Despite this, all OpenGL calls succeeded. I can access the mapped buffer, but it doesn't contain anything meaningful.

QOpenGLFunctions* f = ctx->functions(); 

GLuint framebuffer; 

GLuint depthRenderbuffer; 

GLint prevFbo;

GLenum status = GL_FRAMEBUFFER_COMPLETE;

GLuint texture = static_cast<GLuint>( currentFrame.handle().toInt() );

int texWidth = Util::nextPOT(currentFrame.width());

int texHeight = Util::nextPOT(currentFrame.height());

  

GLCHK(f->glGetIntegerv( GL_FRAMEBUFFER_BINDING, &prevFbo )); 

GLCHK(f->glGenFramebuffers(1, &framebuffer));

GLCHK(f->glBindFramebuffer(GL_FRAMEBUFFER, framebuffer));

GLCHK(glActiveTexture(GL_TEXTURE0));

GLCHK(glBindTexture(GL_TEXTURE_2D, texture));

GLCHK(glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST));

GLCHK(glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST)); 

GLCHK(glEGLImageTargetTexture2DOES(GL_TEXTURE_2D, eglFbImage)); 

GLCHK(f->glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, texture, 0)); 

GLCHK(glBindTexture(GL_TEXTURE_2D, 0)); 

status = glCheckFramebufferStatus(GL_FRAMEBUFFER);

// check fbo status 

GLCHK(f->glFinish()); 

uint8_t *vcsmBuffer; 

VCSM_CACHE_TYPE_T cacheType; 

vcsmBuffer = (uint8_t*)vcsm_lock_cache(vcsm_info.vcsm_handle, VCSM_CACHE_TYPE_HOST, &cacheType);

// print buffer

vcsm_unlock_ptr(vcsmBuffer); 

Is the Texture Id correctly filled by omxh264dec or glupload ?

Can i bind the texture id to a different fbo ?

Why gst_video_frame_map correctly map the video frame content and vcsm do not despite being addressable?

If someone could give a tip on this I would be thankful.

regards.

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Matthew Waters]

Hi,

So, GStreamer also attempts to g_module_open() and g_module_symbol() all OpenGL functions from a specific library as not all OpenGL functions are exported from all OpenGL libraries.  That library name is by default, libGLES2v2 as outlined here: https://gitlab.freedesktop.org/gstreamer/gst-plugins-base/-/blob/master/gst-libs/gst/gl/gstglcontext.c#L111 but can be configured with the meson build option: 'gles2_module_name'.  The same thing occurs for libEGL and can be overridden by the meson build option 'egl_module_name'.

Hope the helps

Cheers
-Matt

On 24/6/20 1:12 pm, Rafael Savignon wrote:

Thanks Matthew,

to confirm it i did a little patch to check if the passed GstMemory was indeed a GstGLMemoryEGL, but apparently it is not, the below assert failed right after  the qt video sink received a new frame.

GstMemory *mem = gst_buffer_peek_memory (buffer, 0);

g_assert (gst_is_gl_memory_egl (mem));

As you suggested, I tried to execute the pipeline 'omxh264dec ! glimagesinkelement', but it ends up giving me an error when calling glGetString(GL_VERSION).

$ gst-launch-1.0 filesrc location=/home/root/test.mp4 ! omxh264dec ! glimagesinkelement

  glcontext gstglcontext_egl.c:464:gst_gl_context_egl_choose_config:<glcontextegl0> chosen EGLConfig                                                                                                  
  glcontext gstglcontext_egl.c:174:gst_gl_context_egl_dump_config:<glcontextegl0> dumping EGLConfig 0x4 with id 0x4 and native visual id 0x8428 of type 0x3038                                        
  glcontext gstglcontext_egl.c:207:gst_gl_context_egl_dump_config:<glcontextegl0> Conformant for OpenGL ES|OpenGL ES 2.x|OpenVG                                                                      
  glcontext gstglcontext_egl.c:240:gst_gl_context_egl_dump_config:<glcontextegl0> Renderable for OpenGL ES|OpenGL ES 2.x|OpenVG                                                                      
  glcontext gstglcontext_egl.c:273:gst_gl_context_egl_dump_config:<glcontextegl0> Surface for window|pbuffer|multisample-resolve-box|swap-behaviour-preserved|vg-alpha-format-pre|vg-colorspace-linear
  glcontext gstglcontext_egl.c:313:gst_gl_context_egl_dump_config:<glcontextegl0> [R, G, B, A] = [8, 8, 8, 0]                                                                                        
  glcontext gstglcontext_egl.c:335:gst_gl_context_egl_dump_config:<glcontextegl0> [D, S] = [24, 0]                                                                                                    
  glcontext gstglcontext_egl.c:346:gst_gl_context_egl_dump_config:<glcontextegl0> Swap interval range is [0, 2147483647]                                                                              
  glcontext gstglcontext_egl.c:360:gst_gl_context_egl_dump_config:<glcontextegl0> PBuffer maximum dimensions are [2048, 2048]. Max pixels are 4194304                                                
  glcontext gstglcontext_egl.c:373:gst_gl_context_egl_dump_config:<glcontextegl0> Multisample buffers: 0 and Samples per pixel: 0                                                                    
  glcontext gstglcontext_egl.c:501:_create_context_with_flags:<glcontextegl0> attempting to create OpenGL ES context version 2.0 flags 0 profile 0                                                    
  glcontext gstglcontext_egl.c:746:gst_gl_context_egl_create_context: gl context created: 1                                                                                                          
  glwindow gstglwindow_dispmanx_egl.c:226:window_resize: resizing invisible window from 0x0 to 16x16                                                                                                
  videosink gstvideosink.c:132:gst_video_sink_center_rect: source is 16x16 dest is 640x480, result is 16x16 with x,y 312x232                                                                        
  glcontext gstglcontext_egl.c:794:gst_gl_context_egl_create_context: Creating EGLSurface from window_handle 0x5fd100                                                                                
  glcontext gstglcontext_egl.c:828:gst_gl_context_egl_create_context: surface created                                                                                                                
  glcontext gstglcontext.c:1247:gst_gl_context_create_thread:<glcontextegl0> created context                                                                                                          
  glcontext gstglcontext.c:750:gst_gl_context_activate:<glcontextegl0> activate:1                                                                                                                    
  glcontext gstglcontext.c:1263:gst_gl_context_create_thread:<glcontextegl0> available GL APIs: gles2                                                                                                
  glcontext gstglcontext.c:1282:gst_gl_context_create_thread:<glcontextegl0> Filling info                                                                                                            
  glcontext gstglcontext.c:1042:gst_gl_context_create:<glcontextegl0> gl thread created                                                                                                              
  glimagesink gstglimagesink.c:1010:_ensure_gl_setup:<glimagesink0> error: glGetString not defined or returned invalid value

After some searching I find out that this problem could be related with the wrong GL lib loading. So I certified that gst-launch-1.0 was loading the correct glGetString version.

My system listed two libraries with the symbol:

$ find . -name \*.so -exec bash -c "nm --defined-only -D {} 2>/dev/null | grep glGetString && echo {}" \;
00008810 T glGetString
./libbrcmGLESv2.so
0000880c T glGetString

./libGLESv2.so

 

According to rpi doc, the corrected one is the brcm. Thus, I ran gdb to check which one is being loaded right before  glGetString(GL_VERSION) was called. gst_gl_display_create_context was chosen to the the point of verification.

 

$ gdb gst-launch-1.0

Thread 1 "gst-launch-1.0" hit Breakpoint 2, 0x7693fa22 in gst_gl_display_create_context () from /usr/lib/libgstgl-1.0.so.0
(gdb) info sharedlibrary
From        To          Syms Read   Shared Object Library
0x76fd5b80  0x76fe96b0  Yes (*)     /lib/ld-linux-armhf.so.3
0x76f0d9d8  0x76f94684  Yes (*)     /usr/lib/libgstreamer-1.0.so.0
0x76e1d7e8  0x76e7850c  Yes (*)     /usr/lib/libglib-2.0.so.0
0x76dc2c40  0x76de9810  Yes (*)     /usr/lib/libgobject-2.0.so.0
0x76d99540  0x76da4b94  Yes (*)     /lib/libpthread.so.0
0x76cbba40  0x76d61ac4  Yes (*)     /lib/libc.so.6
0x76c8dbb0  0x76c8e6dc  Yes (*)     /usr/lib/libgmodule-2.0.so.0
0x76c32148  0x76c59c60  Yes (*)     /lib/libm.so.6
0x76c16b38  0x76c17518  Yes (*)     /lib/libdl.so.2
0x76bb8f48  0x76befd8c  Yes (*)     /usr/lib/libpcre.so.1
0x76ba3360  0x76ba6680  Yes (*)     /usr/lib/libffi.so.7
0x76b85118  0x76b8ff10  Yes (*)     /lib/libgcc_s.so.1
0x76b28520  0x76b59774  Yes (*)     /usr/lib/gstreamer-1.0/libgstcoreelements.so
0x76ac55e8  0x76b012a4  Yes (*)     /usr/lib/libgstbase-1.0.so.0
0x76a77e08  0x76a9ce4c  Yes (*)     /usr/lib/gstreamer-1.0/libgstomx.so
0x769fedd0  0x76a4623c  Yes (*)     /usr/lib/libgstvideo-1.0.so.0
0x76998058  0x769ce23c  Yes (*)     /usr/lib/libgstaudio-1.0.so.0
0x7697ad60  0x7697b910  Yes (*)     /usr/lib/libgstallocators-1.0.so.0
0x769355a0  0x76957ae8  Yes (*)     /usr/lib/libgstgl-1.0.so.0
0x768ccfe8  0x7690893c  Yes (*)     /usr/lib/liborc-0.4.so.0
0x76892618  0x768a9e14  Yes (*)     /usr/lib/libgsttag-1.0.so.0
0x76878a0c  0x76879ed4  Yes (*)     /usr/lib/libbcm_host.so
0x7685114c  0x768642dc  Yes (*)     /usr/lib/libbrcmEGL.so
0x76829928  0x76831dc4  Yes (*)     /usr/lib/libvchostif.so
0x76806dc8  0x76812f2c  Yes (*)     /usr/lib/libbrcmGLESv2.so
0x767e3bc8  0x767ecc88  Yes (*)     /lib/libz.so.1
0x767cd704  0x767cfb30  Yes (*)     /usr/lib/libvchiq_arm.so
0x767b6c4c  0x767b9e28  Yes (*)     /usr/lib/libvcos.so
0x7679fae0  0x767a1f30  Yes (*)     /lib/librt.so.1
0x76764228  0x7677d648  Yes (*)     /usr/lib/gstreamer-1.0/libgstopengl.so
0x76742240  0x76748fb4  Yes (*)     /usr/lib/libgstcontroller-1.0.so.0
0x767135b0  0x7672a9f0  Yes (*)     /usr/lib/libpng16.so.16
0x766d89e0  0x766fb670  Yes (*)     /usr/lib/libjpeg.so.62

It's picking the right lib (libbrcmGLESv2.so) so glGetString is returning a invalid value.

Finally, to validate i coded a simple sample to call glGetString, which surprisingly returned : OpenGL ES 2.0

What could be wrong with glimagesinkelement ?

regards.  

On Sun, Jun 21, 2020 at 11:39 AM Matthew Waters <[hidden email]> wrote:

The OpenGL texture out of omxh264dec on the RPi is wrapped in a GstGLMemoryEGL which already wraps the EGLImage produced by the OMX decoder.  I'm not entirely sure what VCSM is or how it actually relates to all this but I assume that the following pipeline works for you? 'omxh264dec ! glimagesinkelement'  If so, that is already using the EGLImage/OpenGL texture produced by omxh264dec and rendering using OpenGL with glimagesink.  It is very hard in general for the RPi to do anything but only decode and display a 1080p@30 video and even then it can just barely do it.

On 21/6/20 10:22 pm, Rafael Savignon wrote:

Hi all,

I'm developing a Qt5 application to playback hardware decoded h.264 files. But on raspberry pi 3 its struggling to play media greater than 720p. After some investigation of QGstVideoBuffer class I realized that calls to gst_video_frame_map function passing bigger frames were consuming much cpu time ~50ms, which was resulting in the postponing of next frames delivery. So I had to find out a more performatic way to read the video frame content. After some search I found out that a special buffer mechanism could enable applications to map GPU memory directly to addressable process memory, avoiding unnecessary copy (VCSM). Then I took another look at QtMultimedia source code and saw that a OpenGL texture handle was being packaged in  with QGstVideoBuffer, as shown below.

guint textureId = gst_gl_memory_get_texture_id(glmem);

videoBuffer = new QGstVideoBuffer(buffer, m_videoInfo, m_format.handleType(), textureId); 

So I thought that things were starting to get clear. Thus I decided to use vcsm to map the texture and speed things up .

First, on my QAbstractVideoSurface::start implemented method I initialized vcsm and created a EGLimageKHR.

int w = Util::nextPOT(size.width());

int h = Util::nextPOT(size.height());

const EGLint attrib[] = { EGL_IMAGE_PRESERVED_KHR, EGL_TRUE, EGL_NONE, EGL_NONE };

vcsm_info.width = w;

vcsm_info.height = h;

vcsm_init();

eglFbImage = eglCreateImageKHR(eglGetCurrentDisplay(), EGL_NO_CONTEXT, EGL_IMAGE_BRCM_VCSM, &vcsm_info, attrib); 

Second, on my QAbstractVideoSurface::present  implementation method I tried to grab the passed Texture content, but it didn't work. Despite this, all OpenGL calls succeeded. I can access the mapped buffer, but it doesn't contain anything meaningful.

QOpenGLFunctions* f = ctx->functions(); 

GLuint framebuffer; 

GLuint depthRenderbuffer; 

GLint prevFbo;

GLenum status = GL_FRAMEBUFFER_COMPLETE;

GLuint texture = static_cast<GLuint>( currentFrame.handle().toInt() );

int texWidth = Util::nextPOT(currentFrame.width());

int texHeight = Util::nextPOT(currentFrame.height());

  

GLCHK(f->glGetIntegerv( GL_FRAMEBUFFER_BINDING, &prevFbo )); 

GLCHK(f->glGenFramebuffers(1, &framebuffer));

GLCHK(f->glBindFramebuffer(GL_FRAMEBUFFER, framebuffer));

GLCHK(glActiveTexture(GL_TEXTURE0));

GLCHK(glBindTexture(GL_TEXTURE_2D, texture));

GLCHK(glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST));

GLCHK(glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST)); 

GLCHK(glEGLImageTargetTexture2DOES(GL_TEXTURE_2D, eglFbImage)); 

GLCHK(f->glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, texture, 0)); 

GLCHK(glBindTexture(GL_TEXTURE_2D, 0)); 

status = glCheckFramebufferStatus(GL_FRAMEBUFFER);

// check fbo status 

GLCHK(f->glFinish()); 

uint8_t *vcsmBuffer; 

VCSM_CACHE_TYPE_T cacheType; 

vcsmBuffer = (uint8_t*)vcsm_lock_cache(vcsm_info.vcsm_handle, VCSM_CACHE_TYPE_HOST, &cacheType);

// print buffer

vcsm_unlock_ptr(vcsmBuffer); 

Is the Texture Id correctly filled by omxh264dec or glupload ?

Can i bind the texture id to a different fbo ?

Why gst_video_frame_map correctly map the video frame content and vcsm do not despite being addressable?

If someone could give a tip on this I would be thankful.

regards.

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel