VAAPI + meta:GstVideoGLTextureUploadMeta

[Christophe Lafolet]

Hello,

I've troubles with VAAPI and OpenGL :)

I'm developping a Java OpenGL application with dynamic streams and use DecodeBin + AppSink to be notified of buffer reception.


No problem when I disable VAAPI plugin selection and do texture upload myself.

Now I want to use hardware decoding

It's not clear to me which API should I use to shared openGL context with VAAPI

- in this post : http://ystreet00.blogspot.com.au/2015/09/gstreamer-16-and-opengl-contexts.html
the new way should be to set a context on GST_MESSAGE_NEED_CONTEXT reception but the VAAPI plugin seems only verify that the given context contains a VAAPI display and delegate to element_class->set_context() but this callback is unset for VaapiDecode so no context is saved  
Question : why do I need to add a VAAPI display in this context ? DecodeBin is a black box for me and I'm surprised to specify a VAAPI display.

- the old way seems to use allocation query :
AppSink do not provide configurable "signal" to add callback to handle allocation query : I update the AppSink.propose_allocation field directly from Java with my callback and it works ...   yes
So, I add GstVideoGLTextureUploadMeta to the allocation query : "gst.gl.GstGLContext" with a gl_context build with gst_gl_context_new_wrapped(), VAAPI detects my parameters (GST_GL_PLATFORM_GLX, GST_GL_API_OPENGL)
but a SIGSEGV is generated later.
I receive samples containing no raw data (OK), a VideoGLTextureUploadMeta with one texture (OK) but can't upload

Is there a example to use VAAPI + meta:GstVideoGLTextureUploadMeta ?


My config :
Linux + X11 + GLX
gstreamer 1.6.2-1
gstreamer-vaapi 0.7.0
gstopengl not used.

[Victor Jaquez]

Hi Christophe

On 01/17/16 at 11:32am, Christophe Lafolet wrote:

> Hello,
>
> I've troubles with VAAPI and OpenGL :)
>
> I'm developping a Java OpenGL application with dynamic streams and use
> DecodeBin + AppSink to be notified of buffer reception.
>
>
> No problem when I disable VAAPI plugin selection and do texture upload
> myself.
>
> Now I want to use hardware decoding
>
> It's not clear to me which API should I use to shared openGL context with
> VAAPI
>
> - in this post :
> http://ystreet00.blogspot.com.au/2015/09/gstreamer-16-and-opengl-contexts.html
> the new way should be to set a context on GST_MESSAGE_NEED_CONTEXT reception
> but the VAAPI plugin seems only verify that the given context contains a
> VAAPI display and delegate to element_class->set_context() but this callback
> is unset for VaapiDecode so no context is saved  
> Question : why do I need to add a VAAPI display in this context ? DecodeBin
> is a black box for me and I'm surprised to specify a VAAPI display.

The virtual method call to parent class is a requirement for gstreamer 1.7:
https://bugzilla.gnome.org/show_bug.cgi?id=757598#c2

But you are using GStreamer 1.6, that's why that method is not resolved by the
GstElement class.

gstvaapidecode negotiates the GstGLContext in the allocation query because it
needs to know which type of display backend to use in the
GstVideoGLTextureUploadMeta callbacks (GLX or EGL). But it doesn't hold the
context. The meta callbacks are going to be called in the rendering context.

But that's a good catch: perhaps vaapidecode should request for a GstGLContext
through the GstContext mechanism to know earlier the used display backend, and
to avoid the propose_allocation() function in the appsink. Nonetheless, that will
not solve your problem.

The mechanism explained in Matthew's blog post is when you use glimagesink as
a substitute of appsink, using its signal 'client-draw', where you will get a
GstBuffer with a texture to render.

This mechanism is used (experimentally) in WebKitGTK+:

https://github.com/WebKit/webkit/blob/master/Source/WebCore/platform/graphics/gstreamer/MediaPlayerPrivateGStreamerBase.cpp#L727

> - the old way seems to use allocation query :
> AppSink do not provide configurable "signal" to add callback to handle
> allocation query : I update the AppSink.propose_allocation field directly
> from Java with my callback and it works ...   yes
> So, I add GstVideoGLTextureUploadMeta to the allocation query :
> "gst.gl.GstGLContext" with a gl_context build with
> gst_gl_context_new_wrapped(), VAAPI detects my parameters
> (GST_GL_PLATFORM_GLX, GST_GL_API_OPENGL)
> but a SIGSEGV is generated later.
> I receive samples containing no raw data (OK), a VideoGLTextureUploadMeta
> with one texture (OK) but can't upload

That's the way it should work.

The meta callbacks should be called under the application gl context and they
will upload the buffer into the texture:

GstVideoGLTextureUploadMeta* meta;
if ((meta = gst_buffer_get_video_gl_texture_upload_meta(buffer))) {
  if (meta->n_textures == 1) { // BRGx & BGRA formats use only one texture.
    guint ids[4] = { texture.id(), 0, 0, 0 };
    if (gst_video_gl_texture_upload_meta_upload(meta, ids))
      return;
  }
}

Which vaapi backend are you using? Some how I guess that is the vdpau one.

Can you open a bug with the back trace?

> Is there a example to use VAAPI + meta:GstVideoGLTextureUploadMeta ?

the element gstglupload does a good job

http://cgit.freedesktop.org/gstreamer/gst-plugins-bad/tree/gst-libs/gst/gl/gstglupload.c

>
>
> My config :
> Linux + X11 + GLX
> gstreamer 1.6.2-1
> gstreamer-vaapi 0.7.0
> gstopengl not used.


vmjl
_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Christophe Lafolet]

Hello Vector

thanks for your examples that help me to solve my SIGSEGV : I was trying to retrieve a unusefull byte buffer by calling map/unmap on the GstBuffer, seems strange this SIGSEGV ....


but others problems are still present :

 - in file gstvaapivideometa_texture.c ( method gst_buffer_add_texture_upload_meta()),  the code below is not compatible with the gst_buffer_add_video_gl_texture_upload_meta api :
the method need a GstVideoGLTextureType[4] (because a memcpy is made on the 4 elements) and only a GstVideoGLTextureType[1] is given

meta = gst_buffer_add_video_gl_texture_upload_meta (buffer,
      GST_VIDEO_GL_TEXTURE_ORIENTATION_X_NORMAL_Y_NORMAL,
      1, &meta_texture->texture_type, gst_vaapi_texture_upload,
      meta_texture, (GBoxedCopyFunc) meta_texture_copy,
      (GBoxedFreeFunc) meta_texture_free);
 

- in file gstvideometa.c (method gst_buffer_add_video_gl_texture_upload_meta()),
      memcpy (meta->texture_type, texture_type, sizeof (texture_type[0]) * 4);

when there is only one texture (major case), it seems the caller need to feel the GstVideoGLTextureType[4] with a "invalid" value to be defined (GST_VIDEO_GL_TEXTURE_TYPE_INVALID for example)
  GstVideoGLTextureType texture_type[4] =
    {meta_texture->texture_type,
      GST_VIDEO_GL_TEXTURE_TYPE_INVALID,
      GST_VIDEO_GL_TEXTURE_TYPE_INVALID,
      GST_VIDEO_GL_TEXTURE_TYPE_INVALID};


- in gst_vaapi_texture_glx_new_wrapped(), a default gl_api seems to be searched once by calling the method gl_get_current_api() before reception of the query allocation containing my "gst.gl.GstGLContext"
in my case, the GL version is "4.5.0 NVIDIA 358.16", GL_CONTEXT_PROFILE_MASK = 0 (seems to be a bug on nvidia card) : the method gl_get_current_api() can't decode the version and return GST_VAAPI_GL_API_NONE !!
my gl_api (GL_API_OPENGL) contained in the GLContext (in the query allocation) is never read, nor the environment variable GST_GL_API, ........................................


to continue, I modify gl_get_current_api() to return GST_VAAPI_GL_API_OPENGL

- the last problems is
vaapi gstvaapiutils_glx.c:986:gl_unbind_pixmap_object: failed to release pixmap ??


christophe


Le 18/01/2016 11:32, Víctor M. Jáquez L. a écrit :

Hi Christophe

On 01/17/16 at 11:32am, Christophe Lafolet wrote:

Hello,

I've troubles with VAAPI and OpenGL :) 

I'm developping a Java OpenGL application with dynamic streams and use
DecodeBin + AppSink to be notified of buffer reception.


No problem when I disable VAAPI plugin selection and do texture upload
myself.

Now I want to use hardware decoding 

It's not clear to me which API should I use to shared openGL context with
VAAPI

- in this post :
http://ystreet00.blogspot.com.au/2015/09/gstreamer-16-and-opengl-contexts.html
the new way should be to set a context on GST_MESSAGE_NEED_CONTEXT reception
but the VAAPI plugin seems only verify that the given context contains a
VAAPI display and delegate to element_class->set_context() but this callback
is unset for VaapiDecode so no context is saved  
Question : why do I need to add a VAAPI display in this context ? DecodeBin
is a black box for me and I'm surprised to specify a VAAPI display.

The virtual method call to parent class is a requirement for gstreamer 1.7:
https://bugzilla.gnome.org/show_bug.cgi?id=757598#c2

But you are using GStreamer 1.6, that's why that method is not resolved by the
GstElement class.

gstvaapidecode negotiates the GstGLContext in the allocation query because it
needs to know which type of display backend to use in the
GstVideoGLTextureUploadMeta callbacks (GLX or EGL). But it doesn't hold the
context. The meta callbacks are going to be called in the rendering context.

But that's a good catch: perhaps vaapidecode should request for a GstGLContext
through the GstContext mechanism to know earlier the used display backend, and
to avoid the propose_allocation() function in the appsink. Nonetheless, that will
not solve your problem.

The mechanism explained in Matthew's blog post is when you use glimagesink as
a substitute of appsink, using its signal 'client-draw', where you will get a
GstBuffer with a texture to render.

This mechanism is used (experimentally) in WebKitGTK+:

https://github.com/WebKit/webkit/blob/master/Source/WebCore/platform/graphics/gstreamer/MediaPlayerPrivateGStreamerBase.cpp#L727

- the old way seems to use allocation query :
AppSink do not provide configurable "signal" to add callback to handle
allocation query : I update the AppSink.propose_allocation field directly
from Java with my callback and it works ...   yes
So, I add GstVideoGLTextureUploadMeta to the allocation query :
"gst.gl.GstGLContext" with a gl_context build with
gst_gl_context_new_wrapped(), VAAPI detects my parameters
(GST_GL_PLATFORM_GLX, GST_GL_API_OPENGL) 
but a SIGSEGV is generated later.
I receive samples containing no raw data (OK), a VideoGLTextureUploadMeta
with one texture (OK) but can't upload

That's the way it should work.

The meta callbacks should be called under the application gl context and they
will upload the buffer into the texture:

GstVideoGLTextureUploadMeta* meta;
if ((meta = gst_buffer_get_video_gl_texture_upload_meta(buffer))) {
  if (meta->n_textures == 1) { // BRGx & BGRA formats use only one texture.
    guint ids[4] = { texture.id(), 0, 0, 0 };
    if (gst_video_gl_texture_upload_meta_upload(meta, ids))
      return;
  }
}

Which vaapi backend are you using? Some how I guess that is the vdpau one.

Can you open a bug with the back trace?

Is there a example to use VAAPI + meta:GstVideoGLTextureUploadMeta ?

the element gstglupload does a good job

http://cgit.freedesktop.org/gstreamer/gst-plugins-bad/tree/gst-libs/gst/gl/gstglupload.c

My config : 
Linux + X11 + GLX
gstreamer 1.6.2-1
gstreamer-vaapi 0.7.0
gstopengl not used.

vmjl
_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Victor Jaquez]

On 01/23/16 at 11:31pm, Christophe Lafolet wrote:

> Hello Vector
>
> thanks for your examples that help me to solve my SIGSEGV : I was trying to
> retrieve a unusefull byte buffer by calling map/unmap on the GstBuffer,
> seems strange this SIGSEGV ....
>
>
> but others problems are still present :
>
>  - in file gstvaapivideometa_texture.c ( method
> gst_buffer_add_texture_upload_meta()),  the code below is not compatible
> with the gst_buffer_add_video_gl_texture_upload_meta api :
> the method need a /|GstVideoGLTextureType[4] |/|(because a memcpy is made on
> the 4 elements) and only a|/||//|/|GstVideoGLTextureType|/[1] |/|is given|/|
>
> |/meta = gst_buffer_add_video_gl_texture_upload_meta (buffer,
>       GST_VIDEO_GL_TEXTURE_ORIENTATION_X_NORMAL_Y_NORMAL,
>       1, &meta_texture->texture_type, gst_vaapi_texture_upload,
>       meta_texture, (GBoxedCopyFunc) meta_texture_copy,
>       (GBoxedFreeFunc) meta_texture_free);
> - in file gstvideometa.c (method
> gst_buffer_add_video_gl_texture_upload_meta()),
>       memcpy (meta->texture_type, texture_type, sizeof (texture_type[0]) *
> 4);
> when there is only one texture (major case), it seems the caller need to
> feel the /|GstVideoGLTextureType[4]|/ with a "invalid" value to be defined
> (GST_VIDEO_GL_TEXTURE_TYPE_INVALID for example)
>   GstVideoGLTextureType texture_type[4] =
>     {meta_texture->texture_type,
>       GST_VIDEO_GL_TEXTURE_TYPE_INVALID,
>       GST_VIDEO_GL_TEXTURE_TYPE_INVALID,
>       GST_VIDEO_GL_TEXTURE_TYPE_INVALID};

I haven't noticed a problem with it, but yeah, it would be better to fix it,
to avoid a buffer overflow. Good catch!

Do you have a patch?

> - in gst_vaapi_texture_glx_new_wrapped(), a default gl_api seems to be
> searched once by calling the method gl_get_current_api() before reception of
> the query allocation containing my "gst.gl.GstGLContext"
> in my case, the GL version is "4.5.0 NVIDIA 358.16", GL_CONTEXT_PROFILE_MASK
> = 0 (seems to be a bug on nvidia card) : the method gl_get_current_api()
> can't decode the version and return GST_VAAPI_GL_API_NONE !!
> my gl_api (GL_API_OPENGL) contained in the GLContext (in the query
> allocation) is never read, nor the environment variable GST_GL_API,
> ........................................
>
>
> to continue, I modify gl_get_current_api() to return GST_VAAPI_GL_API_OPENGL

Perhaps vaapi element should query for the GstGLContext using the context
mechanism, rather than rely on the allocation query.

>
> - the last problems is
> vaapi gstvaapiutils_glx.c:986:gl_unbind_pixmap_object: failed to release
> pixmap ??

I have saw that with NVIDIA cards, but I haven't dig on it further.

vmjl
_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Christophe Lafolet]

Hello Victor,

"Perhaps vaapi element should query for the GstGLContext using the context
mechanism, rather than rely on the allocation query."

have you planified to do this evolution in vaapi ?

Regards

[Victor Jaquez]

On 05/09/16 at 09:38am, Christophe Lafolet wrote:
> Hello Victor,
>
> "Perhaps vaapi element should query for the GstGLContext using the context
> mechanism, rather than rely on the allocation query."
>
> have you planified to do this evolution in vaapi ?

It is not in my priority list right now. I'm focusing in dmabuf, which I
think, it is a better mechanism for sharing buffers.

vmjl
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Victor Jaquez]

On 05/10/16 at 09:45am, Víctor M. Jáquez L. wrote:

> On 05/09/16 at 09:38am, Christophe Lafolet wrote:
> > Hello Victor,
> >
> > "Perhaps vaapi element should query for the GstGLContext using the context
> > mechanism, rather than rely on the allocation query."
> >
> > have you planified to do this evolution in vaapi ?
>
> It is not in my priority list right now. I'm focusing in dmabuf, which I
> think, it is a better mechanism for sharing buffers.

I opened a bug:

https://bugzilla.gnome.org/show_bug.cgi?id=766206

>
> vmjl
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel