Zero copy omxh264dec to EGLImage

[Gert Glass]

Working on RasperryPI / RDK + gstreamer 1.10.4

I'm looking for a way to zero copy GL textures from the omx h264 decoder from a
separate thread into my own GL context. The pipeline looks like this:

  appsrc ! h264parse ! omxh264dec ! glupload \
  ! appsink caps=\"video/x-raw(memory:GLMemory), format=(string)RGBA

Everything connects and links ok, and when going to PLAY state my appsrc
properly receives GL textures from the pipeline: I see a pool of 3 texture
names being delivered in the new-sample callback - so far so good.

Now I want to take the texture and transfer it to a KHR EGLImage so I can pick
this up in another thread/GL context and use the texture there with
glEGLImageTargetTexture2DOES().

(I am under some tight constraints regarding to EGL contexts and texure IDs. I
can not simple use the texture names received from gstreamer. Basically I need
to 'fake' a TexSubImage2D() call in another thread, injecting texture data
without the application knowing)

Here is my problem:

For some reason mEglCreateImageKHR() fails with:

   "eglCreateImageKHR:  failed to create image for buffer 0x2 target 12465 error 0x3002"

with 0x3002 meaning "EGL_BAD_ACCESS". The Khronos documentation for
mEglCreateImageKHR() gives some reasons for the EGL_BAD_ACCESS, but these
reasons do not seem to apply to my case, as far as I understand. I have looked
into the broadcom GL driver source for the implementation of
mEglCreateImageKHR() on the rPi at

  https://www.graphics-muse.org/archives/pibox/1.0.0/archives/rpi1/gles/gles/
  interface/khronos/ext/egl_khr_image_client.c

but only some basic checking happens here, and the call is transferred
somewhere else using some RPC mechanism, so I'm not able to figure out what is
happening under the hood.

As a test I generated a new texture with some pixel data in the new-sample
calback and pass this to eglCreateImageKHR() instead of the texid received in
the frame:

   glGenTextures(...)
   glBindTexture(...)
   glTexImage2D(...)
   mEglCreateImageKHR(...)

which works as expected: I can transfer the EGLImage to my other context
and use it here.

My question: is there anything 'special' about the texture delivered to the
new-sample callback that makes it different from the texture I glGenTextures()
myself that would make the mEglCreateImageKHR() call fail?

Thank you!

Zevv
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le mercredi 03 octobre 2018 à 10:59 +0200, Zevv Glass a écrit :
> My question: is there anything 'special' about the texture delivered to the
> new-sample callback that makes it different from the texture I glGenTextures()
> myself that would make the mEglCreateImageKHR() call fail?

Yes, they are textures bind to an EGLImage already. I suspect glupload
offers an EGL pool to the decoder (which you would have to do in
appsink if you removed glupload) and hopefully just pass-through that
buffer/memory. You can try and use:

  #include <gst/gl/egl/gstglmemoryegl.h>

You need to check if the memory->allocator is EGL memory allocator, and
if it's the case, you can get the image directly. This is all ultra
specific to RPi.

>
> Thank you!
>
> Zevv
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Gert Glass]

Hi Nicolas, thanks for taking the time to help me out here.

Quoting Nicolas Dufresne (2018-10-03 18:51:48)
 
> Yes, they are textures bind to an EGLImage already. I suspect glupload
> offers an EGL pool to the decoder (which you would have to do in
> appsink if you removed glupload)

That much I figured, and as far as I can tell this it not trivial - so I
was hoping to rely on glupload to handle the gritty details for me.

> and hopefully just pass-through that buffer/memory. You can try and
> use:
>
>   #include <gst/gl/egl/gstglmemoryegl.h>
>
> You need to check if the memory->allocator is EGL memory allocator,

The EGL allocater is indeed being used.

This is what I have now to find the texture name in the new-sample handler
(error handling omitted):

  static GstFlowReturn on_new_sample(GstElement *sink, void *_)
  {
    GstVideoInfo video_info;
    GstVideoFrame frame;

    GstSample *sample = gst_app_sink_pull_sample((GstAppSink*)mVideoSink)
    GstCaps *caps = gst_sample_get_caps(sample);
    gst_video_info_from_caps(&video_info, caps);
    GstBuffer *buffer = gst_sample_get_buffer(sample);
    GstMapFlags map_flags = (GstMapFlags)(GST_MAP_READ|GST_MAP_GL);
    gst_video_frame_map(&frame, &video_info, buffer, map_flags);
    GLuint texid = *((unsigned int*) GST_VIDEO_FRAME_PLANE_DATA(&frame, 0));
    ...
   }

I'm not sure how I would access the underlying EGL image from the
above code, though.

(I also assume the underlying EGL image is reused or destroyed after the
callback finishes, so I will have to take care how I will use these from
the other thread context - but these are problems for later.)

> This is all ultra specific to RPi.

I understood that much, there's quite a lot of #ifdefs in the code with
raspberry specifics.

Thanks,

Zevv
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le mercredi 03 octobre 2018 à 19:11 +0200, Zevv Glass a écrit :

> Hi Nicolas, thanks for taking the time to help me out here.
>
> Quoting Nicolas Dufresne (2018-10-03 18:51:48)
>  
> > Yes, they are textures bind to an EGLImage already. I suspect glupload
> > offers an EGL pool to the decoder (which you would have to do in
> > appsink if you removed glupload)
>
> That much I figured, and as far as I can tell this it not trivial - so I
> was hoping to rely on glupload to handle the gritty details for me.
>
> > and hopefully just pass-through that buffer/memory. You can try and
> > use:
> >
> >   #include <gst/gl/egl/gstglmemoryegl.h>
> >
> > You need to check if the memory->allocator is EGL memory allocator,
>
> The EGL allocater is indeed being used.
>
> This is what I have now to find the texture name in the new-sample handler
> (error handling omitted):
>
>   static GstFlowReturn on_new_sample(GstElement *sink, void *_)
>   {
>     GstVideoInfo video_info;
>     GstVideoFrame frame;
>
>     GstSample *sample = gst_app_sink_pull_sample((GstAppSink*)mVideoSink)
>     GstCaps *caps = gst_sample_get_caps(sample);
>     gst_video_info_from_caps(&video_info, caps);
>     GstBuffer *buffer = gst_sample_get_buffer(sample);
>     GstMapFlags map_flags = (GstMapFlags)(GST_MAP_READ|GST_MAP_GL);
>     gst_video_frame_map(&frame, &video_info, buffer, map_flags);
>     GLuint texid = *((unsigned int*) GST_VIDEO_FRAME_PLANE_DATA(&frame, 0));
>     ...
>    }
>
> I'm not sure how I would access the underlying EGL image from the
> above code, though.

Using GstGLMemoryEGL API I mention:

  mem = gst_buffer_peek_memory (buffer, 0);
  g_assert (gst_is_gl_memory_egl (mem));
  image = gst_gl_memory_egl_get_image ((GstGLMemoryEGL*) mem);
>
> (I also assume the underlying EGL image is reused or destroyed after the
> callback finishes, so I will have to take care how I will use these from
> the other thread context - but these are problems for later.)

Just keep a ref on it (gst_buffer_ref/unref);

>
> > This is all ultra specific to RPi.
>
> I understood that much, there's quite a lot of #ifdefs in the code with
> raspberry specifics.
>
> Thanks,
>
> Zevv

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Zevv Glass-2]

Quoting Nicolas Dufresne (2018-10-03 20:49:24)

> Le mercredi 03 octobre 2018 à 19:11 +0200, Zevv Glass a écrit :
> > I'm not sure how I would access the underlying EGL image from the
> > above code, though.
>
> Using GstGLMemoryEGL API I mention:
>
>   mem = gst_buffer_peek_memory (buffer, 0);
>   g_assert (gst_is_gl_memory_egl (mem));
>   image = gst_gl_memory_egl_get_image ((GstGLMemoryEGL*) mem);

Great, thanks for spelling that out for me!

> > (I also assume the underlying EGL image is reused or destroyed after the
> > callback finishes, so I will have to take care how I will use these from
> > the other thread context - but these are problems for later.)
>
> Just keep a ref on it (gst_buffer_ref/unref);

Clear.

I'll see if I can get things to work like this, I'll report back with
results.


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Gert Glass]

Quoting Nicolas Dufresne (2018-10-03 20:49:24)

> Le mercredi 03 octobre 2018 à 19:11 +0200, Zevv Glass a écrit :
>
> > I'm not sure how I would access the underlying EGL image from the
> > above code, though.
>
> Using GstGLMemoryEGL API I mention:
>
>   mem = gst_buffer_peek_memory (buffer, 0);
>   g_assert (gst_is_gl_memory_egl (mem));
>   image = gst_gl_memory_egl_get_image ((GstGLMemoryEGL*) mem);

Great, thanks for spelling that out for me!

> > (I also assume the underlying EGL image is reused or destroyed after
> > the callback finishes, so I will have to take care how I will use
> > these from the other thread context - but these are problems for
> > later.)
>
> Just keep a ref on it (gst_buffer_ref/unref);

Clear, thank you. I'll see if I can get things to work like this, I will
report back with results.



_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel