How to consume VAAPI VASurface in appsink
Locked
How to consume VAAPI VASurface in appsink
 
|
I have constructed a pipeline for decoding compressed video, with an appsink element at the end. The decoding pipeline is using GPU hardware decoding through libva/gst-vaapi. I am pulling samples in the appsink “new-sample” signal callback,
and have confirmed that the caps include “video/x-raw(VASurface)”. My intent is to display (and potentially process) the framebuffer contained by the VASurface. Can I somehow get access to the VASurface through the GstBuffer contained in the GstSample? If so, then I could get to a DRM prime handle for
GL/CL interop. Thanks, Nico _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
|
|
buffer data can type casted to VASurface.
static void appSinkNewSample(GstElement *psAppSink, gpointer pvUserData) { GstMapInfo sInfo; //Pull the sample GstSample *psSample = gst_app_sink_pull_sample (GST_APP_SINK(psAppSink)); GstBuffer *psBuffer = gst_sample_get_buffer (psSample); gst_buffer_map (psBuffer, &sInfo, GST_MAP_READ); // You can safely type cast sInfo->data to VASurface. gst_buffer_unmap (psBuffer, &sInfo); gst_buffer_unref (psBuffer); return; } -- Sent from: http://gstreamer-devel.966125.n4.nabble.com/ _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
Re: How to consume VAAPI VASurface in appsink
|
|
In reply to this post by Galoppo, Nico
On Wed, 08 Aug 2018 at 23:41, Galoppo, Nico wrote:
> I have constructed a pipeline for decoding compressed video, with an appsink > element at the end. The decoding pipeline is using GPU hardware decoding > through libva/gst-vaapi. I am pulling samples in the appsink "new-sample" > signal callback, and have confirmed that the caps include > "video/x-raw(VASurface)". -> "video/x-raw(memory:VASurface)" > > My intent is to display (and potentially process) the framebuffer contained by > the VASurface. Can I somehow get access to the VASurface through the GstBuffer > contained in the GstSample? If so, then I could get to a DRM prime handle for > GL/CL interop. The VASurface is in a buffer's meta. But the meta's structure is not available. You might look at the element's source and copy the structure and extract it. Nonetheless, if you want the DRM prime handle, I guess there's a direct way to do it, after the vaapipostproc set the caps feature "video/x-raw(memory:DMABuf)", then you get the dmabuf fd with gst_dmabuf_memory_get_fd() https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-libs/html/gst-plugins-base-libs-dmabuf.html vmjl > > Thanks, > > Nico > > _______________________________________________ > gstreamer-devel mailing list > [hidden email] > https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
RE: How to consume VAAPI VASurface in appsink
|
|
In reply to this post by Gst-Geek
Thanks. Is it legal to return from the "new-sample" callback before unmapping & releasing (unref) the handle to the buffer? The goal is to acquire & consume the decompressed buffer in separate threads.
Nico -----Original Message----- From: gstreamer-devel <[hidden email]> On Behalf Of ShilVin Sent: Wednesday, August 08, 2018 9:30 PM To: [hidden email] Subject: Re: How to consume VAAPI VASurface in appsink buffer data can type casted to VASurface. static void appSinkNewSample(GstElement *psAppSink, gpointer pvUserData) { GstMapInfo sInfo; //Pull the sample GstSample *psSample = gst_app_sink_pull_sample (GST_APP_SINK(psAppSink)); GstBuffer *psBuffer = gst_sample_get_buffer (psSample); gst_buffer_map (psBuffer, &sInfo, GST_MAP_READ); // You can safely type cast sInfo->data to VASurface. gst_buffer_unmap (psBuffer, &sInfo); gst_buffer_unref (psBuffer); return; } -- Sent from: http://gstreamer-devel.966125.n4.nabble.com/ _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
RE: How to consume VAAPI VASurface in appsink
|
|
In reply to this post by Victor Jaquez
Thanks!
However, the vaapipostproc doesn't seem to support "video/x-raw(memory:DMABuf)" on its source pad. I tried it out as follows: gst-launch-1.0 -v souphttpsrc location=https://www.freedesktop.org/software/gstreamer-sdk/data/media/sintel_trailer-480p.webm ! matroskademux ! vaapidecodebin ! vaapipostproc ! "video/x-raw(memory:DMABuf)" ! fakesink Is there something I’m missing? Also, if I do find a way to use gst_dmabuf_memory_get_fd(), how do I make sure to keep a reference to the memory so that it doesn't get reused after returning from the 'new-sample' callback? Is simply mapping the buffer sufficient? Thanks, Nico -----Original Message----- From: gstreamer-devel <[hidden email]> On Behalf Of Víctor Jáquez Sent: Thursday, August 09, 2018 8:07 AM To: [hidden email] Subject: Re: How to consume VAAPI VASurface in appsink On Wed, 08 Aug 2018 at 23:41, Galoppo, Nico wrote: > I have constructed a pipeline for decoding compressed video, with an > appsink element at the end. The decoding pipeline is using GPU > hardware decoding through libva/gst-vaapi. I am pulling samples in the appsink "new-sample" > signal callback, and have confirmed that the caps include > "video/x-raw(VASurface)". -> "video/x-raw(memory:VASurface)" > > My intent is to display (and potentially process) the framebuffer > contained by the VASurface. Can I somehow get access to the VASurface > through the GstBuffer contained in the GstSample? If so, then I could > get to a DRM prime handle for GL/CL interop. The VASurface is in a buffer's meta. But the meta's structure is not available. You might look at the element's source and copy the structure and extract it. Nonetheless, if you want the DRM prime handle, I guess there's a direct way to do it, after the vaapipostproc set the caps feature "video/x-raw(memory:DMABuf)", then you get the dmabuf fd with gst_dmabuf_memory_get_fd() https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-libs/html/gst-plugins-base-libs-dmabuf.html vmjl > > Thanks, > > Nico > > _______________________________________________ > gstreamer-devel mailing list > [hidden email] > https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
RE: How to consume VAAPI VASurface in appsink
|
|
In reply to this post by Galoppo, Nico
Hello Shilvin,
I spoke too quickly - there are obviously a few more things missing to go this route: - how do I get access to the VADisplay used by the pipeline? - does sInfo->data point to VASurfaceID, which I can then use with vaDeriveImage to get to the VAImage? (you mentioned sInfo->data pointing to VASurface but only VASurfaceID exists as a type) Thanks! Nico -----Original Message----- From: gstreamer-devel <[hidden email]> On Behalf Of Galoppo, Nico Sent: Thursday, August 09, 2018 8:49 AM To: Discussion of the development of and with GStreamer <[hidden email]> Subject: RE: How to consume VAAPI VASurface in appsink Thanks. Is it legal to return from the "new-sample" callback before unmapping & releasing (unref) the handle to the buffer? The goal is to acquire & consume the decompressed buffer in separate threads. Nico -----Original Message----- From: gstreamer-devel <[hidden email]> On Behalf Of ShilVin Sent: Wednesday, August 08, 2018 9:30 PM To: [hidden email] Subject: Re: How to consume VAAPI VASurface in appsink buffer data can type casted to VASurface. static void appSinkNewSample(GstElement *psAppSink, gpointer pvUserData) { GstMapInfo sInfo; //Pull the sample GstSample *psSample = gst_app_sink_pull_sample (GST_APP_SINK(psAppSink)); GstBuffer *psBuffer = gst_sample_get_buffer (psSample); gst_buffer_map (psBuffer, &sInfo, GST_MAP_READ); // You can safely type cast sInfo->data to VASurface. gst_buffer_unmap (psBuffer, &sInfo); gst_buffer_unref (psBuffer); return; } -- Sent from: http://gstreamer-devel.966125.n4.nabble.com/ _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
Re: How to consume VAAPI VASurface in appsink
|
|
In reply to this post by Galoppo, Nico
On Thu, 09 Aug 2018 at 15:56, Galoppo, Nico wrote:
> Thanks! > > However, the vaapipostproc doesn't seem to support > "video/x-raw(memory:DMABuf)" on its source pad. I tried it out as follows: > > gst-launch-1.0 -v souphttpsrc > location=https://www.freedesktop.org/software/gstreamer-sdk/data/media/sintel_trailer-480p.webm > ! matroskademux ! vaapidecodebin ! vaapipostproc ! > "video/x-raw(memory:DMABuf)" ! fakesink At least the pipeline works for me. Btw, you could remove the vaapipostproc since vaapidecodebin already contains it. > Is there something I’m missing? > > Also, if I do find a way to use gst_dmabuf_memory_get_fd(), how do I make sure > to keep a reference to the memory so that it doesn't get reused after > returning from the 'new-sample' callback? Is simply mapping the buffer > sufficient? You won't map it. While you hold a reference of the GstBuffer structure, hopefully, it won't be reused. You extract the GstMemory from the GstBuffer and then get the fd from it. You can learn how it is done in glupload: https://cgit.freedesktop.org/gstreamer/gst-plugins-base/tree/gst-libs/gst/gl/gstglupload.c#n586 > > Thanks, > > Nico > > > > -----Original Message----- > From: gstreamer-devel <[hidden email]> On Behalf Of Víctor Jáquez > Sent: Thursday, August 09, 2018 8:07 AM > To: [hidden email] > Subject: Re: How to consume VAAPI VASurface in appsink > > On Wed, 08 Aug 2018 at 23:41, Galoppo, Nico wrote: > > I have constructed a pipeline for decoding compressed video, with an > > appsink element at the end. The decoding pipeline is using GPU > > hardware decoding through libva/gst-vaapi. I am pulling samples in the appsink "new-sample" > > signal callback, and have confirmed that the caps include > > "video/x-raw(VASurface)". > > -> "video/x-raw(memory:VASurface)" > > > > > My intent is to display (and potentially process) the framebuffer > > contained by the VASurface. Can I somehow get access to the VASurface > > through the GstBuffer contained in the GstSample? If so, then I could > > get to a DRM prime handle for GL/CL interop. > > The VASurface is in a buffer's meta. But the meta's structure is not available. You might look at the element's source and copy the structure and extract it. > > Nonetheless, if you want the DRM prime handle, I guess there's a direct way to do it, after the vaapipostproc set the caps feature "video/x-raw(memory:DMABuf)", then you get the dmabuf fd with > gst_dmabuf_memory_get_fd() > > https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-libs/html/gst-plugins-base-libs-dmabuf.html > > vmjl > > > > > Thanks, > > > > Nico > > > > > _______________________________________________ > > gstreamer-devel mailing list > > [hidden email] > > https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel > > _______________________________________________ > gstreamer-devel mailing list > [hidden email] > https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel > _______________________________________________ > gstreamer-devel mailing list > [hidden email] > https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
RE: How to consume VAAPI VASurface in appsink
|
|
Is this a recent addition? In my plugin, from gst-inspect (see below), I see that I have version 1.8.3 of the plugin, and the src sink only supports video/x-raw(memory:VASurface), video/x-raw(meta:GstVideoGLTextureUploadMeta), and video/x-raw.
I'm also wondering how to find out the caps beyond "video/x-raw". That's what gst_structure_get_name() tells me. However, I don't know how to get to the part between the parentheses ("memory:VASurface" or "meta:GstVideoGLTextureUploadMeta"). Thanks for the reference to glupload; I'll have a look. Sorry for the rookie questions, I'm definitely new to gstreamer & libva internals. Plugin Details: Name vaapi Description VA-API based elements Filename /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstvaapi.so Version 1.8.3 License LGPL Source module gstreamer-vaapi Source release date 2016-06-09 Binary package gstreamer-vaapi Origin URL http://bugzilla.gnome.org/enter_bug.cgi?product=GStreamer [...] SRC template: 'src' Availability: Always Capabilities: video/x-raw(memory:VASurface) format: { ENCODED, I420, YV12, NV12 } width: [ 1, 2147483647 ] height: [ 1, 2147483647 ] framerate: [ 0/1, 2147483647/1 ] interlace-mode: progressive video/x-raw(meta:GstVideoGLTextureUploadMeta) format: { RGBA, BGRA } width: [ 1, 2147483647 ] height: [ 1, 2147483647 ] framerate: [ 0/1, 2147483647/1 ] video/x-raw format: { I420, YV12, YUY2, UYVY, AYUV, RGBx, BGRx, xRGB, xBGR, RGBA, BGRA, ARGB, ABGR, RGB, BGR, Y41B, Y42B, YVYU, Y444, v210, v216, NV12, NV21, NV16, NV61, NV24, GRAY8, GRAY16_BE, GRAY16_LE, v308, RGB16, BGR16, RGB15, BGR15, UYVP, A420, RGB8P, YUV9, YVU9, IYU1, ARGB64, AYUV64, r210, I420_10LE, I420_10BE, I422_10LE, I422_10BE, Y444_10LE, Y444_10BE, GBR, GBR_10LE, GBR_10BE, NV12_64Z32, A420_10LE, A420_10BE, A422 _10LE, A422_10BE, A444_10LE, A444_10BE } width: [ 1, 2147483647 ] height: [ 1, 2147483647 ] framerate: [ 0/1, 2147483647/1 ] interlace-mode: progressive -----Original Message----- From: gstreamer-devel <[hidden email]> On Behalf Of Víctor Jáquez Sent: Thursday, August 09, 2018 9:47 AM To: Discussion of the development of and with GStreamer <[hidden email]> Subject: Re: How to consume VAAPI VASurface in appsink On Thu, 09 Aug 2018 at 15:56, Galoppo, Nico wrote: > Thanks! > > However, the vaapipostproc doesn't seem to support > "video/x-raw(memory:DMABuf)" on its source pad. I tried it out as follows: > > gst-launch-1.0 -v souphttpsrc > location=https://www.freedesktop.org/software/gstreamer-sdk/data/media > /sintel_trailer-480p.webm ! matroskademux ! vaapidecodebin ! > vaapipostproc ! > "video/x-raw(memory:DMABuf)" ! fakesink At least the pipeline works for me. Btw, you could remove the vaapipostproc since vaapidecodebin already contains it. > Is there something I’m missing? > > Also, if I do find a way to use gst_dmabuf_memory_get_fd(), how do I > make sure to keep a reference to the memory so that it doesn't get > reused after returning from the 'new-sample' callback? Is simply > mapping the buffer sufficient? You won't map it. While you hold a reference of the GstBuffer structure, hopefully, it won't be reused. You extract the GstMemory from the GstBuffer and then get the fd from it. You can learn how it is done in glupload: https://cgit.freedesktop.org/gstreamer/gst-plugins-base/tree/gst-libs/gst/gl/gstglupload.c#n586 > > Thanks, > > Nico > > > > -----Original Message----- > From: gstreamer-devel <[hidden email]> > On Behalf Of Víctor Jáquez > Sent: Thursday, August 09, 2018 8:07 AM > To: [hidden email] > Subject: Re: How to consume VAAPI VASurface in appsink > > On Wed, 08 Aug 2018 at 23:41, Galoppo, Nico wrote: > > I have constructed a pipeline for decoding compressed video, with an > > appsink element at the end. The decoding pipeline is using GPU > > hardware decoding through libva/gst-vaapi. I am pulling samples in the appsink "new-sample" > > signal callback, and have confirmed that the caps include > > "video/x-raw(VASurface)". > > -> "video/x-raw(memory:VASurface)" > > > > > My intent is to display (and potentially process) the framebuffer > > contained by the VASurface. Can I somehow get access to the > > VASurface through the GstBuffer contained in the GstSample? If so, > > then I could get to a DRM prime handle for GL/CL interop. > > The VASurface is in a buffer's meta. But the meta's structure is not available. You might look at the element's source and copy the structure and extract it. > > Nonetheless, if you want the DRM prime handle, I guess there's a > direct way to do it, after the vaapipostproc set the caps feature > "video/x-raw(memory:DMABuf)", then you get the dmabuf fd with > gst_dmabuf_memory_get_fd() > > https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins- > base-libs/html/gst-plugins-base-libs-dmabuf.html > > vmjl > > > > > Thanks, > > > > Nico > > > > > _______________________________________________ > > gstreamer-devel mailing list > > [hidden email] > > https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel > > _______________________________________________ > gstreamer-devel mailing list > [hidden email] > https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel > _______________________________________________ > gstreamer-devel mailing list > [hidden email] > https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
Re: How to consume VAAPI VASurface in appsink
|
|
On Thu, 09 Aug 2018 at 19:34, Galoppo, Nico wrote:
> Is this a recent addition? In my plugin, from gst-inspect (see below), I see > that I have version 1.8.3 of the plugin, and the src sink only supports > video/x-raw(memory:VASurface), video/x-raw(meta:GstVideoGLTextureUploadMeta), > and video/x-raw. That's too old. Update at least to GStreamer 1.12, or 1.14 vmjl _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
RE: How to consume VAAPI VASurface in appsink
|
|
Thank you - I just tried on a vanilla ubuntu 18.04 box. This one does support DMABuf.
Now the following pipeline runs: gst-launch-1.0 -v souphttpsrc location=https://www.freedesktop.org/software/gstreamer-sdk/data/media/sintel_trailer-480p.webm ! matroskademux ! vaapidecodebin ! "video/x-raw(memory:DMABuf)" ! glimagesink However, the output is scrambled. Looks like wrong image frame metadata is passed, perhaps incorrect stride. The following pipeline runs correctly (not buggy), but since I'm not using a capsfilter, it's going through video/x-raw(memory:VASurface) at the glimagesink. gst-launch-1.0 -v souphttpsrc location=https://www.freedesktop.org/software/gstreamer-sdk/data/media/sintel_trailer-480p.webm ! matroskademux ! vaapidecodebin ! glimagesink Nico -----Original Message----- From: gstreamer-devel <[hidden email]> On Behalf Of Víctor Jáquez Sent: Friday, August 10, 2018 1:32 AM To: Discussion of the development of and with GStreamer <[hidden email]> Subject: Re: How to consume VAAPI VASurface in appsink On Thu, 09 Aug 2018 at 19:34, Galoppo, Nico wrote: > Is this a recent addition? In my plugin, from gst-inspect (see below), > I see that I have version 1.8.3 of the plugin, and the src sink only > supports video/x-raw(memory:VASurface), > video/x-raw(meta:GstVideoGLTextureUploadMeta), > and video/x-raw. That's too old. Update at least to GStreamer 1.12, or 1.14 vmjl _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
«
Return to GStreamer-devel
|
1 view|%1 views
| Free forum by Nabble | Edit this page |