Vaapi. The ability to add input types to the decoders plugin: GstVideoGLTextureMeta and DMABuf

[Ivan Fedotov]

Hello!

 

We use the vaapi encoder / decoder vp8. We have a GUI. The gui image is rendered directly into the opengl texture. We need to encode an image from the texture. We have to copy the memory from opengl to the main memory. Then feed to the coder. We need to encode an opengl texture without an intermediate memory copy. Is it possible to add GstVideoGLTextureMeta and DMABuf types to the inputs of the decoder plugins?

Do you plan to add such functionality in the future?

Or we could also do this writing ourselves. Could you give us some information how to properly add this functionality to your library?

 

Ivan

 

 

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Victor Jaquez]

On Tue, 21 Aug 2018 at 20:58, Ivan Fedotov wrote:
> Hello!
>  
> We use the vaapi encoder / decoder vp8. We have a GUI. The gui image is
> rendered directly into the opengl texture. We need to encode an image from the
> texture. We have to copy the memory from opengl to the main memory. Then feed
> to the coder. We need to encode an opengl texture without an intermediate
> memory copy. Is it possible to add GstVideoGLTextureMeta and DMABuf types to
> the inputs of the decoder plugins?

I guess you could use glsrcbin or gldownload to convert a GLMemory based buffer
(with a texture inside) into a dmabuf-based buffer and vaapipostproc would
upload it into a VA surface and then encoding it.

vmjl

> Do you plan to add such functionality in the future?
> Or we could also do this writing ourselves. Could you give us some information
> how to properly add this functionality to your library?
>  
> Ivan
>  
>  

> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Alexander Yanin]

> I guess you could use glsrcbin or gldownload to convert a GLMemory based buffer
> (with a texture inside) into a dmabuf-based buffer and vaapipostproc would
> upload it into a VA surface and then encoding it.

In 1.14 vaapipostproc only has 'video/x-raw(memory:VASurface)' and
'video/x-raw' sink pads, so there is no DMAbuf. Was it added in later
versions of gstreamer?
In case of DMAbuf input, does DMAbuf -> vaapi conversion implies copy operation?
Is there any sense in adding DMAbuf sink pad directly to vaapi encoder
base class?

Alex
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Victor Jaquez]

On Wed, 22 Aug 2018 at 14:03, Alexander Yanin wrote:

> > I guess you could use glsrcbin or gldownload to convert a GLMemory based buffer
> > (with a texture inside) into a dmabuf-based buffer and vaapipostproc would
> > upload it into a VA surface and then encoding it.
>
> In 1.14 vaapipostproc only has 'video/x-raw(memory:VASurface)' and
> 'video/x-raw' sink pads, so there is no DMAbuf. Was it added in later
> versions of gstreamer?
> In case of DMAbuf input, does DMAbuf -> vaapi conversion implies copy operation?
> Is there any sense in adding DMAbuf sink pad directly to vaapi encoder
> base class?

Adding memory:DMABuf caps feature in vaapipostproc sink pad is not
implemented. BUT, dmabuf-based buffers don't need to use that caps feature
*if they are mappable to user-space*.

gldownload can pass dmabuf-based buffers without memory:DMABuf caps feature, as
far as I understand. And vaapipostproc can detect if that buffer is dmabuf-based
and upload it onto a VA surface (no copy is involved).

There is no need to duplicate the code in vaapipostroc in the encoders, when you
can just prepend vaapiproc in the pipeline.

vmjl
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Alexander Yanin]

> Adding memory:DMABuf caps feature in vaapipostproc sink pad is not
> implemented. BUT, dmabuf-based buffers don't need to use that caps feature
> *if they are mappable to user-space*.

> gldownload can pass dmabuf-based buffers without memory:DMABuf caps feature, as
> far as I understand. And vaapipostproc can detect if that buffer is dmabuf-based
> and upload it onto a VA surface (no copy is involved).

> There is no need to duplicate the code in vaapipostroc in the encoders, when you
> can just prepend vaapiproc in the pipeline.

How could gldownload figure out to pass DMABuf instead of raw memory
if the elements negotiated video/x-raw caps? For example, I've tested
this pipeline:

gst-launch-1.0 gltestsrc is-live=true ! glvideomixer ! gldownload !
vaapipostproc ! vaapivp8enc ! webmmux ! filesink location=test.webm

SVG of this pipeline is attached.

It is doubtful that gldownload uses DMABuf in this case.
If I use caps filter with 'memory:DMABuf' specified, negotiation fails.

Alex

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Victor Jaquez]

On Wed, 22 Aug 2018 at 15:12, Alexander Yanin wrote:

> > Adding memory:DMABuf caps feature in vaapipostproc sink pad is not
> > implemented. BUT, dmabuf-based buffers don't need to use that caps feature
> > *if they are mappable to user-space*.
>
> > gldownload can pass dmabuf-based buffers without memory:DMABuf caps feature, as
> > far as I understand. And vaapipostproc can detect if that buffer is dmabuf-based
> > and upload it onto a VA surface (no copy is involved).
>
> > There is no need to duplicate the code in vaapipostroc in the encoders, when you
> > can just prepend vaapiproc in the pipeline.
>
> How could gldownload figure out to pass DMABuf instead of raw memory
> if the elements negotiated video/x-raw caps? For example, I've tested
> this pipeline:

Indeed, there's now a code path in gldownload to verify if the generated buffers
are mappable.

I just quick hacked gldownload

diff --git a/ext/gl/gstgldownloadelement.c b/ext/gl/gstgldownloadelement.c
index aefc289ee..429bcb4d2 100644
--- a/ext/gl/gstgldownloadelement.c
+++ b/ext/gl/gstgldownloadelement.c
@@ -137,14 +137,14 @@ gst_gl_download_element_set_caps (GstBaseTransform * bt, GstCaps * in_caps,
   if (gst_caps_features_contains (features, GST_CAPS_FEATURE_MEMORY_GL_MEMORY)) {
     /* do nothing with the buffer */
 #if GST_GL_HAVE_PLATFORM_EGL && GST_GL_HAVE_DMABUF
-  } else if (gst_caps_features_contains (features,
-          GST_CAPS_FEATURE_MEMORY_DMABUF)) {
+  } else {
     dl->dmabuf_allocator = gst_dmabuf_allocator_new ();
-#endif
-  } else if (gst_caps_features_contains (features,
-          GST_CAPS_FEATURE_MEMORY_SYSTEM_MEMORY)) {
+  }
+#else
+  } else {
     dl->do_pbo_transfers = TRUE;
   }
+#endif
 
   return TRUE;
 }


And it seems to work.

The other option is to implement in vaapipostproc the negotiation of
memory:DMABuf caps feature.

vmjl

>
> gst-launch-1.0 gltestsrc is-live=true ! glvideomixer ! gldownload !
> vaapipostproc ! vaapivp8enc ! webmmux ! filesink location=test.webm
>
> SVG of this pipeline is attached.
>
> It is doubtful that gldownload uses DMABuf in this case.
> If I use caps filter with 'memory:DMABuf' specified, negotiation fails.
>
> Alex

> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le mer. 22 août 2018 07:26, Víctor Jáquez <[hidden email]> a écrit :

On Wed, 22 Aug 2018 at 14:03, Alexander Yanin wrote:
> > I guess you could use glsrcbin or gldownload to convert a GLMemory based buffer
> > (with a texture inside) into a dmabuf-based buffer and vaapipostproc would
> > upload it into a VA surface and then encoding it.
>
> In 1.14 vaapipostproc only has 'video/x-raw(memory:VASurface)' and
> 'video/x-raw' sink pads, so there is no DMAbuf. Was it added in later
> versions of gstreamer?
> In case of DMAbuf input, does DMAbuf -> vaapi conversion implies copy operation?
> Is there any sense in adding DMAbuf sink pad directly to vaapi encoder
> base class?

Adding memory:DMABuf caps feature in vaapipostproc sink pad is not
implemented. BUT, dmabuf-based buffers don't need to use that caps feature
*if they are mappable to user-space*.

gldownload can pass dmabuf-based buffers without memory:DMABuf caps feature, as
far as I understand. And vaapipostproc can detect if that buffer is dmabuf-based
and upload it onto a VA surface (no copy is involved).

Sorry, atm it requires the caps feature and we have issues there since the dmabuf are tiled, event though the modifiers aren't set by mesa. The mesa dmabuf exporter is massively broken, so we'll have to disable this really soon in libgstgl. It worked till around Mesa 1.17, when the tiling and compression got activated by default for all internal texture backend.

There is no need to duplicate the code in vaapipostroc in the encoders, when you
can just prepend vaapiproc in the pipeline.

vmjl
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le mer. 22 août 2018 07:26, Víctor Jáquez <[hidden email]> a écrit :

On Wed, 22 Aug 2018 at 14:03, Alexander Yanin wrote:
> > I guess you could use glsrcbin or gldownload to convert a GLMemory based buffer
> > (with a texture inside) into a dmabuf-based buffer and vaapipostproc would
> > upload it into a VA surface and then encoding it.
>
> In 1.14 vaapipostproc only has 'video/x-raw(memory:VASurface)' and
> 'video/x-raw' sink pads, so there is no DMAbuf. Was it added in later
> versions of gstreamer?
> In case of DMAbuf input, does DMAbuf -> vaapi conversion implies copy operation?
> Is there any sense in adding DMAbuf sink pad directly to vaapi encoder
> base class?

Adding memory:DMABuf caps feature in vaapipostproc sink pad is not
implemented. BUT, dmabuf-based buffers don't need to use that caps feature
*if they are mappable to user-space*.

gldownload can pass dmabuf-based buffers without memory:DMABuf caps feature, as
far as I understand. And vaapipostproc can detect if that buffer is dmabuf-based
and upload it onto a VA surface (no copy is involved).

Sorry, atm it requires the caps feature and we have issues there since the dmabuf are tiled, event though the modifiers aren't set by mesa. The mesa dmabuf exporter is massively broken, so we'll have to disable this really soon in libgstgl. It work till around Mesa 1.17, when the tiling and compression got activated by default for all internal texture backend.

There is no need to duplicate the code in vaapipostroc in the encoders, when you
can just prepend vaapiproc in the pipeline.

vmjl
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Victor Jaquez]

On Wed, 22 Aug 2018 at 08:52, Nicolas Dufresne wrote:

> Le mer. 22 août 2018 07:26, Víctor Jáquez <[hidden email]> a écrit :
>
> > On Wed, 22 Aug 2018 at 14:03, Alexander Yanin wrote:
> > > > I guess you could use glsrcbin or gldownload to convert a GLMemory
> > based buffer
> > > > (with a texture inside) into a dmabuf-based buffer and vaapipostproc
> > would
> > > > upload it into a VA surface and then encoding it.
> > >
> > > In 1.14 vaapipostproc only has 'video/x-raw(memory:VASurface)' and
> > > 'video/x-raw' sink pads, so there is no DMAbuf. Was it added in later
> > > versions of gstreamer?
> > > In case of DMAbuf input, does DMAbuf -> vaapi conversion implies copy
> > operation?
> > > Is there any sense in adding DMAbuf sink pad directly to vaapi encoder
> > > base class?
> >
> > Adding memory:DMABuf caps feature in vaapipostproc sink pad is not
> > implemented. BUT, dmabuf-based buffers don't need to use that caps feature
> > *if they are mappable to user-space*.
> >
> > gldownload can pass dmabuf-based buffers without memory:DMABuf caps
> > feature, as
> > far as I understand. And vaapipostproc can detect if that buffer is
> > dmabuf-based
> > and upload it onto a VA surface (no copy is involved).
> >
>
> Sorry, atm it requires the caps feature and we have issues there since the
> dmabuf are tiled, event though the modifiers aren't set by mesa. The mesa
> dmabuf exporter is massively broken, so we'll have to disable this really
> soon in libgstgl. It work till around Mesa 1.17, when the tiling and
> compression got activated by default for all internal texture backend.

Thus, It is required to add the caps feature in vaapipostroc sink pad and its
machinery.

https://bugzilla.gnome.org/show_bug.cgi?id=797010

vmjl

>
>
> > There is no need to duplicate the code in vaapipostroc in the encoders,
> > when you
> > can just prepend vaapiproc in the pipeline.
> >
> > vmjl
> > _______________________________________________
> > gstreamer-devel mailing list
> > [hidden email]
> > https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
> >

> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le mer. 22 août 2018 08:57, Víctor Jáquez <[hidden email]> a écrit :

On Wed, 22 Aug 2018 at 15:12, Alexander Yanin wrote:
> > Adding memory:DMABuf caps feature in vaapipostproc sink pad is not
> > implemented. BUT, dmabuf-based buffers don't need to use that caps feature
> > *if they are mappable to user-space*.
>
> > gldownload can pass dmabuf-based buffers without memory:DMABuf caps feature, as
> > far as I understand. And vaapipostproc can detect if that buffer is dmabuf-based
> > and upload it onto a VA surface (no copy is involved).
>
> > There is no need to duplicate the code in vaapipostroc in the encoders, when you
> > can just prepend vaapiproc in the pipeline.
>
> How could gldownload figure out to pass DMABuf instead of raw memory
> if the elements negotiated video/x-raw caps? For example, I've tested
> this pipeline:

Indeed, there's now a code path in gldownload to verify if the generated buffers
are mappable.

I just quick hacked gldownload

diff --git a/ext/gl/gstgldownloadelement.c b/ext/gl/gstgldownloadelement.c
index aefc289ee..429bcb4d2 100644
--- a/ext/gl/gstgldownloadelement.c
+++ b/ext/gl/gstgldownloadelement.c
@@ -137,14 +137,14 @@ gst_gl_download_element_set_caps (GstBaseTransform * bt, GstCaps * in_caps,
   if (gst_caps_features_contains (features, GST_CAPS_FEATURE_MEMORY_GL_MEMORY)) {
     /* do nothing with the buffer */
 #if GST_GL_HAVE_PLATFORM_EGL && GST_GL_HAVE_DMABUF
-  } else if (gst_caps_features_contains (features,
-          GST_CAPS_FEATURE_MEMORY_DMABUF)) {
+  } else {
     dl->dmabuf_allocator = gst_dmabuf_allocator_new ();
-#endif
-  } else if (gst_caps_features_contains (features,
-          GST_CAPS_FEATURE_MEMORY_SYSTEM_MEMORY)) {
+  }
+#else
+  } else {
     dl->do_pbo_transfers = TRUE;
   }
+#endif

   return TRUE;
 }


And it seems to work.

VAAPI implements implicit modifiers, pass the same dmabuf to kmssink or waylandsink, you'll see that it breaks. Implicit modifiers are terrible idea which breaks so many use case. The dmabuf currently produced by vaapi Intel driver are only going to render correctly on an Intel GPU, it won't event work on Intel display because the display requires explicit modifiers. VAAPI needs to pass or negotiate the modifiers, meanwhile, we should be really careful with the vaapi dmabufs.

The other option is to implement in vaapipostproc the negotiation of
memory:DMABuf caps feature.

vmjl

>
> gst-launch-1.0 gltestsrc is-live=true ! glvideomixer ! gldownload !
> vaapipostproc ! vaapivp8enc ! webmmux ! filesink location=test.webm
>
> SVG of this pipeline is attached.
>
> It is doubtful that gldownload uses DMABuf in this case.
> If I use caps filter with 'memory:DMABuf' specified, negotiation fails.
>
> Alex


> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Odoacre]

> Thus, It is required to add the caps feature in vaapipostroc sink pad and its
> machinery.
>
> https://bugzilla.gnome.org/show_bug.cgi?id=797010
>
> vmjl

Ok, we will try to add it. Wouldn't it be more prudent to add a DMABuf
feature to the encoder tho? It really seems to me a bit clumsy
workaround to use an unneccessary intermediate element. I see that you
want to avoid duplication, but the post processing element before the
encoder inserted only for copyless buffer exchange seems very unobvious
and excessive.

Freyr
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel