gstglcolorconvert output buffer format

[Bruce Wheaton]

Hey - new to GStreamer, which I’m planning on switching to since it includes (better) versions of a bunch of code I wrote a few years ago (ffmpeg based).Thanks for an awesome library!

So - one thing I need is high bit depth precision, but I’d love to take advantage of glColorConvert and friends. From my reading of gstglcolorconvert.c, an FBO is created, and a texture is attached to it, and that texture is 8-bit per color, unsigned byte:

1910  /* a fake texture is attached to the convert FBO (cannot init without it) */
1911  gl->GenTextures (1, &fake_texture);
1912  gl->BindTexture (GL_TEXTURE_2D, fake_texture);
1913  internal_format =
1914      gst_gl_sized_gl_format_from_gl_format_type (convert->context, GL_RGBA,
1915      GL_UNSIGNED_BYTE);
1916  gl->TexImage2D (GL_TEXTURE_2D, 0, internal_format, out_width, out_height, 0,
1917      GL_RGBA, GL_UNSIGNED_BYTE, NULL);

It does call it a fake texture though, is there somewhere else (I don’t see any more texture generation), not sure if that’s replaced?

Anyway - I can change the code to use float or half float (16-bit float GL format), but I’m not sure how best to fit it into the GStreamer system - still reading up on that. Also, can someone please point me to where/how to submit patches? Here on the list, or in source control?

In my head, I expected that element to look at caps and create an appropriate format based on that?

Thanks for any help,

Bruce Wheaton





_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Matthew Waters]

On 13/01/17 13:14, Bruce Wheaton wrote:

> Hey - new to GStreamer, which I’m planning on switching to since it includes (better) versions of a bunch of code I wrote a few years ago (ffmpeg based).Thanks for an awesome library!
>
> So - one thing I need is high bit depth precision, but I’d love to take advantage of glColorConvert and friends. From my reading of gstglcolorconvert.c, an FBO is created, and a texture is attached to it, and that texture is 8-bit per color, unsigned byte:
>
> 1910  /* a fake texture is attached to the convert FBO (cannot init without it) */
> 1911  gl->GenTextures (1, &fake_texture);
> 1912  gl->BindTexture (GL_TEXTURE_2D, fake_texture);
> 1913  internal_format 1914      gst_gl_sized_gl_format_from_gl_format_type (convert->context, GL_RGBA,
> 1915      GL_UNSIGNED_BYTE);
> 1916  gl->TexImage2D (GL_TEXTURE_2D, 0, internal_format, out_width, out_height, 0,
> 1917      GL_RGBA, GL_UNSIGNED_BYTE, NULL);
>
> It does call it a fake texture though, is there somewhere else (I don’t see any more texture generation), not sure if that’s replaced?

The fake texture is simply there to initialize the FBO, it is replaced
by the actual texture later in
https://cgit.freedesktop.org/gstreamer/gst-plugins-bad/tree/gst-libs/gst/gl/gstglcolorconvert.c#n2473.

> Anyway - I can change the code to use float or half float (16-bit float GL format), but I’m not sure how best to fit it into the GStreamer system - still reading up on that. Also, can someone please point me to where/how to submit patches? Here on the list, or in source control?

Bugs and patches are tracked in gnome bugzilla here:
https://bugzilla.gnome.org/enter_bug.cgi?product=GStreamer.  You can
submit patches there.

> In my head, I expected that element to look at caps and create an appropriate format based on that?

That's exactly how this would work :).

The first step is to add the necessary information to the libgstvideo
library for the float/half-float formats.
Second step is adding the float/half-float textures to gstglformat for
converting between GstVideoFormat and the necessary GL tokens.
Then you can use those formats from glcolorconvert.

Cheers
-Matt

> Thanks for any help,
>
> Bruce Wheaton


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Bruce Wheaton]

Thank you for a detailed response - very kind.

Anyway - I can change the code to use float or half float (16-bit float GL format), but I’m not sure how best to fit it into the GStreamer system - still reading up on that. Also, can someone please point me to where/how to submit patches? Here on the list, or in source control?

Bugs and patches are tracked in gnome bugzilla here:
https://bugzilla.gnome.org/enter_bug.cgi?product=GStreamer.  You can
submit patches there.

In my head, I expected that element to look at caps and create an appropriate format based on that?

That's exactly how this would work :).

Splendid! That’s what I hoped someone would say.

The first step is to add the necessary information to the libgstvideo
library for the float/half-float formats.
Second step is adding the float/half-float textures to gstglformat for
converting between GstVideoFormat and the necessary GL tokens.
Then you can use those formats from glcolorconvert.

Great - there’s a lot of deja-vu looking in there but at least I have some idea of what I’m looking for…

So, in gst-plugins-base, video/video-format.h seems to be where the new format should go?

And it looks like it needs to go in a few places…

Then video-info.c as well? And video-converter.c…. And video scale.c? :-(   Oh boy.

Is a format patch going to get in trouble because it isn’t supported well in software? Scaling and conversions, for instance? 

gstglformat.h seems a little more straightforward… Can probably work that out.

I’ll start on changes and a patch. If I diff against a tagged release in GIT, that works, I presume?

So - more of a can of worms: has there been discussion of compressed image formats? A couple of codecs, notably and most usefully the HAP family are compressed GL formats. I guess I’d have to add support here, or maybe manually handle those frames as a raw format of some kind?

The more complicated is HapQ[A] which is: intraframe only, snappy compression (RLE) on each frame, planar (if alpha is included), GL compressed formats in a custom color space. RCoCg afair.

I’m sure it would have to be a separate patch, as I’m sure there would be a lot of wrinkles to that. Has anyone else been requesting these codecs or formats and handled it already? I suspect libav would handle it through software but that totally defeats the object.

Bruce

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Matthew Waters]

On 14/01/17 06:54, Bruce Wheaton wrote:

Thank you for a detailed response - very kind.

Anyway - I can change the code to use float or half float (16-bit float GL format), but I’m not sure how best to fit it into the GStreamer system - still reading up on that. Also, can someone please point me to where/how to submit patches? Here on the list, or in source control?

Bugs and patches are tracked in gnome bugzilla here:
https://bugzilla.gnome.org/enter_bug.cgi?product=GStreamer.  You can
submit patches there.

In my head, I expected that element to look at caps and create an appropriate format based on that?

That's exactly how this would work :).

Splendid! That’s what I hoped someone would say.

The first step is to add the necessary information to the libgstvideo
library for the float/half-float formats.
Second step is adding the float/half-float textures to gstglformat for
converting between GstVideoFormat and the necessary GL tokens.
Then you can use those formats from glcolorconvert.

Great - there’s a lot of deja-vu looking in there but at least I have some idea of what I’m looking for…

So, in gst-plugins-base, video/video-format.h seems to be where the new format should go?

And it looks like it needs to go in a few places…

Then video-info.c as well? And video-converter.c…. And video scale.c? :-(   Oh boy.

Once you have video-converter, you can convert to something else to scale if necessary (potentially at a loss though).

Is a format patch going to get in trouble because it isn’t supported well in software? Scaling and conversions, for instance? 

gstglformat.h seems a little more straightforward… Can probably work that out.

I’ll start on changes and a patch. If I diff against a tagged release in GIT, that works, I presume?

Sure, we use git format-patch for patches.

So - more of a can of worms: has there been discussion of compressed image formats? A couple of codecs, notably and most usefully the HAP family are compressed GL formats. I guess I’d have to add support here, or maybe manually handle those frames as a raw format of some kind?

The more complicated is HapQ[A] which is: intraframe only, snappy compression (RLE) on each frame, planar (if alpha is included), GL compressed formats in a custom color space. RCoCg afair.

I’m sure it would have to be a separate patch, as I’m sure there would be a lot of wrinkles to that. Has anyone else been requesting these codecs or formats and handled it already? I suspect libav would handle it through software but that totally defeats the object.

No GL compressed format has been implemented in GStreamer AFAIK.  The two options for implementing them involve adding formats (similar to the float/half-float RGBA formats) if they can be used transparently in GstGLMemory as non-compressed textures or creating encoding/decoding GStreamer elements to convert between the different formats.  I haven't looked too closely at the requirements for them either so don't know which would be the best option.

Cheers
-Matt

Bruce

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel