White frames with v4l2 source & GstGLUploadElement

[Gottfried Haider]

Hello,

I am trying to make an application [1] that uses GStreamer's GL API in
a very similar way to the testegl.c [2] example work with v4l2
sources, but I am only getting white frames.

The code previously had a playbin with uri=file:///..., which worked.
Now I am trying uri=v4l2:///dev/video0, which points to a webcam.

Running "gst-launch-1.0 playbin uri=v4l2:///dev/video0" in the
terminal also works, and shows reasonable images from the camera.

Is there anything obvious that I am missing here? (I see the format is
YUY2 all the way to the end of the pipeline, is this perhaps what
needs changing?)

I uploaded a DOT file [3], as well as the debug output [4] when using
the webcam. The same for playing back a regular H.264 encoded video
file as well [5].

Thanks for any leads!

Best
Gottfried


[1] https://github.com/gohai/processing-glvideo/blob/master/src/native/impl.c
[2] https://cgit.freedesktop.org/gstreamer/gst-omx/tree/examples/egl/testegl.c
[3] http://sukzessiv.net/~gohai/gstreamer/v4l2-gl.png or
http://sukzessiv.net/~gohai/gstreamer/v4l2-gl.dot
[4] http://sukzessiv.net/~gohai/gstreamer/v4l2-gl.out
[5] http://sukzessiv.net/~gohai/gstreamer/file-gl.png or
http://sukzessiv.net/~gohai/gstreamer/file-gl.dot
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-4]

Le mardi 26 avril 2016 à 21:58 +0200, Gottfried Haider a écrit :

> I am trying to make an application [1] that uses GStreamer's GL API
> in
> a very similar way to the testegl.c [2] example work with v4l2
> sources, but I am only getting white frames.
>
> The code previously had a playbin with uri=file:///..., which worked.
> Now I am trying uri=v4l2:///dev/video0, which points to a webcam.
>
> Running "gst-launch-1.0 playbin uri=v4l2:///dev/video0" in the
> terminal also works, and shows reasonable images from the camera.
>
> Is there anything obvious that I am missing here? (I see the format
> is
> YUY2 all the way to the end of the pipeline, is this perhaps what
> needs changing?)
>
> I uploaded a DOT file [3], as well as the debug output [4] when using
> the webcam. The same for playing back a regular H.264 encoded video
> file as well [5].
>
> Thanks for any leads!

Sorry that it took a while before someone answer this one. There is
nothing obvious in your pipeline that could explain white frames.
Sounds like GL incompatibility issue. Which GStreamer version are you
using ? What is the paltform / GL stack involved here ?

For the from-file case, you'll notice you are using omxdecoder with
EGLImage support, that is much easyer to render on screen, as you only
have to bind the EGLImage to a texture and display it. While in the
v4l2 case, you have raw image that you need to upload and transform
using shaders.

cheers,
Nicolas
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Gottfried Haider]

Hi Nicolas,

Thanks for looking into this! I was using GStreamer 1.8.1 on a Raspberry Pi with the binary GLES2 driver.

As noted in a separate message to the list ("v4l2src & GstGLUpload"), I could reproduce this problem with the default testegl example in gst-omx as well. While playback of H.264 video files works just fine, there's only a blank frame when I try the v4l2src uri as part of the playbin pipeline.

I think the point that the OMX case is easier than the V4L2 one is an important one. I believe this difference is where things go amiss (perhaps in combination with the difficulties of GL context sharing).

One thing that strikes me as odd in this regard is for the V4L2 case, I am still see the message "Prepare texture for GLMemory", which comes from testegl.c:729 [1]. The testegl example is written to handle three types of buffers it gets handed to: memory:EGLImage, memory:GLMemory and GstVideoGLTextureUploadMeta. The GLMemory case is the only one that doesn't do the upload - it strikes me as odd that this would be the case for the v4l2src. Could it be that some other element in the pipeline is doing the GLUpload for us, but somehow being at odds with the context-sharing, or the fact that it is not Desktop GL?

Best

Gottfried

[1] https://cgit.freedesktop.org/gstreamer/gst-omx/tree/examples/egl/testegl.c#n729

On Sun, May 1, 2016 at 5:43 PM, Nicolas Dufresne <[hidden email]> wrote:

Le mardi 26 avril 2016 à 21:58 +0200, Gottfried Haider a écrit :
> I am trying to make an application [1] that uses GStreamer's GL API
> in
> a very similar way to the testegl.c [2] example work with v4l2
> sources, but I am only getting white frames.
>
> The code previously had a playbin with uri=file:///..., which worked.
> Now I am trying uri=v4l2:///dev/video0, which points to a webcam.
>
> Running "gst-launch-1.0 playbin uri=v4l2:///dev/video0" in the
> terminal also works, and shows reasonable images from the camera.
>
> Is there anything obvious that I am missing here? (I see the format
> is
> YUY2 all the way to the end of the pipeline, is this perhaps what
> needs changing?)
>
> I uploaded a DOT file [3], as well as the debug output [4] when using
> the webcam. The same for playing back a regular H.264 encoded video
> file as well [5].
>
> Thanks for any leads!

Sorry that it took a while before someone answer this one. There is
nothing obvious in your pipeline that could explain white frames.
Sounds like GL incompatibility issue. Which GStreamer version are you
using ? What is the paltform / GL stack involved here ?

For the from-file case, you'll notice you are using omxdecoder with
EGLImage support, that is much easyer to render on screen, as you only
have to bind the EGLImage to a texture and display it. While in the
v4l2 case, you have raw image that you need to upload and transform
using shaders.

cheers,
Nicolas
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le dimanche 01 mai 2016 à 23:19 +0200, Gottfried Haider a écrit :

> One thing that strikes me as odd in this regard is for the V4L2 case,
> I am still see the message "Prepare texture for GLMemory", which
> comes from testegl.c:729 [1]. The testegl example is written to
> handle three types of buffers it gets handed
> to: memory:EGLImage, memory:GLMemory and GstVideoGLTextureUploadMeta.
> The GLMemory case is the only one that doesn't do the upload - it
> strikes me as odd that this would be the case for the v4l2src. Could
> it be that some other element in the pipeline is doing the GLUpload
> for us, but somehow being at odds with the context-sharing, or the
> fact that it is not Desktop GL?

V4L2src does not have any relationship with GL. The element that does
the upload is called glupload. What is likely your issue, is that
someone broke older GLES support while adding version 3 support. Can
you reproduce with simple pipeline like ?

 gst-launch-1.0 v4l2src ! glimagesink

Nicolas


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Gottfried Haider]

> V4L2src does not have any relationship with GL. The element that does
> the upload is called glupload. What is likely your issue, is that
> someone broke older GLES support while adding version 3 support. Can
> you reproduce with simple pipeline like ?

Will test tomorrow morning!

What I meant: I was expecting the testegl code to handle the uploading
of the buffer, since it is clearly written to do exactly that. So I
found it curious that the "raw image" already shows up at the callback
for the "handoff" signal as a GLMemory.

Thanks!
Gottfried
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Gottfried Haider]

> What is likely your issue, is that
> someone broke older GLES support while adding version 3 support. Can
> you reproduce with simple pipeline like ?
>
>  gst-launch-1.0 v4l2src ! glimagesink

This works. Anything else I could try?

Best
Gottfried
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le lundi 02 mai 2016 à 08:43 +0200, Gottfried Haider a écrit :
> > What is likely your issue, is that
> > someone broke older GLES support while adding version 3 support.
> Can
> > you reproduce with simple pipeline like ?
> >
> >  gst-launch-1.0 v4l2src ! glimagesink
>
> This works. Anything else I could try?

Looking at your pipeline again, you probably want to convert the
texture to RGBA before you pass that texture to your Java application.
For this, add glcolorconvert ! capsfilter caps=.... Otherwise the
texture type won't match what a normal application would expect. On
some platform, the texture type we use for intermediate format can't be
rendered. You could also consider using the signal on glimagesink
instead.

Nicolas
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Gottfried Haider]

Nicolas,

> Looking at your pipeline again, you probably want to convert the
> texture to RGBA before you pass that texture to your Java application.
> For this, add glcolorconvert ! capsfilter caps=.... Otherwise the
> texture type won't match what a normal application would expect. On
> some platform, the texture type we use for intermediate format can't be
> rendered. You could also consider using the signal on glimagesink
> instead.

Thanks, glcolorconvert ahead of the capsfilter did the trick!

Best
Gottfried
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel