What's the meaning of video/x-raw(memory:GLMemory) without texture-target?
Locked
What's the meaning of video/x-raw(memory:GLMemory) without texture-target?
 
|
Hey all,
on a Raspberry Pi 3b+ gst-inspect-1.0 shows the following src for omxh264dec: ``` video/x-raw(memory:GLMemory) format: RGBA width: [ 1, 2147483647 ] height: [ 1, 2147483647 ] framerate: [ 0/1, 2147483647/1 ] ``` Note that this is missing a texture-target specification. What kind of memory is this, and how can I leverage it further down? I would like to efficiently change video frames using an OpenGL filter. If we take gleffects-twirl as an example, it expects the following as sink: ``` video/x-raw(memory:GLMemory): format: RGBA width: [ 1, 2147483647 ] height: [ 1, 2147483647 ] framerate: [ 0/1, 2147483647/1 ] texture-target: 2D ``` How can I connect the two together? If I try, to add the filter directly after the decoder in a pipeline, I get: ``` ERROR: from element /GstPipeline:pipeline0/GstOMXH264Dec- omxh264dec:omxh264dec-omxh264dec0: Internal data stream error. Additional debug info: ../../gst-omx-1.16.1/omx/gstomxvideodec.c(1927): gst_omx_video_dec_loop (): / GstPipeline:pipeline0/GstOMXH264Dec-omxh264dec:omxh264dec-omxh264dec0: stream stopped, reason not-negotiated ``` Can someone sched some light on this? How can one post-process the GLMemory in a gl video filter? Is the missing texture-target a bug in omxh264dec, or can I somehow make the raw GLMemory a texture? Or does this mean the frame lives in some other form of non-texture memory that can be displayed, but not otherwise accessed by OpenGL for video filter purposes? Thanks PS: In case you are wondering what we are trying to achieve on a higher level: rpicamsrc -> h264 -> omxh264dec -> glmemory -> custom gl filter to add a non- static overlay to every frame -> tee to display sink and then also a h264- encoded avi record sink -- Milian Wolff | [hidden email] | Senior Software Engineer KDAB (Deutschland) GmbH, a KDAB Group company Tel: +49-30-521325470 KDAB - The Qt, C++ and OpenGL Experts _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
smime.p7s (5K)
Re: What's the meaning of video/x-raw(memory:GLMemory) without texture-target?
|
|
Hi,
By default a 'video/x-raw(memory:GLMemory)' caps without the texture-target field is equivalent to texture-target=2D as that was the default before we differentiated the different OpenGL texture targets. Now elements should be able to negotiate the differences between them correctly however there may be bugs. The GLMemory output from omxh264dec is highly dependant on the platform you are running on and may or may not work correctly. I know that it used to work on the RPi and have not tested any other omx platform myself. The first step to debugging this would be to enable GStreamer debug logging to get an idea of what may be wrong. Cheers -Matt On 22/11/19 3:10 am, Milian Wolff
wrote:
_______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
Re: What's the meaning of video/x-raw(memory:GLMemory) without texture-target?
|
|
On Friday, November 22, 2019 5:12:57 AM CET Matthew Waters wrote:
> Hi, Hey Matt! > By default a 'video/x-raw(memory:GLMemory)' caps without the > texture-target field is equivalent to texture-target=2D as that was the > default before we differentiated the different OpenGL texture targets. > Now elements should be able to negotiate the differences between them > correctly however there may be bugs. OK, thanks - good to know! > The GLMemory output from omxh264dec is highly dependant on the platform > you are running on and may or may not work correctly. I know that it > used to work on the RPi and have not tested any other omx platform myself. What's interesting is that this seems to work fine when I shove the data directly into the glimagesink, i.e.: ``` # GST_DEBUG=3 gst-launch-1.0 rpicamsrc ! "video/x-h264,width=640,height=480" ! h264parse ! omxh264dec ! "video/x-raw(memory:GLMemory)" ! queue ! glimagesinkSetting pipeline to PAUSED ... Pipeline is live and does not need PREROLL ... 0:00:00.488648854 31116 0x167e7b0 FIXME default gstutils.c: 3981:gst_pad_create_stream_id_internal:<rpicamsrc0:src> Creating random stream-id, consider implementing a deterministic way of creating a stream-id Got context from element 'sink': gst.gl.GLDisplay=context, gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayEGL\)\ gldisplayegl0"; Setting pipeline to PLAYING ... New clock: GstSystemClock 0:00:00.894124323 31116 0x167e7b0 ERROR omxvideodec gstomxvideodec.c:2079:gst_omx_video_dec_negotiate:<omxh264dec-omxh264dec0> Empty caps 0:00:00.985837604 31116 0x167e230 FIXME glslstage gstglslstage.c:518:_compile_shader:<glslstage0> vertex shader info log:Compiled 0:00:00.990119375 31116 0x167e230 FIXME glslstage gstglslstage.c:518:_compile_shader:<glslstage1> fragment shader info log:Compiled 0:00:01.013341718 31116 0x167e230 FIXME glslstage gstglslstage.c:518:_compile_shader:<glslstage2> vertex shader info log:Compiled 0:00:01.014671666 31116 0x167e230 FIXME glslstage gstglslstage.c:518:_compile_shader:<glslstage3> fragment shader info log:Compiled ^Chandling interrupt. Interrupt: Stopping pipeline ... Execution ended after 0:00:06.801416612 Setting pipeline to PAUSED ... Setting pipeline to READY ... Setting pipeline to NULL ... Freeing pipeline ... ``` I don't know what the ERROR is trying to tell me, but this pipeline works a charm. > The first step to debugging this would be to enable GStreamer debug > logging to get an idea of what may be wrong. But the following happens when I add the gleffects_twirl filter (either before or after the queue, doesn't matter): ``` # GST_DEBUG=3 gst-launch-1.0 rpicamsrc ! "video/x-h264,width=640,height=480" ! h264parse ! omxh264dec ! "video/x-raw(memory:GLMemory)" ! gleffects_twirl ! queue ! glimagesink Setting pipeline to PAUSED ... Pipeline is live and does not need PREROLL ... 0:00:00.506736041 31143 0x1fed490 FIXME default gstutils.c: 3981:gst_pad_create_stream_id_internal:<rpicamsrc0:src> Creating random stream-id, consider implementing a deterministic way of creating a stream-id Got context from element 'sink': gst.gl.GLDisplay=context, gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayEGL\)\ gldisplayegl0"; Setting pipeline to PLAYING ... New clock: GstSystemClock 0:00:00.882512708 31143 0x1fed490 ERROR omxvideodec gstomxvideodec.c:2079:gst_omx_video_dec_negotiate:<omxh264dec-omxh264dec0> Empty caps 0:00:00.968698020 31143 0x6fd03290 WARN videodecoder gstvideodecoder.c:3783:gst_video_decoder_negotiate_pool:<omxh264dec- omxh264dec0> Subclass failed to decide allocation 0:00:00.969139166 31143 0x6fd03290 WARN GST_PADS gstpad.c: 4231:gst_pad_peer_query:<omxh264dec-omxh264dec0:src> could not send sticky events 0:00:00.969251770 31143 0x6fd03290 WARN videodecoder gstvideodecoder.c:3783:gst_video_decoder_negotiate_pool:<omxh264dec- omxh264dec0> Subclass failed to decide allocation 0:00:00.972893854 31143 0x6fd03290 WARN GST_PADS gstpad.c: 4231:gst_pad_peer_query:<omxh264dec-omxh264dec0:src> could not send sticky events 0:00:00.988252083 31143 0x6fd03290 WARN omxvideodec gstomxvideodec.c:1927:gst_omx_video_dec_loop:<omxh264dec-omxh264dec0> error: Internal data stream error. 0:00:00.988374999 31143 0x6fd03290 WARN omxvideodec gstomxvideodec.c:1927:gst_omx_video_dec_loop:<omxh264dec-omxh264dec0> error: stream stopped, reason not-negotiated 0:00:00.988662604 31143 0x1fed490 WARN basesrc gstbasesrc.c: 3072:gst_base_src_loop:<rpicamsrc0> error: Internal data stream error. 0:00:00.988741718 31143 0x1fed490 WARN basesrc gstbasesrc.c: 3072:gst_base_src_loop:<rpicamsrc0> error: streaming stopped, reason not- negotiated (-4) ERROR: from element /GstPipeline:pipeline0/GstOMXH264Dec- omxh264dec:omxh264dec-omxh264dec0: Internal data stream error. Additional debug info: ../../gst-omx-1.16.1/omx/gstomxvideodec.c(1927): gst_omx_video_dec_loop (): / GstPipeline:pipeline0/GstOMXH264Dec-omxh264dec:omxh264dec-omxh264dec0: stream stopped, reason not-negotiated Execution ended after 0:00:00.478420208 Setting pipeline to PAUSED ... Setting pipeline to READY ... Setting pipeline to NULL ... Freeing pipeline ... ``` Do you have an idea of what's going on here? Thanks > On 22/11/19 3:10 am, Milian Wolff wrote: > > Hey all, > > > > on a Raspberry Pi 3b+ gst-inspect-1.0 shows the following src for > > omxh264dec: > > > > ``` > > > > video/x-raw(memory:GLMemory) > > > > format: RGBA > > > > width: [ 1, 2147483647 ] > > > > height: [ 1, 2147483647 ] > > > > framerate: [ 0/1, 2147483647/1 ] > > > > ``` > > > > Note that this is missing a texture-target specification. What kind of > > memory is this, and how can I leverage it further down? I would like to > > efficiently change video frames using an OpenGL filter. If we take > > gleffects-twirl as an example, it expects the following as sink: > > > > > > ``` > > > > video/x-raw(memory:GLMemory): > > format: RGBA > > > > width: [ 1, 2147483647 ] > > > > height: [ 1, 2147483647 ] > > > > framerate: [ 0/1, 2147483647/1 ] > > > > texture-target: 2D > > > > ``` > > > > How can I connect the two together? If I try, to add the filter directly > > after the decoder in a pipeline, I get: > > > > ``` > > ERROR: from element /GstPipeline:pipeline0/GstOMXH264Dec- > > omxh264dec:omxh264dec-omxh264dec0: Internal data stream error. > > Additional debug info: > > ../../gst-omx-1.16.1/omx/gstomxvideodec.c(1927): gst_omx_video_dec_loop > > (): / > > GstPipeline:pipeline0/GstOMXH264Dec-omxh264dec:omxh264dec-omxh264dec0: > > stream stopped, reason not-negotiated > > ``` > > > > Can someone sched some light on this? How can one post-process the > > GLMemory in a gl video filter? Is the missing texture-target a bug in > > omxh264dec, or can I somehow make the raw GLMemory a texture? Or does > > this mean the frame lives in some other form of non-texture memory that > > can be displayed, but not otherwise accessed by OpenGL for video filter > > purposes? > > > > Thanks > > > > PS: In case you are wondering what we are trying to achieve on a higher > > level: > > > > rpicamsrc -> h264 -> omxh264dec -> glmemory -> custom gl filter to add a > > non- static overlay to every frame -> tee to display sink and then also a > > h264- encoded avi record sink > > > > _______________________________________________ > > gstreamer-devel mailing list > > [hidden email] > > https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel -- Milian Wolff | [hidden email] | Senior Software Engineer KDAB (Deutschland) GmbH, a KDAB Group company Tel: +49-30-521325470 KDAB - The Qt, C++ and OpenGL Experts _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
Re: What's the meaning of video/x-raw(memory:GLMemory) without texture-target?
|
|
Try adding a glupload ! glcolorconvert before the gleffects_twirl
element. glimagesink contains these automatically already. On 25/11/19 8:30 pm, Milian Wolff wrote: > On Friday, November 22, 2019 5:12:57 AM CET Matthew Waters wrote: >> Hi, > Hey Matt! > >> By default a 'video/x-raw(memory:GLMemory)' caps without the >> texture-target field is equivalent to texture-target=2D as that was the >> default before we differentiated the different OpenGL texture targets. >> Now elements should be able to negotiate the differences between them >> correctly however there may be bugs. > OK, thanks - good to know! > >> The GLMemory output from omxh264dec is highly dependant on the platform >> you are running on and may or may not work correctly. I know that it >> used to work on the RPi and have not tested any other omx platform myself. > What's interesting is that this seems to work fine when I shove the data > directly into the glimagesink, i.e.: > > ``` > # GST_DEBUG=3 gst-launch-1.0 rpicamsrc ! "video/x-h264,width=640,height=480" ! > h264parse ! omxh264dec ! "video/x-raw(memory:GLMemory)" ! queue ! > glimagesinkSetting pipeline to PAUSED ... > Pipeline is live and does not need PREROLL ... > 0:00:00.488648854 31116 0x167e7b0 FIXME default gstutils.c: > 3981:gst_pad_create_stream_id_internal:<rpicamsrc0:src> Creating random > stream-id, consider implementing a deterministic way of creating a stream-id > Got context from element 'sink': gst.gl.GLDisplay=context, > gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayEGL\)\ gldisplayegl0"; > Setting pipeline to PLAYING ... > New clock: GstSystemClock > 0:00:00.894124323 31116 0x167e7b0 ERROR omxvideodec > gstomxvideodec.c:2079:gst_omx_video_dec_negotiate:<omxh264dec-omxh264dec0> > Empty caps > 0:00:00.985837604 31116 0x167e230 FIXME glslstage > gstglslstage.c:518:_compile_shader:<glslstage0> vertex shader info > log:Compiled > 0:00:00.990119375 31116 0x167e230 FIXME glslstage > gstglslstage.c:518:_compile_shader:<glslstage1> fragment shader info > log:Compiled > 0:00:01.013341718 31116 0x167e230 FIXME glslstage > gstglslstage.c:518:_compile_shader:<glslstage2> vertex shader info > log:Compiled > 0:00:01.014671666 31116 0x167e230 FIXME glslstage > gstglslstage.c:518:_compile_shader:<glslstage3> fragment shader info > log:Compiled > ^Chandling interrupt. > Interrupt: Stopping pipeline ... > Execution ended after 0:00:06.801416612 > Setting pipeline to PAUSED ... > Setting pipeline to READY ... > Setting pipeline to NULL ... > Freeing pipeline ... > ``` > > I don't know what the ERROR is trying to tell me, but this pipeline works a > charm. > >> The first step to debugging this would be to enable GStreamer debug >> logging to get an idea of what may be wrong. > But the following happens when I add the gleffects_twirl filter (either before > or after the queue, doesn't matter): > > ``` > # GST_DEBUG=3 gst-launch-1.0 rpicamsrc ! "video/x-h264,width=640,height=480" ! > h264parse ! omxh264dec ! "video/x-raw(memory:GLMemory)" ! gleffects_twirl ! > queue ! glimagesink > Setting pipeline to PAUSED ... > Pipeline is live and does not need PREROLL ... > 0:00:00.506736041 31143 0x1fed490 FIXME default gstutils.c: > 3981:gst_pad_create_stream_id_internal:<rpicamsrc0:src> Creating random > stream-id, consider implementing a deterministic way of creating a stream-id > Got context from element 'sink': gst.gl.GLDisplay=context, > gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayEGL\)\ gldisplayegl0"; > Setting pipeline to PLAYING ... > New clock: GstSystemClock > 0:00:00.882512708 31143 0x1fed490 ERROR omxvideodec > gstomxvideodec.c:2079:gst_omx_video_dec_negotiate:<omxh264dec-omxh264dec0> > Empty caps > 0:00:00.968698020 31143 0x6fd03290 WARN videodecoder > gstvideodecoder.c:3783:gst_video_decoder_negotiate_pool:<omxh264dec- > omxh264dec0> Subclass failed to decide allocation > 0:00:00.969139166 31143 0x6fd03290 WARN GST_PADS gstpad.c: > 4231:gst_pad_peer_query:<omxh264dec-omxh264dec0:src> could not send sticky > events > 0:00:00.969251770 31143 0x6fd03290 WARN videodecoder > gstvideodecoder.c:3783:gst_video_decoder_negotiate_pool:<omxh264dec- > omxh264dec0> Subclass failed to decide allocation > 0:00:00.972893854 31143 0x6fd03290 WARN GST_PADS gstpad.c: > 4231:gst_pad_peer_query:<omxh264dec-omxh264dec0:src> could not send sticky > events > 0:00:00.988252083 31143 0x6fd03290 WARN omxvideodec > gstomxvideodec.c:1927:gst_omx_video_dec_loop:<omxh264dec-omxh264dec0> error: > Internal data stream error. > 0:00:00.988374999 31143 0x6fd03290 WARN omxvideodec > gstomxvideodec.c:1927:gst_omx_video_dec_loop:<omxh264dec-omxh264dec0> error: > stream stopped, reason not-negotiated > 0:00:00.988662604 31143 0x1fed490 WARN basesrc gstbasesrc.c: > 3072:gst_base_src_loop:<rpicamsrc0> error: Internal data stream error. > 0:00:00.988741718 31143 0x1fed490 WARN basesrc gstbasesrc.c: > 3072:gst_base_src_loop:<rpicamsrc0> error: streaming stopped, reason not- > negotiated (-4) > ERROR: from element /GstPipeline:pipeline0/GstOMXH264Dec- > omxh264dec:omxh264dec-omxh264dec0: Internal data stream error. > Additional debug info: > ../../gst-omx-1.16.1/omx/gstomxvideodec.c(1927): gst_omx_video_dec_loop (): / > GstPipeline:pipeline0/GstOMXH264Dec-omxh264dec:omxh264dec-omxh264dec0: > stream stopped, reason not-negotiated > Execution ended after 0:00:00.478420208 > Setting pipeline to PAUSED ... > Setting pipeline to READY ... > Setting pipeline to NULL ... > Freeing pipeline ... > ``` > > Do you have an idea of what's going on here? > > Thanks > >> On 22/11/19 3:10 am, Milian Wolff wrote: >>> Hey all, >>> >>> on a Raspberry Pi 3b+ gst-inspect-1.0 shows the following src for >>> omxh264dec: >>> >>> ``` >>> >>> video/x-raw(memory:GLMemory) >>> >>> format: RGBA >>> >>> width: [ 1, 2147483647 ] >>> >>> height: [ 1, 2147483647 ] >>> >>> framerate: [ 0/1, 2147483647/1 ] >>> >>> ``` >>> >>> Note that this is missing a texture-target specification. What kind of >>> memory is this, and how can I leverage it further down? I would like to >>> efficiently change video frames using an OpenGL filter. If we take >>> gleffects-twirl as an example, it expects the following as sink: >>> >>> >>> ``` >>> >>> video/x-raw(memory:GLMemory): >>> format: RGBA >>> >>> width: [ 1, 2147483647 ] >>> >>> height: [ 1, 2147483647 ] >>> >>> framerate: [ 0/1, 2147483647/1 ] >>> >>> texture-target: 2D >>> >>> ``` >>> >>> How can I connect the two together? If I try, to add the filter directly >>> after the decoder in a pipeline, I get: >>> >>> ``` >>> ERROR: from element /GstPipeline:pipeline0/GstOMXH264Dec- >>> omxh264dec:omxh264dec-omxh264dec0: Internal data stream error. >>> Additional debug info: >>> ../../gst-omx-1.16.1/omx/gstomxvideodec.c(1927): gst_omx_video_dec_loop >>> (): / >>> GstPipeline:pipeline0/GstOMXH264Dec-omxh264dec:omxh264dec-omxh264dec0: >>> stream stopped, reason not-negotiated >>> ``` >>> >>> Can someone sched some light on this? How can one post-process the >>> GLMemory in a gl video filter? Is the missing texture-target a bug in >>> omxh264dec, or can I somehow make the raw GLMemory a texture? Or does >>> this mean the frame lives in some other form of non-texture memory that >>> can be displayed, but not otherwise accessed by OpenGL for video filter >>> purposes? >>> >>> Thanks >>> >>> PS: In case you are wondering what we are trying to achieve on a higher >>> level: >>> >>> rpicamsrc -> h264 -> omxh264dec -> glmemory -> custom gl filter to add a >>> non- static overlay to every frame -> tee to display sink and then also a >>> h264- encoded avi record sink >>> >>> _______________________________________________ >>> gstreamer-devel mailing list >>> [hidden email] >>> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel > _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
Re: What's the meaning of video/x-raw(memory:GLMemory) without texture-target?
|
|
On Monday, November 25, 2019 11:20:00 AM CET Matthew Waters wrote:
> Try adding a glupload ! glcolorconvert before the gleffects_twirl > element. glimagesink contains these automatically already. Thanks! This fixes the errors but it leaves me equally puzzled: Most notably, the twirl effect isn't taking effect, i.e. this is my command and debug output now: ``` # GST_DEBUG=3 gst-launch-1.0 rpicamsrc ! "video/x-h264,width=640,height=480" ! h264parse ! omxh264dec ! "video/x-raw(memory:GLMemory)" ! glupload ! gleffects_twirl ! queue ! glimagesink Setting pipeline to PAUSED ... Pipeline is live and does not need PREROLL ... 0:00:00.489548698 381 0x1c45af0 FIXME default gstutils.c: 3981:gst_pad_create_stream_id_internal:<rpicamsrc0:src> Creating random stream-id, consider implementing a deterministic way of creating a stream-id Got context from element 'sink': gst.gl.GLDisplay=context, gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayEGL\)\ gldisplayegl0"; Setting pipeline to PLAYING ... New clock: GstSystemClock 0:00:00.880536979 381 0x1c45af0 ERROR omxvideodec gstomxvideodec.c:2079:gst_omx_video_dec_negotiate:<omxh264dec-omxh264dec0> Empty caps 0:00:01.057018906 381 0x1c2f750 FIXME glslstage gstglslstage.c:518:_compile_shader:<glslstage0> vertex shader info log:Compiled 0:00:01.067645364 381 0x1c2f750 FIXME glslstage gstglslstage.c:518:_compile_shader:<glslstage1> fragment shader info log:Compiled 0:00:01.125377187 381 0x1c2f750 FIXME glslstage gstglslstage.c:518:_compile_shader:<glslstage2> vertex shader info log:Compiled 0:00:01.127863541 381 0x1c2f750 FIXME glslstage gstglslstage.c:518:_compile_shader:<glslstage3> fragment shader info log:Compiled 0:00:01.160143698 381 0x1c2f750 FIXME glslstage gstglslstage.c:518:_compile_shader:<glslstage4> vertex shader info log:Compiled 0:00:01.161840260 381 0x1c2f750 FIXME glslstage gstglslstage.c:518:_compile_shader:<glslstage5> fragment shader info log:Compiled ``` But the camera image is displayed as-is. What's odd is that I seem to be seeing the twirl effect for a very short time when I "Ctrl+C" terminate the `gst-launch` process. If I use `gltestsrc` as a replacement for the source up to `omxh264dec`, I see the twirl effect directly. What's going on here? Secondly, could you please explain to me why the `glupload ! glcolorconvert` is required here? Most notably, `video/x-raw(GLMemory)` should be on the GPU already, why do I need to "upload" it again? And the format is already `RGBA`, so why do I need glcolorconvert? Thanks > On 25/11/19 8:30 pm, Milian Wolff wrote: > > On Friday, November 22, 2019 5:12:57 AM CET Matthew Waters wrote: > >> Hi, > > > > Hey Matt! > > > >> By default a 'video/x-raw(memory:GLMemory)' caps without the > >> texture-target field is equivalent to texture-target=2D as that was the > >> default before we differentiated the different OpenGL texture targets. > >> Now elements should be able to negotiate the differences between them > >> correctly however there may be bugs. > > > > OK, thanks - good to know! > > > >> The GLMemory output from omxh264dec is highly dependant on the platform > >> you are running on and may or may not work correctly. I know that it > >> used to work on the RPi and have not tested any other omx platform > >> myself. > > > > What's interesting is that this seems to work fine when I shove the data > > directly into the glimagesink, i.e.: > > > > ``` > > # GST_DEBUG=3 gst-launch-1.0 rpicamsrc ! > > "video/x-h264,width=640,height=480" ! h264parse ! omxh264dec ! > > "video/x-raw(memory:GLMemory)" ! queue ! > > glimagesinkSetting pipeline to PAUSED ... > > Pipeline is live and does not need PREROLL ... > > 0:00:00.488648854 31116 0x167e7b0 FIXME default > > gstutils.c: > > 3981:gst_pad_create_stream_id_internal:<rpicamsrc0:src> Creating random > > stream-id, consider implementing a deterministic way of creating a > > stream-id Got context from element 'sink': gst.gl.GLDisplay=context, > > gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayEGL\)\ gldisplayegl0"; > > Setting pipeline to PLAYING ... > > New clock: GstSystemClock > > 0:00:00.894124323 31116 0x167e7b0 ERROR omxvideodec > > gstomxvideodec.c:2079:gst_omx_video_dec_negotiate:<omxh264dec-omxh264dec0> > > Empty caps > > 0:00:00.985837604 31116 0x167e230 FIXME glslstage > > gstglslstage.c:518:_compile_shader:<glslstage0> vertex shader info > > log:Compiled > > 0:00:00.990119375 31116 0x167e230 FIXME glslstage > > gstglslstage.c:518:_compile_shader:<glslstage1> fragment shader info > > log:Compiled > > 0:00:01.013341718 31116 0x167e230 FIXME glslstage > > gstglslstage.c:518:_compile_shader:<glslstage2> vertex shader info > > log:Compiled > > 0:00:01.014671666 31116 0x167e230 FIXME glslstage > > gstglslstage.c:518:_compile_shader:<glslstage3> fragment shader info > > log:Compiled > > ^Chandling interrupt. > > Interrupt: Stopping pipeline ... > > Execution ended after 0:00:06.801416612 > > Setting pipeline to PAUSED ... > > Setting pipeline to READY ... > > Setting pipeline to NULL ... > > Freeing pipeline ... > > ``` > > > > I don't know what the ERROR is trying to tell me, but this pipeline works > > a > > charm. > > > >> The first step to debugging this would be to enable GStreamer debug > >> logging to get an idea of what may be wrong. > > > > But the following happens when I add the gleffects_twirl filter (either > > before or after the queue, doesn't matter): > > > > ``` > > # GST_DEBUG=3 gst-launch-1.0 rpicamsrc ! > > "video/x-h264,width=640,height=480" ! h264parse ! omxh264dec ! > > "video/x-raw(memory:GLMemory)" ! gleffects_twirl ! queue ! glimagesink > > Setting pipeline to PAUSED ... > > Pipeline is live and does not need PREROLL ... > > 0:00:00.506736041 31143 0x1fed490 FIXME default > > gstutils.c: > > 3981:gst_pad_create_stream_id_internal:<rpicamsrc0:src> Creating random > > stream-id, consider implementing a deterministic way of creating a > > stream-id Got context from element 'sink': gst.gl.GLDisplay=context, > > gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayEGL\)\ gldisplayegl0"; > > Setting pipeline to PLAYING ... > > New clock: GstSystemClock > > 0:00:00.882512708 31143 0x1fed490 ERROR omxvideodec > > gstomxvideodec.c:2079:gst_omx_video_dec_negotiate:<omxh264dec-omxh264dec0> > > Empty caps > > 0:00:00.968698020 31143 0x6fd03290 WARN videodecoder > > gstvideodecoder.c:3783:gst_video_decoder_negotiate_pool:<omxh264dec- > > omxh264dec0> Subclass failed to decide allocation > > 0:00:00.969139166 31143 0x6fd03290 WARN GST_PADS gstpad.c: > > 4231:gst_pad_peer_query:<omxh264dec-omxh264dec0:src> could not send sticky > > events > > 0:00:00.969251770 31143 0x6fd03290 WARN videodecoder > > gstvideodecoder.c:3783:gst_video_decoder_negotiate_pool:<omxh264dec- > > omxh264dec0> Subclass failed to decide allocation > > 0:00:00.972893854 31143 0x6fd03290 WARN GST_PADS gstpad.c: > > 4231:gst_pad_peer_query:<omxh264dec-omxh264dec0:src> could not send sticky > > events > > 0:00:00.988252083 31143 0x6fd03290 WARN omxvideodec > > gstomxvideodec.c:1927:gst_omx_video_dec_loop:<omxh264dec-omxh264dec0> > > error: Internal data stream error. > > 0:00:00.988374999 31143 0x6fd03290 WARN omxvideodec > > gstomxvideodec.c:1927:gst_omx_video_dec_loop:<omxh264dec-omxh264dec0> > > error: stream stopped, reason not-negotiated > > 0:00:00.988662604 31143 0x1fed490 WARN basesrc > > gstbasesrc.c: 3072:gst_base_src_loop:<rpicamsrc0> error: Internal data > > stream error. 0:00:00.988741718 31143 0x1fed490 WARN > > basesrc gstbasesrc.c: 3072:gst_base_src_loop:<rpicamsrc0> error: > > streaming stopped, reason not- negotiated (-4) > > ERROR: from element /GstPipeline:pipeline0/GstOMXH264Dec- > > omxh264dec:omxh264dec-omxh264dec0: Internal data stream error. > > Additional debug info: > > ../../gst-omx-1.16.1/omx/gstomxvideodec.c(1927): gst_omx_video_dec_loop > > (): / > > GstPipeline:pipeline0/GstOMXH264Dec-omxh264dec:omxh264dec-omxh264dec0: > > stream stopped, reason not-negotiated > > Execution ended after 0:00:00.478420208 > > Setting pipeline to PAUSED ... > > Setting pipeline to READY ... > > Setting pipeline to NULL ... > > Freeing pipeline ... > > ``` > > > > Do you have an idea of what's going on here? > > > > Thanks > > > >> On 22/11/19 3:10 am, Milian Wolff wrote: > >>> Hey all, > >>> > >>> on a Raspberry Pi 3b+ gst-inspect-1.0 shows the following src for > >>> omxh264dec: > >>> > >>> ``` > >>> > >>> video/x-raw(memory:GLMemory) > >>> > >>> format: RGBA > >>> > >>> width: [ 1, 2147483647 ] > >>> > >>> height: [ 1, 2147483647 ] > >>> > >>> framerate: [ 0/1, 2147483647/1 ] > >>> > >>> ``` > >>> > >>> Note that this is missing a texture-target specification. What kind of > >>> memory is this, and how can I leverage it further down? I would like to > >>> efficiently change video frames using an OpenGL filter. If we take > >>> gleffects-twirl as an example, it expects the following as sink: > >>> > >>> > >>> ``` > >>> > >>> video/x-raw(memory:GLMemory): > >>> format: RGBA > >>> > >>> width: [ 1, 2147483647 ] > >>> > >>> height: [ 1, 2147483647 ] > >>> > >>> framerate: [ 0/1, 2147483647/1 ] > >>> > >>> texture-target: 2D > >>> > >>> ``` > >>> > >>> How can I connect the two together? If I try, to add the filter directly > >>> after the decoder in a pipeline, I get: > >>> > >>> ``` > >>> ERROR: from element /GstPipeline:pipeline0/GstOMXH264Dec- > >>> omxh264dec:omxh264dec-omxh264dec0: Internal data stream error. > >>> Additional debug info: > >>> ../../gst-omx-1.16.1/omx/gstomxvideodec.c(1927): gst_omx_video_dec_loop > >>> (): / > >>> GstPipeline:pipeline0/GstOMXH264Dec-omxh264dec:omxh264dec-omxh264dec0: > >>> stream stopped, reason not-negotiated > >>> ``` > >>> > >>> Can someone sched some light on this? How can one post-process the > >>> GLMemory in a gl video filter? Is the missing texture-target a bug in > >>> omxh264dec, or can I somehow make the raw GLMemory a texture? Or does > >>> this mean the frame lives in some other form of non-texture memory that > >>> can be displayed, but not otherwise accessed by OpenGL for video filter > >>> purposes? > >>> > >>> Thanks > >>> > >>> PS: In case you are wondering what we are trying to achieve on a higher > >>> level: > >>> > >>> rpicamsrc -> h264 -> omxh264dec -> glmemory -> custom gl filter to add a > >>> non- static overlay to every frame -> tee to display sink and then also > >>> a > >>> h264- encoded avi record sink > >>> > >>> _______________________________________________ > >>> gstreamer-devel mailing list > >>> [hidden email] > >>> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel -- Milian Wolff | [hidden email] | Senior Software Engineer KDAB (Deutschland) GmbH, a KDAB Group company Tel: +49-30-521325470 KDAB - The Qt, C++ and OpenGL Experts _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
Re: What's the meaning of video/x-raw(memory:GLMemory) without texture-target?
|
|
Now you're at my limit if what I can confidently tell you without
experimenting myself :) I don't know why the twirl effect is not applied unfortunately. I would need to experiment with that to find the bug there. The omx decoder GL zerocopy path for the RPi is unfortunately rather special. In theory you don't need the 'glupload ! glcolorconvert' however, there may be lifetime/format considerations that glupload ! glcolorconvert will paper over. On 25/11/19 9:31 pm, Milian Wolff wrote: > On Monday, November 25, 2019 11:20:00 AM CET Matthew Waters wrote: >> Try adding a glupload ! glcolorconvert before the gleffects_twirl >> element. glimagesink contains these automatically already. > Thanks! This fixes the errors but it leaves me equally puzzled: > > Most notably, the twirl effect isn't taking effect, i.e. this is my command > and debug output now: > > ``` > # GST_DEBUG=3 gst-launch-1.0 rpicamsrc ! "video/x-h264,width=640,height=480" ! > h264parse ! omxh264dec ! "video/x-raw(memory:GLMemory)" ! glupload ! > gleffects_twirl ! queue ! glimagesink > Setting pipeline to PAUSED ... > Pipeline is live and does not need PREROLL ... > 0:00:00.489548698 381 0x1c45af0 FIXME default gstutils.c: > 3981:gst_pad_create_stream_id_internal:<rpicamsrc0:src> Creating random > stream-id, consider implementing a deterministic way of creating a stream-id > Got context from element 'sink': gst.gl.GLDisplay=context, > gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayEGL\)\ gldisplayegl0"; > Setting pipeline to PLAYING ... > New clock: GstSystemClock > 0:00:00.880536979 381 0x1c45af0 ERROR omxvideodec > gstomxvideodec.c:2079:gst_omx_video_dec_negotiate:<omxh264dec-omxh264dec0> > Empty caps > 0:00:01.057018906 381 0x1c2f750 FIXME glslstage > gstglslstage.c:518:_compile_shader:<glslstage0> vertex shader info > log:Compiled > 0:00:01.067645364 381 0x1c2f750 FIXME glslstage > gstglslstage.c:518:_compile_shader:<glslstage1> fragment shader info > log:Compiled > 0:00:01.125377187 381 0x1c2f750 FIXME glslstage > gstglslstage.c:518:_compile_shader:<glslstage2> vertex shader info > log:Compiled > 0:00:01.127863541 381 0x1c2f750 FIXME glslstage > gstglslstage.c:518:_compile_shader:<glslstage3> fragment shader info > log:Compiled > 0:00:01.160143698 381 0x1c2f750 FIXME glslstage > gstglslstage.c:518:_compile_shader:<glslstage4> vertex shader info > log:Compiled > 0:00:01.161840260 381 0x1c2f750 FIXME glslstage > gstglslstage.c:518:_compile_shader:<glslstage5> fragment shader info > log:Compiled > ``` > > But the camera image is displayed as-is. What's odd is that I seem to be > seeing the twirl effect for a very short time when I "Ctrl+C" terminate the > `gst-launch` process. > > If I use `gltestsrc` as a replacement for the source up to `omxh264dec`, I see > the twirl effect directly. What's going on here? > > Secondly, could you please explain to me why the `glupload ! glcolorconvert` > is required here? Most notably, `video/x-raw(GLMemory)` should be on the GPU > already, why do I need to "upload" it again? And the format is already `RGBA`, > so why do I need glcolorconvert? > > Thanks > >> On 25/11/19 8:30 pm, Milian Wolff wrote: >>> On Friday, November 22, 2019 5:12:57 AM CET Matthew Waters wrote: >>>> Hi, >>> Hey Matt! >>> >>>> By default a 'video/x-raw(memory:GLMemory)' caps without the >>>> texture-target field is equivalent to texture-target=2D as that was the >>>> default before we differentiated the different OpenGL texture targets. >>>> Now elements should be able to negotiate the differences between them >>>> correctly however there may be bugs. >>> OK, thanks - good to know! >>> >>>> The GLMemory output from omxh264dec is highly dependant on the platform >>>> you are running on and may or may not work correctly. I know that it >>>> used to work on the RPi and have not tested any other omx platform >>>> myself. >>> What's interesting is that this seems to work fine when I shove the data >>> directly into the glimagesink, i.e.: >>> >>> ``` >>> # GST_DEBUG=3 gst-launch-1.0 rpicamsrc ! >>> "video/x-h264,width=640,height=480" ! h264parse ! omxh264dec ! >>> "video/x-raw(memory:GLMemory)" ! queue ! >>> glimagesinkSetting pipeline to PAUSED ... >>> Pipeline is live and does not need PREROLL ... >>> 0:00:00.488648854 31116 0x167e7b0 FIXME default >>> gstutils.c: >>> 3981:gst_pad_create_stream_id_internal:<rpicamsrc0:src> Creating random >>> stream-id, consider implementing a deterministic way of creating a >>> stream-id Got context from element 'sink': gst.gl.GLDisplay=context, >>> gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayEGL\)\ gldisplayegl0"; >>> Setting pipeline to PLAYING ... >>> New clock: GstSystemClock >>> 0:00:00.894124323 31116 0x167e7b0 ERROR omxvideodec >>> gstomxvideodec.c:2079:gst_omx_video_dec_negotiate:<omxh264dec-omxh264dec0> >>> Empty caps >>> 0:00:00.985837604 31116 0x167e230 FIXME glslstage >>> gstglslstage.c:518:_compile_shader:<glslstage0> vertex shader info >>> log:Compiled >>> 0:00:00.990119375 31116 0x167e230 FIXME glslstage >>> gstglslstage.c:518:_compile_shader:<glslstage1> fragment shader info >>> log:Compiled >>> 0:00:01.013341718 31116 0x167e230 FIXME glslstage >>> gstglslstage.c:518:_compile_shader:<glslstage2> vertex shader info >>> log:Compiled >>> 0:00:01.014671666 31116 0x167e230 FIXME glslstage >>> gstglslstage.c:518:_compile_shader:<glslstage3> fragment shader info >>> log:Compiled >>> ^Chandling interrupt. >>> Interrupt: Stopping pipeline ... >>> Execution ended after 0:00:06.801416612 >>> Setting pipeline to PAUSED ... >>> Setting pipeline to READY ... >>> Setting pipeline to NULL ... >>> Freeing pipeline ... >>> ``` >>> >>> I don't know what the ERROR is trying to tell me, but this pipeline works >>> a >>> charm. >>> >>>> The first step to debugging this would be to enable GStreamer debug >>>> logging to get an idea of what may be wrong. >>> But the following happens when I add the gleffects_twirl filter (either >>> before or after the queue, doesn't matter): >>> >>> ``` >>> # GST_DEBUG=3 gst-launch-1.0 rpicamsrc ! >>> "video/x-h264,width=640,height=480" ! h264parse ! omxh264dec ! >>> "video/x-raw(memory:GLMemory)" ! gleffects_twirl ! queue ! glimagesink >>> Setting pipeline to PAUSED ... >>> Pipeline is live and does not need PREROLL ... >>> 0:00:00.506736041 31143 0x1fed490 FIXME default >>> gstutils.c: >>> 3981:gst_pad_create_stream_id_internal:<rpicamsrc0:src> Creating random >>> stream-id, consider implementing a deterministic way of creating a >>> stream-id Got context from element 'sink': gst.gl.GLDisplay=context, >>> gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayEGL\)\ gldisplayegl0"; >>> Setting pipeline to PLAYING ... >>> New clock: GstSystemClock >>> 0:00:00.882512708 31143 0x1fed490 ERROR omxvideodec >>> gstomxvideodec.c:2079:gst_omx_video_dec_negotiate:<omxh264dec-omxh264dec0> >>> Empty caps >>> 0:00:00.968698020 31143 0x6fd03290 WARN videodecoder >>> gstvideodecoder.c:3783:gst_video_decoder_negotiate_pool:<omxh264dec- >>> omxh264dec0> Subclass failed to decide allocation >>> 0:00:00.969139166 31143 0x6fd03290 WARN GST_PADS gstpad.c: >>> 4231:gst_pad_peer_query:<omxh264dec-omxh264dec0:src> could not send sticky >>> events >>> 0:00:00.969251770 31143 0x6fd03290 WARN videodecoder >>> gstvideodecoder.c:3783:gst_video_decoder_negotiate_pool:<omxh264dec- >>> omxh264dec0> Subclass failed to decide allocation >>> 0:00:00.972893854 31143 0x6fd03290 WARN GST_PADS gstpad.c: >>> 4231:gst_pad_peer_query:<omxh264dec-omxh264dec0:src> could not send sticky >>> events >>> 0:00:00.988252083 31143 0x6fd03290 WARN omxvideodec >>> gstomxvideodec.c:1927:gst_omx_video_dec_loop:<omxh264dec-omxh264dec0> >>> error: Internal data stream error. >>> 0:00:00.988374999 31143 0x6fd03290 WARN omxvideodec >>> gstomxvideodec.c:1927:gst_omx_video_dec_loop:<omxh264dec-omxh264dec0> >>> error: stream stopped, reason not-negotiated >>> 0:00:00.988662604 31143 0x1fed490 WARN basesrc >>> gstbasesrc.c: 3072:gst_base_src_loop:<rpicamsrc0> error: Internal data >>> stream error. 0:00:00.988741718 31143 0x1fed490 WARN >>> basesrc gstbasesrc.c: 3072:gst_base_src_loop:<rpicamsrc0> error: >>> streaming stopped, reason not- negotiated (-4) >>> ERROR: from element /GstPipeline:pipeline0/GstOMXH264Dec- >>> omxh264dec:omxh264dec-omxh264dec0: Internal data stream error. >>> Additional debug info: >>> ../../gst-omx-1.16.1/omx/gstomxvideodec.c(1927): gst_omx_video_dec_loop >>> (): / >>> GstPipeline:pipeline0/GstOMXH264Dec-omxh264dec:omxh264dec-omxh264dec0: >>> stream stopped, reason not-negotiated >>> Execution ended after 0:00:00.478420208 >>> Setting pipeline to PAUSED ... >>> Setting pipeline to READY ... >>> Setting pipeline to NULL ... >>> Freeing pipeline ... >>> ``` >>> >>> Do you have an idea of what's going on here? >>> >>> Thanks >>> >>>> On 22/11/19 3:10 am, Milian Wolff wrote: >>>>> Hey all, >>>>> >>>>> on a Raspberry Pi 3b+ gst-inspect-1.0 shows the following src for >>>>> omxh264dec: >>>>> >>>>> ``` >>>>> >>>>> video/x-raw(memory:GLMemory) >>>>> >>>>> format: RGBA >>>>> >>>>> width: [ 1, 2147483647 ] >>>>> >>>>> height: [ 1, 2147483647 ] >>>>> >>>>> framerate: [ 0/1, 2147483647/1 ] >>>>> >>>>> ``` >>>>> >>>>> Note that this is missing a texture-target specification. What kind of >>>>> memory is this, and how can I leverage it further down? I would like to >>>>> efficiently change video frames using an OpenGL filter. If we take >>>>> gleffects-twirl as an example, it expects the following as sink: >>>>> >>>>> >>>>> ``` >>>>> >>>>> video/x-raw(memory:GLMemory): >>>>> format: RGBA >>>>> >>>>> width: [ 1, 2147483647 ] >>>>> >>>>> height: [ 1, 2147483647 ] >>>>> >>>>> framerate: [ 0/1, 2147483647/1 ] >>>>> >>>>> texture-target: 2D >>>>> >>>>> ``` >>>>> >>>>> How can I connect the two together? If I try, to add the filter directly >>>>> after the decoder in a pipeline, I get: >>>>> >>>>> ``` >>>>> ERROR: from element /GstPipeline:pipeline0/GstOMXH264Dec- >>>>> omxh264dec:omxh264dec-omxh264dec0: Internal data stream error. >>>>> Additional debug info: >>>>> ../../gst-omx-1.16.1/omx/gstomxvideodec.c(1927): gst_omx_video_dec_loop >>>>> (): / >>>>> GstPipeline:pipeline0/GstOMXH264Dec-omxh264dec:omxh264dec-omxh264dec0: >>>>> stream stopped, reason not-negotiated >>>>> ``` >>>>> >>>>> Can someone sched some light on this? How can one post-process the >>>>> GLMemory in a gl video filter? Is the missing texture-target a bug in >>>>> omxh264dec, or can I somehow make the raw GLMemory a texture? Or does >>>>> this mean the frame lives in some other form of non-texture memory that >>>>> can be displayed, but not otherwise accessed by OpenGL for video filter >>>>> purposes? >>>>> >>>>> Thanks >>>>> >>>>> PS: In case you are wondering what we are trying to achieve on a higher >>>>> level: >>>>> >>>>> rpicamsrc -> h264 -> omxh264dec -> glmemory -> custom gl filter to add a >>>>> non- static overlay to every frame -> tee to display sink and then also >>>>> a >>>>> h264- encoded avi record sink >>>>> >>>>> _______________________________________________ >>>>> gstreamer-devel mailing list >>>>> [hidden email] >>>>> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
| Free forum by Nabble | Edit this page |