Text overlays don't work on i.MX6 in certain pipelines

[Toon Heyrman]

I'm building a pipeline like in this image:

<http://gstreamer-devel.966125.n4.nabble.com/file/t378027/pipeline.png>

This is a minimal version that recreates the issue.
The problem is that the text overlay is not shown in this pipeline
This when I use a 'tee' in combination with an overlay element(after tee),
imx encoder and an imx cam input.

I've tested 5 different scenarios and the overlay is only not shown in the
first scenario(same as above image).
Pipeline as in image above -> NOK
Removing the 'tee' -> overlay OK
Moving the overlay element in front of the 'tee' -> OK
Removing the encoder and putting raw data in a file -> OK
Replace the imxv4l2videosrc with a testvideosrv -> OK

The output images are included (open the .mp4 files with 'ffplay ...mp4',
.data file with 'gimp ...data' and select width 1024, height 768)
The text is positioned in the bottom of the image.
image_with_tee.mp4
<http://gstreamer-devel.966125.n4.nabble.com/file/t378027/image_with_tee.mp4>  
image_without_tee.mp4
<http://gstreamer-devel.966125.n4.nabble.com/file/t378027/image_without_tee.mp4>  
image_with_tee_overlay_in_front_of_tee.mp4
<http://gstreamer-devel.966125.n4.nabble.com/file/t378027/image_with_tee_overlay_in_front_of_tee.mp4>  
image_with_tee_without_enc.data
<http://gstreamer-devel.966125.n4.nabble.com/file/t378027/image_with_tee_without_enc.data>  
image_with_tee_testvideo.mp4
<http://gstreamer-devel.966125.n4.nabble.com/file/t378027/image_with_tee_testvideo.mp4>  

Debug information by doing GST_DEBUG=*:5
debug_output_with_tee.log
<http://gstreamer-devel.966125.n4.nabble.com/file/t378027/debug_output_with_tee.log>  
debug_output_without_tee.log
<http://gstreamer-devel.966125.n4.nabble.com/file/t378027/debug_output_without_tee.log>  
debug_output_with_tee_overlay_in_front_of_tee.log
<http://gstreamer-devel.966125.n4.nabble.com/file/t378027/debug_output_with_tee_overlay_in_front_of_tee.log>  
debug_output_with_tee_without_enc.log
<http://gstreamer-devel.966125.n4.nabble.com/file/t378027/debug_output_with_tee_without_enc.log>  
debug_output_with_tee_testvideo.log
<http://gstreamer-devel.966125.n4.nabble.com/file/t378027/debug_output_with_tee_testvideo.log>  

The pipelines I ran:
test-day-with-tee.test-day-with-tee
<http://gstreamer-devel.966125.n4.nabble.com/file/t378027/test-day-with-tee.test-day-with-tee>  
test-day-without-tee.test-day-without-tee
<http://gstreamer-devel.966125.n4.nabble.com/file/t378027/test-day-without-tee.test-day-without-tee>  
test-day-with-tee-overlay-in-front-of-tee.test-day-with-tee-overlay-in-front-of-tee
<http://gstreamer-devel.966125.n4.nabble.com/file/t378027/test-day-with-tee-overlay-in-front-of-tee.test-day-with-tee-overlay-in-front-of-tee>  
test-day-with-tee-without-enc.test-day-with-tee-without-enc
<http://gstreamer-devel.966125.n4.nabble.com/file/t378027/test-day-with-tee-without-enc.test-day-with-tee-without-enc>  
testvideo-with-tee.testvideo-with-tee
<http://gstreamer-devel.966125.n4.nabble.com/file/t378027/testvideo-with-tee.testvideo-with-tee>  

Replacing the multiqueue to two queues or queue2 doesn't help.

Am I doing something wrong or am I missing something essential?



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Toon Heyrman]

Another test case is with a software encoder instead of the hardware encoder,
here it is working(annotation present):
test-day-with-tee-sw-enc.test-day-with-tee-sw-enc
<http://gstreamer-devel.966125.n4.nabble.com/file/t378027/test-day-with-tee-sw-enc.test-day-with-tee-sw-enc>  
debug_output_with_tee_sw_enc.log
<http://gstreamer-devel.966125.n4.nabble.com/file/t378027/debug_output_with_tee_sw_enc.log>  
image_with_tee_sw_enc.mp4
<http://gstreamer-devel.966125.n4.nabble.com/file/t378027/image_with_tee_sw_enc.mp4>  

Also when a lot of annotation has to be done(in my custom element), the
annotation is sometimes shown in the not working scenario.




--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le jeudi 11 janvier 2018 à 03:30 -0700, Toon Heyrman a écrit :
> I've tested 5 different scenarios and the overlay is only not shown
> in the
> first scenario(same as above image).
> Pipeline as in image above -> NOK
> Removing the 'tee' -> overlay OK
> Moving the overlay element in front of the 'tee' -> OK
> Removing the encoder and putting raw data in a file -> OK
> Replace the imxv4l2videosrc with a testvideosrv -> OK

Have you consider filing a bug for gstreamer-imx ?

https://github.com/Freescale/gstreamer-imx

It looks like an hardware specific (or IMX  element specific) issue
with the custom memory handling.

Nicolas
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Toon Heyrman]

> Have you consider filing a bug for gstreamer-imx ?

Yes, here is the link:
https://github.com/Freescale/gstreamer-imx/issues/182 <http://>  



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Toon Heyrman]

The element gamma in the videofilter plugins has the same issue as the
textoverlay.
Adding a delay of 15ms in the gamma element, makes the annotation/filtering
work.

Using imxg2dtextoverlay instead of the gamma or textoverlay works too.
Only this needs RGB and I need I420 for the encoder. Conversions are not an
option in the pipeline due to performance reasons.

Maybe if I modify my own annotation element(based on the gamma element) to
draw with the g2d it could work.
But is there any example on how to draw via a videofilter element with the
g2d?



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le mercredi 17 janvier 2018 à 01:59 -0700, Toon Heyrman a écrit :
> The element gamma in the videofilter plugins has the same issue as
> the
> textoverlay.
> Adding a delay of 15ms in the gamma element, makes the
> annotation/filtering
> work.

That's sounds like a missing cache flush on map. This is on the
gstreamer-imx side though, as this should be handled by the GstMemory
implementation when the buffer is mapped.

>
> Using imxg2dtextoverlay instead of the gamma or textoverlay works
> too.
> Only this needs RGB and I need I420 for the encoder. Conversions are
> not an
> option in the pipeline due to performance reasons.
>
> Maybe if I modify my own annotation element(based on the gamma
> element) to
> draw with the g2d it could work.
> But is there any example on how to draw via a videofilter element
> with the
> g2d?
>
>
>
> --
> Sent from: http://gstreamer-devel.966125.n4.nabble.com/
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Toon Heyrman]

I discovered something extra:
If I use a imxipuvideotransform in front of the videofilter and force the
transform to take a copy of the buffer by setting the output caps a little
bit different (height from 768px to 752px), it works always.
So there is definitely something going wrong with the buffers.

> That's sounds like a missing cache flush on map. This is on the
> gstreamer-imx side though, as this should be handled by the GstMemory
> implementation when the buffer is mapped.

To flush the map shall we use msync?
And is it right after the mmap?
https://github.com/Freescale/gstreamer-imx/blob/master/src/v4l2video/v4l2_buffer_pool.c#L174
<https://github.com/Freescale/gstreamer-imx/blob/master/src/v4l2video/v4l2_buffer_pool.c#L174>  




--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le 18 janv. 2018 8:08 AM, "Toon Heyrman" <[hidden email]> a écrit :

I discovered something extra:
If I use a imxipuvideotransform in front of the videofilter and force the
transform to take a copy of the buffer by setting the output caps a little
bit different (height from 768px to 752px), it works always.
So there is definitely something going wrong with the buffers.

> That's sounds like a missing cache flush on map. This is on the
> gstreamer-imx side though, as this should be handled by the GstMemory
> implementation when the buffer is mapped.

To flush the map shall we use msync?
And is it right after the mmap?
https://github.com/Freescale/gstreamer-imx/blob/master/src/v4l2video/v4l2_buffer_pool.c#L174
<https://github.com/Freescale/gstreamer-imx/blob/master/src/v4l2video/v4l2_buffer_pool.c#L174>

I don't know, this is NXP specific. This could be done right before returning from map. Though, in most other drivers, this is done in the driver implicitly.

--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Eloi Bail]

I don't know, this is NXP specific. This could be done right before returning from map. Though, in most other drivers, this is done in the driver implicitly.

Hi,

I am facing similar issues and am only able to reproduce that using imx4l2src with a high framerate (30 fps).

Having a video filtering elememt, I see that only some frames are processed by transform_frame_ip method of vfilter and sometimes even none of them !

So it means that gstreamer_imx is sometimes not exposing DMA buffers to Gstreamer.

Am I right ?

I also tried to flush the mmapped buffer adding a msync right after the mmap but it does not change anything.

What is very curious is that this issue is only reproducible with a tee:

imx4l2src --> imxvpuenc_h264 -> vfillter -> rtph264pay -> udpsink

                  |-> tee -> fakesink





We are also able to *fix* the issue adding a videotransform with different caps between the src and the encoder. This workaround will copy the buffer calling prepare_output_buffer of the parent class (gstbasetransform). The fps remains good but the resolution is modified...

I definitely believe that the issue is in gstreamer-imx but we are lacking answers from them.

So any help or workaround *clean* are welcomed.

Thanks,

Eloi

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel