NV12 issue in 1.14.1

[tobatrance]

Hi there,

I'm right now shifting my application from 1.8.3 to 1.14.1 as I shift Ubuntu
as well from 16.04 to 18.04. I got some issues with the NV12 format. Back in
1.8.3 almost all my videos came in I420, meanwhile they're by default
encoded in NV12.

I'm in general using this simple pipeline:

filesrc ! decodebin ! appsink

I grab the frames from my appsink and push them on to OpenGL for some
postprocessing before displaying them. There are *some* videos which cause
trouble, the majority runs fine, like bunny e.g., but some don't work the
way they're supposed to which have been working with I420. Said that I can
exclude my shader being wrong. Please note the attached video. This is one
of these working before in I420 but not in NV12.

countdown.mp4
<http://gstreamer-devel.966125.n4.nabble.com/file/t377636/countdown.mp4>  

To me it seems that the strides given by the GstVideoInfo.stride property do
not fit the corresponding GstVideoFrame.data property. For the countdown
video, when using the stride 765 in 1.14.1/NV12 for the Y plane it works
properly. In 1.8.3/I420, GST gives 720, which is the same as the actual
width and it works there. In 1.14.1/NV12 it also gives 720 but as I
mentioned it's not displayed correctly. When changing the value to 765 it
works. I'm using the stride in /glPixelStorei(GL_UNPACK_ROW_LENGTH,
stride);/ when updating the texture with the current sample's buffer.

I got a different issue with the UV plane: The strides there don't work as
well like with the Y plane. But additionally it seems that the plane has an
offset in respect to the Y plane. When swichting off displaying the Y plane
in the shader and just showing the UV plane it seems the UV plane is shifted
to the top. In other words: It seems like the first couple of rows are
missing in the UV plane. The bottom is filled with visual noise.

Appreciate any help. If you need more information let me know.



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le mercredi 25 juillet 2018 à 07:24 -0500, tobatrance a écrit :
> Hi there,
>
> I'm right now shifting my application from 1.8.3 to 1.14.1 as I shift Ubuntu
> as well from 16.04 to 18.04. I got some issues with the NV12 format. Back in
> 1.8.3 almost all my videos came in I420, meanwhile they're by default
> encoded in NV12.

I420 and NV12 is the same format, just a different layout, so it's not
possible to encode in one or the other.

>
> I'm in general using this simple pipeline:
>
> filesrc ! decodebin ! appsink
>
> I grab the frames from my appsink and push them on to OpenGL for some
> postprocessing before displaying them. There are *some* videos which cause
> trouble, the majority runs fine, like bunny e.g., but some don't work the
> way they're supposed to which have been working with I420. Said that I can
> exclude my shader being wrong. Please note the attached video. This is one
> of these working before in I420 but not in NV12.
>
> countdown.mp4
> <http://gstreamer-devel.966125.n4.nabble.com/file/t377636/countdown.mp4> 

This video is profile main, so it's you are not using openh264, maybe
you are using vaapi then, which indeed gives me NV12 by default.

Though, there does not seem to be any issues with the GstVideoMeta when
using glimagesink here:

gst-launch-1.0 souphttpsrc location=http://gstreamer-devel.966125.n4.nabble.com/file/t377636/countdown.mp4 ! queue2 ! qtdemux ! h264parse ! vaapih264dec ! video/x-raw ! glimagesink

This is 1.14.1, you should do the same test to validate.

>
> To me it seems that the strides given by the GstVideoInfo.stride property do
> not fit the corresponding GstVideoFrame.data property. For the countdown
> video, when using the stride 765 in 1.14.1/NV12 for the Y plane it works
> properly. In 1.8.3/I420, GST gives 720, which is the same as the actual
> width and it works . In 1.14.1/NV12 it also gives 720 but as I
> mentioned it's not displayed correctly. When changing the value to 765 it
> works. I'm using the stride in /glPixelStorei(GL_UNPACK_ROW_LENGTH,
> stride);/ when updating the texture with the current sample's buffer.
>
> I got a different issue with the UV plane: The strides there don't work as
> well like with the Y plane. But additionally it seems that the plane has an
> offset in respect to the Y plane. When swichting off displaying the Y plane
> in the shader and just showing the UV plane it seems the UV plane is shifted
> to the top. In other words: It seems like the first couple of rows are
> missing in the UV plane. The bottom is filled with visual noise.
>
> Appreciate any help. If you need more information let me know.

Without any code it would be hard to point to your bugs. This is a bug
in your code of course.

>
>
>
> --
> Sent from: http://gstreamer-devel.966125.n4.nabble.com/
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[tobatrance]

CONTENTS DELETED

The author has deleted this message.

[Nicolas Dufresne-5]

Le jeudi 26 juillet 2018 à 11:22 -0500, tobatrance a écrit :
> I extracted that stuff quite dirty from my code but it gives the same
> error.
> Bunny is working. Countdown is not working. Same code. Please find it
> here.

I don't really have the time to build and test foreign code. But the
upload part seems fine. Have you thought of running API trace on your
code and a glupload pipeline in order to compare ?

>
> gst-glfw-test.zip
> <
> http://gstreamer-devel.966125.n4.nabble.com/file/t377636/gst-glfw-test.zip> 
>  
>
> For running please add  https://github.com/Dav1dde/glad.git
> <https://github.com/Dav1dde/glad.git>   as /glad/ to the /libs/
> folder. Same
> with  https://github.com/glfw/glfw <https://github.com/glfw/glfw>  .
>
> You can edit lines 577, 578 to choose the source.
>
> Please let me know what I'm doing wrong.
>
>
>
> --
> Sent from: http://gstreamer-devel.966125.n4.nabble.com/
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[tobatrance]

Haven't heared about apitrace before. Cool tool!

So I compared the apitrace output from my test app with the apitrace out
from gst-launch-1.0 filesrc ! decodebin ! video/x-raw ! glimagesink for the
same file. The strides used by gst-launch are different from the strides I
get in my test app from the GstVideoInfo struct. In my app I get for both,
the Y plane and the UV plane, 720. gst-launch uses 768 for the Y plane and
384 for the UV plane. Any idea where these might come from? When using those
strides in my app it looks quite good, though the UV plane in my app is
still shifted vertically from the Y plane. I don't really see in the
apitrace from gst-launch how this is done in GL and which Gst parameter is
used for that. Any idea?



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le vendredi 27 juillet 2018 à 06:55 -0500, tobatrance a écrit :
> Haven't heared about apitrace before. Cool tool!
>
> So I compared the apitrace output from my test app with the apitrace out
> from gst-launch-1.0 filesrc ! decodebin ! video/x-raw ! glimagesink for the
> same file. The strides used by gst-launch are different from the strides I
> get in my test app from the GstVideoInfo struct. In my app I get for both,
> the Y plane and the UV plane, 720. gst-launch uses 768 for the Y plane and
> 384 for the UV plane. Any idea where these might come from? When using those

Two options, you get a copy of the frames as you don't advertise
GstVideoMeta support. With appsink, you need a pad probe to handle the
allocation query to do that (see kmscube example software). Or, you
forgot to update your GstVideoInfo from the buffer GstVideoMeta.

> strides in my app it looks quite good, though the UV plane in my app is
> still shifted vertically from the Y plane. I don't really see in the
> apitrace from gst-launch how this is done in GL and which Gst parameter is
> used for that. Any idea?

Vertical shift would mean that you have ignored GstVideoMetas::offset[]
when accessing planes. At least, that's my impression.

>
>
>
> --
> Sent from: http://gstreamer-devel.966125.n4.nabble.com/
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[tobatrance]

So I copied the callback

GstPadProbeReturn appsink_query_cb(GstPad *pad_G_GNUC_UNUSED,
GstPadProbeInfo *_info, gpointer user_data_G_GNUC_UNUSED) {
        GstQuery *query = reinterpret_cast<GstQuery*>(_info->data);

        if(GST_QUERY_TYPE(query) != GST_QUERY_ALLOCATION) {
                return GST_PAD_PROBE_OK;
        }

        gst_query_add_allocation_meta(query, GST_VIDEO_META_API_TYPE, NULL);

        return GST_PAD_PROBE_HANDLED;
}

from kmscube and add the probe while constructing the pipeline

GstPad *pad = gst_element_get_static_pad(videosink, "sink");
gst_pad_add_probe(pad, GST_PAD_PROBE_TYPE_QUERY_DOWNSTREAM,
appsink_query_cb, NULL, NULL);
gst_object_unref(pad);

but still don't get any GstVideoMeta with my buffers. Any hint?



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[tobatrance]

Ok, got it working now.

Like in the kmscube you have to use a capsfilter in front of the sink. The
shift of the UV plane disappeared as well with that.

That filter actually just limits the used format to the desired and does not
do any cpu conversion right?



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le 31 juil. 2018, à 07:11, tobatrance <[hidden email]> a écrit:

> Ok, got it working now.
>
> Like in the kmscube you have to use a capsfilter in front of the
> sink. The
> shift of the UV plane disappeared as well with that.
>
> That filter actually just limits the used format to the desired and
> does not
> do any cpu conversion right?
> NV12

Well, conversion takes place in VPP normally, does that count ?

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel