YUV buffer pstride and rstride clarification needed

[Neil Young]

Hi,

I'm referring to this documentation

https://gstreamer.freedesktop.org/documentation/additional/design/mediatype-video-raw.html?gi-language=c

Up to now I received namely I420 buffers with a size of 1.5*width*height, as expected default YUV size. 

Caps:

video/x-raw, format=(string)I420, width=(int)720, height=(int)540, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)jpeg, colorimetry=(string)bt601, framerate=(fraction)0/1;

Frame buffer size is 583200 byte.

Today for the first time I got this:

Caps:

video/x-raw(memory:VASurface), format=(string)NV12, width=(int)720, height=(int)540, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)0/1

with a frame buffer size of 622080 byte, which is more than 1.5 * width * height.

Now I'm wondering, how to understand the values of rstride and pstride in the above document. Could somebody help please?

TIA

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[David Ing]

I hope someone can correct me if I am wrong, but I am guessing that

• pstride = pixel stride (i.e. the number of bytes per pixel)
• rstride = row stride (i.e. the number of bytes per row of pixels)

Suppose I want to find the pixel (row=y, column=x), then I should look at the following location within the byte array:

• y * rstride + x * pstride

Keep in mind that I am a complete novice ... this is just a guess.

On Mon, Jul 15, 2019 at 2:45 PM Neil Young <[hidden email]> wrote:

Hi,

I'm referring to this documentation

https://gstreamer.freedesktop.org/documentation/additional/design/mediatype-video-raw.html?gi-language=c

Up to now I received namely I420 buffers with a size of 1.5*width*height, as expected default YUV size. 

Caps:

video/x-raw, format=(string)I420, width=(int)720, height=(int)540, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)jpeg, colorimetry=(string)bt601, framerate=(fraction)0/1;

Frame buffer size is 583200 byte.

Today for the first time I got this:

Caps:

video/x-raw(memory:VASurface), format=(string)NV12, width=(int)720, height=(int)540, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)0/1

with a frame buffer size of 622080 byte, which is more than 1.5 * width * height.

Now I'm wondering, how to understand the values of rstride and pstride in the above document. Could somebody help please?

TIA

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le lundi 15 juillet 2019 à 14:52 -0700, David Ing a écrit :
> I hope someone can correct me if I am wrong, but I am guessing that
> pstride = pixel stride (i.e. the number of bytes per pixel)
> rstride = row stride (i.e. the number of bytes per row of pixels)
> Suppose I want to find the pixel (row=y, column=x), then I should look at the following location within the byte array:
> y * rstride + x * pstride
> Keep in mind that I am a complete novice ... this is just a guess.

Well, clearly you know enough to read this, since this is correct
(appart the pstride does not exist for all formats).

>
> On Mon, Jul 15, 2019 at 2:45 PM Neil Young <[hidden email]> wrote:
> > Hi,
> >
> > I'm referring to this documentation
> >
> > https://gstreamer.freedesktop.org/documentation/additional/design/mediatype-video-raw.html?gi-language=c
> >
> > Up to now I received namely I420 buffers with a size of 1.5*width*height, as expected default YUV size.
> >
> > Caps:
> >
> > video/x-raw, format=(string)I420, width=(int)720, height=(int)540, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)jpeg, colorimetry=(string)bt601, framerate=(fraction)0/1;
> >
> > Frame buffer size is 583200 byte.
> >
> > Today for the first time I got this:
> >
> > Caps:
> >
> > video/x-raw(memory:VASurface), format=(string)NV12, width=(int)720, height=(int)540, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)0/1
> >
> > with a frame buffer size of 622080 byte, which is more than 1.5 * width * height.
> >
> > Now I'm wondering, how to understand the values of rstride and pstride in the above document. Could somebody help please?
> >
> > TIA
> > _______________________________________________
> > gstreamer-devel mailing list
> > [hidden email]
> > https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Neil Young]

@David Ing: Thanks for your explanation, which I for some reasons just received with the digest. From the quoted documentation of NV12 and other sources (e.g. https://stackoverflow.com/questions/17487048/nv12-format-and-uv-plane) I still can‘t see, how a frame of 720 by 540 can be delivered as a buffer of 622080 bytes. 
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Thornton, Keith]

I think pstride is probably the scanning stride. I.e. it takes all of the Luma pixel data and every second Chroma pixel byte both horizontally and vertically. This probably doesn’t say anything about the stride in the buffer.

The increase in buffer size is because NV12 encoder often need width and height to be multiples of 16. i.e. 720 * 576. Whereby you can ignore the data in the additional added lines.

 

Von: gstreamer-devel <[hidden email]> Im Auftrag von Neil Young
Gesendet: Montag, 15. Juli 2019 23:45
An: [hidden email]
Betreff: YUV buffer pstride and rstride clarification needed

 

Hi,

I'm referring to this documentation

https://gstreamer.freedesktop.org/documentation/additional/design/mediatype-video-raw.html?gi-language=c

Up to now I received namely I420 buffers with a size of 1.5*width*height, as expected default YUV size. 

Caps:

video/x-raw, format=(string)I420, width=(int)720, height=(int)540, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)jpeg, colorimetry=(string)bt601, framerate=(fraction)0/1;

Frame buffer size is 583200 byte.

Today for the first time I got this:

Caps:

video/x-raw(memory:VASurface), format=(string)NV12, width=(int)720, height=(int)540, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)0/1

with a frame buffer size of 622080 byte, which is more than 1.5 * width * height.

Now I'm wondering, how to understand the values of rstride and pstride in the above document. Could somebody help please?

TIA

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Neil Young]

@Keith Thornton: This explanation makes a lot of sense to me. Thanks.

BTW: I again didn't receive your original post, just via digest. Something is wrong here.


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Neil Young]

In order to finalise this discussion with the proper answer here the solution, so that everybody (who is interested) can benefit:

I need to feed OpenCV with the frame delivered by GStreamer (in native code). The OpenCV::Mat constructor is using a parameter called "step", which was set to AUTO_STEP before (which is NOT, what you would suppose an automatic calculation of the padding, but NO_PADDING as documented). Up to now we were in luck and did only meet devices, which gave frames with a padding of zero, so that AUTO_STEP did work.

https://docs.opencv.org/3.4.1/d3/d63/classcv_1_1Mat.html#a51615ebf17a64c968df0bf49b4de6a3a

Except the connection with a DJI drone via Android _and_ on Ubuntu _and_ GStreamer 1.14.4 (right, these were the corner conditions): This came up with an NV12, 720 x 540 frame with a buffer size of 622080, which is not exactly 3/2*w*h.

So the "step" parameter, which describes the "number of bytes each matrix row occupies, including the padding bytes" is now calculated outside the constructor, like so

step = buffer_length * 2 / 3 / height;

which in the documented case gave a step of 768. So this means, the row consists of 740 "real" pixel bytes plus 48 padding bytes, in other words the frame including padding is 768 * 540.

The proper OpenCV constructor call is now:

inputFrame = cv::Mat(height * 3/2, width, CV_8U, ucBuffer, step);

BTW: The same Android device with the same DJI drone behind, but in front of GStreamer 1.16.0 and macOS Mojave (and the same Java app utilising the GStreamer WebRTCBin plugin) came up with an I420 frame of 720 x 540 and a buffer size of 583200. The step in this case is 720, which is exactly the announced width, so there is no padding and it worked with AUTO_STEP...So it was more or less accidentally, that we came across this padding issue...

Somehow brainsick :)

Thanks for the pointers, which made this solution finally possible.

Regards

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel