Where is the support for NV12 with 64x32 tiling is added ?

[ssshukla26]

I am not much familiar with how and where the support for different video formats are added under gstreamer source.

I am currently looking for GST_VIDEO_FORMAT_NV12_64Z32 (NV12 with 64x32 tiling in zigzag pattern) under gstreamer source, but am not able to find anything.

Can anyone please explain how and where the support for NV12 with 64x32 tiling is added under gstreamer source and which plugin/s is affected by this support ?

[Sebastian Dröge-3]

On Do, 2016-04-28 at 23:44 -0700, ssshukla26 wrote:

> I am not much familiar with how and where the support for different video
> formats are added under gstreamer source. 
>
> I am currently looking for *GST_VIDEO_FORMAT_NV12_64Z32* (/NV12 with 64x32
> tiling in zigzag pattern/) under gstreamer source, but am not able to find
> anything.
>
> Can anyone please explain how and where the support for NV12 with 64x32
> tiling is added under gstreamer source and which plugin/s is affected by
> this support ?

All the relevant code for this is in
  gst-plugins-base/gst-libs/gst/video
, including metadata information for that format and conversion
functions to other video formats.

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[ssshukla26]

Ok. Thanks. I was referring under older source codes, on downloading the latest source am able to get it.

Can you please explain how a frame with nv12 64x32 tile format stores Y, U and V data compare to a nv12 4 x 4 pixel image (as shown below).

Byte Order. Each cell is one byte.
start + 0:  Y'00    Y'01    Y'02    Y'03
start + 4:  Y'10    Y'11    Y'12    Y'13
start + 8:  Y'20    Y'21    Y'22    Y'23
start + 12: Y'30    Y'31    Y'32    Y'33
start + 16: Cb00    Cr00    Cb01    Cr01
start + 20: Cb10    Cr10    Cb11    Cr11

Reference :- http://linuxtv.org/downloads/v4l-dvb-apis/re34.html

[Sebastian Dröge-3]

On Fr, 2016-04-29 at 00:40 -0700, ssshukla26 wrote:
> Ok. Thanks. I was referring under older source codes, on downloading the
> latest source am able to get it.
>
> Can you please explain how a frame with *nv12 64x32 tile* format stores Y, U
> and V data compare to a *nv12* 4 x 4 pixel image (as shown below).

The planes are the same as with NV12, but each plane consists of tiles
of the given size. The ziles are stored in a zig-zag order instead of
linearly, while each pixel inside a tile is stored linearly.

Take a look at the code for details and run through the indexing on a
piece of paper to get the general idea.

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-4]

Le dimanche 01 mai 2016 à 10:18 +0300, Sebastian Dröge a écrit :

> On Fr, 2016-04-29 at 00:40 -0700, ssshukla26 wrote:
> >
> > Ok. Thanks. I was referring under older source codes, on
> > downloading the
> > latest source am able to get it.
> >
> > Can you please explain how a frame with *nv12 64x32 tile* format
> > stores Y, U
> > and V data compare to a *nv12* 4 x 4 pixel image (as shown below).
> The planes are the same as with NV12, but each plane consists of
> tiles
> of the given size. The ziles are stored in a zig-zag order instead of
> linearly, while each pixel inside a tile is stored linearly.
>
> Take a look at the code for details and run through the indexing on a
> piece of paper to get the general idea.

For extra details about this specific format, it was implemented base
on the description provided with the Media Kernel API documentation:

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-4]

Le vendredi 29 avril 2016 à 00:40 -0700, ssshukla26 a écrit :

> Ok. Thanks. I was referring under older source codes, on downloading
> the
> latest source am able to get it.
>
> Can you please explain how a frame with *nv12 64x32 tile* format
> stores Y, U
> and V data compare to a *nv12* 4 x 4 pixel image (as shown below).
>
> Byte Order. Each cell is one byte.
> start + 0:  Y'00    Y'01    Y'02    Y'03
> start + 4:  Y'10    Y'11    Y'12    Y'13
> start + 8:  Y'20    Y'21    Y'22    Y'23
> start + 12: Y'30    Y'31    Y'32    Y'33
> start + 16: Cb00    Cr00    Cb01    Cr01
> start + 20: Cb10    Cr10    Cb11    Cr11
>
> Reference :-  http://linuxtv.org/downloads/v4l-dvb-apis/re34.html
> <http://linuxtv.org/downloads/v4l-dvb-apis/re34.html> 

Note that this format is normal NV12, it is already exposed in
GStreamer V4L2 elements. We also support the V4L2 multi-planar variants
(NV12M, NV21M). NV12MT_16x16 support is not there yet, but should be
near to trivial to add.

cheers,
Nicolas
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-4]

Sorry, I fat fingered a way to send the email instead of pasting the
link.

Le dimanche 01 mai 2016 à 17:13 -0400, Nicolas Dufresne a écrit :

> Le dimanche 01 mai 2016 à 10:18 +0300, Sebastian Dröge a écrit :
> >
> > On Fr, 2016-04-29 at 00:40 -0700, ssshukla26 wrote:
> > >
> > >
> > > Ok. Thanks. I was referring under older source codes, on
> > > downloading the
> > > latest source am able to get it.
> > >
> > > Can you please explain how a frame with *nv12 64x32 tile* format
> > > stores Y, U
> > > and V data compare to a *nv12* 4 x 4 pixel image (as shown
> > > below).
> > The planes are the same as with NV12, but each plane consists of
> > tiles
> > of the given size. The ziles are stored in a zig-zag order instead
> > of
> > linearly, while each pixel inside a tile is stored linearly.
> >
> > Take a look at the code for details and run through the indexing on
> > a
> > piece of paper to get the general idea.
> For extra details about this specific format, it was implemented base
> on the description provided with the Media Kernel API documentation:

http://linuxtv.org/downloads/v4l-dvb-apis/re36.html

In GStreamer, we had to create new rules for tiled format. One of the
required information, specially for non-linear layout, was the number
of tiles in X and Y, which can be bigger the the minimum requires
number of tiles (e.g. subimage). As this is similar to stride, while
stride make no sense for such format, we have overloaded the stride
information and encoded the number of tiles in X in the most
significant 16bit, and the number of tiles in Y in the least
significant 16bits. See macros GST_VIDEO_TILE_X_TILES() and
GST_VIDEO_TILE_Y_TILES() and other GST_VIDEO_TILE_* macros.

cheers,
Nicolas
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[ssshukla26]

Hi,

Thanks. I also studied this link. Can you please clarify two more things.

Suppose am looking at a frame of resolution 1280x720 of "NV12 64x32 Tile" format.

What will be size of each tile ? (as per my understanding its 2048)

How to determine number of Horizontal tiles & Vertical tiles per frame ?

[Nicolas Dufresne-4]

Le lundi 02 mai 2016 à 00:12 -0700, ssshukla26 a écrit :

> Hi,
>
> Thanks. I also studied this  link
> <http://linuxtv.org/downloads/v4l-dvb-apis/re36.html>  . Can you
> please
> clarify two more things. 
>
> Suppose am looking at a frame of resolution *1280x720* of "NV12 64x32
> Tile"
> format. 
>
> What will be size of each tile ? (as per my understanding its 2048) 
>
> How to determine number of Horizontal tiles & Vertical tiles per
> frame ?

Tiles are fixed size, 64x32 bytes (2048 bytes). That also means, 64x32
pixels for the Y plane, and 32x32 for the UV interleaved plane. Because
of the Z flip Z pattern, the number of rows must be a multiple of 2
(you must have flip Z after a Z). So you can figure-out the minimum
required column by doing round_up_128 (width) /64. The number of column
will be the same for the UV plane (just like for the stride in NV12).

A driver may require allocating a larger amount of column. For this
reason, we extrapolate the number of columns using the provided stride.
We expect the provided stride to be a multiple of 128 and to be the
representation of the stride if the buffer was de-tiled keeping the
padding bytes.

For the number of rows, it's going to be round_up (32) / 32 Y plane,
and half this (rounded up) for the UV plane. Again, driver may require
more rows, this is extrapolated from the plane size. The layout is
affected by extra rows.

For 720p, you minimally need:

  Y Plane:   10 x 23
  UV Plane:  10 x 12

Nicolas
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[ssshukla26]

Hi,

As you suggested, I made a C program to change NV12 tile to NV12 standard format as follows, but am not getting desired output. Can you please suggest what am missing ?

convert_NV12TileToNV12.c

#include <stdio.h>
#include <stdlib.h>
#include <fcntl.h>
#include <string.h>
#include <errno.h>

#define ROUND_UP_X(num,x) (((num)+(x-1))&~(x-1))

void copyTile(char **dst,char **src,int tile_size)
{
    memcpy(*dst,*src,tile_size);

    *dst = *dst + tile_size;
    *src = *src + tile_size;
}

void NV12TileToNV12(char* src,char* dst,int tile_size,int wTiles,int hTiles)
{
    int i = 0;      //Loop Variable
    int Z = 0;      //For Z Flip Z Pattern
    char *X = NULL; //Source Even Line
    char *Y = NULL; //Source Odd Line

    //For a proper Z Flip Z pattern two consecutive rows are needed to be process
    for(i = 0; i < (hTiles%2 == 0 ? hTiles : hTiles-1); i=i+2)
    {
        //Put X pointer at start of current row
        X = src;
        //Put Y pointer at start of next row
        Y = X + (tile_size * wTiles);

        for(Z = 0; Z < wTiles/2; Z++)
        {
            if(Z%2 == 0)
            {
                //For Z pattern
                copyTile(&dst,&X,tile_size*2);
                copyTile(&dst,&Y,tile_size*2);
            }
            else
            {
                //For Flip Z Pattern
                copyTile(&dst,&Y,tile_size*2);
                copyTile(&dst,&X,tile_size*2);
            }
        }

        //Move source pointer by two rows
        src = src + (2 * tile_size * wTiles);
    }

    //For linear pattern in last remaining row
    if(hTiles%2 != 0)
        copyTile(&dst,&src,(tile_size * wTiles));
}

int main(int argc, char* argv[])
{
    if(5 == argc)
    {
        if(strcmp(argv[1],argv[4]) != 0)
        {
            int infile = -1;
            if(-1 != (infile = open(argv[1],O_RDONLY)))
            {
                int outfile = -1;
                if(-1 != (outfile = open(argv[4],O_WRONLY | O_CREAT,0775)))
                {
                    int width = atoi(argv[2]);
                    int height = atoi(argv[3]);

                    int tile_size = 64*32;                          //Tile size
                    int wTiles = ROUND_UP_X(width,128)/64;          //Minimum no of columns required
                    int hTiles = ROUND_UP_X(height,32)/32;          //Minimun no of rows required for Y Plane
                    int hTiles_UV = ROUND_UP_X(height/2,32)/32;     //Minimum no of rows required for UV Plane

                    int frame_size = (tile_size * wTiles * hTiles) + (tile_size * wTiles * hTiles_UV);
                    char *src_buf = (char *) calloc(1,frame_size);
                    char *dst_buf = (char *) calloc(1,frame_size);

                    printf("tile_size = %d\nwTiles=%d\nhTiles=%d\nhTiles_UV=%d\nframe_size=%d\n",
                            tile_size,wTiles,hTiles,hTiles_UV,frame_size);

                    int cnt = 1;
                    while(read(infile,src_buf,frame_size))
                    {
                        //Convert Y Plane from NV12 tile to NV12 format
                        NV12TileToNV12(src_buf,dst_buf,tile_size,wTiles,hTiles);
                       
                        //Convert UV Plane from NV12 tile to NV12 format
                        NV12TileToNV12(src_buf+(tile_size * wTiles * hTiles),
                                dst_buf+(tile_size * wTiles * hTiles),
                                (tile_size/2),(wTiles*2),hTiles_UV);

                        if(frame_size != write(outfile,dst_buf,frame_size))
                        {
                            perror("Error writing output file :");
                            break;
                        }

                        printf("No of frames converted : %d\r",cnt++);
                        fflush(stdout);

                        memset(src_buf,'\0',sizeof(char) * frame_size);
                        memset(dst_buf,'\0',sizeof(char) * frame_size);
                    }

                    printf("\rNo of frames converted : %d\n",cnt);
                    fflush(stdout);

                    close(outfile);
                }
                else
                {
                    perror("Unable to open output file :");
                }

                close(infile);
            }
            else
            {
                perror("Unable to open input file :");
            }
        }
        else
        {
            printf("Input file name and output file name can't be same\n");
        }

    }
    else
    {
        printf("Usage: ./convert input_file width height output_file\n");
    }

    return 0;
}

[Sebastian Dröge-3]

On Do, 2016-05-05 at 01:50 -0700, ssshukla26 wrote:
> Hi,
>
> As you suggested, I made a C program to change NV12 tile to NV12 standard
> format as follows, but am not getting desired output. Can you please suggest
> what am missing ?

This has nothing to do with GStreamer anymore. If you want to do the
conversion with GStreamer you can use the GstVideoConverter API from
libgstvideo or use the videoconvert element.

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[ssshukla26]

Ok fine, can you tell me where I can find a similar logic in gstreamer source ? Cause in have to anyhow derive by which ALGO (pixel by pixel code) NV12 tile to NV12 format is converted. Will you help me to find the place where you people have actually put pixel by pixel logic for conversion !

[Sebastian Dröge-3]

On Fr, 2016-05-06 at 02:23 -0700, ssshukla26 wrote:
> Ok fine, can you tell me where I can find a similar logic in gstreamer source
> ? Cause in have to anyhow derive by which ALGO (pixel by pixel code) NV12
> tile to NV12 format is converted. Will you help me to find the place where
> you people have actually put pixel by pixel logic for conversion !

It's all in gst-plugins-base/gst-libs/gst/video . I think there is
currently no fast-path for directly going to NV12 but only for going to
AYUV as an intermediate step, but that's basically the same.

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-4]

Le vendredi 06 mai 2016 à 14:04 +0300, Sebastian Dröge a écrit :

> On Fr, 2016-05-06 at 02:23 -0700, ssshukla26 wrote:
> >
> > Ok fine, can you tell me where I can find a similar logic in
> > gstreamer source
> > ? Cause in have to anyhow derive by which ALGO (pixel by pixel
> > code) NV12
> > tile to NV12 format is converted. Will you help me to find the
> > place where
> > you people have actually put pixel by pixel logic for conversion !
> It's all in gst-plugins-base/gst-libs/gst/video . I think there is
> currently no fast-path for directly going to NV12 but only for going
> to
> AYUV as an intermediate step, but that's basically the same.

We still have a denser version in android media plugin. There is bunch
of different method you'll find in Android vendor trees too.

https://cgit.freedesktop.org/gstreamer/gst-plugins-
bad/tree/sys/androidmedia/gstamc.c#n2460
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-4]

Sorry, I fat fingered a way to send the email instead of pasting the
link.

Le dimanche 01 mai 2016 à 17:13 -0400, Nicolas Dufresne a écrit :

> Le dimanche 01 mai 2016 à 10:18 +0300, Sebastian Dröge a écrit :
> >
> > On Fr, 2016-04-29 at 00:40 -0700, ssshukla26 wrote:
> > >
> > >
> > > Ok. Thanks. I was referring under older source codes, on
> > > downloading the
> > > latest source am able to get it.
> > >
> > > Can you please explain how a frame with *nv12 64x32 tile* format
> > > stores Y, U
> > > and V data compare to a *nv12* 4 x 4 pixel image (as shown
> > > below).
> > The planes are the same as with NV12, but each plane consists of
> > tiles
> > of the given size. The ziles are stored in a zig-zag order instead
> > of
> > linearly, while each pixel inside a tile is stored linearly.
> >
> > Take a look at the code for details and run through the indexing on
> > a
> > piece of paper to get the general idea.
> For extra details about this specific format, it was implemented base
> on the description provided with the Media Kernel API documentation:

http://linuxtv.org/downloads/v4l-dvb-apis/re36.html

In GStreamer, we had to create new rules for tiled format. One of the
required information, specially for non-linear layout, was the number
of tiles in X and Y, which can be bigger the the minimum requires
number of tiles (e.g. subimage). As this is similar to stride, while
stride make no sense for such format, we have overloaded the stride
information and encoded the number of tiles in X in the most
significant 16bit, and the number of tiles in Y in the least
significant 16bits. See 
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[ssshukla26]

I just solved the tile conversion. Please refer this link for NV12Tile to NV12 conversion.

[Nicolas Dufresne-4]

Le jeudi 16 juin 2016 à 23:43 -0700, ssshukla26 a écrit :
> I just solved the tile conversion. Please refer this  link
> <https://github.com/ssshukla26/NV12Tile-To-NV12-Conversion>   for
> NV12Tile
> to NV12 conversion.

This repository is not public. When you say you solved it, what do you
mean ? It already worked in software in GStreamer.

regards,
Nicolas
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[ssshukla26]

I have to take the repo down because I was updating a patch for ffmpeg and wasn't sure that the logic will work out of the box. Now the repo is public.

Sorry to use word "solved" I mean I pasted the pixel by pixel logic in a C file so that anyone can understand the conversion logic from NV12Tile to NV12, and yes it already solved in gstreamer.

Thanks.