proposal: support for row-stride in gstreamer

[Clark, Rob]

Hi gstreamer folks,

The following is a proposal for how to add row-stride (and possibly  
some related changes) to gstreamer.  I gave a couple of possible  
examples of where this would be useful, but it is probably not  
exhaustive.  Please let me know if you see any cases that I missed, or  
details that I overlooked, etc.



Use-cases:
----------
  + display hardware with special constraints on image dimensions, for  
example
    if the output buffer must have dimensions that are a power of two
  + zero-copy cropping of image / videoframe (at least for interleaved  
color
    formats.. more on this later)

One example to think about is rendering onto a 3d surface.  In some  
cases, graphics hardware could require that the surface dimensions are  
a power of 2.  In this case, you would want the vsink to allocate a  
buffer with a rowstride that is the next larger power of 2 from the  
image width.


Another example to think about is video stabilization.  In this use  
case, you would ask the camera to capture an oversized frame.  Your  
vstab algorithm would calculate an x,y offset of the stabilized  
image.  But if the decoder understands rowstride, you do not need to  
actually copy the image buffer.  Say, just to pick some numbers, you  
want your final output to be 640x480, and you want your oversized  
frame to be +20% in each dimension (768x576):

    +--------+           +-------+           +------+
    | camera |---------->| vstab |---------->| venc |
    +--------+ width=768 +-------+ width=640 +------+
              height=576          height=480
           rowstride=768       rowstride=768

In the case of an interleaved color format (RGB, UYVY, etc), you could  
simply increment the 'data' pointer in the buffer by (y*rowstride)+x.  
No memcpy() required.  As long as the video encoder respects the  
rowstride, it will see the stabilized frame correctly.



Proposal:
---------

In all cases that I can think of, the row-stride will not be changing  
dynamically.  So this parameter can be negotiated thru caps  
negotiation in the same way as image width/height, colorformat, etc.  
However, we need to know conclusively that there is no element in the  
pipeline that cares about the image format, but does not understand  
"rowstride", so we cannot use existing type strings (ex. "video/x-raw-
yuv").  And, at least in the cases that I can think of, the video sink  
will dictate the row-stride.  So upstream caps-renegotiation will be  
used to arrive at the final "rowstride" value.

For media types, I propose to continue using existing strings for non-
stride-aware element caps, ex. "video/x-raw-yuv".  For stride-aware  
elements, they can support a second format, ex. "video/x-raw-yuv-
strided", "image/x-raw-rgb-strided", etc (ie. append "-strided" to  
whatever the existing string is).  In the case that a strided format  
is negotiated, it is required for there to also be a "rowstride" entry  
in the final negotiated caps.

question: in general, most elements supporting rowstride will have no  
constraint on what particular rowstride values are supported.  Do they  
just list "rowstride=[0-4294967295]" in their caps template?  The  
video sink allocating the buffer will likely have some constraints on  
rowstride, although this will be a function of the width (for example,  
round the width up to next power of two).

We will implement some sort of GstRowStrideTransform element to  
interface between stride-aware and non-stride-aware elements.




Non-Interleaved Color Formats:
------------------------------

So, everything I've said so far about zero-copy cropping works until  
you start considering planar/semi-planar color formats which do not  
have equal size planes.  For example, consider NV12:  the offset to  
add to the Y plane is (y*rowstride)+x, but the offset to add to UV  
plane is ((y/2)*rowstride)+x.  There are only three ways that I can  
think of to deal with this (listed in order of my preference):

  1) add fields to GstBuffer to pass additional pointers to the other  
color
     planes within the same GstBuffer
  2) add a field(s) to GstBuffer to pass an offset..  either an x,y  
offset, or a
     single value that is (y*rowstride)+x.  Either way, the various  
elements in
     the pipeline can use this to calculate the start of the  
individual planes
     of data.
  3) pass individual planes of a single image as separate  
GstBuffer's.. but I'm
     not a huge fan of this because now every element needs to have  
some sort
     of "I've got Y, but I'm waiting for UV" state.

I'm not sure if anyone has any thoughts about which of these three  
approaches is preferred.  Or any alternative ideas?



BR,
-Rob


------------------------------------------------------------------------------
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

[Clark, Rob]

btw, I noticed the proposal about GstBuffer metadata:

  <http://cgit.freedesktop.org/gstreamer/gstreamer/tree/docs/design/draft-buffer2.txt 
 >

this provides a nice way to handle x,y vstab coordinates mentioned  
below.

In the cases that I can think of, rowstride would not change from  
buffer to buffer, so I would propose to handle this in caps  
negotation, as proposed originally.  But I would like to use per-
buffer meta-data to handle the per-buffer cropping/panning.

Comments?

BR,
-R


On Jul 2, 2009, at 6:43 PM, Clark, Rob wrote:

> Hi gstreamer folks,
>
> The following is a proposal for how to add row-stride (and possibly
> some related changes) to gstreamer.  I gave a couple of possible
> examples of where this would be useful, but it is probably not
> exhaustive.  Please let me know if you see any cases that I missed, or
> details that I overlooked, etc.
>
>
>
> Use-cases:
> ----------
>  + display hardware with special constraints on image dimensions, for
> example
>    if the output buffer must have dimensions that are a power of two
>  + zero-copy cropping of image / videoframe (at least for interleaved
> color
>    formats.. more on this later)
>
> One example to think about is rendering onto a 3d surface.  In some
> cases, graphics hardware could require that the surface dimensions are
> a power of 2.  In this case, you would want the vsink to allocate a
> buffer with a rowstride that is the next larger power of 2 from the
> image width.
>
>
> Another example to think about is video stabilization.  In this use
> case, you would ask the camera to capture an oversized frame.  Your
> vstab algorithm would calculate an x,y offset of the stabilized
> image.  But if the decoder understands rowstride, you do not need to
> actually copy the image buffer.  Say, just to pick some numbers, you
> want your final output to be 640x480, and you want your oversized
> frame to be +20% in each dimension (768x576):
>
>    +--------+           +-------+           +------+
>    | camera |---------->| vstab |---------->| venc |
>    +--------+ width=768 +-------+ width=640 +------+
>              height=576          height=480
>           rowstride=768       rowstride=768
>
> In the case of an interleaved color format (RGB, UYVY, etc), you could
> simply increment the 'data' pointer in the buffer by (y*rowstride)+x.
> No memcpy() required.  As long as the video encoder respects the
> rowstride, it will see the stabilized frame correctly.
>
>
>
> Proposal:
> ---------
>
> In all cases that I can think of, the row-stride will not be changing
> dynamically.  So this parameter can be negotiated thru caps
> negotiation in the same way as image width/height, colorformat, etc.
> However, we need to know conclusively that there is no element in the
> pipeline that cares about the image format, but does not understand
> "rowstride", so we cannot use existing type strings (ex. "video/x-raw-
> yuv").  And, at least in the cases that I can think of, the video sink
> will dictate the row-stride.  So upstream caps-renegotiation will be
> used to arrive at the final "rowstride" value.
>
> For media types, I propose to continue using existing strings for non-
> stride-aware element caps, ex. "video/x-raw-yuv".  For stride-aware
> elements, they can support a second format, ex. "video/x-raw-yuv-
> strided", "image/x-raw-rgb-strided", etc (ie. append "-strided" to
> whatever the existing string is).  In the case that a strided format
> is negotiated, it is required for there to also be a "rowstride" entry
> in the final negotiated caps.
>
> question: in general, most elements supporting rowstride will have no
> constraint on what particular rowstride values are supported.  Do they
> just list "rowstride=[0-4294967295]" in their caps template?  The
> video sink allocating the buffer will likely have some constraints on
> rowstride, although this will be a function of the width (for example,
> round the width up to next power of two).
>
> We will implement some sort of GstRowStrideTransform element to
> interface between stride-aware and non-stride-aware elements.
>
>
>
>
> Non-Interleaved Color Formats:
> ------------------------------
>
> So, everything I've said so far about zero-copy cropping works until
> you start considering planar/semi-planar color formats which do not
> have equal size planes.  For example, consider NV12:  the offset to
> add to the Y plane is (y*rowstride)+x, but the offset to add to UV
> plane is ((y/2)*rowstride)+x.  There are only three ways that I can
> think of to deal with this (listed in order of my preference):
>
>  1) add fields to GstBuffer to pass additional pointers to the other
> color
>     planes within the same GstBuffer
>  2) add a field(s) to GstBuffer to pass an offset..  either an x,y
> offset, or a
>     single value that is (y*rowstride)+x.  Either way, the various
> elements in
>     the pipeline can use this to calculate the start of the
> individual planes
>     of data.
>  3) pass individual planes of a single image as separate
> GstBuffer's.. but I'm
>     not a huge fan of this because now every element needs to have
> some sort
>     of "I've got Y, but I'm waiting for UV" state.
>
> I'm not sure if anyone has any thoughts about which of these three
> approaches is preferred.  Or any alternative ideas?
>
>
>
> BR,
> -Rob
>

------------------------------------------------------------------------------
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

[Jan Schmidt-6]

On Tue, 2009-07-21 at 20:44 -0500, Rob Clark wrote:

> btw, I noticed the proposal about GstBuffer metadata:
>
>   <http://cgit.freedesktop.org/gstreamer/gstreamer/tree/docs/design/draft-buffer2.txt 
>  >
>
> this provides a nice way to handle x,y vstab coordinates mentioned  
> below.
>
> In the cases that I can think of, rowstride would not change from  
> buffer to buffer, so I would propose to handle this in caps  
> negotation, as proposed originally.  But I would like to use per-
> buffer meta-data to handle the per-buffer cropping/panning.

Exactly what I've had in mind for a while - and exactly what we
discussed at the recent "GStreamer 1.0" summit in Gran Canaria.

Now, we just need to find the time to implement stuff, and I need to
write up my notes from GCDS in a more digestible form and post them.

Cheers,
Jan.
 

>
> Comments?
>
> BR,
> -R
>
>
> On Jul 2, 2009, at 6:43 PM, Clark, Rob wrote:
>
> > Hi gstreamer folks,
> >
> > The following is a proposal for how to add row-stride (and possibly
> > some related changes) to gstreamer.  I gave a couple of possible
> > examples of where this would be useful, but it is probably not
> > exhaustive.  Please let me know if you see any cases that I missed, or
> > details that I overlooked, etc.
> >
> >
> >
> > Use-cases:
> > ----------
> >  + display hardware with special constraints on image dimensions, for
> > example
> >    if the output buffer must have dimensions that are a power of two
> >  + zero-copy cropping of image / videoframe (at least for interleaved
> > color
> >    formats.. more on this later)
> >
> > One example to think about is rendering onto a 3d surface.  In some
> > cases, graphics hardware could require that the surface dimensions are
> > a power of 2.  In this case, you would want the vsink to allocate a
> > buffer with a rowstride that is the next larger power of 2 from the
> > image width.
> >
> >
> > Another example to think about is video stabilization.  In this use
> > case, you would ask the camera to capture an oversized frame.  Your
> > vstab algorithm would calculate an x,y offset of the stabilized
> > image.  But if the decoder understands rowstride, you do not need to
> > actually copy the image buffer.  Say, just to pick some numbers, you
> > want your final output to be 640x480, and you want your oversized
> > frame to be +20% in each dimension (768x576):
> >
> >    +--------+           +-------+           +------+
> >    | camera |---------->| vstab |---------->| venc |
> >    +--------+ width=768 +-------+ width=640 +------+
> >              height=576          height=480
> >           rowstride=768       rowstride=768
> >
> > In the case of an interleaved color format (RGB, UYVY, etc), you could
> > simply increment the 'data' pointer in the buffer by (y*rowstride)+x.
> > No memcpy() required.  As long as the video encoder respects the
> > rowstride, it will see the stabilized frame correctly.
> >
> >
> >
> > Proposal:
> > ---------
> >
> > In all cases that I can think of, the row-stride will not be changing
> > dynamically.  So this parameter can be negotiated thru caps
> > negotiation in the same way as image width/height, colorformat, etc.
> > However, we need to know conclusively that there is no element in the
> > pipeline that cares about the image format, but does not understand
> > "rowstride", so we cannot use existing type strings (ex. "video/x-raw-
> > yuv").  And, at least in the cases that I can think of, the video sink
> > will dictate the row-stride.  So upstream caps-renegotiation will be
> > used to arrive at the final "rowstride" value.
> >
> > For media types, I propose to continue using existing strings for non-
> > stride-aware element caps, ex. "video/x-raw-yuv".  For stride-aware
> > elements, they can support a second format, ex. "video/x-raw-yuv-
> > strided", "image/x-raw-rgb-strided", etc (ie. append "-strided" to
> > whatever the existing string is).  In the case that a strided format
> > is negotiated, it is required for there to also be a "rowstride" entry
> > in the final negotiated caps.
> >
> > question: in general, most elements supporting rowstride will have no
> > constraint on what particular rowstride values are supported.  Do they
> > just list "rowstride=[0-4294967295]" in their caps template?  The
> > video sink allocating the buffer will likely have some constraints on
> > rowstride, although this will be a function of the width (for example,
> > round the width up to next power of two).
> >
> > We will implement some sort of GstRowStrideTransform element to
> > interface between stride-aware and non-stride-aware elements.
> >
> >
> >
> >
> > Non-Interleaved Color Formats:
> > ------------------------------
> >
> > So, everything I've said so far about zero-copy cropping works until
> > you start considering planar/semi-planar color formats which do not
> > have equal size planes.  For example, consider NV12:  the offset to
> > add to the Y plane is (y*rowstride)+x, but the offset to add to UV
> > plane is ((y/2)*rowstride)+x.  There are only three ways that I can
> > think of to deal with this (listed in order of my preference):
> >
> >  1) add fields to GstBuffer to pass additional pointers to the other
> > color
> >     planes within the same GstBuffer
> >  2) add a field(s) to GstBuffer to pass an offset..  either an x,y
> > offset, or a
> >     single value that is (y*rowstride)+x.  Either way, the various
> > elements in
> >     the pipeline can use this to calculate the start of the
> > individual planes
> >     of data.
> >  3) pass individual planes of a single image as separate
> > GstBuffer's.. but I'm
> >     not a huge fan of this because now every element needs to have
> > some sort
> >     of "I've got Y, but I'm waiting for UV" state.
> >
> > I'm not sure if anyone has any thoughts about which of these three
> > approaches is preferred.  Or any alternative ideas?
> >
> >
> >
> > BR,
> > -Rob
> >
>
>
> ------------------------------------------------------------------------------
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>

--
Jan Schmidt <[hidden email]>



------------------------------------------------------------------------------
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

[Clark, Rob]

On Jul 22, 2009, at 3:23 AM, Jan Schmidt wrote:

> On Tue, 2009-07-21 at 20:44 -0500, Rob Clark wrote:
>> btw, I noticed the proposal about GstBuffer metadata:
>>
>>  <http://cgit.freedesktop.org/gstreamer/gstreamer/tree/docs/design/draft-buffer2.txt
>>>
>>
>> this provides a nice way to handle x,y vstab coordinates mentioned
>> below.
>>
>> In the cases that I can think of, rowstride would not change from
>> buffer to buffer, so I would propose to handle this in caps
>> negotation, as proposed originally.  But I would like to use per-
>> buffer meta-data to handle the per-buffer cropping/panning.
>
> Exactly what I've had in mind for a while - and exactly what we
> discussed at the recent "GStreamer 1.0" summit in Gran Canaria.
>
> Now, we just need to find the time to implement stuff, and I need to
> write up my notes from GCDS in a more digestible form and post them.
>

I would be definitely interested to see the notes from GCDS..  I heard  
second-hand some proposal about caps-negotiation, but would like to  
know more..

 From my end, the relatively short term need is just for rowstride.  
Fortunately, I think this should not be such a big code-change as not  
all elements need to support it.  We'll add support for it in gst-
openmax video encoder/decoder elements, and at least v4l2sink (which,  
btw, I have working now in my private fork of gst-plugins-good[1],  
although I need to do a bit more testing and it could use some eyes).  
I would be interested to eventually add this to xvimagesink too,  
although I'm not sure yet when I'll get to this, so if someone has a  
more short-term interest in this, please don't wait for me ;-)

We will anyways implement a transform element that can convert from  
strided to non-strided buffer, which would be useful between elements  
requiring stride and those that don't support it (and also, just for  
debugging, to dump out raw decoded frames without stride to a filesrc,  
and visa versa).

For slightly longer term (ie. between now and December-ish), if there  
are ways to divide up the work for per-buffer meta-data, maybe we can  
help with some parts of the implementation?  I haven't though about it  
too much, but if it there is a clean way to break it up into sub-
tasks, I'm all for trying to help if we can.  At a minimum, I'll have  
some weekend and evening time where I'd be happy to help.


BR,
-R

[1] see: <http://github.com/robclark/gst-plugins-good/tree/4065c9e1ff927604d9a52c944eeee1047a24d283/sys/v4l2 
 >

> Cheers,
> Jan.
>
>>
>> Comments?
>>
>> BR,
>> -R
>>
>>
>> On Jul 2, 2009, at 6:43 PM, Clark, Rob wrote:
>>
>>> Hi gstreamer folks,
>>>
>>> The following is a proposal for how to add row-stride (and possibly
>>> some related changes) to gstreamer.  I gave a couple of possible
>>> examples of where this would be useful, but it is probably not
>>> exhaustive.  Please let me know if you see any cases that I  
>>> missed, or
>>> details that I overlooked, etc.
>>>
>>>
>>>
>>> Use-cases:
>>> ----------
>>> + display hardware with special constraints on image dimensions, for
>>> example
>>>   if the output buffer must have dimensions that are a power of two
>>> + zero-copy cropping of image / videoframe (at least for interleaved
>>> color
>>>   formats.. more on this later)
>>>
>>> One example to think about is rendering onto a 3d surface.  In some
>>> cases, graphics hardware could require that the surface dimensions  
>>> are
>>> a power of 2.  In this case, you would want the vsink to allocate a
>>> buffer with a rowstride that is the next larger power of 2 from the
>>> image width.
>>>
>>>
>>> Another example to think about is video stabilization.  In this use
>>> case, you would ask the camera to capture an oversized frame.  Your
>>> vstab algorithm would calculate an x,y offset of the stabilized
>>> image.  But if the decoder understands rowstride, you do not need to
>>> actually copy the image buffer.  Say, just to pick some numbers, you
>>> want your final output to be 640x480, and you want your oversized
>>> frame to be +20% in each dimension (768x576):
>>>
>>>   +--------+           +-------+           +------+
>>>   | camera |---------->| vstab |---------->| venc |
>>>   +--------+ width=768 +-------+ width=640 +------+
>>>             height=576          height=480
>>>          rowstride=768       rowstride=768
>>>
>>> In the case of an interleaved color format (RGB, UYVY, etc), you  
>>> could
>>> simply increment the 'data' pointer in the buffer by (y*rowstride)
>>> +x.
>>> No memcpy() required.  As long as the video encoder respects the
>>> rowstride, it will see the stabilized frame correctly.
>>>
>>>
>>>
>>> Proposal:
>>> ---------
>>>
>>> In all cases that I can think of, the row-stride will not be  
>>> changing
>>> dynamically.  So this parameter can be negotiated thru caps
>>> negotiation in the same way as image width/height, colorformat, etc.
>>> However, we need to know conclusively that there is no element in  
>>> the
>>> pipeline that cares about the image format, but does not understand
>>> "rowstride", so we cannot use existing type strings (ex. "video/x-
>>> raw-
>>> yuv").  And, at least in the cases that I can think of, the video  
>>> sink
>>> will dictate the row-stride.  So upstream caps-renegotiation will be
>>> used to arrive at the final "rowstride" value.
>>>
>>> For media types, I propose to continue using existing strings for  
>>> non-
>>> stride-aware element caps, ex. "video/x-raw-yuv".  For stride-aware
>>> elements, they can support a second format, ex. "video/x-raw-yuv-
>>> strided", "image/x-raw-rgb-strided", etc (ie. append "-strided" to
>>> whatever the existing string is).  In the case that a strided format
>>> is negotiated, it is required for there to also be a "rowstride"  
>>> entry
>>> in the final negotiated caps.
>>>
>>> question: in general, most elements supporting rowstride will have  
>>> no
>>> constraint on what particular rowstride values are supported.  Do  
>>> they
>>> just list "rowstride=[0-4294967295]" in their caps template?  The
>>> video sink allocating the buffer will likely have some constraints  
>>> on
>>> rowstride, although this will be a function of the width (for  
>>> example,
>>> round the width up to next power of two).
>>>
>>> We will implement some sort of GstRowStrideTransform element to
>>> interface between stride-aware and non-stride-aware elements.
>>>
>>>
>>>
>>>
>>> Non-Interleaved Color Formats:
>>> ------------------------------
>>>
>>> So, everything I've said so far about zero-copy cropping works until
>>> you start considering planar/semi-planar color formats which do not
>>> have equal size planes.  For example, consider NV12:  the offset to
>>> add to the Y plane is (y*rowstride)+x, but the offset to add to UV
>>> plane is ((y/2)*rowstride)+x.  There are only three ways that I can
>>> think of to deal with this (listed in order of my preference):
>>>
>>> 1) add fields to GstBuffer to pass additional pointers to the other
>>> color
>>>    planes within the same GstBuffer
>>> 2) add a field(s) to GstBuffer to pass an offset..  either an x,y
>>> offset, or a
>>>    single value that is (y*rowstride)+x.  Either way, the various
>>> elements in
>>>    the pipeline can use this to calculate the start of the
>>> individual planes
>>>    of data.
>>> 3) pass individual planes of a single image as separate
>>> GstBuffer's.. but I'm
>>>    not a huge fan of this because now every element needs to have
>>> some sort
>>>    of "I've got Y, but I'm waiting for UV" state.
>>>
>>> I'm not sure if anyone has any thoughts about which of these three
>>> approaches is preferred.  Or any alternative ideas?
>>>
>>>
>>>
>>> BR,
>>> -Rob
>>>
>>
>>
>> ------------------------------------------------------------------------------
>> _______________________________________________
>> gstreamer-devel mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>>
>
>
> --
> Jan Schmidt <[hidden email]>
>
>
>
> ------------------------------------------------------------------------------
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>

------------------------------------------------------------------------------
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

[Thomas Vander Stichele]

On Wed, 2009-07-22 at 09:23 +0100, Jan Schmidt wrote:

> On Tue, 2009-07-21 at 20:44 -0500, Rob Clark wrote:
> > btw, I noticed the proposal about GstBuffer metadata:
> >
> >   <http://cgit.freedesktop.org/gstreamer/gstreamer/tree/docs/design/draft-buffer2.txt 
> >  >
> >
> > this provides a nice way to handle x,y vstab coordinates mentioned  
> > below.
> >
> > In the cases that I can think of, rowstride would not change from  
> > buffer to buffer, so I would propose to handle this in caps  
> > negotation, as proposed originally.  But I would like to use per-
> > buffer meta-data to handle the per-buffer cropping/panning.
>
> Exactly what I've had in mind for a while - and exactly what we
> discussed at the recent "GStreamer 1.0" summit in Gran Canaria.
>
> Now, we just need to find the time to implement stuff, and I need to
> write up my notes from GCDS in a more digestible form and post them.

Hey ! You mean my notes ? Thief !

Uh, I meant, hey Jan, have you massaged my notes in something normal
people can understand too yet ? Let me know if you need help.

Thomas


--
Nobody cares when you're gone
--
GStreamer - bringing multimedia to your desktop
http://gstreamer.freedesktop.org/



------------------------------------------------------------------------------
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

[Clark, Rob]

so, my next steps (at least my current thinking about next steps) are  
to start adding some rowstride related stuff in gst-plugins-base:


gst-libs/gst/video/video.c:
1) add gst_video_format_get_size_strided()
2) change gst_video_format_parse_caps() to understand strided caps,  
and add a gst_video_format_parse_caps_strided() function  (it seems  
there are enough places already using gst_video_format_parse_caps()  
that I probably don't want to change the signature of this function)

and then add a GstStrideTransform element under gst/stride


Comments?  Suggestions?  Objections?


BR,
-R


On Jul 22, 2009, at 3:23 AM, Jan Schmidt wrote:

> On Tue, 2009-07-21 at 20:44 -0500, Rob Clark wrote:
>> btw, I noticed the proposal about GstBuffer metadata:
>>
>>  <http://cgit.freedesktop.org/gstreamer/gstreamer/tree/docs/design/draft-buffer2.txt
>>>
>>
>> this provides a nice way to handle x,y vstab coordinates mentioned
>> below.
>>
>> In the cases that I can think of, rowstride would not change from
>> buffer to buffer, so I would propose to handle this in caps
>> negotation, as proposed originally.  But I would like to use per-
>> buffer meta-data to handle the per-buffer cropping/panning.
>
> Exactly what I've had in mind for a while - and exactly what we
> discussed at the recent "GStreamer 1.0" summit in Gran Canaria.
>
> Now, we just need to find the time to implement stuff, and I need to
> write up my notes from GCDS in a more digestible form and post them.
>
> Cheers,
> Jan.
>
>>
>> Comments?
>>
>> BR,
>> -R
>>
>>
>> On Jul 2, 2009, at 6:43 PM, Clark, Rob wrote:
>>
>>> Hi gstreamer folks,
>>>
>>> The following is a proposal for how to add row-stride (and possibly
>>> some related changes) to gstreamer.  I gave a couple of possible
>>> examples of where this would be useful, but it is probably not
>>> exhaustive.  Please let me know if you see any cases that I  
>>> missed, or
>>> details that I overlooked, etc.
>>>
>>>
>>>
>>> Use-cases:
>>> ----------
>>> + display hardware with special constraints on image dimensions, for
>>> example
>>>   if the output buffer must have dimensions that are a power of two
>>> + zero-copy cropping of image / videoframe (at least for interleaved
>>> color
>>>   formats.. more on this later)
>>>
>>> One example to think about is rendering onto a 3d surface.  In some
>>> cases, graphics hardware could require that the surface dimensions  
>>> are
>>> a power of 2.  In this case, you would want the vsink to allocate a
>>> buffer with a rowstride that is the next larger power of 2 from the
>>> image width.
>>>
>>>
>>> Another example to think about is video stabilization.  In this use
>>> case, you would ask the camera to capture an oversized frame.  Your
>>> vstab algorithm would calculate an x,y offset of the stabilized
>>> image.  But if the decoder understands rowstride, you do not need to
>>> actually copy the image buffer.  Say, just to pick some numbers, you
>>> want your final output to be 640x480, and you want your oversized
>>> frame to be +20% in each dimension (768x576):
>>>
>>>   +--------+           +-------+           +------+
>>>   | camera |---------->| vstab |---------->| venc |
>>>   +--------+ width=768 +-------+ width=640 +------+
>>>             height=576          height=480
>>>          rowstride=768       rowstride=768
>>>
>>> In the case of an interleaved color format (RGB, UYVY, etc), you  
>>> could
>>> simply increment the 'data' pointer in the buffer by (y*rowstride)
>>> +x.
>>> No memcpy() required.  As long as the video encoder respects the
>>> rowstride, it will see the stabilized frame correctly.
>>>
>>>
>>>
>>> Proposal:
>>> ---------
>>>
>>> In all cases that I can think of, the row-stride will not be  
>>> changing
>>> dynamically.  So this parameter can be negotiated thru caps
>>> negotiation in the same way as image width/height, colorformat, etc.
>>> However, we need to know conclusively that there is no element in  
>>> the
>>> pipeline that cares about the image format, but does not understand
>>> "rowstride", so we cannot use existing type strings (ex. "video/x-
>>> raw-
>>> yuv").  And, at least in the cases that I can think of, the video  
>>> sink
>>> will dictate the row-stride.  So upstream caps-renegotiation will be
>>> used to arrive at the final "rowstride" value.
>>>
>>> For media types, I propose to continue using existing strings for  
>>> non-
>>> stride-aware element caps, ex. "video/x-raw-yuv".  For stride-aware
>>> elements, they can support a second format, ex. "video/x-raw-yuv-
>>> strided", "image/x-raw-rgb-strided", etc (ie. append "-strided" to
>>> whatever the existing string is).  In the case that a strided format
>>> is negotiated, it is required for there to also be a "rowstride"  
>>> entry
>>> in the final negotiated caps.
>>>
>>> question: in general, most elements supporting rowstride will have  
>>> no
>>> constraint on what particular rowstride values are supported.  Do  
>>> they
>>> just list "rowstride=[0-4294967295]" in their caps template?  The
>>> video sink allocating the buffer will likely have some constraints  
>>> on
>>> rowstride, although this will be a function of the width (for  
>>> example,
>>> round the width up to next power of two).
>>>
>>> We will implement some sort of GstRowStrideTransform element to
>>> interface between stride-aware and non-stride-aware elements.
>>>
>>>
>>>
>>>
>>> Non-Interleaved Color Formats:
>>> ------------------------------
>>>
>>> So, everything I've said so far about zero-copy cropping works until
>>> you start considering planar/semi-planar color formats which do not
>>> have equal size planes.  For example, consider NV12:  the offset to
>>> add to the Y plane is (y*rowstride)+x, but the offset to add to UV
>>> plane is ((y/2)*rowstride)+x.  There are only three ways that I can
>>> think of to deal with this (listed in order of my preference):
>>>
>>> 1) add fields to GstBuffer to pass additional pointers to the other
>>> color
>>>    planes within the same GstBuffer
>>> 2) add a field(s) to GstBuffer to pass an offset..  either an x,y
>>> offset, or a
>>>    single value that is (y*rowstride)+x.  Either way, the various
>>> elements in
>>>    the pipeline can use this to calculate the start of the
>>> individual planes
>>>    of data.
>>> 3) pass individual planes of a single image as separate
>>> GstBuffer's.. but I'm
>>>    not a huge fan of this because now every element needs to have
>>> some sort
>>>    of "I've got Y, but I'm waiting for UV" state.
>>>
>>> I'm not sure if anyone has any thoughts about which of these three
>>> approaches is preferred.  Or any alternative ideas?
>>>
>>>
>>>
>>> BR,
>>> -Rob
>>>
>>
>>
>> ------------------------------------------------------------------------------
>> _______________________________________________
>> gstreamer-devel mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>>
>
>
> --
> Jan Schmidt <[hidden email]>
>
>
>
> ------------------------------------------------------------------------------
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

[Jan Schmidt-6]

Hi,

On Thu, 2009-07-30 at 11:45 -0500, Rob Clark wrote:

> so, my next steps (at least my current thinking about next steps) are  
> to start adding some rowstride related stuff in gst-plugins-base:
>
>
> gst-libs/gst/video/video.c:
> 1) add gst_video_format_get_size_strided()
> 2) change gst_video_format_parse_caps() to understand strided caps,  
> and add a gst_video_format_parse_caps_strided() function  (it seems  
> there are enough places already using gst_video_format_parse_caps()  
> that I probably don't want to change the signature of this function)
>
> and then add a GstStrideTransform element under gst/stride

I haven't put enough thought into designing a stride system to say
anything for certain. Particularly, I'm not sure whether we'll be able
to safely and successfully integrate it into the 0.10 series without
thinking about it harder.

I am pretty sure, however, that adding a separate stride adjust element
is the wrong way to go. None of the existing pipelines will include it,
so it would no be useful without changes to the applications/playbin
etc.

My hunch is that it will be better to add a new 'video/x-raw-yuv-full'
format and add stride as a new parameter, and adjust ffmpegcolorspace to
support conversions to/from the unstrided/strided formats.

- Jan.

> Comments?  Suggestions?  Objections?
>
>
> BR,
> -R
>
>
> On Jul 22, 2009, at 3:23 AM, Jan Schmidt wrote:
>
> > On Tue, 2009-07-21 at 20:44 -0500, Rob Clark wrote:
> >> btw, I noticed the proposal about GstBuffer metadata:
> >>
> >>  <http://cgit.freedesktop.org/gstreamer/gstreamer/tree/docs/design/draft-buffer2.txt
> >>>
> >>
> >> this provides a nice way to handle x,y vstab coordinates mentioned
> >> below.
> >>
> >> In the cases that I can think of, rowstride would not change from
> >> buffer to buffer, so I would propose to handle this in caps
> >> negotation, as proposed originally.  But I would like to use per-
> >> buffer meta-data to handle the per-buffer cropping/panning.
> >
> > Exactly what I've had in mind for a while - and exactly what we
> > discussed at the recent "GStreamer 1.0" summit in Gran Canaria.
> >
> > Now, we just need to find the time to implement stuff, and I need to
> > write up my notes from GCDS in a more digestible form and post them.
> >
> > Cheers,
> > Jan.
> >
> >>
> >> Comments?
> >>
> >> BR,
> >> -R
> >>
> >>
> >> On Jul 2, 2009, at 6:43 PM, Clark, Rob wrote:
> >>
> >>> Hi gstreamer folks,
> >>>
> >>> The following is a proposal for how to add row-stride (and possibly
> >>> some related changes) to gstreamer.  I gave a couple of possible
> >>> examples of where this would be useful, but it is probably not
> >>> exhaustive.  Please let me know if you see any cases that I  
> >>> missed, or
> >>> details that I overlooked, etc.
> >>>
> >>>
> >>>
> >>> Use-cases:
> >>> ----------
> >>> + display hardware with special constraints on image dimensions, for
> >>> example
> >>>   if the output buffer must have dimensions that are a power of two
> >>> + zero-copy cropping of image / videoframe (at least for interleaved
> >>> color
> >>>   formats.. more on this later)
> >>>
> >>> One example to think about is rendering onto a 3d surface.  In some
> >>> cases, graphics hardware could require that the surface dimensions  
> >>> are
> >>> a power of 2.  In this case, you would want the vsink to allocate a
> >>> buffer with a rowstride that is the next larger power of 2 from the
> >>> image width.
> >>>
> >>>
> >>> Another example to think about is video stabilization.  In this use
> >>> case, you would ask the camera to capture an oversized frame.  Your
> >>> vstab algorithm would calculate an x,y offset of the stabilized
> >>> image.  But if the decoder understands rowstride, you do not need to
> >>> actually copy the image buffer.  Say, just to pick some numbers, you
> >>> want your final output to be 640x480, and you want your oversized
> >>> frame to be +20% in each dimension (768x576):
> >>>
> >>>   +--------+           +-------+           +------+
> >>>   | camera |---------->| vstab |---------->| venc |
> >>>   +--------+ width=768 +-------+ width=640 +------+
> >>>             height=576          height=480
> >>>          rowstride=768       rowstride=768
> >>>
> >>> In the case of an interleaved color format (RGB, UYVY, etc), you  
> >>> could
> >>> simply increment the 'data' pointer in the buffer by (y*rowstride)
> >>> +x.
> >>> No memcpy() required.  As long as the video encoder respects the
> >>> rowstride, it will see the stabilized frame correctly.
> >>>
> >>>
> >>>
> >>> Proposal:
> >>> ---------
> >>>
> >>> In all cases that I can think of, the row-stride will not be  
> >>> changing
> >>> dynamically.  So this parameter can be negotiated thru caps
> >>> negotiation in the same way as image width/height, colorformat, etc.
> >>> However, we need to know conclusively that there is no element in  
> >>> the
> >>> pipeline that cares about the image format, but does not understand
> >>> "rowstride", so we cannot use existing type strings (ex. "video/x-
> >>> raw-
> >>> yuv").  And, at least in the cases that I can think of, the video  
> >>> sink
> >>> will dictate the row-stride.  So upstream caps-renegotiation will be
> >>> used to arrive at the final "rowstride" value.
> >>>
> >>> For media types, I propose to continue using existing strings for  
> >>> non-
> >>> stride-aware element caps, ex. "video/x-raw-yuv".  For stride-aware
> >>> elements, they can support a second format, ex. "video/x-raw-yuv-
> >>> strided", "image/x-raw-rgb-strided", etc (ie. append "-strided" to
> >>> whatever the existing string is).  In the case that a strided format
> >>> is negotiated, it is required for there to also be a "rowstride"  
> >>> entry
> >>> in the final negotiated caps.
> >>>
> >>> question: in general, most elements supporting rowstride will have  
> >>> no
> >>> constraint on what particular rowstride values are supported.  Do  
> >>> they
> >>> just list "rowstride=[0-4294967295]" in their caps template?  The
> >>> video sink allocating the buffer will likely have some constraints  
> >>> on
> >>> rowstride, although this will be a function of the width (for  
> >>> example,
> >>> round the width up to next power of two).
> >>>
> >>> We will implement some sort of GstRowStrideTransform element to
> >>> interface between stride-aware and non-stride-aware elements.
> >>>
> >>>
> >>>
> >>>
> >>> Non-Interleaved Color Formats:
> >>> ------------------------------
> >>>
> >>> So, everything I've said so far about zero-copy cropping works until
> >>> you start considering planar/semi-planar color formats which do not
> >>> have equal size planes.  For example, consider NV12:  the offset to
> >>> add to the Y plane is (y*rowstride)+x, but the offset to add to UV
> >>> plane is ((y/2)*rowstride)+x.  There are only three ways that I can
> >>> think of to deal with this (listed in order of my preference):
> >>>
> >>> 1) add fields to GstBuffer to pass additional pointers to the other
> >>> color
> >>>    planes within the same GstBuffer
> >>> 2) add a field(s) to GstBuffer to pass an offset..  either an x,y
> >>> offset, or a
> >>>    single value that is (y*rowstride)+x.  Either way, the various
> >>> elements in
> >>>    the pipeline can use this to calculate the start of the
> >>> individual planes
> >>>    of data.
> >>> 3) pass individual planes of a single image as separate
> >>> GstBuffer's.. but I'm
> >>>    not a huge fan of this because now every element needs to have
> >>> some sort
> >>>    of "I've got Y, but I'm waiting for UV" state.
> >>>
> >>> I'm not sure if anyone has any thoughts about which of these three
> >>> approaches is preferred.  Or any alternative ideas?
> >>>
> >>>
> >>>
> >>> BR,
> >>> -Rob
> >>>
> >>
> >>
> >> ------------------------------------------------------------------------------
> >> _______________________________________________
> >> gstreamer-devel mailing list
> >> [hidden email]
> >> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
> >>
> >
> >
> > --
> > Jan Schmidt <[hidden email]>
> >
> >
> >
> > ------------------------------------------------------------------------------
> > _______________________________________________
> > gstreamer-devel mailing list
> > [hidden email]
> > https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
> >
>
>
> ------------------------------------------------------------------------------
> Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
> trial. Simplify your report design, integration and deployment - and focus on
> what you do best, core application coding. Discover what's new with
> Crystal Reports now.  http://p.sf.net/sfu/bobj-july
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>

--
Jan Schmidt <[hidden email]>


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

[Clark, Rob]

On Aug 4, 2009, at 4:49 AM, Jan Schmidt wrote:

> Hi,
>
> On Thu, 2009-07-30 at 11:45 -0500, Rob Clark wrote:
>> so, my next steps (at least my current thinking about next steps) are
>> to start adding some rowstride related stuff in gst-plugins-base:
>>
>>
>> gst-libs/gst/video/video.c:
>> 1) add gst_video_format_get_size_strided()
>> 2) change gst_video_format_parse_caps() to understand strided caps,
>> and add a gst_video_format_parse_caps_strided() function  (it seems
>> there are enough places already using gst_video_format_parse_caps()
>> that I probably don't want to change the signature of this function)
>>
>> and then add a GstStrideTransform element under gst/stride
>
> I haven't put enough thought into designing a stride system to say
> anything for certain. Particularly, I'm not sure whether we'll be able
> to safely and successfully integrate it into the 0.10 series without
> thinking about it harder.
>
> I am pretty sure, however, that adding a separate stride adjust  
> element
> is the wrong way to go. None of the existing pipelines will include  
> it,
> so it would no be useful without changes to the applications/playbin
> etc.
>
> My hunch is that it will be better to add a new 'video/x-raw-yuv-full'
> format and add stride as a new parameter, and adjust  
> ffmpegcolorspace to
> support conversions to/from the unstrided/strided formats.

Hmm.. my idea for stridetransform was mainly just an element for  
testing and debugging with manually constructed gst-launch pipelines.  
We need some way to run codecs both with and without rowstide and  
verifying the output.  For example:


   ... ! decoder ! video/x-raw-
yuv,format=(fourcc)YUY2,width=320,height=240,framerate=30/1 ! filesink  
location=file1.dat

and then

   ... ! decoder ! video/x-raw-yuv-
strided
,format
=(fourcc)YUY2,width=320,height=240,rowstride=700,framerate=30/1 !  
stridetransform ! video/x-raw-
yuv,format=(fourcc)YUY2,width=320,height=240,framerate=30/1 ! filesink  
location=file2.dat


But if you think this should be part of a normal pipeline, then I  
think it would make sense to merge in with the colorspace conversion,  
so you have one memory copy instead of two.

But do you think this is required?  At least in the cases that I have  
in mind, the video sink will also support non-strided buffers, but  
will be falling back to a less optimal mechanism.


------

btw, at the risk of starting a bikeshed discussion, is 'video/x-raw-
yuv-full' preferred to 'video/x-raw-yuv-strided'?  So far I've been  
using the latter, but I don't mind changing the code that I've written  
so far.  Should the functions added to video.c have the _full suffix  
instead of _strided (ie. gst_video_format_parse_caps_full() instead of  
gst_video_format_parse_caps_strided())?

If we go with -full, are there any other fields we should add to the  
caps at the same time?  Offhand, the only thing I can think of that is  
a mandatory field would be rowstride, but I haven't thought too much  
about, for example, interlaced buffers.

------

fyi, I've begun making some changes in my private tree at http://github.com/robclark/gst-plugins-base 
  ... but nothing that I've changed so far is set in stone.  But we do  
have to start adding support in the codec and other elements that  
we'll use pretty soon, so comments and suggestions now are greatly  
appreciated.  We can stay on our own tree, or a special rowstride  
branch, for now if integration to master is post-0.10.  But it would  
be nice to not have to *completely* re-write things later ;-)



BR,
-R






------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

[Nie Jun]

    I agree to use x-raw-yuv-stride, x-raw-rgb-stride to mark the
buffer and pad capability. However, stridetransform element may impact
the other elements and application usage. It will be big effort to
adopt it.

    I have a propose that may not be mature neither. Comments are welcome.
    Elements that support stride, or sub region of video buffers will
support both x-raw-yuv-stride and x-raw-yuv. If successfully
negotiated with downstream element, its src_pad will use
gst_video_buffer that derived from gst_buffer. gst_video_buffer
contains the offset/stride information.

    If element fails to negotiate with downstream element with
x-raw-yuv-stride, it will try x-raw-yuv as current model. Memory is
copied from subregion to new malloced memory and push to downstream.

    In this way, we can keep compatible with current framework. All
effort introduced by supporting subregion/stride is limited to the
elements that want to support it.

Jun

2009/8/4, Rob Clark <[hidden email]>:

>
> On Aug 4, 2009, at 4:49 AM, Jan Schmidt wrote:
>
>> Hi,
>>
>> On Thu, 2009-07-30 at 11:45 -0500, Rob Clark wrote:
>>> so, my next steps (at least my current thinking about next steps) are
>>> to start adding some rowstride related stuff in gst-plugins-base:
>>>
>>>
>>> gst-libs/gst/video/video.c:
>>> 1) add gst_video_format_get_size_strided()
>>> 2) change gst_video_format_parse_caps() to understand strided caps,
>>> and add a gst_video_format_parse_caps_strided() function  (it seems
>>> there are enough places already using gst_video_format_parse_caps()
>>> that I probably don't want to change the signature of this function)
>>>
>>> and then add a GstStrideTransform element under gst/stride
>>
>> I haven't put enough thought into designing a stride system to say
>> anything for certain. Particularly, I'm not sure whether we'll be able
>> to safely and successfully integrate it into the 0.10 series without
>> thinking about it harder.
>>
>> I am pretty sure, however, that adding a separate stride adjust
>> element
>> is the wrong way to go. None of the existing pipelines will include
>> it,
>> so it would no be useful without changes to the applications/playbin
>> etc.
>>
>> My hunch is that it will be better to add a new 'video/x-raw-yuv-full'
>> format and add stride as a new parameter, and adjust
>> ffmpegcolorspace to
>> support conversions to/from the unstrided/strided formats.
>
>
> Hmm.. my idea for stridetransform was mainly just an element for
> testing and debugging with manually constructed gst-launch pipelines.
> We need some way to run codecs both with and without rowstide and
> verifying the output.  For example:
>
>
>    ... ! decoder ! video/x-raw-
> yuv,format=(fourcc)YUY2,width=320,height=240,framerate=30/1 ! filesink
> location=file1.dat
>
> and then
>
>    ... ! decoder ! video/x-raw-yuv-
> strided
> ,format
> =(fourcc)YUY2,width=320,height=240,rowstride=700,framerate=30/1 !
> stridetransform ! video/x-raw-
> yuv,format=(fourcc)YUY2,width=320,height=240,framerate=30/1 ! filesink
> location=file2.dat
>
>
> But if you think this should be part of a normal pipeline, then I
> think it would make sense to merge in with the colorspace conversion,
> so you have one memory copy instead of two.
>
> But do you think this is required?  At least in the cases that I have
> in mind, the video sink will also support non-strided buffers, but
> will be falling back to a less optimal mechanism.
>
>
> ------
>
> btw, at the risk of starting a bikeshed discussion, is 'video/x-raw-
> yuv-full' preferred to 'video/x-raw-yuv-strided'?  So far I've been
> using the latter, but I don't mind changing the code that I've written
> so far.  Should the functions added to video.c have the _full suffix
> instead of _strided (ie. gst_video_format_parse_caps_full() instead of
> gst_video_format_parse_caps_strided())?
>
> If we go with -full, are there any other fields we should add to the
> caps at the same time?  Offhand, the only thing I can think of that is
> a mandatory field would be rowstride, but I haven't thought too much
> about, for example, interlaced buffers.
>
> ------
>
> fyi, I've begun making some changes in my private tree at
> http://github.com/robclark/gst-plugins-base
>   ... but nothing that I've changed so far is set in stone.  But we do
> have to start adding support in the codec and other elements that
> we'll use pretty soon, so comments and suggestions now are greatly
> appreciated.  We can stay on our own tree, or a special rowstride
> branch, for now if integration to master is post-0.10.  But it would
> be nice to not have to *completely* re-write things later ;-)
>
>
>
> BR,
> -R
>
>
>
>
>
>
> ------------------------------------------------------------------------------
> Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
> trial. Simplify your report design, integration and deployment - and focus
> on
> what you do best, core application coding. Discover what's new with
> Crystal Reports now.  http://p.sf.net/sfu/bobj-july
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

[Clark, Rob]

Hi Jun,

I would like to avoid subclassing GstBuffer for this.. since a lot of  
videosink (and other) elements already subclass GstBuffer for various  
purposes.  Having to both subclass GstBuffer and GstVideoBuffer would  
make a mess.

If there was some case where the stride could be changing from frame  
to frame, the buffer metadata proposal would solve this:

   http://cgit.freedesktop.org/gstreamer/gstreamer/tree/docs/design/draft-buffer2.txt

but, at least in the cases I can think of, the stride would not be  
changing from frame to frame.

I'm currently working on implementing support for it in gst-openmax  
video decoder elements.. which at least should serve as a good  
reference to the changes to make in other video decoder/encoder  
elements.  I think it won't be so complicated, although the decoder  
element would need to support re-negotiating the caps when it  
pad_alloc's a buffer from the video sink element.  But this shouldn't  
be a big deal.

----

as far as stridetransform.. I wouldn't expect it to be part of a  
"normal" pipeline.  I'm using it for now for testing.  If the  
consensus is that normal pipelines might need some stride  
transformation, I think it should be combined with colorspace  
conversion, to avoid multiple passes over the decoded video frame.  
(But I've not had a chance to implement this yet.)

BR,
-R


On Aug 11, 2009, at 12:37 AM, Jim Nie wrote:

>    I agree to use x-raw-yuv-stride, x-raw-rgb-stride to mark the
> buffer and pad capability. However, stridetransform element may impact
> the other elements and application usage. It will be big effort to
> adopt it.
>
>    I have a propose that may not be mature neither. Comments are  
> welcome.
>    Elements that support stride, or sub region of video buffers will
> support both x-raw-yuv-stride and x-raw-yuv. If successfully
> negotiated with downstream element, its src_pad will use
> gst_video_buffer that derived from gst_buffer. gst_video_buffer
> contains the offset/stride information.
>
>    If element fails to negotiate with downstream element with
> x-raw-yuv-stride, it will try x-raw-yuv as current model. Memory is
> copied from subregion to new malloced memory and push to downstream.
>
>    In this way, we can keep compatible with current framework. All
> effort introduced by supporting subregion/stride is limited to the
> elements that want to support it.
>
> Jun
>
> 2009/8/4, Rob Clark <[hidden email]>:
>>
>> On Aug 4, 2009, at 4:49 AM, Jan Schmidt wrote:
>>
>>> Hi,
>>>
>>> On Thu, 2009-07-30 at 11:45 -0500, Rob Clark wrote:
>>>> so, my next steps (at least my current thinking about next steps)  
>>>> are
>>>> to start adding some rowstride related stuff in gst-plugins-base:
>>>>
>>>>
>>>> gst-libs/gst/video/video.c:
>>>> 1) add gst_video_format_get_size_strided()
>>>> 2) change gst_video_format_parse_caps() to understand strided caps,
>>>> and add a gst_video_format_parse_caps_strided() function  (it seems
>>>> there are enough places already using gst_video_format_parse_caps()
>>>> that I probably don't want to change the signature of this  
>>>> function)
>>>>
>>>> and then add a GstStrideTransform element under gst/stride
>>>
>>> I haven't put enough thought into designing a stride system to say
>>> anything for certain. Particularly, I'm not sure whether we'll be  
>>> able
>>> to safely and successfully integrate it into the 0.10 series without
>>> thinking about it harder.
>>>
>>> I am pretty sure, however, that adding a separate stride adjust
>>> element
>>> is the wrong way to go. None of the existing pipelines will include
>>> it,
>>> so it would no be useful without changes to the applications/playbin
>>> etc.
>>>
>>> My hunch is that it will be better to add a new 'video/x-raw-yuv-
>>> full'
>>> format and add stride as a new parameter, and adjust
>>> ffmpegcolorspace to
>>> support conversions to/from the unstrided/strided formats.
>>
>>
>> Hmm.. my idea for stridetransform was mainly just an element for
>> testing and debugging with manually constructed gst-launch pipelines.
>> We need some way to run codecs both with and without rowstide and
>> verifying the output.  For example:
>>
>>
>>   ... ! decoder ! video/x-raw-
>> yuv,format=(fourcc)YUY2,width=320,height=240,framerate=30/1 !  
>> filesink
>> location=file1.dat
>>
>> and then
>>
>>   ... ! decoder ! video/x-raw-yuv-
>> strided
>> ,format
>> =(fourcc)YUY2,width=320,height=240,rowstride=700,framerate=30/1 !
>> stridetransform ! video/x-raw-
>> yuv,format=(fourcc)YUY2,width=320,height=240,framerate=30/1 !  
>> filesink
>> location=file2.dat
>>
>>
>> But if you think this should be part of a normal pipeline, then I
>> think it would make sense to merge in with the colorspace conversion,
>> so you have one memory copy instead of two.
>>
>> But do you think this is required?  At least in the cases that I have
>> in mind, the video sink will also support non-strided buffers, but
>> will be falling back to a less optimal mechanism.
>>
>>
>> ------
>>
>> btw, at the risk of starting a bikeshed discussion, is 'video/x-raw-
>> yuv-full' preferred to 'video/x-raw-yuv-strided'?  So far I've been
>> using the latter, but I don't mind changing the code that I've  
>> written
>> so far.  Should the functions added to video.c have the _full suffix
>> instead of _strided (ie. gst_video_format_parse_caps_full() instead  
>> of
>> gst_video_format_parse_caps_strided())?
>>
>> If we go with -full, are there any other fields we should add to the
>> caps at the same time?  Offhand, the only thing I can think of that  
>> is
>> a mandatory field would be rowstride, but I haven't thought too much
>> about, for example, interlaced buffers.
>>
>> ------
>>
>> fyi, I've begun making some changes in my private tree at
>> http://github.com/robclark/gst-plugins-base
>>  ... but nothing that I've changed so far is set in stone.  But we do
>> have to start adding support in the codec and other elements that
>> we'll use pretty soon, so comments and suggestions now are greatly
>> appreciated.  We can stay on our own tree, or a special rowstride
>> branch, for now if integration to master is post-0.10.  But it would
>> be nice to not have to *completely* re-write things later ;-)
>>
>>
>>
>> BR,
>> -R

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

[Clark, Rob]

Just to update..  I have pipelines with rowstride working fine.  The  
approach is relatively straightforward and not too revolutionary.  
Just normal caps negotiation, which I guess should not be too  
surprising.

I did add some utility functions and macros in libgstvideo to help  
with the caps building/parsing.  And I did add a patch on  
gst_pad_fixate_caps() (see below).  Other than gst_pad_fixate_caps,  
the changes are all in gst-plugins-base.  You can find my git tree:

   git://github.com/robclark/gst-plugins-base.git
   http://github.com/robclark/gst-plugins-base/commits/master

(I agree with earlier comments that stride-transformation should be  
supported in ffmpegcolorspace..  but this part I've not had a chance  
to implement yet.)


------------------------------------------

Essentially what is required is for a video playback pipeline:

1) srcpad on video decoders, sinkpad on video sinks, etc, should add x-
raw-yuv-strided to their template caps:

   static GstStaticPadTemplate src_template =
         GST_STATIC_PAD_TEMPLATE ("src",
                 GST_PAD_SRC,
                 GST_PAD_ALWAYS,
                 GST_STATIC_CAPS (GST_VIDEO_CAPS_YUV_STRIDED (
                         "{ I420, YUY2, UYVY }", "[ 0, max ]"))
         );

2) elements like the video decoders should implement a _get_caps()  
function for the srcpad which returns two equivalent structures, video/
x-raw-yuv and video/x-raw-yuv-strided (the later with a rowstride  
field).  It might be worthwhile to add a helper function in  
libgstvideo for this.

3) And in the _set_caps() function for video decoder srcpad should do  
something like:

     if (gst_video_format_parse_caps_strided (caps,
             &format, &width, &height, &rowstride)) {

       ... configure decoder width/height/fourcc ...

       if (rowstride) {
         ... configure decoder rowstride ...
       }
     }

and of course the video decoder should pad_alloc() their buffers to  
give the video sink element an opportunity to dictate the rowstride  
that it would prefer to use, if any.

4) To have *some* caps when allocating the first buffer, I used  
something like:

     new_caps = gst_caps_intersect (gst_pad_get_caps (srcpad),
            gst_pad_peer_get_caps (srcpad));

     if (!gst_caps_is_fixed (new_caps)) {
       gst_caps_do_simplify (new_caps);
       gst_pad_fixate_caps (srcpad, new_caps);
     }

     gst_pad_set_caps (srcpad, new_caps);


Now here is where I ran into a small issue that I think is best fixed  
in gst_pad_fixate_caps().  If the caps consists of multiple structs  
(such as strided and regular non-strided caps), gst_pad_fixate_caps()  
will only fixate the individual structs, and not choose a single  
struct to make the caps fixed.  I made a small change to remove all  
but the first struct:


------------------------------------------
  gst_pad_fixate_caps (GstPad * pad, GstCaps * caps)
  {
    GstPadFixateCapsFunction fixatefunc;
-  guint n, len;
+  guint len;

    g_return_if_fail (GST_IS_PAD (pad));
    g_return_if_fail (caps != NULL);
@@ -2359,11 +2359,15 @@ gst_pad_fixate_caps (GstPad * pad, GstCaps *  
caps)

    /* default fixation */
    len = gst_caps_get_size (caps);
-  for (n = 0; n < len; n++) {
-    GstStructure *s = gst_caps_get_structure (caps, n);
+  if (len > 0) {
+    GstStructure *s = gst_caps_get_structure (caps, 0);

      gst_structure_foreach (s, gst_pad_default_fixate, s);
    }
+
+  while (len > 1) {
+    gst_caps_remove_structure (caps, --len);
+  }
  }
------------------------------------------


I'm curious if this is the correct solution, or if there is some use-
case for a function which fixates the individual structs but does not  
make the caps fixed?



BR,
-R


On Aug 11, 2009, at 2:03 PM, Clark, Rob wrote:

> Hi Jun,
>
> I would like to avoid subclassing GstBuffer for this.. since a lot of
> videosink (and other) elements already subclass GstBuffer for various
> purposes.  Having to both subclass GstBuffer and GstVideoBuffer would
> make a mess.
>
> If there was some case where the stride could be changing from frame
> to frame, the buffer metadata proposal would solve this:
>
>   http://cgit.freedesktop.org/gstreamer/gstreamer/tree/docs/design/draft-buffer2.txt
>
> but, at least in the cases I can think of, the stride would not be
> changing from frame to frame.
>
> I'm currently working on implementing support for it in gst-openmax
> video decoder elements.. which at least should serve as a good
> reference to the changes to make in other video decoder/encoder
> elements.  I think it won't be so complicated, although the decoder
> element would need to support re-negotiating the caps when it
> pad_alloc's a buffer from the video sink element.  But this shouldn't
> be a big deal.
>
> ----
>
> as far as stridetransform.. I wouldn't expect it to be part of a
> "normal" pipeline.  I'm using it for now for testing.  If the
> consensus is that normal pipelines might need some stride
> transformation, I think it should be combined with colorspace
> conversion, to avoid multiple passes over the decoded video frame.
> (But I've not had a chance to implement this yet.)
>
> BR,
> -R
>
>
> On Aug 11, 2009, at 12:37 AM, Jim Nie wrote:
>
>>   I agree to use x-raw-yuv-stride, x-raw-rgb-stride to mark the
>> buffer and pad capability. However, stridetransform element may  
>> impact
>> the other elements and application usage. It will be big effort to
>> adopt it.
>>
>>   I have a propose that may not be mature neither. Comments are
>> welcome.
>>   Elements that support stride, or sub region of video buffers will
>> support both x-raw-yuv-stride and x-raw-yuv. If successfully
>> negotiated with downstream element, its src_pad will use
>> gst_video_buffer that derived from gst_buffer. gst_video_buffer
>> contains the offset/stride information.
>>
>>   If element fails to negotiate with downstream element with
>> x-raw-yuv-stride, it will try x-raw-yuv as current model. Memory is
>> copied from subregion to new malloced memory and push to downstream.
>>
>>   In this way, we can keep compatible with current framework. All
>> effort introduced by supporting subregion/stride is limited to the
>> elements that want to support it.
>>
>> Jun
>>
>> 2009/8/4, Rob Clark <[hidden email]>:
>>>
>>> On Aug 4, 2009, at 4:49 AM, Jan Schmidt wrote:
>>>
>>>> Hi,
>>>>
>>>> On Thu, 2009-07-30 at 11:45 -0500, Rob Clark wrote:
>>>>> so, my next steps (at least my current thinking about next steps)
>>>>> are
>>>>> to start adding some rowstride related stuff in gst-plugins-base:
>>>>>
>>>>>
>>>>> gst-libs/gst/video/video.c:
>>>>> 1) add gst_video_format_get_size_strided()
>>>>> 2) change gst_video_format_parse_caps() to understand strided  
>>>>> caps,
>>>>> and add a gst_video_format_parse_caps_strided() function  (it  
>>>>> seems
>>>>> there are enough places already using  
>>>>> gst_video_format_parse_caps()
>>>>> that I probably don't want to change the signature of this
>>>>> function)
>>>>>
>>>>> and then add a GstStrideTransform element under gst/stride
>>>>
>>>> I haven't put enough thought into designing a stride system to say
>>>> anything for certain. Particularly, I'm not sure whether we'll be
>>>> able
>>>> to safely and successfully integrate it into the 0.10 series  
>>>> without
>>>> thinking about it harder.
>>>>
>>>> I am pretty sure, however, that adding a separate stride adjust
>>>> element
>>>> is the wrong way to go. None of the existing pipelines will include
>>>> it,
>>>> so it would no be useful without changes to the applications/
>>>> playbin
>>>> etc.
>>>>
>>>> My hunch is that it will be better to add a new 'video/x-raw-yuv-
>>>> full'
>>>> format and add stride as a new parameter, and adjust
>>>> ffmpegcolorspace to
>>>> support conversions to/from the unstrided/strided formats.
>>>
>>>
>>> Hmm.. my idea for stridetransform was mainly just an element for
>>> testing and debugging with manually constructed gst-launch  
>>> pipelines.
>>> We need some way to run codecs both with and without rowstide and
>>> verifying the output.  For example:
>>>
>>>
>>>  ... ! decoder ! video/x-raw-
>>> yuv,format=(fourcc)YUY2,width=320,height=240,framerate=30/1 !
>>> filesink
>>> location=file1.dat
>>>
>>> and then
>>>
>>>  ... ! decoder ! video/x-raw-yuv-
>>> strided
>>> ,format
>>> =(fourcc)YUY2,width=320,height=240,rowstride=700,framerate=30/1 !
>>> stridetransform ! video/x-raw-
>>> yuv,format=(fourcc)YUY2,width=320,height=240,framerate=30/1 !
>>> filesink
>>> location=file2.dat
>>>
>>>
>>> But if you think this should be part of a normal pipeline, then I
>>> think it would make sense to merge in with the colorspace  
>>> conversion,
>>> so you have one memory copy instead of two.
>>>
>>> But do you think this is required?  At least in the cases that I  
>>> have
>>> in mind, the video sink will also support non-strided buffers, but
>>> will be falling back to a less optimal mechanism.
>>>
>>>
>>> ------
>>>
>>> btw, at the risk of starting a bikeshed discussion, is 'video/x-raw-
>>> yuv-full' preferred to 'video/x-raw-yuv-strided'?  So far I've been
>>> using the latter, but I don't mind changing the code that I've
>>> written
>>> so far.  Should the functions added to video.c have the _full suffix
>>> instead of _strided (ie. gst_video_format_parse_caps_full() instead
>>> of
>>> gst_video_format_parse_caps_strided())?
>>>
>>> If we go with -full, are there any other fields we should add to the
>>> caps at the same time?  Offhand, the only thing I can think of that
>>> is
>>> a mandatory field would be rowstride, but I haven't thought too much
>>> about, for example, interlaced buffers.
>>>
>>> ------
>>>
>>> fyi, I've begun making some changes in my private tree at
>>> http://github.com/robclark/gst-plugins-base
>>> ... but nothing that I've changed so far is set in stone.  But we do
>>> have to start adding support in the codec and other elements that
>>> we'll use pretty soon, so comments and suggestions now are greatly
>>> appreciated.  We can stay on our own tree, or a special rowstride
>>> branch, for now if integration to master is post-0.10.  But it would
>>> be nice to not have to *completely* re-write things later ;-)
>>>
>>>
>>>
>>> BR,
>>> -R
>
>

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel