GStreamer-devel

sending extra data along with buffers

Classic

List

Threaded

11 messages Options

Stefan Sauer

sending extra data along with buffers

hi,

I am looking for a way to send some private data along with a buffer.
During data capture I get some metadata about capture settings and I
would like to store/analyze them later along with the data stream. I
tried to add a payload item to caps and put a GstBuffer there. But
this seems to be painful and I am afraid that it will cause
renegotiation as the content changes for each buffer. Has anyone been
doing something similar?

alternative (better approach) seems to be using a buffer subclass, right?

Attached is my current test. Its a bit hackish and fails. One thing I
was wondering is that fakesrc send buffer with caps=NULL. A bug?

Stefan

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

capspayload.c (3K) Download Attachment

Edward Hervey

Re: sending extra data along with buffers

Administrator

Hi,

Another aproach would be to have a second stream for your metadata :)

It gives you the added bonus of
* being able to use the non-modified (audio or video) stream with
existing plugins,
* be able to modify, if needed, your metadata stream with new elements
that know how to handle it
* create/modify elements that need both streams (using timestamps of
buffer to correlate the raw buffers to the associated metadata
* Not modifying GStreamer core for this

Edward

On Mon, 2008-01-14 at 10:05 +0100, Stefan Kost wrote:

> hi,
>
> I am looking for a way to send some private data along with a buffer.
> During data capture I get some metadata about capture settings and I
> would like to store/analyze them later along with the data stream. I
> tried to add a payload item to caps and put a GstBuffer there. But
> this seems to be painful and I am afraid that it will cause
> renegotiation as the content changes for each buffer. Has anyone been
> doing something similar?
>
> alternative (better approach) seems to be using a buffer subclass, right?
>
> Attached is my current test. Its a bit hackish and fails. One thing I
> was wondering is that fakesrc send buffer with caps=NULL. A bug?
>
> Stefan
>
> -------------------------------------------------------------------------
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services for
> just about anything Open Source.
> http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
> _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

Edward Hervey

Re: sending extra data along with buffers

Administrator

Also forgot, doing it that way, if you have a element creating those
extra metadata but you don't want to use them, you just don't connect to
that metadata pad, which minimizes the overload of that metadata in the
whole pipeline

On Mon, 2008-01-14 at 11:27 +0100, Edward Hervey wrote:

> Hi,
>
> Another aproach would be to have a second stream for your metadata :)
>
> It gives you the added bonus of
> * being able to use the non-modified (audio or video) stream with
> existing plugins,
> * be able to modify, if needed, your metadata stream with new elements
> that know how to handle it
> * create/modify elements that need both streams (using timestamps of
> buffer to correlate the raw buffers to the associated metadata
> * Not modifying GStreamer core for this
>
> Edward
>
> On Mon, 2008-01-14 at 10:05 +0100, Stefan Kost wrote:
> > hi,
> >
> > I am looking for a way to send some private data along with a buffer.
> > During data capture I get some metadata about capture settings and I
> > would like to store/analyze them later along with the data stream. I
> > tried to add a payload item to caps and put a GstBuffer there. But
> > this seems to be painful and I am afraid that it will cause
> > renegotiation as the content changes for each buffer. Has anyone been
> > doing something similar?
> >
> > alternative (better approach) seems to be using a buffer subclass, right?
> >
> > Attached is my current test. Its a bit hackish and fails. One thing I
> > was wondering is that fakesrc send buffer with caps=NULL. A bug?
> >
> > Stefan
> >
> > -------------------------------------------------------------------------
> > Check out the new SourceForge.net Marketplace.
> > It's the best place to buy or sell services for
> > just about anything Open Source.
> > http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
> > _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>

Stefan Sauer

Re: sending extra data along with buffers

In reply to this post by Stefan Sauer

Hi,

another alternative - a tag-event. it has flag serialized. I need to
check if the sink gets the event before it gets the buffer, when the
source send the
event before it pushes the buffer. That is if the source sends e1, b1,
e2, b2, ... the sink gets it in exactly the same order.

Stefan

Jan Schmidt-2

Re: sending extra data along with buffers

In reply to this post by Stefan Sauer

> hi,
>
> I am looking for a way to send some private data along with a buffer.
> During data capture I get some metadata about capture settings and I would
> like to store/analyze them later along with the data stream. I tried to add
> a payload item to caps and put a GstBuffer there. But this seems to be
> painful and I am afraid that it will cause renegotiation as the content
> changes for each buffer. Has anyone been doing something similar?
>
> alternative (better approach) seems to be using a buffer subclass, right?
>
> Attached is my current test. Its a bit hackish and fails. One thing I was
> wondering is that fakesrc send buffer with caps=NULL. A bug?
>

>From time to time, I've thought about adding something like this directly
to GstBuffer, for carrying information whose presence been negotiated in
the caps, but whose content should be allowed to change without
renegotiating.

Examples:
* Pan and scan (video subregion) position information
* interlacing markers (top field, bottom field)
* theora granulepos, instead of slapping it in the 'offset' field
* MPEG Decoding timestamps, which we currently never send along.

My thought was to add a GstStructure to GstBuffer which is allowed to be
NULL.

Wim has expressed concern that it would cause performance issues, and it
would certainly require extending various bits of GStreamer to support
properly (buffer serialisation/deserialisation in GDP, for example). Also,
the semantics of how to handle the 'extradata' when transforming a buffer
need to be clearly defined, ie the extradata has to be clearly specified
by the caps attached to the buffer, and there needs to be a way for it to
be handled by elements that don't understand the extradata they've been
given. It can't just be a dumping ground in which to place arbitrary data,
or I don't think it can be made to work.

J.

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

Albert Costa

Re: sending extra data along with buffers

In reply to this post by Stefan Sauer

Hi,

I'm quite interested in the topic about adding extra data to buffers. I'm currently working on a project where I have to analyse video frames, extract some piece of info out of it (like detection of features), and perform some overlaying of these data over the video frame.

That is, I absolutely needed to pass some information along with each buffer... And these information are not size-invariant : for example, the data are 'number of detected features + ROI for each feature' (and other stuff...), so it changes from frame to frame.

So I could not put it into caps or tags, I had to find a way to solve the issue.

What I did (the easiest but not prettiest... but I lacked time) is develop an input plugin that takes a common x-raw-rgb type flow as input. It analyses the frame, and then creates an output buffer that contains the frame data and my own specific data. It is sent to src pad as 'ANY' type (because the size of my buffer is no longer coherent with raw-rgb caps).

Then my next plugin will take that as input. Knowing the length of each field, I extract the original frame, my data, and perform some overlaying on my frame. Finally, I send again a 'raw-rgb' on the src pad, to be linked to a directdrawsink:

gst-inspect filesrc location=myfile.avi ! decodebin ! ffmpegcolorspace ! myvideo_analyser ! myvideo_overlay ! ffmpegcolorspace ! directdrawsink

That's the 'easy' way, but it is not very nice-looking, nor 'generic' (e.g. I cannot use my specific data with other base gstreamer plugins).

I was thinking once of splitting my original buffer frame and my data over 2 distincts output pads. But then again for my data I cannot do much (it has to by 'ANY' type, unless I create a new specific mime which I don't know how to do)... Sending events with my data for each buffer doesn't look safe. I've done some other work where events where not received synchronously (ie sent event1, buffer1, event2, buffer2 but received event1, buffer1, buffer2, event2...).

The idea of adding a GstStructure (or something similar) inside a GstBuffer seems to be the most interesting thing, at least with regard to my own experience. Would be null for general case, and user-filled for... user purpose.

Regards,

Albert

Ne gardez plus qu'une seule adresse mail ! Copiez vos mails vers Yahoo! Mail
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

Stefan Sauer

Re: sending extra data along with buffers

In reply to this post by Jan Schmidt-2

Hi,

Jan Schmidt schrieb:

> <quote who="Stefan Kost">
>
>> hi,
>>
>> I am looking for a way to send some private data along with a buffer.
>> During data capture I get some metadata about capture settings and I would
>> like to store/analyze them later along with the data stream. I tried to add
>> a payload item to caps and put a GstBuffer there. But this seems to be
>> painful and I am afraid that it will cause renegotiation as the content
>> changes for each buffer. Has anyone been doing something similar?
>>
>> alternative (better approach) seems to be using a buffer subclass, right?
>>
>> Attached is my current test. Its a bit hackish and fails. One thing I was
>> wondering is that fakesrc send buffer with caps=NULL. A bug?
>>
>
>>From time to time, I've thought about adding something like this directly
> to GstBuffer, for carrying information whose presence been negotiated in
> the caps, but whose content should be allowed to change without
> renegotiating.
>
> Examples:
> * Pan and scan (video subregion) position information
> * interlacing markers (top field, bottom field)
> * theora granulepos, instead of slapping it in the 'offset' field
> * MPEG Decoding timestamps, which we currently never send along.
>
> My thought was to add a GstStructure to GstBuffer which is allowed to be
> NULL.
>
> Wim has expressed concern that it would cause performance issues, and it
> would certainly require extending various bits of GStreamer to support
> properly (buffer serialisation/deserialisation in GDP, for example). Also,
> the semantics of how to handle the 'extradata' when transforming a buffer
> need to be clearly defined, ie the extradata has to be clearly specified
> by the caps attached to the buffer, and there needs to be a way for it to
> be handled by elements that don't understand the extradata they've been
> given. It can't just be a dumping ground in which to place arbitrary data,
> or I don't think it can be made to work.
>
> J.
>

Maybe some GstTaglist features can help. I am thinking of the merge-modes here.
Atleast this is needed when mixing data.

No idea how to express constraints that would render the data useless. E.g.
muxing two streams removes the 'frame' context from video frames. Thus the muxer
would not copy/merge the extra-data to outgoing buffers.

I am not so much concerned with performance. In most cases it would not be used.
Freeing buffers would need to release the taglst if its not NULL. Copying
buffers need to copy it.

As we have hierarchical structures now, we could probably require that the
producer uses its element name as a root tag. This way the taglist could carry
multiple data-sets identified in a unique way. Elements that can use the data
need to know the name of the element that created it.

I'll think more about it - may be we can use our shiny new wiki to draft it :)

Stefan

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

Nie Jun

Re: sending extra data along with buffers

I am interested in video subregion information. I had plan to add caps to do this. Now you may help to estimate the feasibility of this plan.
Adding 5 caps, x, y, subwidth, subheight, first data byte offset in original video. substream tests these caps, if they are zeros, then ignores subregion information. Otherwise we operate subregion according to these information.

2008/1/16, Stefan Kost <[hidden email]>:

Hi,

Jan Schmidt schrieb:

> <quote who="Stefan Kost">
>
>> hi,
>>
>> I am looking for a way to send some private data along with a buffer.
>> During data capture I get some metadata about capture settings and I would
>> like to store/analyze them later along with the data stream. I tried to add
>> a payload item to caps and put a GstBuffer there. But this seems to be
>> painful and I am afraid that it will cause renegotiation as the content
>> changes for each buffer. Has anyone been doing something similar?
>>
>> alternative (better approach) seems to be using a buffer subclass, right?
>>
>> Attached is my current test. Its a bit hackish and fails. One thing I was
>> wondering is that fakesrc send buffer with caps=NULL. A bug?
>>
>
>>From time to time, I've thought about adding something like this directly
> to GstBuffer, for carrying information whose presence been negotiated in
> the caps, but whose content should be allowed to change without
> renegotiating.
>
> Examples:
>   * Pan and scan (video subregion) position information
>   * interlacing markers (top field, bottom field)
>   * theora granulepos, instead of slapping it in the 'offset' field
>   * MPEG Decoding timestamps, which we currently never send along.
>
> My thought was to add a GstStructure to GstBuffer which is allowed to be
> NULL.
>
> Wim has expressed concern that it would cause performance issues, and it
> would certainly require extending various bits of GStreamer to support
> properly (buffer serialisation/deserialisation in GDP, for example). Also,
> the semantics of how to handle the 'extradata' when transforming a buffer
> need to be clearly defined, ie the extradata has to be clearly specified
> by the caps attached to the buffer, and there needs to be a way for it to
> be handled by elements that don't understand the extradata they've been
> given. It can't just be a dumping ground in which to place arbitrary data,
> or I don't think it can be made to work.
>
> J.
>
Maybe some GstTaglist features can help. I am thinking of the merge-modes here.
Atleast this is needed when mixing data.

No idea how to express constraints that would render the data useless. E.g.
muxing two streams removes the 'frame' context from video frames. Thus the muxer
would not copy/merge the extra-data to outgoing buffers.

I am not so much concerned with performance. In most cases it would not be used.
Freeing buffers would need to release the taglst if its not NULL. Copying
buffers need to copy it.

As we have hierarchical structures now, we could probably require that the
producer uses its element name as a root tag. This way the taglist could carry
multiple data-sets identified in a unique way. Elements that can use the data
need to know the name of the element that created it.

I'll think more about it - may be we can use our shiny new wiki to draft it :)

Stefan

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

Stefan Sauer

Re: sending extra data along with buffers

In reply to this post by Stefan Sauer

hi,

I've put a summary on the wiki:
http://gstreamer.freedesktop.org/wiki/BufferMetadata

please add your use-cases and ideas.

Stefan

Quoting Stefan Kost <[hidden email]>:

> Hi,
>
> Jan Schmidt schrieb:
>> <quote who="Stefan Kost">
>>
>>> hi,
>>>
>>> I am looking for a way to send some private data along with a buffer.
>>> During data capture I get some metadata about capture settings and I would
>>> like to store/analyze them later along with the data stream. I tried to add
>>> a payload item to caps and put a GstBuffer there. But this seems to be
>>> painful and I am afraid that it will cause renegotiation as the content
>>> changes for each buffer. Has anyone been doing something similar?
>>>
>>> alternative (better approach) seems to be using a buffer subclass, right?
>>>
>>> Attached is my current test. Its a bit hackish and fails. One thing I was
>>> wondering is that fakesrc send buffer with caps=NULL. A bug?
>>>
>>
>>> From time to time, I've thought about adding something like this directly
>> to GstBuffer, for carrying information whose presence been negotiated in
>> the caps, but whose content should be allowed to change without
>> renegotiating.
>>
>> Examples:
>> * Pan and scan (video subregion) position information
>> * interlacing markers (top field, bottom field)
>> * theora granulepos, instead of slapping it in the 'offset' field
>> * MPEG Decoding timestamps, which we currently never send along.
>>
>> My thought was to add a GstStructure to GstBuffer which is allowed to be
>> NULL.
>>
>> Wim has expressed concern that it would cause performance issues, and it
>> would certainly require extending various bits of GStreamer to support
>> properly (buffer serialisation/deserialisation in GDP, for example). Also,
>> the semantics of how to handle the 'extradata' when transforming a buffer
>> need to be clearly defined, ie the extradata has to be clearly specified
>> by the caps attached to the buffer, and there needs to be a way for it to
>> be handled by elements that don't understand the extradata they've been
>> given. It can't just be a dumping ground in which to place arbitrary data,
>> or I don't think it can be made to work.
>>
>> J.
>>
> Maybe some GstTaglist features can help. I am thinking of the
> merge-modes here.
> Atleast this is needed when mixing data.
>
> No idea how to express constraints that would render the data useless. E.g.
> muxing two streams removes the 'frame' context from video frames.
> Thus the muxer
> would not copy/merge the extra-data to outgoing buffers.
>
> I am not so much concerned with performance. In most cases it would
> not be used.
> Freeing buffers would need to release the taglst if its not NULL. Copying
> buffers need to copy it.
>
> As we have hierarchical structures now, we could probably require that the
> producer uses its element name as a root tag. This way the taglist
> could carry
> multiple data-sets identified in a unique way. Elements that can use the data
> need to know the name of the element that created it.
>
> I'll think more about it - may be we can use our shiny new wiki to
> draft it :)
>
> Stefan
>
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2008.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>

Jan Schmidt-6

Re: sending extra data along with buffers

Stefan Kost wrote:
> hi,
>
> I've put a summary on the wiki:
> http://gstreamer.freedesktop.org/wiki/BufferMetadata
>
> please add your use-cases and ideas.
>

I don't quite know how to have a dialogue on a wiki page. Seems like a
good way to document a design, but a poor way to generate it.

Regarding what's there in the wiki at the moment:

I particularly dislike the idea of using the element name to identify
data. It's pretty much exactly what I said I *don't* think the design
should be - a dumping ground in which to place arbitrary data.

I think any data that requires intimate knowledge of the actual element
generating it belongs in a private element event or message.

On the other hand, all the examples in the wiki look to me like things
that can be considered tied to the caps - that is, they are extra
descriptions of the content of the buffer, and knowledge of how to
interpret them can be derived from the caps, or some extra property in
the caps.

I also think that we should adopt a policy that any such 'extradata' (I
want a better name, people!) is free to be dropped by any element that
doesn't explicitly support it. I want whitelisting, rather than in any
way expecting elements to pass through the data unless they know better.

J.

> Stefan
>
> Quoting Stefan Kost <[hidden email]>:
>
>> Hi,
>>
>> Jan Schmidt schrieb:
>>
>>> <quote who="Stefan Kost">
>>>
>>>
>>>> hi,
>>>>
>>>> I am looking for a way to send some private data along with a buffer.
>>>> During data capture I get some metadata about capture settings and I would
>>>> like to store/analyze them later along with the data stream. I tried to add
>>>> a payload item to caps and put a GstBuffer there. But this seems to be
>>>> painful and I am afraid that it will cause renegotiation as the content
>>>> changes for each buffer. Has anyone been doing something similar?
>>>>
>>>> alternative (better approach) seems to be using a buffer subclass, right?
>>>>
>>>> Attached is my current test. Its a bit hackish and fails. One thing I was
>>>> wondering is that fakesrc send buffer with caps=NULL. A bug?
>>>>
>>>>
>>>> From time to time, I've thought about adding something like this directly
>>>>
>>> to GstBuffer, for carrying information whose presence been negotiated in
>>> the caps, but whose content should be allowed to change without
>>> renegotiating.
>>>
>>> Examples:
>>> * Pan and scan (video subregion) position information
>>> * interlacing markers (top field, bottom field)
>>> * theora granulepos, instead of slapping it in the 'offset' field
>>> * MPEG Decoding timestamps, which we currently never send along.
>>>
>>> My thought was to add a GstStructure to GstBuffer which is allowed to be
>>> NULL.
>>>
>>> Wim has expressed concern that it would cause performance issues, and it
>>> would certainly require extending various bits of GStreamer to support
>>> properly (buffer serialisation/deserialisation in GDP, for example). Also,
>>> the semantics of how to handle the 'extradata' when transforming a buffer
>>> need to be clearly defined, ie the extradata has to be clearly specified
>>> by the caps attached to the buffer, and there needs to be a way for it to
>>> be handled by elements that don't understand the extradata they've been
>>> given. It can't just be a dumping ground in which to place arbitrary data,
>>> or I don't think it can be made to work.
>>>
>>> J.
>>>
>>>
>> Maybe some GstTaglist features can help. I am thinking of the
>> merge-modes here.
>> Atleast this is needed when mixing data.
>>
>> No idea how to express constraints that would render the data useless. E.g.
>> muxing two streams removes the 'frame' context from video frames.
>> Thus the muxer
>> would not copy/merge the extra-data to outgoing buffers.
>>
>> I am not so much concerned with performance. In most cases it would
>> not be used.
>> Freeing buffers would need to release the taglst if its not NULL. Copying
>> buffers need to copy it.
>>
>> As we have hierarchical structures now, we could probably require that the
>> producer uses its element name as a root tag. This way the taglist
>> could carry
>> multiple data-sets identified in a unique way. Elements that can use the data
>> need to know the name of the element that created it.
>>
>> I'll think more about it - may be we can use our shiny new wiki to
>> draft it :)
>>
>> Stefan
>>
>>
>> -------------------------------------------------------------------------
>> This SF.net email is sponsored by: Microsoft
>> Defy all challenges. Microsoft(R) Visual Studio 2008.
>> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
>> _______________________________________________
>> gstreamer-devel mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>>
>>
>
>
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2008.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>
>

Stefan Sauer

Re: sending extra data along with buffers

Hi,

Quoting Jan Schmidt <[hidden email]>:

> Stefan Kost wrote:
>> hi,
>>
>> I've put a summary on the wiki:
>> http://gstreamer.freedesktop.org/wiki/BufferMetadata
>>
>> please add your use-cases and ideas.
>>
>
> I don't quite know how to have a dialogue on a wiki page. Seems like a
> good way to document a design, but a poor way to generate it.
>

Yes having the dialog here is good. Still it would be good to update
the wiki page at the same time to have the running draft there.
>
> Regarding what's there in the wiki at the moment:
>
> I particularly dislike the idea of using the element name to identify
> data. It's pretty much exactly what I said I *don't* think the design
> should be - a dumping ground in which to place arbitrary data.
>
> I think any data that requires intimate knowledge of the actual element
> generating it belongs in a private element event or message.

In the past days I wrote a test programm that uses tag events and it
shows that they are indeed in sync with the buffer flow. Means one
element sends both buffers and tag-events out on its src pad. Then the
connected peer element will receive them in the same order. (tried
with two basetransform elements).
I'll make that a bit clearer in the docs after the freeze.

This definitely makes events the better mechanism for realy private stuff.
>
> On the other hand, all the examples in the wiki look to me like things
> that can be considered tied to the caps - that is, they are extra
> descriptions of the content of the buffer, and knowledge of how to
> interpret them can be derived from the caps, or some extra property in
> the caps.

What about caps like
video/whatever, width=320, height=240,...,extradata=(fields,subregion);

that would mean that the extradata has two structures named 'fields'
and 'subregion'. Regarding the name 'extradata', it is still caps
data. Only not essential for all plugins.

>
> I also think that we should adopt a policy that any such 'extradata' (I
> want a better name, people!) is free to be dropped by any element that
> doesn't explicitly support it. I want whitelisting, rather than in any
> way expecting elements to pass through the data unless they know better.
>
An element that e.g. just changes image contrast would/should not need
to drop the interlaced field info. I means that we need to define
those subsets, their name and purpose.

Stefan

>
> J.
>
>> Stefan
>>
>> Quoting Stefan Kost <[hidden email]>:
>>
>>> Hi,
>>>
>>> Jan Schmidt schrieb:
>>>
>>>> <quote who="Stefan Kost">
>>>>
>>>>
>>>>> hi,
>>>>>
>>>>> I am looking for a way to send some private data along with a buffer.
>>>>> During data capture I get some metadata about capture settings
>>>>> and I would
>>>>> like to store/analyze them later along with the data stream. I
>>>>> tried to add
>>>>> a payload item to caps and put a GstBuffer there. But this seems to be
>>>>> painful and I am afraid that it will cause renegotiation as the content
>>>>> changes for each buffer. Has anyone been doing something similar?
>>>>>
>>>>> alternative (better approach) seems to be using a buffer subclass, right?
>>>>>
>>>>> Attached is my current test. Its a bit hackish and fails. One thing I was
>>>>> wondering is that fakesrc send buffer with caps=NULL. A bug?
>>>>>
>>>>> From time to time, I've thought about adding something
>>>>> like this directly
>>>>>
>>>> to GstBuffer, for carrying information whose presence been negotiated in
>>>> the caps, but whose content should be allowed to change without
>>>> renegotiating.
>>>>
>>>> Examples:
>>>> * Pan and scan (video subregion) position information
>>>> * interlacing markers (top field, bottom field)
>>>> * theora granulepos, instead of slapping it in the 'offset' field
>>>> * MPEG Decoding timestamps, which we currently never send along.
>>>>
>>>> My thought was to add a GstStructure to GstBuffer which is allowed to be
>>>> NULL.
>>>>
>>>> Wim has expressed concern that it would cause performance issues, and it
>>>> would certainly require extending various bits of GStreamer to support
>>>> properly (buffer serialisation/deserialisation in GDP, for example). Also,
>>>> the semantics of how to handle the 'extradata' when transforming a buffer
>>>> need to be clearly defined, ie the extradata has to be clearly specified
>>>> by the caps attached to the buffer, and there needs to be a way for it to
>>>> be handled by elements that don't understand the extradata they've been
>>>> given. It can't just be a dumping ground in which to place arbitrary data,
>>>> or I don't think it can be made to work.
>>>>
>>>> J.
>>>>
>>>>
>>> Maybe some GstTaglist features can help. I am thinking of the
>>> merge-modes here.
>>> Atleast this is needed when mixing data.
>>>
>>> No idea how to express constraints that would render the data useless. E.g.
>>> muxing two streams removes the 'frame' context from video frames.
>>> Thus the muxer
>>> would not copy/merge the extra-data to outgoing buffers.
>>>
>>> I am not so much concerned with performance. In most cases it
>>> would not be used.
>>> Freeing buffers would need to release the taglst if its not NULL. Copying
>>> buffers need to copy it.
>>>
>>> As we have hierarchical structures now, we could probably require that the
>>> producer uses its element name as a root tag. This way the taglist
>>> could carry
>>> multiple data-sets identified in a unique way. Elements that can
>>> use the data
>>> need to know the name of the element that created it.
>>>
>>> I'll think more about it - may be we can use our shiny new wiki to
>>> draft it :)
>>>
>>> Stefan
>>>
>>>
>>> -------------------------------------------------------------------------
>>> This SF.net email is sponsored by: Microsoft
>>> Defy all challenges. Microsoft(R) Visual Studio 2008.
>>> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
>>> _______________________________________________
>>> gstreamer-devel mailing list
>>> [hidden email]
>>> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>>>
>>>
>>
>>
>>
>> -------------------------------------------------------------------------
>> This SF.net email is sponsored by: Microsoft
>> Defy all challenges. Microsoft(R) Visual Studio 2008.
>> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
>> _______________________________________________
>> gstreamer-devel mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>>
>>