Correspondence between appsrc input and appsink output buffers

[benjamin.hepp]

Hi everyone,

I'm new to gstreamer (it's great!) so I very much appreciate any help.

I basically have the following setup:
Custom video-source -> appsrc -> encoding -> appsink -> send over
network together with other information that corresponds to each frame
(e.g. GPS).
On the server I receive buffer and associated information -> appsrc ->
decoding -> appsink.

Now in both cases I push a buffer into an appsrc that has some
associated information. Is there any way to link this information again
to a buffer coming out of the appsink?

I now that theoretically there is no guarantee that one buffer out of
the appsink is one frame, but in my case it is. So on the client side I
can just assume that no frame is ever dropped (assuming I never hit any
buffer limit).
However, on the receiving side the decoder sometimes needs to drop some
frame until it can start decoding and I don't know how many it drops. So
I'm in the situation that I cannot link the information to the buffers
coming out of the appsink.
Unfortunately I can't use the buffer offset or PTS information because
this is just reset by the encoder/decoder.

Is there any other way to establish these correspondences?

Cheers
Benjamin

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On Sun, 2016-11-06 at 02:35 +0100, [hidden email] wrote:

> Hi everyone,
>
> I'm new to gstreamer (it's great!) so I very much appreciate any help.
>
> I basically have the following setup:
> Custom video-source -> appsrc -> encoding -> appsink -> send over 
> network together with other information that corresponds to each frame 
> (e.g. GPS).
> On the server I receive buffer and associated information -> appsrc -> 
> decoding -> appsink.
>
> Now in both cases I push a buffer into an appsrc that has some 
> associated information. Is there any way to link this information again 
> to a buffer coming out of the appsink?
>
> I now that theoretically there is no guarantee that one buffer out of 
> the appsink is one frame, but in my case it is. So on the client side I 
> can just assume that no frame is ever dropped (assuming I never hit any 
> buffer limit).
> However, on the receiving side the decoder sometimes needs to drop some 
> frame until it can start decoding and I don't know how many it drops. So 
> I'm in the situation that I cannot link the information to the buffers 
> coming out of the appsink.
> Unfortunately I can't use the buffer offset or PTS information because 
> this is just reset by the encoder/decoder.
>
> Is there any other way to establish these correspondences?

You could mark buffers with a GstMeta (make sure to use new enough
GStreamer, and a GstMeta that has no tags) or use serialized events
between each buffer as markers (not very efficient).

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[benjamin.hepp]

Hi Sebastian,

thanks for this tip, it works like a charm. I wouldn't have figured out that I can use GstMeta for this.

One note for others who might try to do something similar: Depending on the pipeline one has to lower the refcount of the destination buffer to 1 within the meta transformation function. Otherwise the buffer cannot be mapped writable and thus the GstMeta fails to be attached to the buffer. This seems to be an ok workaround though.

Cheers
Benjamin

On 07.11.2016 01:36, Sebastian Dröge wrote:

On Sun, 2016-11-06 at 02:35 +0100, [hidden email] wrote:

Hi everyone,

I'm new to gstreamer (it's great!) so I very much appreciate any help.

I basically have the following setup:
Custom video-source -> appsrc -> encoding -> appsink -> send over 
network together with other information that corresponds to each frame 
(e.g. GPS).
On the server I receive buffer and associated information -> appsrc -> 
decoding -> appsink.

Now in both cases I push a buffer into an appsrc that has some 
associated information. Is there any way to link this information again 
to a buffer coming out of the appsink?

I now that theoretically there is no guarantee that one buffer out of 
the appsink is one frame, but in my case it is. So on the client side I 
can just assume that no frame is ever dropped (assuming I never hit any 
buffer limit).
However, on the receiving side the decoder sometimes needs to drop some 
frame until it can start decoding and I don't know how many it drops. So 
I'm in the situation that I cannot link the information to the buffers 
coming out of the appsink.
Unfortunately I can't use the buffer offset or PTS information because 
this is just reset by the encoder/decoder.

Is there any other way to establish these correspondences?

You could mark buffers with a GstMeta (make sure to use new enough
GStreamer, and a GstMeta that has no tags) or use serialized events
between each buffer as markers (not very efficient).

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On Tue, 2016-11-08 at 20:04 -0800, Benjamin Hepp wrote:

> Hi Sebastian,
>
> thanks for this tip, it works like a charm. I wouldn't have figured
> out that I can use GstMeta for this.
>
> One note for others who might try to do something similar: Depending
> on the pipeline one has to lower the refcount of the destination
> buffer to 1 within the meta transformation function. Otherwise the
> buffer cannot be mapped writable and thus the GstMeta fails to be
> attached to the buffer. This seems to be an ok workaround though.

That's generally not a good idea and should cause crashes or other
problems over time. You always have to do the reference counting
correct.


If a buffer is not writable, you need to copy it (the buffer, not the
actual memory inside it). See gst_buffer_make_writable() for automating
that task.

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[benjamin.hepp]

On 09.11.2016 08:49, Sebastian Dröge wrote:

> That's generally not a good idea and should cause crashes or other
> problems over time. You always have to do the reference counting
> correct.

Of course, after decreasing the ref count to 1 and adding the GstMeta to
the buffer one has to increase the ref count to the previous value.

> If a buffer is not writable, you need to copy it (the buffer, not the
> actual memory inside it). See gst_buffer_make_writable() for automating
> that task.

I don't see how that can work for the GstMeta transformation function:
***_meta_transform (GstBuffer * transbuf, GstMeta * meta,
     GstBuffer * buffer, GQuark type, gpointer data)

You can very well copy the transbuf buffer and add the GstMeta to the
copy but then how could you replace transbuf with it?

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On Wed, 2016-11-09 at 22:24 +0100, [hidden email] wrote:
> On 09.11.2016 08:49, Sebastian Dröge wrote:
>
> > That's generally not a good idea and should cause crashes or other
> > problems over time. You always have to do the reference counting
> > correct.
>
> Of course, after decreasing the ref count to 1 and adding the GstMeta
> to the buffer one has to increase the ref count to the previous
> value.

Nothing prevents another thread from calling unref() for its own
reference at the same time, which then would cause the buffer to be
destroyed as you removed all additional references.

Also other code might be iterating the metas at this very moment, and
if a new one is added then random things could happen.

> > If a buffer is not writable, you need to copy it (the buffer, not
> > the
> > actual memory inside it). See gst_buffer_make_writable() for
> > automating
> > that task.
>
> I don't see how that can work for the GstMeta transformation
> function:
> ***_meta_transform (GstBuffer * transbuf, GstMeta * meta,
>      GstBuffer * buffer, GQuark type, gpointer data)
>
> You can very well copy the transbuf buffer and add the GstMeta to
> the 
> copy but then how could you replace transbuf with it?

If the buffer is not writable in this method, then that's a bug in
basetransform somewhere.

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel