Pool of variable-size buffers

[Petr Kulhavy]

Hi,

I'm trying to implement an efficient buffer management for zero-copy UDP
reception, where the data is provided in chunks of memory preallocated
by the kernel.
The unfortunate heavy-weight nature of GStreamer buffers, where every
buffer is a set of objects, is a bit clashing with the "efficient" part
of my task.

So I have the buffers of memory with data and now need to implement a
pool of GstBuffer and GstMemory objects in order to be able to hand over
the data down the pipeline. Kind of pool of empty buffers, where I can
then hang whatever chunk of memory I need.
The buffer pool abstraction is based on the fact that all buffers are of
equal size. This is unfortunately not what I can guarantee.

Is there some existing abstraction I can use?

Thanks
Petr
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Olivier Crête-3]

Hi,

Using a GstBufferPool is a valid option. One thing you can do is to
create entirely empty buffers (with no memories attached). You will
then need to implement a custom reset_memory vfunc on the bufferpool
to reset the memory tag flag and separate the memories from the buffer
before calling up. Then you can create a custom GstAllocator that you
can use to get the GstMemory objects from, you can also pre-allocate
those (one for each of your buffers), then you can attach them to the
GstBuffer from the pool when you want to send something down the
pipeline. So your entire use-case can be done without doing a single
allocation!

Olivier


On Mon, 2016-12-05 at 20:00 +0100, Petr Kulhavy wrote:

> Hi,
>
> I'm trying to implement an efficient buffer management for zero-copy
> UDP 
> reception, where the data is provided in chunks of memory
> preallocated 
> by the kernel.
> The unfortunate heavy-weight nature of GStreamer buffers, where
> every 
> buffer is a set of objects, is a bit clashing with the "efficient"
> part 
> of my task.
>
> So I have the buffers of memory with data and now need to implement
> a 
> pool of GstBuffer and GstMemory objects in order to be able to hand
> over 
> the data down the pipeline. Kind of pool of empty buffers, where I
> can 
> then hang whatever chunk of memory I need.
> The buffer pool abstraction is based on the fact that all buffers are
> of 
> equal size. This is unfortunately not what I can guarantee.
>
> Is there some existing abstraction I can use?
>
> Thanks
> Petr
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

--
Olivier Crête
[hidden email]
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Petr Kulhavy]

Hi Olivier,

thank you for the detailed explanation. Doing this without a single
allocation is exactly what I'm looking for.
I have a few more questions:

* Can I integrate the pool creation into the caps negotiation, i.e.
_decide_allocation() and possibly reuse a pool provided by a downstream
element ?

* Can I just pass 0 as size to gst_buffer_pool_config_set_params() and
gst_query_add_allocation_pool() even though the doc says size must be >0
? ;-)

* How can I pre-allocate the GstMemory objects prior to knowing the
memory location and size? I don't see a function to replace or
manipulate the GstMapInfo within GstMemory.


Thanks
Petr

On 06/12/16 00:39, Olivier Crête wrote:

> Hi,
>
> Using a GstBufferPool is a valid option. One thing you can do is to
> create entirely empty buffers (with no memories attached). You will
> then need to implement a custom reset_memory vfunc on the bufferpool
> to reset the memory tag flag and separate the memories from the buffer
> before calling up. Then you can create a custom GstAllocator that you
> can use to get the GstMemory objects from, you can also pre-allocate
> those (one for each of your buffers), then you can attach them to the
> GstBuffer from the pool when you want to send something down the
> pipeline. So your entire use-case can be done without doing a single
> allocation!
>
> Olivier
>
>
> On Mon, 2016-12-05 at 20:00 +0100, Petr Kulhavy wrote:
>> Hi,
>>
>> I'm trying to implement an efficient buffer management for zero-copy
>> UDP
>> reception, where the data is provided in chunks of memory
>> preallocated
>> by the kernel.
>> The unfortunate heavy-weight nature of GStreamer buffers, where
>> every
>> buffer is a set of objects, is a bit clashing with the "efficient"
>> part
>> of my task.
>>
>> So I have the buffers of memory with data and now need to implement
>> a
>> pool of GstBuffer and GstMemory objects in order to be able to hand
>> over
>> the data down the pipeline. Kind of pool of empty buffers, where I
>> can
>> then hang whatever chunk of memory I need.
>> The buffer pool abstraction is based on the fact that all buffers are
>> of
>> equal size. This is unfortunately not what I can guarantee.
>>
>> Is there some existing abstraction I can use?
>>
>> Thanks
>> Petr
>> _______________________________________________
>> gstreamer-devel mailing list
>> [hidden email]
>> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Olivier Crête-3]

Hi,

On Tue, 2016-12-06 at 18:36 +0100, Petr Kulhavy wrote:
> * Can I integrate the pool creation into the caps negotiation, i.e. 
> _decide_allocation() and possibly reuse a pool provided by a
> downstream 
> element ?

You want to implement your own pool subclass, so you can't use the
downstream one.

> * Can I just pass 0 as size to gst_buffer_pool_config_set_params()
> and 
> gst_query_add_allocation_pool() even though the doc says size must be
> >0 
> ? ;-)

gst_query_add_allocation_pool() is for downstream elements to send
their pool upstream, so it wouldn't be used in this case. Same for the
config params, since your pool may not be doing the allocation, you an
ignore all of those APIs.

> * How can I pre-allocate the GstMemory objects prior to knowing the 
> memory location and size? I don't see a function to replace or 
> manipulate the GstMapInfo within GstMemory.

Yes, you could, you'd need to do your own GstAllocator/GstMemory
subclasses, then you can control what's in it.. That said, I think it
makes little sense to pre-allocate the GstMemory before you allocate
the actual memory.

Olivier

>
> Thanks
> Petr
>
> On 06/12/16 00:39, Olivier Crête wrote:
> > Hi,
> >
> > Using a GstBufferPool is a valid option. One thing you can do is to
> > create entirely empty buffers (with no memories attached). You will
> > then need to implement a custom reset_memory vfunc on the
> > bufferpool
> > to reset the memory tag flag and separate the memories from the
> > buffer
> > before calling up. Then you can create a custom GstAllocator that
> > you
> > can use to get the GstMemory objects from, you can also pre-
> > allocate
> > those (one for each of your buffers), then you can attach them to
> > the
> > GstBuffer from the pool when you want to send something down the
> > pipeline. So your entire use-case can be done without doing a
> > single
> > allocation!
> >
> > Olivier
> >
> >
> > On Mon, 2016-12-05 at 20:00 +0100, Petr Kulhavy wrote:
> > > Hi,
> > >
> > > I'm trying to implement an efficient buffer management for zero-
> > > copy
> > > UDP
> > > reception, where the data is provided in chunks of memory
> > > preallocated
> > > by the kernel.
> > > The unfortunate heavy-weight nature of GStreamer buffers, where
> > > every
> > > buffer is a set of objects, is a bit clashing with the
> > > "efficient"
> > > part
> > > of my task.
> > >
> > > So I have the buffers of memory with data and now need to
> > > implement
> > > a
> > > pool of GstBuffer and GstMemory objects in order to be able to
> > > hand
> > > over
> > > the data down the pipeline. Kind of pool of empty buffers, where
> > > I
> > > can
> > > then hang whatever chunk of memory I need.
> > > The buffer pool abstraction is based on the fact that all buffers
> > > are
> > > of
> > > equal size. This is unfortunately not what I can guarantee.
> > >
> > > Is there some existing abstraction I can use?
> > >
> > > Thanks
> > > Petr
> > > _______________________________________________
> > > gstreamer-devel mailing list
> > > [hidden email]
> > > https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>
>
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>

--
Olivier Crête
[hidden email]
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel