Reading and Modifying Buffers in a Pipeline

[mhassan]

Hi. I am new to gstreamer and trying to figure out a way to read buffers from a video source, modifying the buffers in some way (e.g. draw overlay information) and output the buffers to a video or filesink. I am using the gstreamer-rust bindings and have not been able to find a solution to this problem. Reading on the forums, I tried to add a pad probe at the source element of the following pipeline as a test by getting the static pad: 

"videotestsrc ! autovideosink"

I am able to read the buffer from "probe_info.data" :

src_pad.add_probe(gst::PadProbeType::BUFFER, |_, probe_info| {

___READ BUFFER___

}

However, if I try to map the buffer as writable by using gst::buffer::BufferRef::map_writeable()

it doesn't allow me to do so with the error "cannot borrow as mutable". At this point I am completely lost and unable to find an alternative to read and modify a buffer element through a pad probe. Any help or suggestion would be appreciated to help me overcome this.

Thanks,

Muhammad Hassan

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On Tue, 2020-11-10 at 12:24 +0500, Hassan Muhammad wrote:

Hi. I am new to gstreamer and trying to figure out a way to read buffers from a video source, modifying the buffers in some way (e.g. draw overlay information) and output the buffers to a video or filesink. I am using the gstreamer-rust bindings and have not been able to find a solution to this problem. Reading on the forums, I tried to add a pad probe at the source element of the following pipeline as a test by getting the static pad: 

"videotestsrc ! autovideosink"

I am able to read the buffer from "probe_info.data" :

src_pad.add_probe(gst::PadProbeType::BUFFER, |_, probe_info| {

___READ BUFFER___

}

However, if I try to map the buffer as writable by using gst::buffer::BufferRef::map_writeable()

it doesn't allow me to do so with the error "cannot borrow as mutable". At this point I am completely lost and unable to find an alternative to read and modify a buffer element through a pad probe. Any help or suggestion would be appreciated to help me overcome this.

The following should work

    src_pad.add_probe(gst::PadProbeType::BUFFER, |_, probe_info| {

        if let Some(gst::PadProbeData::Buffer(ref mut buffer)) = probe_info.data {

            let buffer = buffer.make_mut();

            let map = buffer.map_writable().unwrap();

            [do things with `map`] 

        }

        gst::PadProbeReturn::Ok

    });

You need to get a mutable reference to the `gst::Buffer` from the `probe_info`, and then make sure that its content is actually writable and get a `&mut gst::BufferRef` to modify it via `make_mut()`.

Then you can map it writable, change the PTS/DTS etc.

However for modifying buffers (e.g. to add an overlay), it would be better to write a new element for that or make use of the `overlaycomposition` element.

For the former you can find various examples in gst-plugins-rs, for the latter you can find an example here:

    https://gitlab.freedesktop.org/gstreamer/gstreamer-rs/-/blob/master/examples/src/bin/overlay-composition.rs

-- 

Sebastian Dröge, Centricular Ltd · https://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[mhassan]

Thank you for your prompt response. It has been really helpful. I have taken a look over the plugin writing tutorial that you have referenced and it seems to me that this may be the most appropriate solution to modifying buffers in a non-intrusive manner i.e. within an element between source and sink pads. One thing I'd like to ask is, would mapping buffer read/writable and performing some operation on the buffer (i.e. like you did converting BGRx colorspace to GRAY8) impose any significant performance penalty on the pipeline when specifically rendering HD or FHD videos?

Sorry if this seems like a silly question, I'm actually new to gstreamer and only starting to get the hang of how it works. Thanks again for your help.

On Tue, Nov 10, 2020 at 6:00 PM Sebastian Dröge <[hidden email]> wrote:

On Tue, 2020-11-10 at 12:24 +0500, Hassan Muhammad wrote:

Hi. I am new to gstreamer and trying to figure out a way to read buffers from a video source, modifying the buffers in some way (e.g. draw overlay information) and output the buffers to a video or filesink. I am using the gstreamer-rust bindings and have not been able to find a solution to this problem. Reading on the forums, I tried to add a pad probe at the source element of the following pipeline as a test by getting the static pad: 

"videotestsrc ! autovideosink"

I am able to read the buffer from "probe_info.data" :

src_pad.add_probe(gst::PadProbeType::BUFFER, |_, probe_info| {

___READ BUFFER___

}

However, if I try to map the buffer as writable by using gst::buffer::BufferRef::map_writeable()

it doesn't allow me to do so with the error "cannot borrow as mutable". At this point I am completely lost and unable to find an alternative to read and modify a buffer element through a pad probe. Any help or suggestion would be appreciated to help me overcome this.

The following should work

    src_pad.add_probe(gst::PadProbeType::BUFFER, |_, probe_info| {

        if let Some(gst::PadProbeData::Buffer(ref mut buffer)) = probe_info.data {

            let buffer = buffer.make_mut();

            let map = buffer.map_writable().unwrap();

            [do things with `map`] 

        }

        gst::PadProbeReturn::Ok

    });

You need to get a mutable reference to the `gst::Buffer` from the `probe_info`, and then make sure that its content is actually writable and get a `&mut gst::BufferRef` to modify it via `make_mut()`.

Then you can map it writable, change the PTS/DTS etc.

However for modifying buffers (e.g. to add an overlay), it would be better to write a new element for that or make use of the `overlaycomposition` element.

For the former you can find various examples in gst-plugins-rs, for the latter you can find an example here:

    https://gitlab.freedesktop.org/gstreamer/gstreamer-rs/-/blob/master/examples/src/bin/overlay-composition.rs

-- 

Sebastian Dröge, Centricular Ltd · https://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On Thu, 2020-11-12 at 11:39 +0500, Hassan Muhammad wrote:

Thank you for your prompt response. It has been really helpful. I have taken a look over the plugin writing tutorial that you have referenced and it seems to me that this may be the most appropriate solution to modifying buffers in a non-intrusive manner i.e. within an element between source and sink pads.

It would at least be the cleanest solution. You also don't have to write a full plugin but can also do just an element as part of your application.

One thing I'd like to ask is, would mapping buffer read/writable and performing some operation on the buffer (i.e. like you did converting BGRx colorspace to GRAY8) impose any significant performance penalty on the pipeline when specifically rendering HD or FHD videos?

Sorry if this seems like a silly question, I'm actually new to gstreamer and only starting to get the hang of how it works. Thanks again for your help.

Mapping a buffer writable might cause it to be copied if the buffer's memory is also in use elsewhere. Apart from that, if you need to modify the individual pixels then there's not much else you can do.

By using the overlaycomposition element this could potentially be more efficient if the overlayed frames end up in an element (e.g. a sink) that can handle the overlaying separately.

-- 

Sebastian Dröge, Centricular Ltd · https://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel