Minimalistic GstAggregator

[tlpl]

Hi Everyone,

I am currently writing GStreamer (and Nvidia's DeepStream) plugins for the
project I am working on. I have managed to successfully create plugin that
receives one buffer (called original buffer) which is a single video frame,
creates new buffer based on the original one and pushes both downstream from
two separate src pads. Next, I am doing some calculations on new buffer and
attach some custom meta data to it. As a final step I want to create new
element with exactly two sink pads that takes the new buffer (after
calculations mentioned) and original buffer, and then pushes old buffer
downstream. Easy, right?

I started implementation of the new plugin using GstAggregator class, for
GStreamer 1.14.5 (have to use this version because of Nvidia's DeepStream).
I did review "compositor" and "audio_mixer" plugins, but those are not 100%
clear to me and are too broad in general. Nevertheless, the logic inside the
"_aggregate" function in my plugin is not really point of concern - the
problem I am facing is that, even though the code compiles and the plugin
technically work, original buffer which I am pushing downstream "becomes"
16x16 px on the output rather than preserving it's original width and height
of the video frame. For testing I am using my custom Gstreamer application.
I am reading from file and writing output to file with following GStreamer
elements in pipeline:

filesrc -> qtdemux -> h264parse -> nvv4l2decoder -> (DeepStream part) ->
videoconvert -> x264enc -> qtmux -> filesink

I am aware that this is GStreamer devel mailing list and You do not deal
with DeepStream problem, but I really do feel like the problem lies
somewhere in my implementation of GstAggregator, either in caps or in the
way I am sending buffer downstream. It is not clear to me how
GstAggregatorPad class interacts with GstAggregator too. Here are links to
the minimalist version of the source code I currently have:
- https://gist.github.com/tl-oke/0c3afc281b958b139e54eb5048f23d00
- https://gist.github.com/tl-oke/e071f9a9dff79f8204aea4773334f959

Needless to say: I don't even know if GstAggregator is the way to go,
especially that I always be using exactly 2 sink pads in the exact order
that I am using (1st sink pad for new buffer which is then dropped, and 2nd
sink pad for original buffer which is then pushed downstream). Maybe the
problem lies not in implementation but in the pipeline (I have tested that
if I will get rid of this element the resulting output video has correct /
original width and height). I am stuck on it for 2 weeks and any kind of
help / tip / reference to read would be greatly appreciated.

Best,
Tom




--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Matthew Waters]

Hi,

So, 16x16 output from aggregator sounds like some default caps handling
somewhere.  You need to implement the
GstAggregatorClass::update_src_caps and
GstAggregatorClass::fixate_src_caps methods in your class.  In master
and 1.18, you can override the entire negotiation process by overriding
GstAggregatorClass::negotiate.  For video, you probably want to look at
how compositor and glvideomixer implement their negotiation.  Just as a
warning: without conversion internal to the element between all possible
input formats, negotiation on multiple input streams is inherently racy
and may choose incompatible formats or settings.

Side note, the reason for 16x16 is that the default fixation
(gst_caps_fixate()) function will choose the lowest value for fields of
a range type.  For width and height, the lowest value is constrained by
x264enc which has a minimum input size of 16x16.

Cheers
-Matt

On 2/5/20 9:54 pm, tlpl wrote:

> Hi Everyone,
>
> I am currently writing GStreamer (and Nvidia's DeepStream) plugins for the
> project I am working on. I have managed to successfully create plugin that
> receives one buffer (called original buffer) which is a single video frame,
> creates new buffer based on the original one and pushes both downstream from
> two separate src pads. Next, I am doing some calculations on new buffer and
> attach some custom meta data to it. As a final step I want to create new
> element with exactly two sink pads that takes the new buffer (after
> calculations mentioned) and original buffer, and then pushes old buffer
> downstream. Easy, right?
>
> I started implementation of the new plugin using GstAggregator class, for
> GStreamer 1.14.5 (have to use this version because of Nvidia's DeepStream).
> I did review "compositor" and "audio_mixer" plugins, but those are not 100%
> clear to me and are too broad in general. Nevertheless, the logic inside the
> "_aggregate" function in my plugin is not really point of concern - the
> problem I am facing is that, even though the code compiles and the plugin
> technically work, original buffer which I am pushing downstream "becomes"
> 16x16 px on the output rather than preserving it's original width and height
> of the video frame. For testing I am using my custom Gstreamer application.
> I am reading from file and writing output to file with following GStreamer
> elements in pipeline:
>
> filesrc -> qtdemux -> h264parse -> nvv4l2decoder -> (DeepStream part) ->
> videoconvert -> x264enc -> qtmux -> filesink
>
> I am aware that this is GStreamer devel mailing list and You do not deal
> with DeepStream problem, but I really do feel like the problem lies
> somewhere in my implementation of GstAggregator, either in caps or in the
> way I am sending buffer downstream. It is not clear to me how
> GstAggregatorPad class interacts with GstAggregator too. Here are links to
> the minimalist version of the source code I currently have:
> - https://gist.github.com/tl-oke/0c3afc281b958b139e54eb5048f23d00
> - https://gist.github.com/tl-oke/e071f9a9dff79f8204aea4773334f959
>
> Needless to say: I don't even know if GstAggregator is the way to go,
> especially that I always be using exactly 2 sink pads in the exact order
> that I am using (1st sink pad for new buffer which is then dropped, and 2nd
> sink pad for original buffer which is then pushed downstream). Maybe the
> problem lies not in implementation but in the pipeline (I have tested that
> if I will get rid of this element the resulting output video has correct /
> original width and height). I am stuck on it for 2 weeks and any kind of
> help / tip / reference to read would be greatly appreciated.
>
> Best,
> Tom
>
>
>
>
> --
> Sent from: http://gstreamer-devel.966125.n4.nabble.com/
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel