[videocrop/compositor] Change output of videocrop just in time using GstVideoCropMeta metadata

[Arjen Veenhuizen]

My goal is to create a composited image with a low resolution/low quality full frame and one or multiple high quality cut-outs, neatly aligned and scaled to properly fit the low resolution full frame. (See [1] for an impression)

I have roughly the following pipeline wrapped in python code:

                                                        - [queue] - [videocrop] - [          ]
                                                       /                          [          ]
[videotestsrc] - [capsfilter framerate=10/1] - [tee] -    [encoder] - [decoder] - [compositor] - [capsfilter framerate=25/1] - [xvimagesink]
                                                       \                          [          ]
                                                        - [queue] - [videocrop] - [          ]

I use videocrop to crop a specific region per frame. This region (x, y, width, height), however, also changes for each frame. I cannot rely on out of band pad-probes and change the videocrop element properties (left, right, top, bottom) in a pad probe callback due to overhead so I have (and want to) do this in-band. Therefore I have modified the videocrop element to accept GstVideoCropMeta metadata and use that to define the output per frame. The videocrop element will append GstvideoCropMeta metadata to its output buffer as well instructing the modified compositor element on how to position and scale its input frame on its sinkpad.

Problem 1:
If I understand things correctly, I am already too late to set the output frame width/height when I am parsing the input buffer in the videocrop element. If I would still output a frame with the required width/height (as signalled in the GstVideoCropMeta metadata), I would potentially start writing in unallocated memory (because my output frame is simply not big enough) causing visual distortions and sooner or later a segfault. As a partial work-around, I signal the width and height of the next output frame one frame in advance so that the src pad can negotiate the proper caps on time but this is far from ideal. Is there a way to properly solve this problem?

Problem 2:
In case the input framerate is equal to the output framerate, things run relatively smoothly using the workaround above. However, if the input framerate is lower than the output framerate, I notice all kinds of visual glitches in the output video. I am not an expert on the exact inner-workings of the compositor element, but this is my assessment so far:

I change the output caps of videocrop src pad 10 times per second, but the compositor element queries the sink pad linked to the videocrop element 25 times per second(?). Since I change the src pad caps just after a frame is output on the src pad, the compositor element would (at the next interval) query an incorrectly sized frame (because the src pad properties already changed but no new frame was rendered on the videocrop src pad). This way, the videocrop element expects a bigger/smaller frame on its sink pad than actually available. Is this assessment correct or am I making a mistake?

Any insight on how to tackle this problem?

On a side note, perhaps problem 1 is related to the issue raised by Nicolas Dufresne recently [2].

[1] https://www.dropbox.com/s/szavq5cbtrmtvj1/DynamicCropper.jpg?dl=0
[2] http://gstreamer-devel.966125.n4.nabble.com/Re-gst-plugins-good-videocrop-mark-crop-properties-as-mutable-in-playing-state-td4677731.html

[Nicolas Dufresne-4]

Le dimanche 29 mai 2016 à 10:10 -0700, Arjen Veenhuizen a écrit :
> I use videocrop to crop a specific region per frame. This region (x, y,
> width, height), however, also changes for each frame. I cannot rely on out
> of band pad-probes and change the videocrop element properties (left, right,
> top, bottom) in a pad probe callback due to overhead so I have (and want to)
> do this in-band. Therefore I have modified the videocrop element to accept
> GstVideoCropMeta metadata and use that to define the output per frame. The
> videocrop element will append GstvideoCropMeta metadata to its output buffer
> as well instructing the modified compositor element on how to position and
> scale its input frame on its sinkpad. 

That's a great change, hope you'll share this, it would be really nice.

>
> Problem 1: 
> If I understand things correctly, I am already too late to set the output
> frame width/height when I am parsing the input buffer in the videocrop
> element. If I would still output a frame with the required width/height (as
> signalled in the GstVideoCropMeta metadata), I would potentially start
> writing in unallocated memory (because my output frame is simply not big
> enough) causing visual distortions and sooner or later a segfault. As a
> partial work-around, I signal the width and height of the next output frame
> one frame in advance so that the src pad can negotiate the proper caps on
> time but this is far from ideal. Is there a way to properly solve this
> problem?

It's a general bug in the way videocrop negotiate. The final
negotiation can be moved to when the frame is to be produces. This
would at the same time remove few races that occurs when we change
properties during the negotiation process.

>
> Problem 2:
> In case the input framerate is equal to the output framerate, things run
> relatively smoothly using the workaround above. However, if the input
> framerate is lower than the output framerate, I notice all kinds of visual
> glitches in the output video. I am not an expert on the exact inner-workings
> of the compositor element, but this is my assessment so far:

Interesting, seems like a seperate bug. Looks like a buffer being
released, but still used.

>
> I change the output caps of videocrop src pad 10 times per second, but the
> compositor element queries the sink pad linked to the videocrop element 25
> times per second(?). Since I change the src pad caps just after a frame is
> output on the src pad, the compositor element would (at the next interval)
> query an incorrectly sized frame (because the src pad properties already
> changed but no new frame was rendered on the videocrop src pad). This way,
> the videocrop element expects a bigger/smaller frame on its sink pad than
> actually available. Is this assessment correct or am I making a mistake?

A VideoCropMeta should be applied independently from the properties. In
fact, the properties should apply on top of the meta. You might have a
bug in your implementation here.

Hope this helps, if you need more precise guidance, share some code.

regards,
Nicolas
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel