Calling set_property for each buffer (video frame) from probe or signal handler?

[mkonstapel]

Hi all,

What I am trying to achieve, is to do "digital pan/zoom" on live video
frames using a glvideomixer. For each frame, I want to set xpos, ypos,
width, and height on the glvideomixer sink (input) pad - ideally, right
before each buffer is processed. However, when I do this in a buffer
probe callback on the sink pad, I occasionally get incorrect frames,
where some of the properties have updated and others have not. For
example, the xpos might have been updated while the ypos, height and
width are still the old values, resulting in a distorted frame. Note
that I am using the Python bindings, so I don't think I have access to a
`set_property` that can set multiple properties in one call, as would be
possible from C.

My workaround so far is to set the properties from a probe on the
*source* pad instead. That way, the property changes have one frame
duration (40 ms, at 25 fps) to be applied. That appears to be working,
and I can live with the extra one frame delay, but I am still curious
why this is required. I would expect the buffer not to be delivered to
the input pad until the probe callback has returned, and the
set_property calls to have completed.

Is this an issue specifically with glvideomixer? Are set_property calls
allowed from a probe callback? If not, what about a signal handler?
Would it be better to insert an identity element before the mixer and
call set_property from its handoff signal?

Alternatively, I could do the set_property calls from the main thread,
but then I don't see any way to synchronize them against the
glvideomixer's redraws; there's no way to do locking to ensure all four
properties are updated at once, is there?

Kind regards,
Michiel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Matthew Waters]

There is a samples-selected signal on glvideomixerelement (in 1.18+)
where you can get a callback at the correct place to set the necessary
pad information after selection but before composition which should work
for this case.

See e.g.
https://gstreamer.freedesktop.org/documentation/base/gstaggregator.html?gi-language=c#GstAggregator::samples-selected 
<https://gstreamer.freedesktop.org/documentation/base/gstaggregator.html?gi-language=c#GstAggregator::samples-selected>.

Cheers
-Matt

On 12/11/20 2:42 am, Michiel Konstapel wrote:

> Hi all,
>
> What I am trying to achieve, is to do "digital pan/zoom" on live video
> frames using a glvideomixer. For each frame, I want to set xpos, ypos,
> width, and height on the glvideomixer sink (input) pad - ideally,
> right before each buffer is processed. However, when I do this in a
> buffer probe callback on the sink pad, I occasionally get incorrect
> frames, where some of the properties have updated and others have not.
> For example, the xpos might have been updated while the ypos, height
> and width are still the old values, resulting in a distorted frame.
> Note that I am using the Python bindings, so I don't think I have
> access to a `set_property` that can set multiple properties in one
> call, as would be possible from C.
>
> My workaround so far is to set the properties from a probe on the
> *source* pad instead. That way, the property changes have one frame
> duration (40 ms, at 25 fps) to be applied. That appears to be working,
> and I can live with the extra one frame delay, but I am still curious
> why this is required. I would expect the buffer not to be delivered to
> the input pad until the probe callback has returned, and the
> set_property calls to have completed.
>
> Is this an issue specifically with glvideomixer? Are set_property
> calls allowed from a probe callback? If not, what about a signal
> handler? Would it be better to insert an identity element before the
> mixer and call set_property from its handoff signal?
>
> Alternatively, I could do the set_property calls from the main thread,
> but then I don't see any way to synchronize them against the
> glvideomixer's redraws; there's no way to do locking to ensure all
> four properties are updated at once, is there?
>
> Kind regards,
> Michiel
>
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[mkonstapel]

Thanks Matt!

Works like a charm, and I don't think I'd ever have found that one :) So just to clarify for others: it's on the *glvideomixerelement*, not on the glvideomixer *bin*. To hook it up: glvideomixer.get_by_name("mixer").connect("samples-selected", callback)

Kind regards,
Michiel

On 12-11-2020 07:06, Matthew Waters wrote:

There is a samples-selected signal on glvideomixerelement (in 1.18+) where you can get a callback at the correct place to set the necessary pad information after selection but before composition which should work for this case.

See e.g. https://gstreamer.freedesktop.org/documentation/base/gstaggregator.html?gi-language=c#GstAggregator::samples-selected <https://gstreamer.freedesktop.org/documentation/base/gstaggregator.html?gi-language=c#GstAggregator::samples-selected>.

Cheers
-Matt

On 12/11/20 2:42 am, Michiel Konstapel wrote:

Hi all,

What I am trying to achieve, is to do "digital pan/zoom" on live video frames using a glvideomixer. For each frame, I want to set xpos, ypos, width, and height on the glvideomixer sink (input) pad - ideally, right before each buffer is processed. However, when I do this in a buffer probe callback on the sink pad, I occasionally get incorrect frames, where some of the properties have updated and others have not. For example, the xpos might have been updated while the ypos, height and width are still the old values, resulting in a distorted frame. Note that I am using the Python bindings, so I don't think I have access to a `set_property` that can set multiple properties in one call, as would be possible from C.

My workaround so far is to set the properties from a probe on the *source* pad instead. That way, the property changes have one frame duration (40 ms, at 25 fps) to be applied. That appears to be working, and I can live with the extra one frame delay, but I am still curious why this is required. I would expect the buffer not to be delivered to the input pad until the probe callback has returned, and the set_property calls to have completed.

Is this an issue specifically with glvideomixer? Are set_property calls allowed from a probe callback? If not, what about a signal handler? Would it be better to insert an identity element before the mixer and call set_property from its handoff signal?

Alternatively, I could do the set_property calls from the main thread, but then I don't see any way to synchronize them against the glvideomixer's redraws; there's no way to do locking to ensure all four properties are updated at once, is there?

Kind regards,
Michiel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel