BaseTransform with live video

[Ralf Sippl]

Hello,

I have a set of plugins written in Python. A transform element
(BaseTransform subclass) analyzes video frames and outputs data as a stream
of serialized Python objects (custom application/x-mydatastream caps;
serialization is done in do_prepare_output_buffer()). A second element, an
aggregator, takes data from this stream and merges it with the initial video
stream (coming from a tee element), drawing overlays on top of it.

Basically, this works fine, but the resulting stream is only as fast as the
slowest element in the pipeline, the transform element which runs the
analysis. For live video input, this means that frames are being dropped.
Ideally, I'd like the video to be played at full framerate, with the
overlays displayed when available, or based on cached data. Is this
possible?



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Mathieu Duponchelle]

If your transform element is too slow for real time operation, it should handle
QoS events and skip ahead. Out of curiosity, are you performing the transformation
in pure python? If so, it would probably be wise to optimize the critical bits out to
a C extension for example, as pure python for heavy duty image manipulation is
bound to have very poor performance :)

On 4/10/19 3:21 PM, rs99 wrote:

> Hello,
>
> I have a set of plugins written in Python. A transform element
> (BaseTransform subclass) analyzes video frames and outputs data as a stream
> of serialized Python objects (custom application/x-mydatastream caps;
> serialization is done in do_prepare_output_buffer()). A second element, an
> aggregator, takes data from this stream and merges it with the initial video
> stream (coming from a tee element), drawing overlays on top of it.
>
> Basically, this works fine, but the resulting stream is only as fast as the
> slowest element in the pipeline, the transform element which runs the
> analysis. For live video input, this means that frames are being dropped.
> Ideally, I'd like the video to be played at full framerate, with the
> overlays displayed when available, or based on cached data. Is this
> possible?
>
>
>
> --
> Sent from: http://gstreamer-devel.966125.n4.nabble.com/
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Ralf Sippl]

Mathieu Duponchelle wrote
> If your transform element is too slow for real time operation, it should
> handle
> QoS events and skip ahead.

Thanks for the reply. "Skip ahead" is a bit vague :). I tried doing this via
a gap event, but it doesn't work as desired, because the transform element
can't anticipate precisely how long the transformation will take, and the
gap event needs to contain the gap duration. Especially when having more
than one transform pipeline 'branch', the results get quite ugly...

I believe the aggregator isn't the right element class for this. An element
(maybe a BaseTransform?) with always pads for in and out video, and request
sink pads for each transform branch of the pipeline might work better.

Sure, pure Python is too slow for this, but  even with C / C++, most tasks
cannot be done in realtime, so I believe my question remains valid
regardless of the implementation.



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Mathieu Duponchelle]

A gap event is not required to have a duration. However I assume you could
calculate one the following way: when the received QoS event shows that your
transform element is too late, you interrupt whatever transformation you were
performing, then drop input buffers until you've caught up with the desired position
(this implies having a large enough queue in front of your transform element),
and only send the gap event at that point, would that work?

On 4/12/19 10:28 AM, rs99 wrote:

> Mathieu Duponchelle wrote
>> If your transform element is too slow for real time operation, it should
>> handle
>> QoS events and skip ahead.
> Thanks for the reply. "Skip ahead" is a bit vague :). I tried doing this via
> a gap event, but it doesn't work as desired, because the transform element
> can't anticipate precisely how long the transformation will take, and the
> gap event needs to contain the gap duration. Especially when having more
> than one transform pipeline 'branch', the results get quite ugly...
>
> I believe the aggregator isn't the right element class for this. An element
> (maybe a BaseTransform?) with always pads for in and out video, and request
> sink pads for each transform branch of the pipeline might work better.
>
> Sure, pure Python is too slow for this, but  even with C / C++, most tasks
> cannot be done in realtime, so I believe my question remains valid
> regardless of the implementation.
>
>
>
> --
> Sent from: http://gstreamer-devel.966125.n4.nabble.com/
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel