GStreamer-devel

Slow plugin skipping frames when needed

Classic

List

Threaded

4 messages Options

MathieuP

Slow plugin skipping frames when needed

Hi,

I am currently working on a "slow" plugin, meant to analyze as many frames
of a live stream as possible and drop the rest to make sure I am not falling
behind. Let's say my hardware can deal with 10 frames per second and I have
a live stream of 30 FPS in input, I am looking for a way to keep only 1 out
of 3 frames. I want my plugin to be the driving force of this frame rate,
because I can't come up with a magic number for the frame rate that will
work on any PC / input video.

I thought about dynamically updating the frame rate CAPS of my plugin and
let an upstream videorate plugin handle the frame rate changes. However, I
feel like this would make the adjustments happen too late: if the first
frame takes 1 sec to process, I'll already have 29 frames queued up to catch
up before I get a filtered input.

From the docs, I feel like supporting pull mode would be perfect for my use
case. This way, I could easily come up with 2 different pipelines: one that
would handle each frame but fall behind quickly, and one that would drop
unhandled frames and pull the latest frame every time we have processing
power available.
However, I couldn't come up with a pipeline that actually does this. I
thought putting a "queue max-size-buffers=1 leaky=downstream" upstream would
do the trick, however this guy doesn't support pull mode according to my
tests.

I also thought about putting an upstream valve and update its "drop"
property dynamically, however I feel like this is bad architecture because
my plugin would need to be aware of its upstream plugin, by specifically
looking for this valve. Actually, I don't even know if this is even possible
since I haven't tested it yet, but it just feels wrong.

Has anyone ever done something like this? I'd love to have some help about
the architecture I should use for this particular use case.

--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

MathieuP

Re: Slow plugin skipping frames when needed

Latest news is, I decided to create my own plugin to achieve something
similar to what queue2 can do when using a file or a ring-buffer: work in
Push mode in its sink pad, and in Pull mode on its src pad.

My issue is that my entire pipeline works well in Push mode, but when I use
Pull mode, the upstream guy doesn't Push anything. I tried replacing my
plugin with queue2, and the upstream Push works perfectly fine, so I'd
assume the problem is with my "custom queue" plugin.

Looking at the source code, I can't see any additional step queue2 does
during query/activatemode phases. I feel like it might be a threading issue,
where my Pull plugin does start its own thread as expected via
gst_pad_start_task, but this somehow disables the default behaviour of the
upstream plugins looping on a thread provided by the gstreamer framework
itself.

Has anyone ever encountered something like this?

MathieuP wrote

> Hi,
>
> I am currently working on a "slow" plugin, meant to analyze as many frames
> of a live stream as possible and drop the rest to make sure I am not
> falling
> behind. Let's say my hardware can deal with 10 frames per second and I
> have
> a live stream of 30 FPS in input, I am looking for a way to keep only 1
> out
> of 3 frames. I want my plugin to be the driving force of this frame rate,
> because I can't come up with a magic number for the frame rate that will
> work on any PC / input video.
>
> I thought about dynamically updating the frame rate CAPS of my plugin and
> let an upstream videorate plugin handle the frame rate changes. However, I
> feel like this would make the adjustments happen too late: if the first
> frame takes 1 sec to process, I'll already have 29 frames queued up to
> catch
> up before I get a filtered input.
>
> From the docs, I feel like supporting pull mode would be perfect for my
> use
> case. This way, I could easily come up with 2 different pipelines: one
> that
> would handle each frame but fall behind quickly, and one that would drop
> unhandled frames and pull the latest frame every time we have processing
> power available.
> However, I couldn't come up with a pipeline that actually does this. I
> thought putting a "queue max-size-buffers=1 leaky=downstream" upstream
> would
> do the trick, however this guy doesn't support pull mode according to my
> tests.
>
> I also thought about putting an upstream valve and update its "drop"
> property dynamically, however I feel like this is bad architecture because
> my plugin would need to be aware of its upstream plugin, by specifically
> looking for this valve. Actually, I don't even know if this is even
> possible
> since I haven't tested it yet, but it just feels wrong.
>
> Has anyone ever done something like this? I'd love to have some help about
> the architecture I should use for this particular use case.
>
>
>
> --
> Sent from: http://gstreamer-devel.966125.n4.nabble.com/
> _______________________________________________
> gstreamer-devel mailing list

> gstreamer-devel@.freedesktop

> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

Tim Müller

Re: Slow plugin skipping frames when needed

On Mon, 2018-11-12 at 21:59 -0600, MathieuP wrote:

Hi Mathieu,

Some more context would be good.

Pull mode usually only works for byte-based streams that can do random
access (theoretically it should also be possible to do pull mode
sequentially on a stream but so far this is only something the API
provides, I'm not aware of anything actually implementing that).

Pull mode always needs the co-operation of the upstream elements, and
there are very few elements that support it, and none of them video-
specific.

I think you will probably have to find a solution that works with push
mode, unless you read input from a file with raw video data or so.

Is your analysis plugin a pass-through filter of some sort, e.g. could
it just pass through video frames immediately as they come in at full
framerate, but only analyse a fraction of them?

Or did you want to take video in, and output data of a different kind
for each analysed frame, so the output would be at a lower/variable
frame rate?

Cheers
Tim

--
Tim Müller, Centricular Ltd - http://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

MathieuP

Re: Slow plugin skipping frames when needed

Hi Tim, thanks for the answer.

This plugin is not a pass-through plugin. For my use case, I think it's
better to drop frames I don't have time to analyze instead of outputting
everything using a video-filter, but only analyzing a bunch of them. For a
fluid streaming experience, I assume I should still be able to achieve this
by using a tee and synchronizing the two streams based the timestamps in the
end of the pipeline. I haven't tested this yet, though.

About the fact that all upstream plugins should work in Pull mode too, I
feel like there has to be a way around this, because this is exactly what
queue2 can do ("converting" Push to Pull when using a ring buffer). However,
I could not find any other plugin doing the same thing, so this one looks
like the exception.

In the end, I ended up handling this queue internally and spawning a new
task on the srcpad to make everything work in Push mode.
I feel like it is not as clean as I expected it would be in the end, because
this plugin is now responsible for too many unrelated operations (handling a
queue and analyzing images).
Also, I plan on creating other "analyzer" plugins in the future, so I would
have loved to be able to extract this "queue" part to reuse it, but I gave
up on this idea for now.

Tim Müller wrote

> On Mon, 2018-11-12 at 21:59 -0600, MathieuP wrote:
>
> Hi Mathieu,
>
> Some more context would be good.
>
> Pull mode usually only works for byte-based streams that can do random
> access (theoretically it should also be possible to do pull mode
> sequentially on a stream but so far this is only something the API
> provides, I'm not aware of anything actually implementing that).
>
> Pull mode always needs the co-operation of the upstream elements, and
> there are very few elements that support it, and none of them video-
> specific.
>
> I think you will probably have to find a solution that works with push
> mode, unless you read input from a file with raw video data or so.
>
> Is your analysis plugin a pass-through filter of some sort, e.g. could
> it just pass through video frames immediately as they come in at full
> framerate, but only analyse a fraction of them?
>
> Or did you want to take video in, and output data of a different kind
> for each analysed frame, so the output would be at a lower/variable
> frame rate?
>
> Cheers
> Tim
>
> --
> Tim Müller, Centricular Ltd - http://www.centricular.com
>
> _______________________________________________
> gstreamer-devel mailing list