Support for keyframe-only trickmode

[gimmeamilk]

My application needs to support keyframe-only high-speed trickmode, both forward and reverse. So to get reasonable performance I need to ensure only keyframes are pushed from as far upstream as possible.

I was thinking of making a "KeyframeFilter" plugin that sits just downstream from the demuxer and, when enabled, only passes down the keyframes. (it would check the GST_BUFFER_FLAG_DELTA_UNIT on the incoming buffers to decide which were keyframes).

Is this a reasonable solution? I couldn't find any support for this within the demuxers themselves (specifically qtdemux), so I suppose this is the next best thing in terms of efficiency. Or has a better way already been implemented?

Thanks.

[Tim-Philipp Müller-2]

On Wed, 2012-11-21 at 13:56 -0800, gimmeamilk wrote:

> My application needs to support keyframe-only high-speed trickmode, both
> forward and reverse. So to get reasonable performance I need to ensure only
> keyframes are pushed from as far upstream as possible.
>
> I was thinking of making a "KeyframeFilter" plugin that sits just downstream
> from the demuxer and, when enabled, only passes down the keyframes. (it
> would check the GST_BUFFER_FLAG_DELTA_UNIT on the incoming buffers to decide
> which were keyframes).
>
> Is this a reasonable solution? I couldn't find any support for this within
> the demuxers themselves (specifically qtdemux), so I suppose this is the
> next best thing in terms of efficiency. Or has a better way already been
> implemented?

There's a SKIP seek flag that's supposed to be for this. You could just
implement that in the demuxer(s) in question, shouldn't be too hard,
hopefully. (1.0 please).

You can also drop select buffers using a pad probe, fwiw.

Cheers
 -Tim

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[gimmeamilk]

Tim-Philipp Müller-2 wrote

There's a SKIP seek flag that's supposed to be for this. You could just
implement that in the demuxer(s) in question, shouldn't be too hard,
hopefully. (1.0 please).

Thanks, I will look at implementing this long-term. I think I need finer control over the decision making though, eg for my app, anything up to 4x speed MUST push all frames and anything faster MUST push only I-frames.

[Tim-Philipp Müller-2]

On Wed, 2012-11-21 at 15:21 -0800, gimmeamilk wrote:

> Thanks, I will look at implementing this long-term. I think I need finer
> control over the decision making though, eg for my app, anything up to 4x
> speed MUST push all frames and anything faster MUST push only I-frames.

Your app is the one changing the rate (via a seek request), so you can
just set the flag or not set it according to whatever rate you request,
no?

Cheers
 -Tim

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[gimmeamilk]

Tim-Philipp Müller-2 wrote

Your app is the one changing the rate (via a seek request), so you can
just set the flag or not set it according to whatever rate you request,
no?

True... :)

[Robert Swain-2]

On Wednesday, November 21, 2012, Tim-Philipp Müller wrote:

On Wed, 2012-11-21 at 13:56 -0800, gimmeamilk wrote:

> My application needs to support keyframe-only high-speed trickmode, both
> forward and reverse. So to get reasonable performance I need to ensure only
> keyframes are pushed from as far upstream as possible.
>
> I was thinking of making a "KeyframeFilter" plugin that sits just downstream
> from the demuxer and, when enabled, only passes down the keyframes. (it
> would check the GST_BUFFER_FLAG_DELTA_UNIT on the incoming buffers to decide
> which were keyframes).
>
> Is this a reasonable solution? I couldn't find any support for this within
> the demuxers themselves (specifically qtdemux), so I suppose this is the
> next best thing in terms of efficiency. Or has a better way already been
> implemented?

There's a SKIP seek flag that's supposed to be for this. You could just
implement that in the demuxer(s) in question, shouldn't be too hard,
hopefully. (1.0 please).

Is the SKIP seek just a tweak for the demuxer if you know better than to just let QoS do the adaptation in decodes...?

Best regards,

Rob

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Wim Taymans]

On 11/22/2012 10:03 AM, Rob wrote:

> On Wednesday, November 21, 2012, Tim-Philipp Müller wrote:
>
>     On Wed, 2012-11-21 at 13:56 -0800, gimmeamilk wrote:
>
>     > My application needs to support keyframe-only high-speed
>     trickmode, both
>     > forward and reverse. So to get reasonable performance I need to
>     ensure only
>     > keyframes are pushed from as far upstream as possible.
>     >
>     > I was thinking of making a "KeyframeFilter" plugin that sits
>     just downstream
>     > from the demuxer and, when enabled, only passes down the
>     keyframes. (it
>     > would check the GST_BUFFER_FLAG_DELTA_UNIT on the incoming
>     buffers to decide
>     > which were keyframes).
>     >
>     > Is this a reasonable solution? I couldn't find any support for
>     this within
>     > the demuxers themselves (specifically qtdemux), so I suppose
>     this is the
>     > next best thing in terms of efficiency. Or has a better way
>     already been
>     > implemented?
>
>     There's a SKIP seek flag that's supposed to be for this. You could
>     just
>     implement that in the demuxer(s) in question, shouldn't be too hard,
>     hopefully. (1.0 please).
>
>
> Is the SKIP seek just a tweak for the demuxer if you know better than
> to just let QoS do the adaptation in decodes...?

 From the part-trickmodes.txt desgin doc:

  When the SKIP flag is set, frames can be dropped in the elements. If S
is the
  speedup factor, a good algorithm for implementing frame skipping is to
send audio in
  chunks of Nms (usualy 300ms is good) and then skip ((S-1) * Nns) of
audio data.
  For the video we send only the keyframes in the (S * Nns) interval. In
this
  case, the demuxer would scale the timestamps and would set an applied
rate of S.

I would like to see this implemented at some point.

Wim
>
> Best regards,
> Rob
>
>
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Tim-Philipp Müller-2]

On Thu, 2012-11-22 at 10:03 +0100, Rob wrote:

> Is the SKIP seek just a tweak for the demuxer if you know better than
> to just let QoS do the adaptation in decodes...?

It depends a bit where your choke points are at those speeds. If you
process things at 8x, 16x or 32x, you might find that the
demuxer/parsers start using way too much CPU, or that data can't be read
fast enough from storage/network, or that all those unaligned memcpys
are really starting to bite. In those cases, the demuxer might need to
avoid unnecessary processing to make things work at all.

 Cheers
  -Tim


_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel