Video decoder performance and QoS

[Yi-Lung Tsai]

Our team currently develop an HEVC decode library and want to integrate it into GStreamer pipeline.

Unfortunately we encounter a performance issue.

We test a video with frame rate 29.97.

In our test-bed, the HEVC library could process video stream under FPS 33~34 on average.

This performance seems enough on the target machine.

Then we develop a video decoder plugin to leverage our HEVC library.

The performance is measured in "playbin" with "fpsdisplaysink".

FPS message is shown every 500ms.

From the beginning, FPS value is around 29.8~30.1.

After two seconds, the FPS starts declining and never goes back to 20.

There is a drop mechanism in "gst_video_decoder_clip_and_push_buf".

I guess this issue is related to QoS.

My question is which direction should I try to get performance improvement.

Thanks.

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Tim Müller]

On Wed, 2017-01-04 at 11:25 +0800, Yi-Lung Tsai wrote:

> Our team currently develop an HEVC decode library and want to
> integrate it into GStreamer pipeline.
> Unfortunately we encounter a performance issue.
>
> We test a video with frame rate 29.97.
> In our test-bed, the HEVC library could process video stream under
> FPS 33~34 on average.
> This performance seems enough on the target machine.
>
> Then we develop a video decoder plugin to leverage our HEVC library.
> The performance is measured in "playbin" with "fpsdisplaysink".
> FPS message is shown every 500ms.
> From the beginning, FPS value is around 29.8~30.1.
> After two seconds, the FPS starts declining and never goes back to
> 20.
>
> There is a drop mechanism in "gst_video_decoder_clip_and_push_buf".
> I guess this issue is related to QoS.

You could first check if that's actually the case.

There might be other reasons for poor performance (e.g. additional
frame copies/allocations or such).

Or fpsdisplaysink's text rendering being slow (use text-overlay=false).

Most sinks (not fpsdisplaysink though) have a "qos" property, which
will be true by default for most video sinks and false for most other
sinks.

You could try e.g.

 ... ! fpsdisplaysink text-overlay=false video-sink="fakesink"

(^^^ measures pure decoder throughput)


 ... ! fpsdisplaysink text-overlay=false video-sink="fakesink sync=true
max-lateness=20000000"

(^^^ like above but syncing to the clock and configure more like a
videosink, but without QoS enabled)


 ... ! fpsdisplaysink text-overlay=false video-sink="fakesink sync=true
max-lateness=20000000 qos=true"

(^^^ like above but syncing to the clock and configure more like a
videosink, with QoS enabled)

Cheers
 -Tim

--
Tim Müller, Centricular Ltd - http://www.centricular.com
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Yi-Lung Tsai]

On Wed, Jan 4, 2017 at 6:07 PM, Tim Müller <[hidden email]> wrote:

On Wed, 2017-01-04 at 11:25 +0800, Yi-Lung Tsai wrote:

> Our team currently develop an HEVC decode library and want to
> integrate it into GStreamer pipeline.
> Unfortunately we encounter a performance issue.
>
> We test a video with frame rate 29.97.
> In our test-bed, the HEVC library could process video stream under
> FPS 33~34 on average.
> This performance seems enough on the target machine.
>
> Then we develop a video decoder plugin to leverage our HEVC library.
> The performance is measured in "playbin" with "fpsdisplaysink".
> FPS message is shown every 500ms.
> From the beginning, FPS value is around 29.8~30.1.
> After two seconds, the FPS starts declining and never goes back to
> 20.
>
> There is a drop mechanism in "gst_video_decoder_clip_and_push_buf".
> I guess this issue is related to QoS.

You could first check if that's actually the case.

There might be other reasons for poor performance (e.g. additional
frame copies/allocations or such).

Or fpsdisplaysink's text rendering being slow (use text-overlay=false).

Most sinks (not fpsdisplaysink though) have a "qos" property, which
will be true by default for most video sinks and false for most other
sinks.

You could try e.g.

 ... ! fpsdisplaysink text-overlay=false video-sink="fakesink"

In this case, average fps could exceed 30 to the end.

(^^^ measures pure decoder throughput)


 ... ! fpsdisplaysink text-overlay=false video-sink="fakesink sync=true
max-lateness=20000000"

In this case, average fps could also exceed 30 to the end.

(^^^ like above but syncing to the clock and configure more like a
videosink, but without QoS enabled)


 ... ! fpsdisplaysink text-overlay=false video-sink="fakesink sync=true
max-lateness=20000000 qos=true"

In this case, the measure result is unstable.

Sometimes average fps ranges from 27 to 30.

But sometimes average fps is down to around 20.

(^^^ like above but syncing to the clock and configure more like a
videosink, with QoS enabled)

Cheers
 -Tim

Here is also an interesting found.

In case 3 above, there is no help if the "max-size-buffers" of queue between decoder and video sink is set to 60.

But in my real playback pipeline, average fps would be 29.8~30.1 if "max-size-buffers" is set to 60 as well.

According to testing results with "fakesink" above, the key point would be QoS.

However, in my real playback, enlarging max size of queue seems an option to get better performance.

Is turning QoS a good option?

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Tim Müller]

Hi,

> >  ... ! fpsdisplaysink text-overlay=false video-sink="fakesink
> > sync=true
> > max-lateness=20000000"
> >
>
> In this case, average fps could also exceed 30 to the end. 
> >
> >
> >  ... ! fpsdisplaysink text-overlay=false video-sink="fakesink
> > sync=true
> > max-lateness=20000000 qos=true"
> >
>
> In this case, the measure result is unstable.
> Sometimes average fps ranges from 27 to 30.
> But sometimes average fps is down to around 20. 
> > (^^^ like above but syncing to the clock and configure more like a
> > videosink, with QoS enabled)
> >
> >
> Here is also an interesting found.
> In case 3 above, there is no help if the "max-size-buffers" of queue
> between decoder and video sink is set to 60.
> But in my real playback pipeline, average fps would be 29.8~30.1 if
> "max-size-buffers" is set to 60 as well.

Mind that the lowest max-* limit wins, so if you decode an large video
then the byte size might easily kick in (e.g. 10MB =~ a few frames @
1080p).

I don't know why it would behave differently though.

It might also be interesting to look at what QoS events basesink sends
exactly - how much frames are late, etc.


> According to testing results with "fakesink" above, the key point
> would be QoS.
> However, in my real playback, enlarging max size of queue seems an
> option to get better performance.

A queue between decoder and videosink is certainly a good idea.


> Is turning QoS a good option?

It depends a bit on your device/environment. The goal is to recover
when decoding/rendering are too slow for a short time for some reason.
If you don't need that then you can disable QoS of course. The sink
will still throw away and not render buffers that are way too late in
that case.

Cheers
 -Tim

--
Tim Müller, Centricular Ltd - http://www.centricular.com
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Yi-Lung Tsai]

Thanks for the QoS opinion.

I found enlarging queue size could fit my needs and I will improve from this direction.

I also took a look at “playsink”.

The function “gen_video_chain” creates a queue for video samples and set “max-size-buffers” to 3.

And “gen_audio_chain” does not set any limits on audio queue.

I think “max-size-buffers=3” is too small for video samples.

--

Yi-Lung (Bruce) Tsai

[hidden email]

On Jan 6, 2017, at 5:53 PM, Tim Müller <[hidden email]> wrote:

Hi,

 ... ! fpsdisplaysink text-overlay=false video-sink="fakesink
sync=true
max-lateness=20000000"

In this case, average fps could also exceed 30 to the end. 

 ... ! fpsdisplaysink text-overlay=false video-sink="fakesink
sync=true
max-lateness=20000000 qos=true"

In this case, the measure result is unstable.
Sometimes average fps ranges from 27 to 30.
But sometimes average fps is down to around 20. 

(^^^ like above but syncing to the clock and configure more like a
videosink, with QoS enabled)

Here is also an interesting found.
In case 3 above, there is no help if the "max-size-buffers" of queue
between decoder and video sink is set to 60.
But in my real playback pipeline, average fps would be 29.8~30.1 if
"max-size-buffers" is set to 60 as well.

Mind that the lowest max-* limit wins, so if you decode an large video
then the byte size might easily kick in (e.g. 10MB =~ a few frames @
1080p).

I don't know why it would behave differently though.

It might also be interesting to look at what QoS events basesink sends
exactly - how much frames are late, etc.

According to testing results with "fakesink" above, the key point
would be QoS.
However, in my real playback, enlarging max size of queue seems an
option to get better performance.

A queue between decoder and videosink is certainly a good idea.

Is turning QoS a good option?

It depends a bit on your device/environment. The goal is to recover
when decoding/rendering are too slow for a short time for some reason.
If you don't need that then you can disable QoS of course. The sink
will still throw away and not render buffers that are way too late in
that case.

Cheers
-Tim

--
Tim Müller, Centricular Ltd - http://www.centricular.com
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel