Using GPU to decode video on Android

[ivan-perez]

Hello

I'm receiving on my Android mobile phone a video via RTP/UDP, but I've seen that CPU usage is very high, nearly 100% of one core is being used to decode the video. On high-end devices the video is played smoothly, but on middle and low-end devices I barely get 1 frame per second.

This is my pipeline:

udpsrc port=5004 caps=application/x-rtp,payload=96,clock-rate=90000,encoding-name=H264 \
    ! rtpjitterbuffer drop-on-latency=true max-dropout-time=500 \
    ! rtph264depay \
    ! video/x-h264,width=480,height=360,framerate=30/1,profile=baseline,stream-format=avc,alignment=au \
    ! h264parse \
    ! decodebin \
    ! videoconvert \
    ! autovideosink

And this one is the pipeline on the server side (in this case I'm using a Raspberry):

gst-launch-1.0 -e -v fdsrc \
  ! video/x-h264,width=480,height=360,framerate=30/1,profile=baseline,stream-format=avc,alignment=au \
  ! h264parse \
  ! rtph264pay \
      config-interval=1 \
      pt=96 \
  ! udpsink \
      sync=false \
      host=192.168.1.5 \
      port=5004 \
      bind-address=192.168.1.2 \
      bind-port=5004

I think that this can be fixed if I could use GPU decoding instead of software decoding, but I haven't found any information about it on the docs.

So my question is: How can I decode a H264 video (or whatever other format, I don't mind to use another one if I get better results) using hardware acceleration on Android?

Thanks!

Kind regards.

[Matthew Waters]

On 20/01/17 03:12, ivan-perez wrote:

> Hello
>
> I'm receiving on my Android mobile phone a video via RTP/UDP, but I've seen
> that CPU usage is very high, nearly 100% of one core is being used to decode
> the video. On high-end devices the video is played smoothly, but on middle
> and low-end devices I barely get 1 frame per second.
>
> This is my pipeline:
>
> udpsrc port=5004
> caps=application/x-rtp,payload=96,clock-rate=90000,encoding-name=H264 \
>     ! rtpjitterbuffer drop-on-latency=true max-dropout-time=500 \
>     ! rtph264depay \
>     !
> video/x-h264,width=480,height=360,framerate=30/1,profile=baseline,stream-format=avc,alignment=au \

This caps filter shouldn't be needed and may be causing the h264
bitstream to be converted from byte-stream to avc and back to byte-stream.

>     ! h264parse \
>     ! decodebin \
>     ! videoconvert \
>     ! autovideosink
> And this one is the pipeline on the server side (in this case I'm using a
> Raspberry):
>
> gst-launch-1.0 -e -v fdsrc \
>   !
> video/x-h264,width=480,height=360,framerate=30/1,profile=baseline,stream-format=avc,alignment=au
> \

These h264/avc caps are missing the codec_data field.

>   ! h264parse \
>   ! rtph264pay \
>       config-interval=1 \
>       pt=96 \
>   ! udpsink \
>       sync=false \
>       host=192.168.1.5 \
>       port=5004 \
>       bind-address=192.168.1.2 \
>       bind-port=5004
>
> I think that this can be fixed if I could use GPU decoding instead of
> software decoding, but I haven't found any information about it on the docs.
>
> So my question is: How can I decode a H264 video (or whatever other format,
> I don't mind to use another one if I get better results) using hardware
> acceleration on Android?

Using decodebin will automatically use the available hardware decoders
if they are available.

Perhaps you mean though, zerocopy through to display which requires
extra code in the decodebin ! glimagesink case that playbin contains.
Specifically, the autoplug-query signal needs to implemented for the
caps and context queries between decodebin and glimagesink.

Cheers
-Matt

> Thanks!
>
> Kind regards.


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[qqk]

As I know, GPU can not decode H264.
The hardware acceleration that can do that is the video codec hardware unit.
you can check that if your CPU support the hardware acceleration.

[ivan-perez]

Matthew Waters wrote

On 20/01/17 03:12, ivan-perez wrote:
> Hello
>
> I'm receiving on my Android mobile phone a video via RTP/UDP, but I've seen
> that CPU usage is very high, nearly 100% of one core is being used to decode
> the video. On high-end devices the video is played smoothly, but on middle
> and low-end devices I barely get 1 frame per second.
>
> This is my pipeline:
>
> udpsrc port=5004
> caps=application/x-rtp,payload=96,clock-rate=90000,encoding-name=H264 \
>     ! rtpjitterbuffer drop-on-latency=true max-dropout-time=500 \
>     ! rtph264depay \
>     !
> video/x-h264,width=480,height=360,framerate=30/1,profile=baseline,stream-format=avc,alignment=au \

This caps filter shouldn't be needed and may be causing the h264
bitstream to be converted from byte-stream to avc and back to byte-stream.

>     ! h264parse \
>     ! decodebin \
>     ! videoconvert \
>     ! autovideosink
> And this one is the pipeline on the server side (in this case I'm using a
> Raspberry):
>
> gst-launch-1.0 -e -v fdsrc \
>   !
> video/x-h264,width=480,height=360,framerate=30/1,profile=baseline,stream-format=avc,alignment=au
> \

These h264/avc caps are missing the codec_data field.

>   ! h264parse \
>   ! rtph264pay \
>       config-interval=1 \
>       pt=96 \
>   ! udpsink \
>       sync=false \
>       host=192.168.1.5 \
>       port=5004 \
>       bind-address=192.168.1.2 \
>       bind-port=5004
>
> I think that this can be fixed if I could use GPU decoding instead of
> software decoding, but I haven't found any information about it on the docs.
>
> So my question is: How can I decode a H264 video (or whatever other format,
> I don't mind to use another one if I get better results) using hardware
> acceleration on Android?

Using decodebin will automatically use the available hardware decoders
if they are available.

Perhaps you mean though, zerocopy through to display which requires
extra code in the decodebin ! glimagesink case that playbin contains.
Specifically, the autoplug-query signal needs to implemented for the
caps and context queries between decodebin and glimagesink.

Cheers
-Matt

> Thanks!
>
> Kind regards.


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel


signature.asc (527 bytes) <http://gstreamer-devel.966125.n4.nabble.com/attachment/4681573/0/signature.asc>

Hello Matthew,

Thank you very much for your answer. I've removed the caps filter on the client (Android) side and I get far more better performance than before. Now the video on the low-end device starts to play smoother. I'm getting about 7-10 fps and CPU is not at 100% anymore (now it sticks around 50-60%). I've also removed the "h264parse" and "videoconvert" elements, without any drawbacks other than a litter better performance.

So this is my pipeline at this moment:

udpsrc port=5004 caps=application/x-rtp,payload=96,clock-rate=90000,encoding-name=H264
    ! rtpjitterbuffer drop-on-latency=true max-dropout-time=500
    ! rtph264depay
    ! decodebin
    ! autovideosink

I'm still not sure if the video decoding is using hardware acceleration. This phone plays H264 videos at 1080p smoothly. I'm still getting some messages about poor performance on my phone (before the caps removal I got a lot of them, now I get only a few and only at the beginning):

> gstbasesink.c:2834:gst_base_sink_is_too_late:<sink> warning: A lot of buffers are being dropped.
> gstbasesink.c:2834:gst_base_sink_is_too_late:<sink> warning: There may be a timestamping problem, or this computer is too slow.


Now, for the server side, how can I know what codec_data field should I use? When I run the pipeline I see "codec_data=(buffer)01640028ffe1000e27640028ac2b40c84ff700f1226a01000528ee025cb0" on the standard output, but I have no idea of what I have to use on that field.


Finally, I've replaced "autovideosink" into "glimagesink" and I see no performance gain, so at the moment I am going to continue using the first one. What do you mean about "playbin"? I can't manage to use it properly on my pipeline. The docs says "playbin" has no pads, so I can't figure out how I can insert it on the pipeline.

Thanks again.

[ivan-perez]

qqk wrote

As I know, GPU can not decode H264.
The hardware acceleration that can do that is the video codec hardware unit.
you can check that if your CPU support the hardware acceleration.

Hello,

I've confirmed using a debugging tool that the GPU doesn't decode video (at least on my phone). Is there any way to check if GStreamer is using any type of hardware acceleration? I can perfectly play videos in 1080p stored on my device using the default video player which comes in the phone (and I've checked that the GPU is not doing that work), but I don't know why my video, at only 360p, can't be played smoothly with GStreamer.

Thanks!