Hardware decoding on android using GStreamer

[yusuf.husainy]

Hello everyone,

The company I work for is developing an application on Android using GStreamer. The application is to receive a mpegts stream over udp via wifi and then to live playback it on the tablet as well as to record simultaneously.

The video stream received will be from an endoscopy machine streamin fullHD 1920x1080p at 60fps. The requirement is to playback this live on the tablet. The tablet to be used is a Samsung Galaxy Tab2. According to the specs of this tablet, it says, it is capable of playing back fullHD.

Currently, to have a prototype of the application, I have modified Android-Tutorial3 for receiving the stream, using the standard plugins provided by GStreamerSDK. I also have used video overlays, to make the GStreamerSurfaceView, occupy the whole screen of the tablet.

Currently, GStreamerSDK does not support any of the gstreamer provided hardware accelerated plugins, so I am unable to playback fullHD at 60fps.

Hence, for proceeding ahead, I used lower resolution video streaming of 1280x720(the tablet's native screen resolution), and also used 800x600. Both of these, I tried at 30fps, and also 10fps.

However, I assume, as they or not fullHD at 60fps, they would stream without any need for hardware acceleration. But, this does load the CPU and the video is crappy and funky.

I would appreciate any suggestions or advice for achieving this kind of functionality, knowing the limitations of GStreamerSDK.

Thanks and Regards,
Yusuf Husainy

[Andoni Morales]

2013/6/19 yusuf.husainy <[hidden email]>

Hello everyone,

The company I work for is developing an application on Android using
GStreamer. The application is to receive a mpegts stream over udp via wifi
and then to live playback it on the tablet as well as to record
simultaneously.

The video stream received will be from an endoscopy machine streamin fullHD
1920x1080p at 60fps. The requirement is to playback this live on the tablet.
The tablet to be used is a Samsung Galaxy Tab2. According to the specs of
this tablet, it says, it is capable of playing back fullHD.

Currently, to have a prototype of the application, I have modified
Android-Tutorial3 for receiving the stream, using the standard plugins
provided by GStreamerSDK. I also have used video overlays, to make the
GStreamerSurfaceView, occupy the whole screen of the tablet.

Currently, GStreamerSDK does not support any of the gstreamer provided
hardware accelerated plugins, so I am unable to playback fullHD at 60fps.

Hence, for proceeding ahead, I used lower resolution video streaming of
1280x720(the tablet's native screen resolution), and also used 800x600. Both
of these, I tried at 30fps, and also 10fps.

However, I assume, as they or not fullHD at 60fps, they would stream without
any need for hardware acceleration. But, this does load the CPU and the
video is crappy and funky.

I would appreciate any suggestions or advice for achieving this kind of
functionality, knowing the limitations of GStreamerSDK.

Hi Yusuf,

In theory the androidmedia plugin, which uses the MediaCodec API, should be capable of handling the same decoding profiles and levels supported by your device. The first step is to check that your are actually using the androidmedia plugin, which is only available for API level >= 16. The second step is to get a log from the application and check the capabilities exposed by the decoder, to see which levels and profiles are reported to be supported.

Cheers,

Andoni  

Thanks and Regards,
Yusuf Husainy



--
View this message in context: http://gstreamer-devel.966125.n4.nabble.com/Hardware-decoding-on-android-using-GStreamer-tp4660630.html
Sent from the GStreamer-devel mailing list archive at Nabble.com.
_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

--
Andoni Morales Alastruey

LongoMatch:The Digital Coach
http://www.longomatch.ylatuya.es

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[yusuf.husainy]

Thanks Andoni,

Is androidmedia plugin a decoder in itself, or is it set of different plugins?

Also, I could not find some examples on androidmedia used in GStreamerSDK. If you know some example code that I can refer to understand, that would be wonderful.

What I think is, my pipeline should be something like this--
udpsrc caps="" ! tsdemux ! queue ! androidmedia(ffdec_h264?) ! eglglessink

Also, this may seem silly, but I do know how to do logging on Desktop Linux machines for GStreamer, but I am not that comfortable as yet with android environment, so how would I go about logging stuff in android similar to what I would see if I would do "$gst-launch -v ...".

Then this would help me getting to know the caps of the decoder and would help me analyze my problem.

I appreciate the help very much.

Regards,
Yusuf Husainy.

[Sebastian Dröge]

On Mi, 2013-06-19 at 14:03 -0700, yusuf.husainy wrote:

> Thanks Andoni,
>
> Is androidmedia plugin a decoder in itself, or is it set of different
> plugins?
>
> Also, I could not find some examples on androidmedia used in GStreamerSDK.
> If you know some example code that I can refer to understand, that would be
> wonderful.
>
> What I think is, my pipeline should be something like this--
> udpsrc caps="" ! tsdemux ! queue ! androidmedia(ffdec_h264?) ! eglglessink

It's a plugin similar to gst-omx or gst-libav, which dynamically
registers a lot of elements as available on the system. It's not easy to
guess the actual names of the elements because they depend on the name
of the codecs on your specific device.

If you use decodebin they will be used automatically if possible. Their
names start with amcviddec- and amcauddec-

> Also, this may seem silly, but I do know how to do logging on Desktop Linux
> machines for GStreamer, but I am not that comfortable as yet with android
> environment, so how would I go about logging stuff in android similar to
> what I would see if I would do "$gst-launch -v ...".
>
> Then this would help me getting to know the caps of the decoder and would
> help me analyze my problem.

Check the gst-launch code, what it does there is connecting to the
"deep-notify" signal, which causes all these things to be printed.

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Andoni Morales]

2013/6/19 yusuf.husainy <[hidden email]>

Thanks Andoni,

Is androidmedia plugin a decoder in itself, or is it set of different
plugins?

Also, I could not find some examples on androidmedia used in GStreamerSDK.
If you know some example code that I can refer to understand, that would be
wonderful.

What I think is, my pipeline should be something like this--
udpsrc caps="" ! tsdemux ! queue ! androidmedia(ffdec_h264?) ! eglglessink

Also, this may seem silly, but I do know how to do logging on Desktop Linux
machines for GStreamer, but I am not that comfortable as yet with android
environment, so how would I go about logging stuff in android similar to
what I would see if I would do "$gst-launch -v ...".

GStreamer debug logs are redirected to the Android console,  you can use gst_debug_set_default_threshold to configure the debug levels being printed.

Note there are 2 variants of the SDK, the release one, with debug logs disabled, and the debug one with gst debug enabled.

Cheers,

Andoni

Then this would help me getting to know the caps of the decoder and would
help me analyze my problem.

I appreciate the help very much.

Regards,
Yusuf Husainy.



--
View this message in context: http://gstreamer-devel.966125.n4.nabble.com/Hardware-decoding-on-android-using-GStreamer-tp4660630p4660634.html

Sent from the GStreamer-devel mailing list archive at Nabble.com.
_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

--
Andoni Morales Alastruey

LongoMatch:The Digital Coach
http://www.longomatch.ylatuya.es

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[yusuf.husainy]

Hi,

Thanks Andoni and Sebastian.
I did some work and found out the decoder type used in my device. In the /etc/media-codec.xml file, there is a decoder omx-googleh264encoder used for decoding h264 frames.

So, I think I should use the amcviddec-omxgoogleh264decoder element as my decoder. Now, the pipeline I am using on my receiver is:

data->pipeline = gst_parse_launch("udpsrc caps=\"video/mpegts, systemstream=true, packet-size=188\" ! tsdemux ! queue ! amcviddec-omxgoogleh264decoder ! eglglessink", &error);

The log that I get is : http://pastebin.com/p0EJ8r15

Also. I have not transmitted any audio in my stream, only video is transmitted. I would appreciate very much if someone could help me with this. As soon as I press the play button, I get the error: Internal data flow error from udpsrc. This is shown in the above log file too.

Regards,
Yusuf Husainy.

[Nicolas Dufresne-3]

Le samedi 22 juin 2013 à 06:10 -0700, yusuf.husainy a écrit :

data->pipeline = gst_parse_launch("udpsrc caps=\"video/mpegts,
systemstream=true, packet-size=188\" ! tsdemux ! queue !
amcviddec-omxgoogleh264decoder ! eglglessink", &error);

The log that I get is : http://pastebin.com/p0EJ8r15

The error is not-linked, are you sure you don't need h264parse before your decoder ?

Nicolas
_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[yusuf.husainy]

Hi Nicolas,

Thanks for the reply.

I am using an h264parse element at the transmitter, which is capturing H264frames from C920. So, the H264 bitstream is already split up into logical units, this would mean that there wouldn't be a requirement of h264parse at the receiver also, right?, I mean I don't know, should I be using it?

Also, in the /etc/media-codec.xml file, it states that the type of the media stream is video/avc, however, the video stream I have shows, it is video/x-h264, should this be an issue I should be looking at?

Also, the receiver pipeline works fine if used on a Linux Desktop PC, without a h264parse element.

Regards,
Yusuf Husainy.

[Nicolas Dufresne-3]

Le samedi 22 juin 2013 à 13:31 -0700, yusuf.husainy a écrit :

I am using an h264parse element at the transmitter, which is capturing
H264frames from C920. So, the H264 bitstream is already split up into
logical units, this would mean that there wouldn't be a requirement of
h264parse at the receiver also, right?, I mean I don't know, should I be
using it?

Also, in the /etc/media-codec.xml file, it states that the type of the media
stream is video/avc, however, the video stream I have shows, it is
video/x-h264, should this be an issue I should be looking at?

H264 parse will convert the format as needed, or complete the caps (which seems to be your case). Since 1.0, it is required and was already recommended to always have it in 0.10.

Nicolas
_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[krakorar]

Hi Yusuf,

The C920 is H.264 SVC (Scalable Video Coding) and your codec indicates that it supports H.264 AVC (Advanced Video Coding).  Your decoder may not be able to decode the H.264 SVC video produced by the C920.

Best Regards,

Rob

On Sat, Jun 22, 2013 at 5:43 PM, Nicolas Dufresne <[hidden email]> wrote:

Le samedi 22 juin 2013 à 13:31 -0700, yusuf.husainy a écrit :

I am using an h264parse element at the transmitter, which is capturing
H264frames from C920. So, the H264 bitstream is already split up into
logical units, this would mean that there wouldn't be a requirement of
h264parse at the receiver also, right?, I mean I don't know, should I be
using it?

Also, in the /etc/media-codec.xml file, it states that the type of the media
stream is video/avc, however, the video stream I have shows, it is
video/x-h264, should this be an issue I should be looking at?

H264 parse will convert the format as needed, or complete the caps (which seems to be your case). Since 1.0, it is required and was already recommended to always have it in 0.10.

Nicolas

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

--
Rob Krakora
MessageNet Systems
101 East Carmel Dr. Suite 105
Carmel, IN 46032
[phone] Ext 212
[phone] Fax

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[yusuf.husainy]

Hi,

Thanks for the replies everyone.

I worked further and was able to remove the not linking error. It seems, as Nicolas said, h264parse was essential.

I am able to compile the application and it launches successfully on my tablet.


When I press the play button:
Now, I get this error:
06-26 00:04:56.927: D/GStreamer+tutorial-3(7201): 0:00:05.920807000 0x5a65c320 jni/tutorial-3.c:88:set_ui_message Setting message to: Error received from element amcvideodec-omxgoogleh264decoder0: GStreamer encountered a general supporting library error.

A more detailed log of the application as shown on the logcat of the Eclipse IDE:http://pastebin.com/EX8sgcEp

So it seems that the amcviddec-omxgoogleh264decoder element cannot dequeue the input data as well as GStreamer encounters a library error.

Also, I ran the H264 from C920 to another linux desktop, and used H264/AVC ffdec_h264 element, and it worked great. SO what I think is, it is not a problem if I use the C920 H264/SVC or AVC, or I did not understand it correctly.

I really appreciate everyone's help.

Regards,
Yusuf Husainy.

[krakorar]

Hi Yusuf,

The H264 decoder wrapped in the ffdec_h264 plugin supports AVC and SVC.  I have used it as well as a Fluendo software and hardware accelerated plugins and they do as well.  Your decoded stated that it support AVC explicitly and did not make mention of SVC, but that does not mean that it may not also support SVC as well.  If video does not decode you may want to take a look at the codec.  :)

Best Regards,

Rob

On Tue, Jun 25, 2013 at 2:47 PM, yusuf.husainy <[hidden email]> wrote:

Hi,

Thanks for the replies everyone.

I worked further and was able to remove the not linking error. It seems, as
Nicolas said, h264parse was essential.

I am able to compile the application and it launches successfully on my
tablet.


When I press the play button:
Now, I get this error:
06-26 00:04:56.927: D/GStreamer+tutorial-3(7201): 0:00:05.920807000
0x5a65c320 jni/tutorial-3.c:88:set_ui_message Setting message to: Error
received from element amcvideodec-omxgoogleh264decoder0: GStreamer
encountered a general supporting library error.

A more detailed log of the application as shown on the logcat of the Eclipse
IDE:http://pastebin.com/EX8sgcEp

So it seems that the amcviddec-omxgoogleh264decoder element cannot dequeue
the input data as well as GStreamer encounters a library error.

Also, I ran the H264 from C920 to another linux desktop, and used H264/AVC
ffdec_h264 element, and it worked great. SO what I think is, it is not a
problem if I use the C920 H264/SVC or AVC, or I did not understand it
correctly.

I really appreciate everyone's help.

Regards,
Yusuf Husainy.





--
View this message in context: http://gstreamer-devel.966125.n4.nabble.com/Hardware-decoding-on-android-using-GStreamer-tp4660630p4660711.html
Sent from the GStreamer-devel mailing list archive at Nabble.com.
_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

--
Rob Krakora
MessageNet Systems
101 East Carmel Dr. Suite 105
Carmel, IN 46032
[phone] Ext 212
[phone] Fax

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[yusuf.husainy]

Hello all,

I analysed further and found that if I use the ffdec_h264 element instead of the amcviddec-omxgoogleh264parse element, then the following pipeline works:
data->pipeline = gst_parse_launch("filesrc location=/storage/sdcard0/Movies/c48000.mp4 ! qtdemux ! queue ! h264parse ! ffdec_h264 ! eglglessink sync=0", &error);

However, if iuse this, it fails:
data->pipeline = gst_parse_launch("filesrc location=/storage/sdcard0/Movies/c48000.mp4 ! qtdemux ! queue ! h264parse ! amcviddec-omxgoogleh264decoder ! eglglessink sync=0", &error);

Now, the log I get is:http://pastebin.com/9kpCiEwL
This is the log the logcat shows in eclipse.

It does not link the elements because it is not negotiated,  this is what I understand from the log file.

How can I know the exact caps needed by amcviddec-omxgoogleh264decoder element so that this would work?

Also, in this case, the .mp4 file is a 1920x1080 at 60fps and has video/x-h264 (avc) type.

the element amcviddec-omxgoogleh264decoder shows the type as video/avc in the /etc/media-codecs.xml file

It seems the h264parse would have the h264/avc type, but is this because amcviddec-omxgoogleh264decoder has video/avc and the file has video/x-h264?
I mean is this some nomenclature kind of issue, because the h264 in the file is avc format, however the caps show video/x-h264, so the sink caps of amcviddec-omxgoogleh264decoder shou;d be negotiated and the pipeline should run.

I don't completely understand what I should do to make amcviddec-omxgoogleh264decoder work in my gstreamer pipeline.

Also, I have Android 4.1.2 and am using API-16.

I would appreciate any body who has worked on  amcviddec-omxgoogleh264decoder before or who has any kind of idea.

Regards,
Yusuf Husainy.

[krakorar]

What is the output of "gst-inspect amcviddec-omxgoogleh264decoder"? What is the output of the pipeline below in the log.txt file?

GST_DEBUG=amcviddec-omxgoogleh264decoder:5 GST_DEBUG_NO_COLOR=TRUE gst-launch filesrc

location=/storage/sdcard0/Movies/c48000.mp4 ! qtdemux ! queue ! h264parse !
amcviddec-omxgoogleh264decoder ! eglglessink sync=FALSE &>log.txt

On Tue, Jul 2, 2013 at 12:51 PM, yusuf.husainy <[hidden email]> wrote:

Hello all,

I analysed further and found that if I use the ffdec_h264 element instead of
the amcviddec-omxgoogleh264parse element, then the following pipeline works:
data->pipeline = gst_parse_launch("filesrc
location=/storage/sdcard0/Movies/c48000.mp4 ! qtdemux ! queue ! h264parse !
ffdec_h264 ! eglglessink sync=0", &error);

However, if iuse this, it fails:
data->pipeline = gst_parse_launch("filesrc
location=/storage/sdcard0/Movies/c48000.mp4 ! qtdemux ! queue ! h264parse !
amcviddec-omxgoogleh264decoder ! eglglessink sync=0", &error);

Now, the log I get is:http://pastebin.com/9kpCiEwL
This is the log the logcat shows in eclipse.

It does not link the elements because it is not negotiated,  this is what I
understand from the log file.

How can I know the exact caps needed by amcviddec-omxgoogleh264decoder
element so that this would work?

Also, in this case, the .mp4 file is a 1920x1080 at 60fps and has
video/x-h264 (avc) type.

the element amcviddec-omxgoogleh264decoder shows the type as video/avc in
the /etc/media-codecs.xml file

It seems the h264parse would have the h264/avc type, but is this because
amcviddec-omxgoogleh264decoder has video/avc and the file has video/x-h264?
I mean is this some nomenclature kind of issue, because the h264 in the file
is avc format, however the caps show video/x-h264, so the sink caps of
amcviddec-omxgoogleh264decoder shou;d be negotiated and the pipeline should
run.

I don't completely understand what I should do to make
amcviddec-omxgoogleh264decoder work in my gstreamer pipeline.

Also, I have Android 4.1.2 and am using API-16.

I would appreciate any body who has worked on
amcviddec-omxgoogleh264decoder before or who has any kind of idea.

Regards,
Yusuf Husainy.





--
View this message in context: http://gstreamer-devel.966125.n4.nabble.com/Hardware-decoding-on-android-using-GStreamer-tp4660630p4660866.html

Sent from the GStreamer-devel mailing list archive at Nabble.com.
_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

--
Rob Krakora
MessageNet Systems
101 East Carmel Dr. Suite 105
Carmel, IN 46032
[phone] Ext 212
[phone] Fax

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[yusuf.husainy]

Hi,

I had thought of using gst-inspect or the command line launch of the pipeline you have given, but I am using the GStreamerSDK for development on Android, which does not come with a package like gstreamer-tools or something similar.

Also, I don't know how to use a gst-inspect like command in Android for gstreamer.

I appreciate everyone's response and help too.

Regards,
Yusuf Husainy

[krakorar]

According to GStreamer SDK documentation, gst-launch and gst-inspect are present.

On Jul 3, 2013 9:28 AM, "yusuf.husainy" <[hidden email]> wrote:

Hi,

I had thought of using gst-inspect or the command line launch of the
pipeline you have given, but I am using the GStreamerSDK for development on
Android, which does not come with a package like gstreamer-tools or
something similar.

Also, I don't know how to use a gst-inspect like command in Android for
gstreamer.

I appreciate everyone's response and help too.

Regards,
Yusuf Husainy



--
View this message in context: http://gstreamer-devel.966125.n4.nabble.com/Hardware-decoding-on-android-using-GStreamer-tp4660630p4660878.html
Sent from the GStreamer-devel mailing list archive at Nabble.com.
_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[krakorar]

Google "gst-inspect on Android" and you will see threads describing how to use it on an Android rooted device with the GStreamer SDK.

On Jul 3, 2013 10:23 AM, "Robert Krakora" <[hidden email]> wrote:

According to GStreamer SDK documentation, gst-launch and gst-inspect are present.

On Jul 3, 2013 9:28 AM, "yusuf.husainy" <[hidden email]> wrote:

Hi,

I had thought of using gst-inspect or the command line launch of the
pipeline you have given, but I am using the GStreamerSDK for development on
Android, which does not come with a package like gstreamer-tools or
something similar.

Also, I don't know how to use a gst-inspect like command in Android for
gstreamer.

I appreciate everyone's response and help too.

Regards,
Yusuf Husainy



--
View this message in context: http://gstreamer-devel.966125.n4.nabble.com/Hardware-decoding-on-android-using-GStreamer-tp4660630p4660878.html
Sent from the GStreamer-devel mailing list archive at Nabble.com.
_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[krakorar]

http://gstreamer-devel.966125.n4.nabble.com/How-to-use-gst-inspect-on-Android-td4660612.html

[krakorar]

http://gstreamer-devel.966125.n4.nabble.com/How-to-use-gst-inspect-on-Android-td4660612.html

[ivan-perez]

Hi Sebastian

I'm having trouble with searching for the right pipeline to decode a H264 video on my phone. I think the problem is very similar to the explained on this thread, so I decided to post right here instead of opening a new one.

This is the media_codecs.xml file: http://pastebin.com/9KxHDpE3

And this is what I get on the error logs when I use `amcviddec-omxgoogleh264decoder` or `amcviddec-omxmarvellvideohwcoda7542decoder` (on the mobile screen I get "GStreamer encountered a general supporting library error"): http://pastebin.com/raw/H5FDiqTQ

Can you please help to decide what is the right element for amcviddec-XXX given the XML from my Android phone?

Thank you in advance!