Problem when binding Android Surface to glimagesink

[Wei Lee]

Hi everyone,

I am new to GStreamer. I am developing an Android app which could play video using GStreamer. I have a SurfaceTexture to link with video sink on the GStreamer side. First, I get a Surface from SurfaceTexture and pass it to the native GStreamer codes just like this tutorial. The differences are that I use GStreamer 1.0 rather than 0.1 and I modified my pipeline to

"videotestsrc ! videoconvert ! glimagesink"

for simple testing. However, I cannot see the correct result. All I see is all blank.

But if I link a Surface which is from a SurfaceView just like the tutorial, I could see the result correctly.
Also, if I bind the Surface not with GStreamer codes but with Android MediaPlayer object by setSurface() method of it, the result is fine too.

After using an OpenGL ES Debugger on Android, I found out that when I link Surface with glimagesink , it will somehow trigger eglCreateContext() and put data in that Context rather than the original one. I guess that is the reason why I get blank view. All the data is drawn on a context which is unseen.

Could anybody give me some hints to fix the problem ?
Many thanks.

Best regards,
Wei

[Sebastian Dröge-3]

On Do, 2016-05-05 at 05:52 -0700, Wei Lee wrote:

> Hi everyone,
>
> I am new to GStreamer. I am developing an Android app which could play video
> using GStreamer. I have a SurfaceTexture to link with video sink on the
> GStreamer side. First, I get a Surface from SurfaceTexture and pass it to
> the native GStreamer codes just like  this tutorial
>  
> . The differences are that I use GStreamer 1.0 rather than 0.1 and I
> modified my pipeline to 
>
> "videotestsrc ! videoconvert ! glimagesink"
>
> for simple testing. However, I cannot see the correct result. All I see is
> all blank.
>
> But if I link a Surface which is from a SurfaceView just like the tutorial,
> I could see the result correctly.
> Also, if I bind the Surface not with GStreamer codes but with Android
> MediaPlayer object by setSurface() method of it, the result is fine too.
>
> After using an OpenGL ES Debugger on Android, I found out that when I link
> Surface with glimagesink , it will somehow trigger eglCreateContext() and
> put data in that Context rather than the original one. I guess that is the
> reason why I get blank view. All the data is drawn on a context which is
> unseen.
>
> Could anybody give me some hints to fix the problem ?

Can you share your code? It's expected that glimagesink is using its
own GL context for rendering into the surface and not any of the
application contexts.

You can find an example that works with 1.x here:
https://cgit.freedesktop.org/~slomo/gst-sdk-tutorials/tree/gst-sdk/tutorials/android-tutorial-3

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Wei Lee]

Hi Sebastian,

Thanks for your reply. The following is my GStreamer code.

http://paste.ofcode.org/3npqJKDwgavsxnGvL8ws5k


Basically, I follow the tutorial and does not modify a lot. And it could
work well if I use SurfaceView and get Surface just like the tutorial does.

However, since I want to play video using Google cardboard SDK, I use an
OpenGL ES engine on Android called Rajawali. It gives me a Surface which is
created by a SurfaceTexture. If I use the Surface like this:

SufaceTexture surfaceTexture; // Created by engine
Surface surface = new Surface(surfaceTexture);
nativeSurfaceInit(surface);

In this case, it will draw the content on a new Context which is not
rendered by the engine.
However, if I use Surface by Android MediaPlayer like this:

SufaceTexture surfaceTexture; // Created by engine
Surface surface = new Surface(surfaceTexture);
MediaPlayer mp = MediaPlayer.create(mContext, R.raw.test);
mp.setSurface(surface);
mp.start();

It will not create a new context and render on the current context so I can
see the result.
It seems glimagesink will create a new context to draw by default.
Is it possible to do some operations to make glimagesink draw content on
original context just like the MediaPlayer does?

Thanks a lot for your help.

Best regards,
Wei




--
View this message in context: http://gstreamer-devel.966125.n4.nabble.com/Problem-when-binding-Android-Surface-to-glimagesink-tp4677346p4677361.html
Sent from the GStreamer-devel mailing list archive at Nabble.com.
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On Fr, 2016-05-06 at 01:42 -0700, Wei Lee wrote:

> Hi Sebastian, 
>
> Thanks for your reply. The following is my GStreamer code. 
>
> http://paste.ofcode.org/3npqJKDwgavsxnGvL8ws5k
>
>
> Basically, I follow the tutorial and does not modify a lot. And it
> could work well if I use SurfaceView and get Surface just like the
> tutorial does. 
>
> However, since I want to play video using Google cardboard SDK, I use
> an OpenGL ES engine on Android called Rajawali. It gives me a Surface
> which is created by a SurfaceTexture. If I use the Surface like
> this: 
>
> SufaceTexture surfaceTexture; // Created by engine 
> Surface surface = new Surface(surfaceTexture); 
> nativeSurfaceInit(surface); 
>
> In this case, it will draw the content on a new Context which is not
> rendered by the engine. 
> However, if I use Surface by Android MediaPlayer like this: 
>
> SufaceTexture surfaceTexture; // Created by engine 
> Surface surface = new Surface(surfaceTexture); 
> MediaPlayer mp = MediaPlayer.create(mContext, R.raw.test); 
> mp.setSurface(surface); 
> mp.start(); 
>
> It will not create a new context and render on the current context so
> I can see the result. 
> It seems glimagesink will create a new context to draw by default. 
> Is it possible to do some operations to make glimagesink draw content
> on original context just like the MediaPlayer does? 

If you can get access to the context that should be used, and also can
use that in your own thread or otherwise have a way of using it, sure.

Take a look at how gtkglsink or qmlvideosink or caopengllayersink are
working. Something similar can be done in your case too then.

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Wei Lee]

Thanks a lot for your information.

My purpose is to render video on an Android Surface which is created from SurfaceTexture.

I guess I am not the first one trying to do this.

Is there any related examples/tutorials or elements already exists for this purpose?

I have found that there is an plugin called androidmedia which could do zero copy rendering on android.

https://bugzilla.gnome.org/show_bug.cgi?id=731204

But after some surveys, since I know so little about OpenGL stuffs, I still have no idea how to use it.

And not sure if it is designed to solve problem similar as mine, either.

I am really appreciate for any help.

Thanks.

​

[Sebastian Dröge-3]

On Fr, 2016-05-06 at 09:43 -0700, Wei Lee wrote:
> Thanks a lot for your information.
>
> My purpose is to render video on an Android Surface which is created
> from SurfaceTexture.
> I guess I am not the first one trying to do this.
> Is there any related examples/tutorials or elements already exists
> for this purpose?

Not really, other than the elements I mentioned in my last mail. To
implement this you need to get some understanding of OpenGL ES, EGL and
the Android APIs in any case.

For your specific case, what you have is a SurfaceTexture and you want
things to be rendered in there in the GL context provided by the
texture? So one question that you'll need to answer then is from which
thread you're actually allowed to make that GL context "current" and do
the rendering. If you have some example code (even in Java) that does
rendering to such a SurfaceTexture, that can help you to understand how
things can be done with GStreamer, in combination with looking at the
code of the elements I mentioned before.

> I have found that there is an plugin called androidmedia which could
> do zero copy rendering on android.
> https://bugzilla.gnome.org/show_bug.cgi?id=731204
> But after some surveys, since I know so little about OpenGL stuffs, I
> still have no idea how to use it.
> And not sure if it is designed to solve problem similar as mine,
> either.

The androidmedia plugin gives access to Android specific elements, in
detail video decoders (which can do zerocopy rendering in GLES) and a
camera source element. Both unrelated to what you want.

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[pedjaman]

So i have same problem like creator of this thread.
I'm rendering to SurfaceTexture from android MediaPlayer, VLC, ExoPLayer, but gStreamer is not rendering to it. Sound goes, but no video rendered.
Also onFrameAvailable(SurfaceTexture surface) meaning that new frame is ready is fired, meaning that gStreamer rendered something but it black.

I've used

https://github.com/sdroege/gst-player

as a base and just not getting surface from SurfaceView. I was using SurfaceTexture

                mSurface = new SurfaceTexture(mTextureID);
                mSurface.setOnFrameAvailableListener(this);
                Surface surface = new Surface(mSurface);
                mGstMediaPlayer.setSurface(surface);

Any tips?

[Matthew Waters]

On 13/09/16 07:25, pedjaman wrote:

> So i have same problem like creator of this thread.
> I'm rendering to SurfaceTexture from android MediaPlayer, VLC, ExoPLayer,
> but gStreamer is not rendering to it. Sound goes, but no video rendered.
> Also onFrameAvailable(SurfaceTexture surface) meaning that new frame is
> ready is fired, meaning that gStreamer rendered something but it black.
>
> I've used
>
> https://github.com/sdroege/gst-player
> <https://github.com/sdroege/gst-player>  
>
> as a base and just not getting surface from SurfaceView. I was using
> SurfaceTexture
>
>                 mSurface = new SurfaceTexture(mTextureID);
>                 mSurface.setOnFrameAvailableListener(this);
>                 Surface surface = new Surface(mSurface);
>                 mGstMediaPlayer.setSurface(surface);
>
> Any tips?

glimagesink doesn't support rendering into a SurfaceTexture and a new
video sink element would be needed for that functionality.  The GL part
would look very similar to the existing gtkglsink/qmlglsink elements
with only the platform specific stuff changing.

For zerocopy decoding, the decoder will output to it's own
SurfaceTexture for the external-oes texture that glimagesink will
happily render to a non-SurfaceTexture.

What exactly are you looking to do with your SurfaceTexture?

Cheers
-Matt


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On Di, 2016-09-13 at 09:20 +1000, Matthew Waters wrote:

> On 13/09/16 07:25, pedjaman wrote:
> >
> > So i have same problem like creator of this thread.
> > I'm rendering to SurfaceTexture from android MediaPlayer, VLC, ExoPLayer,
> > but gStreamer is not rendering to it. Sound goes, but no video rendered.
> > Also onFrameAvailable(SurfaceTexture surface) meaning that new frame is
> > ready is fired, meaning that gStreamer rendered something but it black.
> >
> > I've used
> >
> > https://github.com/sdroege/gst-player
> > > > <https://github.com/sdroege/gst-player>  
> >
> > as a base and just not getting surface from SurfaceView. I was using
> > SurfaceTexture
> >
> >                 mSurface = new SurfaceTexture(mTextureID);
> >                 mSurface.setOnFrameAvailableListener(this);
> >                 Surface surface = new Surface(mSurface);
> >                 mGstMediaPlayer.setSurface(surface);
> >
> > Any tips?
>
> glimagesink doesn't support rendering into a SurfaceTexture and a new
> video sink element would be needed for that functionality.  The GL part
> would look very similar to the existing gtkglsink/qmlglsink elements
> with only the platform specific stuff changing.

And for that please file a bug, ideally even working on that yourself
if it's something you need :)

There's already a similar bug for Android, but that way it wouldn't
solve your specific problem:
  https://bugzilla.gnome.org/show_bug.cgi?id=766044

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[pedjaman]

Problem is I'm not sure it is a bug. I believe I'm not using it well and was hopping of some direction where to look at :)

It is the same Surface class like the one used in Player demo in call Player.setSurface(). So GStreamer doesn't need separate code.
I can see that gstreamer internally decode video by using openGL as I can see frames in Tracer for openGL ES.
I also can see some additional context created which is I guess used by gstreamer. So it might be something with different contexts affecting this for example.

Maybe there is a sink which is rendering to opengl texture? That would also help.

[Sebastian Dröge-3]

On Di, 2016-09-13 at 04:34 -0700, pedjaman wrote:

> Problem is I'm not sure it is a bug. I believe I'm not using it well and was
> hopping of some direction where to look at :)
>
> It is the same Surface class like the one used in Player demo in call
> Player.setSurface(). So GStreamer doesn't need separate code.
> I can see that gstreamer internally decode video by using openGL as I can
> see frames in Tracer for openGL ES.
> I also can see some additional context created which is I guess used by
> gstreamer. So it might be something with different contexts affecting this
> for example.

The problem is that the way this works currently, you need a
ANativeWindow backing the Surface that is passed to GstPlayer (or as
the window-handle to glimagesink). Anything else won't work, and AFAIU
you pass something different there (a SurfaceTexture).

The solution to this would be to make an Android specific sink that
directly works with SurfaceTextures, which is not a "bug" but a feature
request... however we track feature requests in Bugzilla too, so
technically it is a "bug" ;)

> Maybe there is a sink which is rendering to opengl texture? That would also
> help.

glimagesink is internally, and you can build something that renders to
a texture with the signals on glimagesink. Nonetheless, a specific sink
that directly works on SurfaceTextures is going to work more reliable
and in the end will be easier.

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[pedjaman]

Thank you for your reply.

I'm not passing SurfaceTexture. I'm passing Surface which is constructed from SurfaceTexture. Therefore i get ANativeWindow.

But anyways, this works now :)

As I explained in other post, Surface passed this way is reported as 1px X 1px. So GStreamer rendered on single pixel. After forcing size of SurfaceTexture (and therefore size of Surface constructed from it) GStreamer renders quite well :)

I hope this is useful for someone.

Thank you