Help me in writing a player based on gstreamer for raspberry pi

[munezbn.dev]

Hi All,

I am planing to write a generic player for raspberry pi based on gstreamer. I am new to gstreamer and still learning stuffs. I am using custom buildroot image without X11.

My first idea is to create a player using playbin. Once that is done i will work on creating own pipeline by adding elements on the fly..

So far i am able to create player based on playbin but  the issue is displaying in fullscreen. I have searched a lot and looks like this is a known problem with glimagesink and so far didnt find any solution.

Solutions which I have tried.

1] Using Gtk and drawingarea and set this window handle using prepare_window_handle message.
    The first issue with this was about getting prepare_window_handle message from glimagesink element. This was known issue and I was able to find the patch. After applying patch I was able to get prepare_winodw_handle message. But when I set window ID of drawing area it doesn't work. Same code works with ximagesing on a fedora machine.. After searching a bit found that this is not possible as glimagesink uses dispmanx and it is not compatible yet ??

2] Thinking of using glcolorscale/glupload  but not sure if it will be hardware accelerated. And also, i am  not sure about aspect ratio.

My plan is to develop a gstreamer based player for raspberry with UI. Kindly guide me.

Thanks & Regards
Munez

[Sebastian Dröge-3]

On Thu, 2016-09-29 at 06:37 -0700, munezbn.dev wrote:
> Hi All,
>
> I am planing to write a generic player for raspberry pi based on gstreamer.
> I am new to gstreamer and still learning stuffs. I am using custom buildroot
> image without X11.
>
> My first idea is to create a player using playbin. Once that is done i will
> work on creating own pipeline by adding elements on the fly..

Why would you like to create a custom pipeline instead of using
playbin? If you're planning a player application, playbin is exactly
what you want to use in 99% of the cases.

Also take a look at GstPlayer, which might make your life even easier:
https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-bad-libs/html/gst-plugins-bad-libs-gstplayer.html#

> So far i am able to create player based on playbin but  the issue is
> displaying in fullscreen. I have searched a lot and looks like this is a
> known problem with glimagesink and so far didnt find any solution.
>
> Solutions which I have tried.
>
> 1] Using Gtk and drawingarea and set this window handle using
> prepare_window_handle message.
>     The first issue with this was about getting prepare_window_handle
> message from glimagesink element. I was able to find the patch and use it.
> After that i was able to get prepare_winodw_handle message. But when I set
> window ID of drawing area it doesn't work. Same code works with ximagesing
> on a fedora machine.. After searching a bit found that this is not possible
> as glimagesink uses dispmanx and it is not compatible yet ??
>
> 2] Thinking of using glcolorscale/glupload  but not sure if it will be
> hardware accelerated also not sure about aspect ratio.

GL is hardware accelerated on the RPi and is what you should use. Just
make sure that it uses the RPi GL implementation and not mesa.

Also there is no support for hardware accelerated GL in X11 on the RPi,
so you will have to use a UI toolkit that uses dispmanx or GL for
rendering.

> My plan is to develop a gstreamer based player for raspberry with UI. Kindly
> guide me. 

Which UI toolkit do you want to use? On that depends every other
answer.

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[munezbn.dev]

Sebastian Dröge-3 wrote

Why would you like to create a custom pipeline instead of using
playbin? If you're planning a player application, playbin is exactly
what you want to use in 99% of the cases.

The main purpose for me is learning and understanding gstreamer. Thats why I want come out of playbin.
And im not sure if we have control over pipeline and elements created by playbin ? For example if we have queue2 element for buffering, can we have custom values for buffering size ? How can we modify property values of elements added by playbin ?
Also, if i create my own plugin in future for xyz purpose, can I tell playbin to add my new element in between pipeline ?
 

Also take a look at GstPlayer, which might make your life even easier:
https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-bad-libs/html/gst-plugins-bad-libs-gstplayer.html#

Looks interesting. Is it available in buildroot ? Or is it part of Gstreamer ?

GL is hardware accelerated on the RPi and is what you should use. Just
make sure that it uses the RPi GL implementation and not mesa.

Also there is no support for hardware accelerated GL in X11 on the RPi,
so you will have to use a UI toolkit that uses dispmanx or GL for
rendering.

Thanks for the information. I will look into dispmanx

Which UI toolkit do you want to use? On that depends every other
answer.

I was thinkiing of GTK. but is there any other widget libraries which uses dispmanx ? Or you can suggest me any.. I am open to try :)

Thanks & Regards
Munez

[Sebastian Dröge-3]

On Fri, 2016-09-30 at 23:15 -0700, munezbn.dev wrote:
> Sebastian Dröge-3 wrote
> >
> > Why would you like to create a custom pipeline instead of using
> > playbin? If you're planning a player application, playbin is exactly
> > what you want to use in 99% of the cases.
>
> The main purpose for me is learning and understanding gstreamer. Thats why I
> want come out of playbin. 

If your goal is understanding, then that's fine. For any real use
playbin is the better choice.

> And im not sure if we have control over pipeline and elements created by
> playbin ? For example if we have queue2 element for buffering, can we have
> custom values for buffering size ?

Check the two buffer-* properties on playbin

>  How can we modify property values of elements added by playbin ?

It depends on which elements, but the idea is that you don't have to
modify any specific element properties for most cases and only
properties on playbin. playbin should set anything up properly for
playback.

> Also, if i create my own plugin in future for xyz purpose, can I tell
> playbin to add my new element in between pipeline ? 

What kind of element? There are various ways of doing this. For
sources, demuxers, parsers and decoders the standard machinery with
element rank, classification and pad template caps would make your
elements used.

For specific video/audio sinks, filters, etc. there are properties on playbin to adjust this.

> > Also take a look at GstPlayer, which might make your life even easier:
> > https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-bad-libs/html/gst-plugins-bad-libs-gstplayer.html#
>
> Thanks I have a look into that

It's a more high-level API around playbin that makes writing player
applications even easier. It does a lot of stuff for you already.

> > Which UI toolkit do you want to use? On that depends every other
> > answer.
>
> I was thinkiing of GTK. but is there any other widget libraries which uses
> dispmanx ? Or you can suggest me any.. I am open to try :)

AFAIK GTK has no dispmanx support on GTK, it just uses X11, which is
not optimal for your use case. Qt would be an option, or anything
rendering the UI in GL and can use EGL (like next GTK release is most
likely going to be able to do).

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[munezbn.dev]

Thanks for the suggestions Sebastian.

For now I have started with Playbin based player. I am using Raspberry Pi3. I am able to play BigBuckBunny from pendrive and also able to play webm content sintel  from gstreamer sample link.

Since I am using PI3, glimagesink has been used which uses dispmanx, so the video is playing at the center of the screen with actual size. How can we make it fullscreen without using GTK ? I have opened a thread for this. But so far no luck.

http://gstreamer-devel.966125.n4.nabble.com/GLimagesink-in-a-widget-tp4673122p4679852.html


Im not able to use glcolorscale also.

Thanks

[Sebastian Dröge-3]

On Mon, 2016-10-03 at 03:08 -0700, munezbn.dev wrote:

> Thanks for the suggestions Sebastian.
>
> For now I have started with Playbin based player. I am using Raspberry Pi3.
> I am able to play BigBuckBunny from pendrive and also able to play webm
> content sintel  from gstreamer sample link.
>
> Since I am using PI3, glimagesink has been used which uses dispmanx, so the
> video is playing at the center of the screen with actual size. How can we
> make it fullscreen without using GTK ? I have opened a thread for this. But
> so far no luck.
>
> http://gstreamer-devel.966125.n4.nabble.com/GLimagesink-in-a-widget-tp4673122p4679852.html

You'll have to use the dispmanx API directly for that, and explicitly
provide a dispmanx layer as "window handle" to glimagesink via the
GstVideoOverlay interface.

You can then set up the dispmanx layer in any way you want.

> Im not able to use glcolorscale also. 

glcolorscale is used inside glimagesink already. Why do you want to use
it explicitly?

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[munezbn.dev]

Sebastian Dröge-3 wrote

On Mon, 2016-10-03 at 03:08 -0700, munezbn.dev wrote:
> Thanks for the suggestions Sebastian.
>
> For now I have started with Playbin based player. I am using Raspberry Pi3.
> I am able to play BigBuckBunny from pendrive and also able to play webm
> content sintel  from gstreamer sample link.
>
> Since I am using PI3, glimagesink has been used which uses dispmanx, so the
> video is playing at the center of the screen with actual size. How can we
> make it fullscreen without using GTK ? I have opened a thread for this. But
> so far no luck.
>
> http://gstreamer-devel.966125.n4.nabble.com/GLimagesink-in-a-widget-tp4673122p4679852.html

You'll have to use the dispmanx API directly for that, and explicitly
provide a dispmanx layer as "window handle" to glimagesink via the
GstVideoOverlay interface.

You can then set up the dispmanx layer in any way you want.

I have started looking into dispmanx code. will try this.

glcolorscale is used inside glimagesink already. Why do you want to use
it explicitly?

I thought glcolorscale is also used for scaling the video so i was thinking of using it to change the width and height of video to match screen resolution to play it in fullscreen. But anyway i think right way to play in full screen is by using dispmanx API and to create GstVideo overlay.
 
I will try this today.

I had one more question related to gst_player.. Is there any api to change the a aspect ratio of the video ?

Thanks & Regards
Munez

[Sebastian Dröge-3]

On Wed, 2016-10-05 at 04:42 -0700, munezbn.dev wrote:
> 
>
> I thought glcolorscale is also used for scaling the video so i was
> thinking
> of using it to change the width and height of video to match screen
> resolution to play it in fullscreen. But anyway i think right way to
> play in
> full screen is by using dispmanx API and to create GstVideo overlay.
>  

glimagesink renders into a GL surface or in the case of the RPi into an
EGLNativeWindow. You externally define the size of that, and then
glimagesink tries to fill it (by default keeping aspect ratio).

> I had one more question related to gst_player.. Is there any api to
> change
> the a aspect ratio of the video ? 

See https://bugzilla.gnome.org/show_bug.cgi?id=765310

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[munezbn.dev]

Hi Sebastian,

I went through Dispmanx and was able to use examples and play with them.

I went through gstvideooverlay page and found the sample code for setting custom window using prepare_window_hanlde.  I had tried this earlier with GTK on Fedora Machine and i was able to set custom window.

I tried same thing but instead of creating GTk/X11 I created ES_DISPMANX_WINDOW_T  window (1920x1080) using dispmanx api and set it using gst_video_overlay_set_window_handle().  But still no luck.. It still plays with defualt window with 720 resolution at the center of screen.

Is there any example or sample code which i can refer ? Do i have to set ES_DISPMANX_WINDOW_T   as window handle or something else ?

Thanks & Regards
Munez

[Sebastian Dröge-3]

On Thu, 2016-10-06 at 03:02 -0700, munezbn.dev wrote:

> Hi Sebastian,
>
> I went through Dispmanx and was able to use examples and play with them.
>
> I went through gstvideooverlay page and found the sample code for setting
> custom window using prepare_window_hanlde.  I had tried this earlier with
> GTK on Fedora Machine and i was able to set custom window.
>
> I tried same thing but instead of creating GTk/X11 I created
> ES_DISPMANX_WINDOW_T  window (1920x1080) using dispmanx api and set it using
> gst_video_overlay_set_window_handle().  But still no luck.. It still plays
> with defualt window with 720 resolution at the center of screen.
>
> Is there any example or sample code which i can refer ? Do i have to set
> ES_DISPMANX_WINDOW_T   as window handle or something else ?

Can you share your code, ideally in a way that is compileable and
allows to reproduce your problem?

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[munezbn.dev]

Sebastian Dröge-3 wrote

Can you share your code, ideally in a way that is compileable and
allows to reproduce your problem?

Yes sure.  But I have figured out the issue and not sure if it is a bug gstglwindow_dispmanx_egl.c

The gst_gl_window_dispmanx_egl_set_window_handle() is empty and does nothing.

static void
gst_gl_window_dispmanx_egl_set_window_handle (GstGLWindow * window,
    guintptr handle)
{
}

I filled this function to assign the window handle (Not sure if it is the right way to do it) and i was able to move the window from my app :).  But still no luck with fullscreen :( ..  The are two issues here.

1] After set_window is called and custom user window is set (1920x1080), I see a call for setting preffered window size with video frame resolution (1280x720), which will call window_resize() function and resize and centers the user custom window.

2] I did try to resize my custom window again using dispmanx call vc_dispmanx_element_change_attributes () but it resulted in garbage video..  I am pretty sure this about src_rect and dest_rect related issue. The src_rect which I had created had 1920x1080 resolution but now due to window resize() it has changed to 1280x720 .. But application doesn't know about this because the resize has happened inside player like explained above 1] ..  So app somehow needs to know what is the current src_rect info so that in can scale to any dest_rect offcourse this is as per my understanding and i might be wrong..

I will share the code soon for all my changes.

Thanks & Regards
Munez

[munezbn.dev]

I have raised a Bug and have praposed a patch as well as you can find my test code over there.
https://bugzilla.gnome.org/show_bug.cgi?id=772608


I have a basic question. How can i find video resolution before creating a window ? I want to create a window with resolution matching with video resolution and set it in set_window_handle.

[munezbn.dev]

munezbn.dev wrote

I have raised a Bug and have praposed a patch as well as you can find my test code over there.
https://bugzilla.gnome.org/show_bug.cgi?id=772608


I have a basic question. How can i find video resolution before creating a window ? I want to create a window with resolution matching with video resolution and set it in set_window_handle.

Never mind. I found it.

[munezbn.dev]

Hi,

I wen ahead with gstPlayer and uploaded sample code at

https://github.com/munezbn/dispmanx-gst-play.git

Thanks & Regards
Munez