Gstreamer + Clutter + Wayland
Locked
 
|
Hello everybody,
I tried to run Gstreamer rtsp video playback on Raspberry Pi embedded into Clutter stage in Wayland, but the video seems very slow. We compiled Wayland/Weston this way: https://www.collabora.com/news-and-blog/blog/2016/06/03/running-weston-on-a-raspbian/ In order to get clutter library with Wayland support, we also recompiled libclutter-1.0, clutter-gst and clutter-gtk In order to take advantage of hardware video deconding, I was forced to recompile gst-plugins-bad as well as gst-omx suited for Raspberry Pi. The result is much slower performance compared to the same application running on top of X11. I would like to point out that the source of all these libraries were the same version as packages in Raspbian repository in order to keep compatibility with the other libraries' versions installed in the system. We also tried to avoid installing as many libraries which were about to install mesa related stuff as we already installed Mesa from sources. Does anyone have any idea how to find where the bottleneck of the application is? We found out that standalone Gstreamer element Waylandsink gives reasonable results when playing pipeline rtsp video from rtspsrc element sinking to Waylandsink omxh264dec, but utilitizing Clutter gives very poor performance. On X11, the same application with same versions of all these libraries runs reasonably fast. Maybe we should recompile all these libraries from up-to-date sources in order to get newest versions, but I guess finding the part which slows down the entire system would probably be the key for the solution, but does anyone know how to profile it? -- Sent from: http://gstreamer-devel.966125.n4.nabble.com/ _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
Re: Gstreamer + Clutter + Wayland
|
|
Le jeudi 28 mars 2019 à 10:57 -0500, horai a écrit :
> Hello everybody, > > I tried to run Gstreamer rtsp video playback on Raspberry Pi embedded into > Clutter stage in Wayland, but the video seems very slow. > We compiled Wayland/Weston this way: > https://www.collabora.com/news-and-blog/blog/2016/06/03/running-weston-on-a-raspbian/ > > In order to get clutter library with Wayland support, we also recompiled > libclutter-1.0, clutter-gst and clutter-gtk > In order to take advantage of hardware video deconding, I was forced to > recompile gst-plugins-bad as well as gst-omx suited for Raspberry Pi. > The result is much slower performance compared to the same application > running on top of X11. > I would like to point out that the source of all these libraries were the > same version as packages in Raspbian repository in order to keep > compatibility with the other libraries' versions installed in the system. > We also tried to avoid installing as many libraries which were about to > install mesa related stuff as we already installed Mesa from sources. > Does anyone have any idea how to find where the bottleneck of the > application is? We found out that standalone Gstreamer element Waylandsink > gives reasonable results when playing pipeline rtsp video from rtspsrc > element sinking to Waylandsink omxh264dec, but utilitizing Clutter gives > very poor performance. omxh264dec zero-copy solution is not supported by the Mesa VC4 driver, only the brcm GLES driver supports this. I know they are working on a V4L2 driver that speaks mmal in the kernel, with DMABuf plumbed through mmal in order to allow using standard DMABuf import path into GL. That being said, patches where submitted to cluttersink (which is not part of GStreamer project), but was never merged due to lack of proper support in CoGL/Clutter. Clutter is basically left without maintenance these days. The right solution is what is coming next, but I have no idea when this will happen, or if there is a way to pick dev version here and there to make it work. > On X11, the same application with same versions of all these libraries runs > reasonably fast. Maybe we should recompile all these libraries from > up-to-date sources in order to get newest versions, but I guess finding the > part which slows down the entire system would probably be the key for the > solution, but does anyone know how to profile it? Without a compositor, X11 rendering should be faster indeed. Profiling and trying to get closer in performance is mainly left to you. To speak for my part, I am waiting on DMABuf support to resume any renderer work on this target platform. > > > > > -- > Sent from: http://gstreamer-devel.966125.n4.nabble.com/ > _______________________________________________ > gstreamer-devel mailing list > [hidden email] > https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
|
|
Nicolas Dufresne-5 wrote
> Le jeudi 28 mars 2019 à 10:57 -0500, horai a écrit : > > > omxh264dec zero-copy solution is not supported by the Mesa VC4 driver, > only the brcm GLES driver supports this. I know they are working on a > V4L2 driver that speaks mmal in the kernel, with DMABuf plumbed through > mmal in order to allow using standard DMABuf import path into GL. That > being said, patches where submitted to cluttersink (which is not part > of GStreamer project), but was never merged due to lack of proper > support in CoGL/Clutter. Clutter is basically left without maintenance > these days. > > The right solution is what is coming next, but I have no idea when this > will happen, or if there is a way to pick dev version here and there to > make it work. Dear sir, Would be so kind and help me understand that matter clearly? I had the impression that omxh264dec zero-copy solution is related to decoding part only and I thought that this process is not dependent on the display backend. From your words I understand that the zero-copy slow down problem of MESA VC4 is present in X11 as well as in Wayland, moreover I think that Wayland does not work without VC4 at all.So, I understand that if my application would be using a sink utilizing VC4 (in my case clutter stage in GTK panel), I would definitely face the zero-copy problem. But this is not dependent on X11 or Wayland. Based upon my experience,when I started using VC4 openGL driver in X11, I experienced huge performance boost in my application (gstreamer + GTK + clutter), therefore I blamed clutter as the bottleneck in my application and I somehow presumed that clutter switched from utilizing OpenGL ES 2.0 to OpenGL driver internally and increased performance. I do not fully understand the structure of VC4, I understand VC4 as a driver of Broadcom videocore GPU driver exposing the hardware to software via OpenGL ES 1.0/2.0 API as well as OpenGL 2.1 API (on a hardware level it is OpenGL ES 2.0 hardware renderer and the OpenGL is just a software emulation layer for OpenGL 2.1 specification functions which are missing in OpenGL ES 2.0, this could be achieved only because of the fact that OpenGL ES 2.0 was derived from OpenGL 2.1 therefore they are very much close to each other) and due to the fact that Clutter supports OpenGL as well as OpenGL ES(either 1.0/2.0) Clutter can choose between OpenGL and OpenGL ES, because the libclutter library could be compiled with OpenGL (and I assume that Raspbian has binary library compiled with both flags enabled during configuration process) as well as OpenGL ES support. Therefore I presumed that when starting using VC4, Clutter layer in my application stopped using OpenGL ES and switched to OpenGL and that increased the performance or maybe my understanding of the matter could be completely wrong and VC4 could be simply much more optimized than original Broadcom OpenGL ES 2.0 binary driver and therefore Clutter could be internally using OpenGL ES 2.0 (no OpenGL) but VC4 is much faster then original Broadcom Open GL ES 2.0 binary driver but lacks zero-copy solution which has no relation to Clutter it relates to omx codec encoding/decoding video speaking in gtreamer words for omxh264dec element. In short despite the fact I am not using zero-copy solution because of VC4 usage I gained performance boost in video rendering (displaying) via Clutter layer when using VC4. My problem was, why do I then experince such a slow rendering when using Wayland with Clutter? Please, would you be so kind and bring some more light to my understanding of the matter as I presumed that omxh264dec has no relation to VC4 OpenGL ES 2.0, I assumed that is does only video decoding (for example h264) on GPU, but I understand that for decoding it also needs to use some part of VC4 driver because GPU driver is responsible for video processing as well as for video or image displaying but I assumed that encoding and displaying use totally different VC4 APIs. I do not know what is being used from VC4 for video encoding/decoding but I assumed that for displaying (in my case clutter stage embedded into GTK panel on top of Wayland or X11 utilizing VC4) OpenGL2.1 or OpenGLES 2.0 API is used specifically talking about RPI 3B+ and Broadcom BCM2837 GPU) -- Sent from: http://gstreamer-devel.966125.n4.nabble.com/ _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
|
|
In reply to this post by Nicolas Dufresne-5
Would you also be so kind and recommend us hardware platform which is most
widely used and nicely integrated with Gstreamer or possibly with Clutter? Thank you very much for all your effort -- Sent from: http://gstreamer-devel.966125.n4.nabble.com/ _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
Re: Gstreamer + Clutter + Wayland
|
|
In reply to this post by horai
Le sam. 6 avr. 2019 06 h 10, horai <[hidden email]> a écrit : Nicolas Dufresne-5 wrote I've just stumbled across this thread, and it seems that v4l2 mem2mem codec has landed already, and GStreamer 1.14 has been ship for this purpose. Basically means you should stop using OMX and try v4l2h264dec element instead. Cluttersink does not support dmabuf, so it will still require a bit of copies, but with that driver you save a dma copy from vcore. I hope it will compensate. You could also try your luck with adding glupload in front, but I'm not sure what is the state of the legacy gluploadmeta that this would use. When you moved to X11/Clutter what you did was to turn software color conversion into GPU accelerated conversion and that certainly give a boost. When you then move to using a compositor (over X11 or Wayland) you start rendering each window twice, once offscreen and finally composed to screen. This requires more GPU but I would also suspect more memory (see kernel parameter gpu_mem). Try increasing that amount of memory, something like 128M, and see if that helps.
_______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
|
|
Dear sir,
thank you very much. I compiled gstreamer-1.0, gstreamer plugins base as well as good. All are of version 1.14. But I am facing the issue described here: http://gstreamer-devel.966125.n4.nabble.com/How-to-build-v4l2videodec-td4669823.html I tried to clear registry. If you would be so kind and give me a hint what might the problem might be? Actually when browsing source code directory after 'make' procedure, I encountered these files: ./sys/v4l2/.libs/libgstvideo4linux2_la-gstv4l2h264enc.o ./sys/v4l2/gstv4l2h264enc.h ./sys/v4l2/.deps/libgstvideo4linux2_la-gstv4l2h264enc.Plo ./sys/v4l2/libgstvideo4linux2_la-gstv4l2h264enc.lo ./sys/v4l2/gstv4l2h264enc.c But I see no presense of v4l2h264dec. -- Sent from: http://gstreamer-devel.966125.n4.nabble.com/ _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
|
|
I also did git checkout 1.14.4
The decoder should start to work from 1.14.4 -- Sent from: http://gstreamer-devel.966125.n4.nabble.com/ _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
Re: Gstreamer + Clutter + Wayland
|
|
Le jeudi 11 avril 2019 à 23:16 -0500, horai a écrit :
> I also did git checkout 1.14.4 > The decoder should start to work from 1.14.4 Someone need to check if the upstream build is valid, they might have disabled V4L2 probe, which would disable CODECs support. To be carried over to the RPI community at this stage. > > > > -- > Sent from: http://gstreamer-devel.966125.n4.nabble.com/ > _______________________________________________ > gstreamer-devel mailing list > [hidden email] > https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
signature.asc (201 bytes) |
|
Hello sir,
I made it !!!! Thank you very much for great and excellent support. Just upgrade to newer kernel was the only thing needed. I am going to test the performance. -- Sent from: http://gstreamer-devel.966125.n4.nabble.com/ _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
Re: Gstreamer + Clutter + Wayland
|
|
Le ven. 12 avr. 2019 17 h 10, horai <[hidden email]> a écrit : Hello sir, Oh great, that shows how brand new this is :-)
_______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
|
|
Dear sir,
Thank you once more. I tested the element v4l2h264dec with this simple pipeline: gst-launch-1.0 -e rtspsrc location="rtsp://10.0.0.9:8555/test" latency=0 ! rtph264depay ! h264parse ! v4l2h264dec ! videoconvert ! waylandsink (or clutterautovideosink) I compared it to: gst-launch-1.0 -e rtspsrc location="rtsp://10.0.0.9:8555/test" latency=0 ! rtph264depay ! h264parse ! omxh264dec ! videoconvert ! waylandsink (or clutterautovideosink) It works, but the performance load was approximately the same. Anyway, what I do not fully understand is why, when compiling gst-omx, it is requesting header from libraspberrypi-dev which seem to be header files for libraspberrypi0. If I understand correctly libraspberrypi0 are proprietary GPU drivers from Broadcom supporting zero-copy. Well, I am I right stating that omx compiled agains this headers would have the zero-copy feature? This way I don't fully understand how it could switch from Broadcom binary to VC4 the one running the Weston is running on top: [10:52:28.444] GL version: OpenGL ES 2.0 Mesa 19.1.0-devel (git-2b7d5c3217) [10:52:28.445] GLSL version: OpenGL ES GLSL ES 1.0.16 [10:52:28.445] GL vendor: Broadcom [10:52:28.445] GL renderer: VC4 V3D 2.1 Anyway, I will continue trying to find solution for the slow performance. Even the before-mentioned pipeline is rather sluggish (but excellent delay). Clutter is a problem, maybe I experience what you have already mentioned that it is not maintained anymore ,as Wayland support is labeled as experimental after ./autogen.sh process. Anyway I compiled COGL with OpenGL (GL) as well as OpenGL ES 2.0 (GLES2) support the same I did with clutter and assumed that just swapping drivers: CLUTTER_DRIVER=gles2|gl could make some difference. But no way. Anyway, as always I blame myself as there are tons of switches (EGL, KMS etc. etc.) during compiling and requires a lot more testing. If only there would be a Clutter cookbook :) -- Sent from: http://gstreamer-devel.966125.n4.nabble.com/ _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
|
|
In reply to this post by horai
Anyway,
I would like to ask a peculiar question? Does it make any sense to try to run realtime video stream or RPI3B+ utilizing the combination of Wayland/Weston on top of VC4 as graphical stack and video rendering in GTK Window using Clutter stage utilizing COGL (GLES 2.0) decoding the stream either with v4l2h264dec? The only thing I wanted to gain was delay increase and performance boost compared to the same application I am running on top of X11 but unluckily I am facing something totally different despite all the promising videos and information on Web. Moreover I encountered few people stating that Wayland support was dropped from Raspberry (or Raspbian) foundation in favor of X11, maybe they know why they decided like this (of course if these rumors are right). I am not that skilled to see whether there are totally obvious obstacles why not to even dare to use Wayland for this purpose, actually Wayland on Raspberry. -- Sent from: http://gstreamer-devel.966125.n4.nabble.com/ _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
Re: Gstreamer + Clutter + Wayland
|
|
In reply to this post by horai
Le ven. 19 avr. 2019 15 h 25, horai <[hidden email]> a écrit : Dear sir, Nice to see it's drop in replacement. I was not clear enough, I'm expecting better performance with glimagesink over Mesa. Anyway, what I do not fully understand is why, when compiling gst-omx, it is Both OMX and Broadcom GL requires calling the infamous brcm_init() function. There is also some Broadcom GL code in gst-omx, but it won't be used on Mesa GL. This way I don't fully understand how it For waylandsink, was the videoconvert required ? The best performance would be achieved if Weston accept the colorformat and allow dmabuf FD being passed. For glimagesink, you should also try forcing GLES 2, since it's will be closer to what the HW can do. GST_GL_API=gles2
_______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
Re: Gstreamer + Clutter + Wayland
|
|
In reply to this post by horai
Le ven. 19 avr. 2019 15 h 40, horai <[hidden email]> a écrit : Anyway, I would like to say yes, hence the reason I trying to help. I don't really understand why it is so slow compared to X11. Normally Wayland bring tear free videos, performance is generally similar. The only thing I wanted to gain was delay increase and performance boost That one is confusing. Wayland protocol does not require GL. So at some point, a dispmanx (Broadcom 2d interface) renderer has been added to Weston. It worked quite well, but never became official graphic interface. It quickly became abandoned code and was later dropped by Weston maintainers. Now that VC4 mesa driver is there, the generic GL renderer in Weston can be used. The only down side, is that Weston does not yet support using the 50 overlays available on the RPi display driver. Support for that would give massive boost to waylandsink use cases. I am pretty sure this is something coming and RPi should will be a good platform to support this. maybe they know why they decided like this (of course if these rumors are _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
|
|
In reply to this post by Nicolas Dufresne-5
Nicolas Dufresne-5 wrote
> For glimagesink, you should also try forcing GLES 2, since it's will be > closer to what the HW can do. GST_GL_API=gles2 Thank you,you are right, no need for videoconvert, anyway, the only way I was able to run glimasink was; gst-launch-1.0 -e rtspsrc location="rtsp://10.0.0.9:8555/test" latency=0 ! rtph264depay ! h264parse ! v4l2h264dec capture-io-mode=2 ! glupload ! glimagesink (presence glupload does not make much difference) With: export GST_GL_API=gles2 I also tried export GST_GL_PLATFORM=egl But the performance is worse than waylandsink moreover the window is very small. Just to make myself sure Gstreamer base is using VC4 MESA driver for libgstopengl.so , I recompiled the gst_plugin_base with: nice -20 ./autogen.sh --enable-introspection=no CPPFLAGS="-I/include" LDFLAGS="-L/libs" Just to make sure the configuration process would search for anything relse elated to GL|GLES2 in the directories where my own MESA build is installed. (/include, /libs), I also exported all the necessary paths for *.pc files for GL|GLES2 : export PKG_CONFIG_PATH=/lib/pkgconfig/:/usr/share/pkgconfig/:/usr/lib/arm-linux-gnueabihf/:/usr/local/lib/pkgconfig/:/usr/lib/arm-linux-gnueabihf/pkgconfig/:/usr/lib/arm-linux-gnueabihf/lib/pkgconfig/ During configure procedure it shows that OpenGL|OpenGL ES 2.0 was detected. I regret one thing, when following this tutorial: https://www.collabora.com/news-and-blog/blog/2016/06/03/running-weston-on-a-raspbian/ I omitted this: sudo apt-get build-dep mesa, just to make sure no mesa related stuff from repository is installed. I did not realize that these drivers are probably older versions of MESA VC4 and will be anyway replaced in the next step during the installation procedure of up to date MESA and maybe now I am missing something very important. If I do: sudo apt-get build-dep mesa now, it makes my Weston installation unable to run and I have to uninstall all the packages. If I understand your words correctly, there is no more need for use of the old Broadcom binary driver in Gstreamer under Wayland anymore, right? Only some header files are necessary for omx-gst compilation (if I really need to use omxh264* instead of v4l2h264*). I assumed that it is even not possible to use that old binary Broadcom driver in Wayland/Weston as in Wayland build guide they claim: "Broadcom (Raspberry Pi): Requires open source driver VC4 (Linux and Mesa), can be used with the DRM-backend" Therefore I assumed Wayland/Weston both run on VC4 and anything else on top of trying to take advantage of hardware acceleration like glimagesink must be compiled with inclusion of VC4 headers and linkage of VC4 libraries. Please let me know if I am wrong. Just to note, for full recompilation of Gstreamer I lately followed this tutorial: https://lists.freedesktop.org/archives/gstreamer-openmax/2013-March/000724.html With minor changes like disabling introspection or replacing flags related to /opt/vc directory containing stuff related to that old Broadcom driver. -- Sent from: http://gstreamer-devel.966125.n4.nabble.com/ _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel |
«
Return to GStreamer-devel
|
1 view|%1 views
| Free forum by Nabble | Edit this page |