Video Hackfest conclusions
Locked
 
|
Hi,
As you may be aware, we held a video hackfest last week in Barcelona. Developers met to discuss how best to improve GPU support for video applications. See http://gstreamer.freedesktop.org/wiki/VideoHackfest for more details. In particular, you might be interested in the notes some people took while the hackfest was going on. These can be found at http://gstreamer.freedesktop.org/wiki/VideoHackfest/Notes What follows is the results we arrived at after the hackfest. They are taken from http://gstreamer.freedesktop.org/wiki/VideoHackfest/Conclusions but certainly deserve wide circulation. We're interested in any comments (or questions) you might have about them, so please don't heistate to reply. If you do reply, please make sure to reduce the recipient list appropriately. Cheers, Benjamin, on behalf of all hackfest paticipants. == YUV in pixman == It has been a goal for a while to make video a first-class citizen in the image stack. A concrete API proposal on how to integrate video into pixman was reached and reviewed. Links: * http://gstreamer.freedesktop.org/wiki/VideoHackfest/YCbCr_Formats * http://gstreamer.freedesktop.org/wiki/VideoHackfest/pixman Actions: * GStreamer developers to provide a specification of important YUV formats * Write implementation == YUV in Cairo == To make it easy for applications to use these different formats, the Cairo API needs to be extended to allow them. A proposal does exist and has been reviewed, details need to be finalized. Links: * http://gstreamer.freedesktop.org/wiki/VideoHackfest/Cairo Actions: * Finalize proposal * Write implementation == locking in Cairo == GStreamer takes a lot of freedom in deciding which threads to schedule elements in. Most hardware backends however require proper serialization of commands. The current gst-plugins-cairo and gst-plugins-gl code provide different, but ugly ways to achieve this. Cairo's internal thread safety guarantees match these requirements very well. But the different Cairo backends don't always keep these guarantees. Interaction with these locking mechanisms from outside applications is not possible yet either. Links: * http://gstreamer.freedesktop.org/wiki/VideoHackfest/Cairo Actions: * Improve Cairo backend implementations, in particular X11 and GL * Expose Cairo's locking API to allow interaction with it * Write testcases to squash bugs == locking in Mesa == The GLX specification does not allow binding the same GLX context in multiple threads at the same time. This is a requirement for both Cairo's and GStreamer's threading model. An extension was proposed and initial code developed to support this requirement in the same way as Apple's GL does by default. Windows does not support this and would require potentially expensive fallback code. Links: * http://people.freedesktop.org/~anholt/MESA_multithread_makecurrent.spec Actions: * Get review for suggested extension and include it in future Mesa releases * Make Windows developers investigate the situation == switch GStreamer to Cairo as default video transport model == The current approach to handling video in GStreamer is very outdated. It does not allow hw-accelerated buffers nor does it provide a unified API to modify video buffers, which leads to fragmentation and duplication. It was agreed that using Cairo to provide an abstraction as a drawing API and abstraction over different backends was a good idea. Links: * http://gstreamer.freedesktop.org/wiki/VideoHackfest/Notes * http://cgit.freedesktop.org/~company/gst-plugins-cairo Actions: * Rework gst-plugins-cairo to match improvements listed in previous points * Get more review on API and to avoid regressions * Merge into gst-plugins-base * Switch plugins gradually to use new Cairo code == gst-plugins-gl == The general consensus was that gst-plugins-gl is a hack. It was necessary in the past to get things to work, but is not a good way forward. However, the functionality or performance provided by the current elements needs to continue working. Developers shared the opinion that gst-plugins-cairo with cairo-gl surfaces is the best way to achieve this. Links: * http://gstreamer.freedesktop.org/wiki/VideoHackfest/Notes Actions: * Get review of cairo-gl/gst-plugins-cairo code from gst-plugins-gl developers * Port gst-plugins-gl elements to use Cairo early to ensure required features are available == XRenderPutImage == Currently there is no way to upload video data to the X server for later use. The XV extension is not sufficient for anything more complicated than a simple video player. Links: * http://gstreamer.freedesktop.org/wiki/VideoHackfest/Cairo Actions: * Figure out the best way to move YUV data into hw-accelerated Cairo surfaces (GL vs X) * Possibly draft an extension to XRender to handle this in the no-GL case == JIT in Pixman == ORC was investigated as a potential JIT for pixman. A lot of talk has happened, but no clear conclusions were reached. The idea to share a JIT with Mesa's GLSL stack was also brought up. Links: * http://gstreamer.freedesktop.org/wiki/VideoHackfest/Notes Actions: * Continue investigating JIT options * Prototype ORC integration into Pixman == Video decoding Acceleration == Hardware video decoding is a feature that everybody is very excited about and wants to integrate into GStreamer as soon as possible. But developers feel that the current APIs available - the focus was on VAAPI and VDPAU - to do hardware acceleration fail to integrate well. No developer managed to create even demo-quality integration with gst-plugins-cairo, even though quite a few tried. In particular, both VDPAU and VAAPI... ... do not integrate with existing frameworks Both libraries fail to use (or allow easy conversion to) existing objects (GEM handles, GL textures, ...) but only provide access to an encapsulation. This makes it hard for a flexible framework like GStreamer to make use of its features without limiting itself. ... do not have a clear scope In particular VDPAU advertises itself as a video playback framework using hardware acceleration. Unfortunately, they both limit themselves to only the formats and functionalities supported by the hardware and see fullscreen video playback as the single usecase. This needlessly complicates integration with existing frameworks ... provide a very complicated API For decoding video, an API like provided by libtheora, libmpeg2 or even ffmpeg is very simple and allows easy integration with both the demuxing and the postprocessing/display side. VAAPI and VDPAU require complex setups and preprocessing of the data to work properly. ... overlap with GL functionality It was felt that a lot of processing features provided by these libraries are equally well available using shaders in GL or by providing GL extensions. It was unclear why it is necessary to use a separate way to do this. Links: * http://gstreamer.freedesktop.org/wiki/VideoHackfest/Notes * http://gstreamer.freedesktop.org/wiki/VideoHackfest?action=AttachFile&do=view&target=VAinGST.pdf Actions: * Get in contact with developers of the APIs * Encourage revision of public * Work closer together in integrating the provided functionality into GStreamer = Timeline = The end goal of all of this is to get the code into users' hands quickly, but allow a smooth and non-disrupting upgrade. Desired dates have been put forth to achieve this: * End of January: release new versions of Cairo and Pixman that contain the new APIs reuired by GStreamer * End of March: X server and Mesa releases are due - make sure the required Mesa extension is part of this. Ideally XrenderPutImage would be included, too. * April/May, after next Fedora/Ubuntu releases: Merge Cairo support into gst-plugins-base and start porting elements to it. Encourage application developers (browser, Flash players) to make use of the new APIs * October: another Fedora/Ubuntu release that switches all users to the new APIs ------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/gstreamer-devel |
Re: Video Hackfest conclusions
|
|
On 11/24/2009 12:47 PM, Benjamin Otte wrote:
> = Timeline = > The end goal of all of this is to get the code into users' hands > quickly, but allow a smooth and non-disrupting upgrade. Desired dates > have been put forth to achieve this: > * End of January: release new versions of Cairo and Pixman that > contain the new APIs reuired by GStreamer > * End of March: X server and Mesa releases are due - make sure the > required Mesa extension is part of this. Ideally XrenderPutImage would > be included, too. > * April/May, after next Fedora/Ubuntu releases: Merge Cairo support > into gst-plugins-base and start porting elements to it. Encourage > application developers (browser, Flash players) to make use of the new > APIs > * October: another Fedora/Ubuntu release that switches all users to > the new APIs thanks for the conclusion. one thing which would be very useful (for me and may be others). Fedora/Ubuntu is a very fast moving target, but i understood it as a good target for gstreamer. although some people (like me) would like to use a more robust os like rhel/centos. but currently it's not possible to update gstreamer to newer version on rhel/centos since gstreamer depend on glib/gtk which is too old in rhel for newer gstreamer. as (probably) rhel-6 will be out in 2010 it means that version of glib/gtk/cairo which will be in rhel-6 will remain for many years in the rhel/centos-6. so if i dare to ask that gtk/glib/cairo version in rhel-6 should have to enough for these features. ie. rise the required gtk/glib/cairo version in the current gstreamer to be enough for later version. otherwise these features can't be used in rhel-6 for many years. thanks in advance. regards. -- Levente "Si vis pacem para bellum!" ------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/gstreamer-devel |
|
|
In reply to this post by Lawrence Auster-2
Message: 1
Date: Tue, 24 Nov 2009 12:47:38 +0100 From: Benjamin Otte <[hidden email]> Subject: [gst-devel] Video Hackfest conclusions To: [hidden email], Cairo list <[hidden email]> Cc: [hidden email], [hidden email], [hidden email], [hidden email] Message-ID: <[hidden email] > Ben: Thanks for your very insightful summary of the Video Hackfest and thanks to the GStreamer team for a very well architected and extendible piece of software. Recently ffmpeg and MPlayer introduced VDPAU support which works very well and allows one to employ lower cost CPUs in Linux based machines that are to be used primarily for multimedia purposes. Can anything be learned from their integration of VDPAU that might benefit the GStreamer team? My end goal is to employ GStreamer with VDPAU support for video decode with Fluendo codecs for audio decode. Best Regards, -- Rob Krakora Senior Software Engineer MessageNet Systems 101 East Carmel Dr. Suite 105 Carmel, IN 46032 (317)566-1677 Ext. 206 (317)663-0808 Fax ------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/gstreamer-devel |
Re: Video Hackfest conclusions
|
|
In reply to this post by Lawrence Auster-2
Hi,
> == gst-plugins-gl == > The general consensus was that gst-plugins-gl is a hack. It was > necessary in the past to get things to work, but is not a good way > forward. However, the functionality or performance provided by the > current elements needs to continue working. Developers shared the > opinion that gst-plugins-cairo with cairo-gl surfaces is the best way > to achieve this. > Links: > * http://gstreamer.freedesktop.org/wiki/VideoHackfest/Notes > Actions: > * Get review of cairo-gl/gst-plugins-cairo code from gst-plugins-gl developers > * Port gst-plugins-gl elements to use Cairo early to ensure required > features are available I am wondering what then would be the preferred way to display video within 3D scene? I am not familiar with Cairo (so maybe my assumption is wrong) but as I understand it is more 2D oriented. Will it be like: gstreamer -> cairo(image?)surface -> gltexture or somehow different? I understand internal implementation problems mentioned in the notes, but I have to say that it is rather convenient now with gst-plugins-gl to get the decoded frame as ready to use GL texture. Thank you, Andrey. ------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/gstreamer-devel |
|
Administrator
|
On Wed, 2009-11-25 at 08:47 -0800, Andrey Nechypurenko wrote:
> Hi, > > > > > == gst-plugins-gl == > > The general consensus was that gst-plugins-gl is a hack. It was > > necessary in the past to get things to work, but is not a good way > > forward. However, the functionality or performance provided by the > > current elements needs to continue working. Developers shared the > > opinion that gst-plugins-cairo with cairo-gl surfaces is the best way > > to achieve this. > > Links: > > * http://gstreamer.freedesktop.org/wiki/VideoHackfest/Notes > > Actions: > > * Get review of cairo-gl/gst-plugins-cairo code from gst-plugins-gl developers > > * Port gst-plugins-gl elements to use Cairo early to ensure required > > features are available > > I am wondering what then would be the preferred way to display video > within 3D scene? I am not familiar with Cairo (so maybe my assumption is > wrong) but as I understand it is more 2D oriented. Will it be like: > gstreamer -> cairo(image?)surface -> gltexture or somehow different? > > I understand internal implementation problems mentioned in the notes, but > I have to say that it is rather convenient now with gst-plugins-gl to get the > decoded frame as ready to use GL texture. The whole point is that you will be able to get a GL texture from the cairo buffer. That's the work that needs to be undertaken, after that all the gst-plugins-gl elements will *also* be able to talk video/x-cairo. Edward > > Thank you, > Andrey. > > > > > > ------------------------------------------------------------------------------ > Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day > trial. Simplify your report design, integration and deployment - and focus on > what you do best, core application coding. Discover what's new with > Crystal Reports now. http://p.sf.net/sfu/bobj-july > _______________________________________________ > gstreamer-devel mailing list > [hidden email] > https://lists.sourceforge.net/lists/listinfo/gstreamer-devel ------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/gstreamer-devel |
Re: Video Hackfest conclusions
|
|
In reply to this post by Farkas Levente
On 11/25/2009 05:56 AM, Farkas Levente wrote:
> thanks for the conclusion. one thing which would be very useful (for me > and may be others). > Fedora/Ubuntu is a very fast moving target, but i understood it as a > good target for gstreamer. although some people (like me) would like to > use a more robust os like rhel/centos. but currently it's not possible > to update gstreamer to newer version on rhel/centos since gstreamer > depend on glib/gtk which is too old in rhel for newer gstreamer. > as (probably) rhel-6 will be out in 2010 it means that version of > glib/gtk/cairo which will be in rhel-6 will remain for many years in the > rhel/centos-6. > so if i dare to ask that gtk/glib/cairo version in rhel-6 should have to > enough for these features. ie. rise the required gtk/glib/cairo version > in the current gstreamer to be enough for later version. otherwise these > features can't be used in rhel-6 for many years. > thanks in advance. As far as I know RHEL6 will be based on Fedora 12 which was released a few days ago. behdad > regards. ------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/gstreamer-devel |
Re: Video Hackfest conclusions
|
|
In reply to this post by Edward Hervey
Thank you Edward for the explanations.
Andrey. ----- Original Message ---- From: Edward Hervey <[hidden email]> To: Discussion of the development of GStreamer <[hidden email]> Sent: Wed, November 25, 2009 5:53:28 PM Subject: Re: [gst-devel] Video Hackfest conclusions On Wed, 2009-11-25 at 08:47 -0800, Andrey Nechypurenko wrote: > Hi, > > > > > == gst-plugins-gl == > > The general consensus was that gst-plugins-gl is a hack. It was > > necessary in the past to get things to work, but is not a good way > > forward. However, the functionality or performance provided by the > > current elements needs to continue working. Developers shared the > > opinion that gst-plugins-cairo with cairo-gl surfaces is the best way > > to achieve this. > > Links: > > * http://gstreamer.freedesktop.org/wiki/VideoHackfest/Notes > > Actions: > > * Get review of cairo-gl/gst-plugins-cairo code from gst-plugins-gl developers > > * Port gst-plugins-gl elements to use Cairo early to ensure required > > features are available > > I am wondering what then would be the preferred way to display video > within 3D scene? I am not familiar with Cairo (so maybe my assumption is > wrong) but as I understand it is more 2D oriented. Will it be like: > gstreamer -> cairo(image?)surface -> gltexture or somehow different? > > I understand internal implementation problems mentioned in the notes, but > I have to say that it is rather convenient now with gst-plugins-gl to get the > decoded frame as ready to use GL texture. The whole point is that you will be able to get a GL texture from the cairo buffer. That's the work that needs to be undertaken, after that all the gst-plugins-gl elements will *also* be able to talk video/x-cairo. Edward > > Thank you, > Andrey. > > > > > > ------------------------------------------------------------------------------ > Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day > trial. Simplify your report design, integration and deployment - and focus on > what you do best, core application coding. Discover what's new with > Crystal Reports now. http://p.sf.net/sfu/bobj-july > _______________________________________________ > gstreamer-devel mailing list > [hidden email] > https://lists.sourceforge.net/lists/listinfo/gstreamer-devel ------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/gstreamer-devel ------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/gstreamer-devel |
|
|
In reply to this post by Lawrence Auster-2
Hi,
I have a comment about this: (>From http://gstreamer.freedesktop.org/wiki/VideoHackfest/Conclusions:) "The GLX specification does not allow binding the same GLX context in multiple threads at the same time. This is a requirement for both Cairo's and GStreamer's threading model. An extension was proposed and initial code developed to support this requirement in the same way as Apple's GL does by default. Windows does not support this and would require potentially expensive fallback code." (from http://gstreamer.freedesktop.org/wiki/VideoHackfest/Notes) "Eric: One thing I'd like is if two neighboring elements could determine whether they are running in the same thread" I already asked gstreamer developers in the past to add support for this but this is not an easy task and only useful for opengl right now. The result would be to remove this : (from http://gstreamer.freedesktop.org/wiki/VideoHackfest/Notes) "So what gst-plugins-gl does is to proxy all OpenGL communication through a single thread associated with the context" For example in the pipeline: videotestsrc ! glupload ! glfilterA ! queue ! glfilterB ! glimagesink It would have one opengl context for glupload ! glfilterA since their chain func is running in the same thread (if the user does not manually push buffer itself form an other thread, I do not know all the possible case but we could ensure that they are running in the same thread). This opengl context would be shared with the opengl context used by glfilterB ! glimagesink Moreover, thoses 2 opengl contexst (shared together) could work at the same time (they would work on different textures at the same time) And so no need the overhead due to the gl thread is avoid, (and the property "each opengl context is made current only one time" is kept. Note that this property is already true in the current gst-plugins-gl implementation) I have already demonstrated the validity of using OpenGL context sharing in the implementation of the glmixer base class. The glmosaic element (which will be renamed to glmixercube) is a glmixer. (availaible in gst-plugins-gl/git) For example in the following pipeline: videotestsrc ! glupload ! glfilterA ! queue ! glmixer name=m ! glfilterC ! glimagesink glupload ! glfilterB ! queue ! .m There are 3 opengl contexts shared together. (there is a glthread per opengl context) One used by glupload ! glfilterA, one other for glupload ! glfilterB, and a last one used by glmixer ! glfilterC ! glimagesink Finally, if we would have the following support: "Eric: One thing I'd like is if two neighboring elements could determine whether they are running in the same thread" We could avoid the use of the a thread per opengl context. Moreover it would not need what you suggested: "The GLX specification does not allow binding the same GLX context in multiple threads at the same time" I mean, maybe binding the same GLX context in multiple threads at the same time is equivalent to have several opengl contexts shared together and each one used in only one thread (a gstreamer streaming thread). If this is really equivalent (at least for our use) maybe it would be more portable to use the context sharing solution, because the feature is already there. Actually I think using opengl context sharing is more efficient because several context can be used at the same time. (not possible with the same context in multiple-thread. Well it depends on the implementation that you are going to do) I could help in the future about that: (from http://gstreamer.freedesktop.org/wiki/VideoHackfest/Conclusions) "Actions: Sincerely
Julien Isorce
2009/11/24 Benjamin Otte <[hidden email]> Hi, ------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/gstreamer-devel |
|
Administrator
|
In reply to this post by krakorar
Hi,
On Wed, 2009-11-25 at 09:46 -0500, Robert Krakora wrote: > > Ben: > > Thanks for your very insightful summary of the Video Hackfest and > thanks to the GStreamer team for a very well architected and > extendible piece of software. Recently ffmpeg and MPlayer introduced > VDPAU support which works very well and allows one to employ lower > cost CPUs in Linux based machines that are to be used primarily for > multimedia purposes. Can anything be learned from their integration > of VDPAU that might benefit the GStreamer team? My end goal is to > employ GStreamer with VDPAU support for video decode with Fluendo > codecs for audio decode. If you want to use proprietary codecs/plugins, just ask the provider of those codecs (I'm pretty sure Fluendo has closed-source vdpau plugins). If you want an open-source solution, there are already plugins in gst-plugins-bad that wrap VDPAU. Right now only mpeg2 is present, but Anton (who has been doing that work) just got git commit access, so one should expect a lot of progress in that regards within the next month or so. Our main concern about VDPAU and VAAPI ... is that those two API rely on "you shall use our API to do everything", which is a very narrow-minded view of modern multimedia usage. We need to be able to (1) only use parts of those API (like only the decoding part) and (2) be able to efficiently (without bringing back to main memory) use the decoded content. Yours, Edward > > Best Regards, > > -- > Rob Krakora > Senior Software Engineer > MessageNet Systems > 101 East Carmel Dr. Suite 105 > Carmel, IN 46032 > (317)566-1677 Ext. 206 > (317)663-0808 Fax > > ------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/gstreamer-devel |
|
|
In reply to this post by Lawrence Auster-2
Sorry for the late reply to this. I'm curious how closed captioning
and/or subtitles will be rolled into this. Do you video folks have any thoughts? I'm also curious if we might also want to consider some sort of gconf setting that says "give me closed captioning" that any video player can check and honor. Will Benjamin Otte wrote: > Hi, > > As you may be aware, we held a video hackfest last week in Barcelona. > Developers met to discuss how best to improve GPU support for video > applications. See http://gstreamer.freedesktop.org/wiki/VideoHackfest > for more details. In particular, you might be interested in the notes > some people took while the hackfest was going on. These can be found > at http://gstreamer.freedesktop.org/wiki/VideoHackfest/Notes > > What follows is the results we arrived at after the hackfest. They are > taken from http://gstreamer.freedesktop.org/wiki/VideoHackfest/Conclusions > but certainly deserve wide circulation. We're interested in any > comments (or questions) you might have about them, so please don't > heistate to reply. > > If you do reply, please make sure to reduce the recipient list appropriately. > > Cheers, > Benjamin, > > on behalf of all hackfest paticipants. > > > == YUV in pixman == > It has been a goal for a while to make video a first-class citizen in > the image stack. A concrete API proposal on how to integrate video > into pixman was reached and reviewed. > Links: > * http://gstreamer.freedesktop.org/wiki/VideoHackfest/YCbCr_Formats > * http://gstreamer.freedesktop.org/wiki/VideoHackfest/pixman > Actions: > * GStreamer developers to provide a specification of important YUV formats > * Write implementation > > == YUV in Cairo == > To make it easy for applications to use these different formats, the > Cairo API needs to be extended to allow them. A proposal does exist > and has been reviewed, details need to be finalized. > Links: > * http://gstreamer.freedesktop.org/wiki/VideoHackfest/Cairo > Actions: > * Finalize proposal > * Write implementation > > == locking in Cairo == > GStreamer takes a lot of freedom in deciding which threads to schedule > elements in. Most hardware backends however require proper > serialization of commands. The current gst-plugins-cairo and > gst-plugins-gl code provide different, but ugly ways to achieve this. > Cairo's internal thread safety guarantees match these requirements > very well. But the different Cairo backends don't always keep these > guarantees. Interaction with these locking mechanisms from outside > applications is not possible yet either. > Links: > * http://gstreamer.freedesktop.org/wiki/VideoHackfest/Cairo > Actions: > * Improve Cairo backend implementations, in particular X11 and GL > * Expose Cairo's locking API to allow interaction with it > * Write testcases to squash bugs > > == locking in Mesa == > The GLX specification does not allow binding the same GLX context in > multiple threads at the same time. This is a requirement for both > Cairo's and GStreamer's threading model. An extension was proposed and > initial code developed to support this requirement in the same way as > Apple's GL does by default. Windows does not support this and would > require potentially expensive fallback code. > Links: > * http://people.freedesktop.org/~anholt/MESA_multithread_makecurrent.spec > Actions: > * Get review for suggested extension and include it in future Mesa releases > * Make Windows developers investigate the situation > > == switch GStreamer to Cairo as default video transport model == > The current approach to handling video in GStreamer is very outdated. > It does not allow hw-accelerated buffers nor does it provide a unified > API to modify video buffers, which leads to fragmentation and > duplication. It was agreed that using Cairo to provide an abstraction > as a drawing API and abstraction over different backends was a good > idea. > Links: > * http://gstreamer.freedesktop.org/wiki/VideoHackfest/Notes > * http://cgit.freedesktop.org/~company/gst-plugins-cairo > Actions: > * Rework gst-plugins-cairo to match improvements listed in previous points > * Get more review on API and to avoid regressions > * Merge into gst-plugins-base > * Switch plugins gradually to use new Cairo code > > == gst-plugins-gl == > The general consensus was that gst-plugins-gl is a hack. It was > necessary in the past to get things to work, but is not a good way > forward. However, the functionality or performance provided by the > current elements needs to continue working. Developers shared the > opinion that gst-plugins-cairo with cairo-gl surfaces is the best way > to achieve this. > Links: > * http://gstreamer.freedesktop.org/wiki/VideoHackfest/Notes > Actions: > * Get review of cairo-gl/gst-plugins-cairo code from gst-plugins-gl developers > * Port gst-plugins-gl elements to use Cairo early to ensure required > features are available > > == XRenderPutImage == > Currently there is no way to upload video data to the X server for > later use. The XV extension is not sufficient for anything more > complicated than a simple video player. > Links: > * http://gstreamer.freedesktop.org/wiki/VideoHackfest/Cairo > Actions: > * Figure out the best way to move YUV data into hw-accelerated Cairo > surfaces (GL vs X) > * Possibly draft an extension to XRender to handle this in the no-GL case > > == JIT in Pixman == > ORC was investigated as a potential JIT for pixman. A lot of talk has > happened, but no clear conclusions were reached. The idea to share a > JIT with Mesa's GLSL stack was also brought up. > Links: > * http://gstreamer.freedesktop.org/wiki/VideoHackfest/Notes > Actions: > * Continue investigating JIT options > * Prototype ORC integration into Pixman > > == Video decoding Acceleration == > Hardware video decoding is a feature that everybody is very excited > about and wants to integrate into GStreamer as soon as possible. But > developers feel that the current APIs available - the focus was on > VAAPI and VDPAU - to do hardware acceleration fail to integrate well. > No developer managed to create even demo-quality integration with > gst-plugins-cairo, even though quite a few tried. In particular, both > VDPAU and VAAPI... > > ... do not integrate with existing frameworks > Both libraries fail to use (or allow easy conversion to) existing > objects (GEM handles, GL textures, ...) but only provide access to an > encapsulation. This makes it hard for a flexible framework like > GStreamer to make use of its features without limiting itself. > > ... do not have a clear scope > In particular VDPAU advertises itself as a video playback framework > using hardware acceleration. Unfortunately, they both limit themselves > to only the formats and functionalities supported by the hardware and > see fullscreen video playback as the single usecase. This needlessly > complicates integration with existing frameworks > > ... provide a very complicated API > For decoding video, an API like provided by libtheora, libmpeg2 or > even ffmpeg is very simple and allows easy integration with both the > demuxing and the postprocessing/display side. VAAPI and VDPAU require > complex setups and preprocessing of the data to work properly. > > ... overlap with GL functionality > It was felt that a lot of processing features provided by these > libraries are equally well available using shaders in GL or by > providing GL extensions. It was unclear why it is necessary to use a > separate way to do this. > > Links: > * http://gstreamer.freedesktop.org/wiki/VideoHackfest/Notes > * http://gstreamer.freedesktop.org/wiki/VideoHackfest?action=AttachFile&do=view&target=VAinGST.pdf > Actions: > * Get in contact with developers of the APIs > * Encourage revision of public > * Work closer together in integrating the provided functionality into GStreamer > > = Timeline = > The end goal of all of this is to get the code into users' hands > quickly, but allow a smooth and non-disrupting upgrade. Desired dates > have been put forth to achieve this: > * End of January: release new versions of Cairo and Pixman that > contain the new APIs reuired by GStreamer > * End of March: X server and Mesa releases are due - make sure the > required Mesa extension is part of this. Ideally XrenderPutImage would > be included, too. > * April/May, after next Fedora/Ubuntu releases: Merge Cairo support > into gst-plugins-base and start porting elements to it. Encourage > application developers (browser, Flash players) to make use of the new > APIs > * October: another Fedora/Ubuntu release that switches all users to > the new APIs > _______________________________________________ > desktop-devel-list mailing list > [hidden email] > http://mail.gnome.org/mailman/listinfo/desktop-devel-list ------------------------------------------------------------------------------ Join us December 9, 2009 for the Red Hat Virtual Experience, a free event focused on virtualization and cloud computing. Attend in-depth sessions from your desk. Your couch. Anywhere. http://p.sf.net/sfu/redhat-sfdev2dev _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/gstreamer-devel |
«
Return to GStreamer-devel
|
1 view|%1 views
| Free forum by Nabble | Edit this page |