Google Summer of Code: advanced OpenGL video mixer/renderer
Locked
Google Summer of Code: advanced OpenGL video mixer/renderer
 
|
Hello!
My name is Attila Afra and I'm a 3rd year CS student. I would like to participate in Google Summer of Code with the following project: I'm planning to write an OpenGL accelerated video mixer/renderer library: AXVR (Accelerated Cross-platform Video Renderer). Its main features will be: * all calculations done using OpenGL 2.1 and GLSL * support for many planar and packed video formats * correct color space conversion taking into account chroma siting, primaries, nominal range, etc. * asynchronous high speed video uploading using pixel buffer objects * high quality resampling (including chroma upsampling): bicubic, Lanczos * pixel-based motion and edge adaptive de-interlacing with automatic inverse telecine supporting arbitrary cadences * 3D noise filter * color correction * video mixing using alpha blending I will write the library in C++, but of course I want to create a C binding too. I want it to be as generic as possible in order to be usable in several multimedia frameworks and players. Because of this, it will handle only the rendering of individual frames and it will not be restricted to OpenGL (possible other backends would be: CUDA, Direct3D, etc.). This library would represent the first part of the project. The second part would be a gstreamer plugin which would use this library. This wouldn't be my first open source project. My first project was FilmShrink (http://filmshrink.sourceforge.net/), a DVD-Video transcoding application, which I wrote 3-4 years ago. The second one is quite recent (I've just uploaded it): a small, cross-platform, OpenGL-based GPGPU library called GPUC (http://code.google.com/p/gpuc/). I also did research on 3D graphics (real-time atmospheric scattering using OpenGL/GLSL). What do you think about this project? I can give more details about it, because I've made some good progress on the design. Regards, Attila ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/gstreamer-devel |
Re: Google Summer of Code: advanced OpenGL video mixer/renderer
|
|
Hi Attila,
> I'm planning to write an OpenGL accelerated video mixer/renderer library: AXVR > (Accelerated Cross-platform Video Renderer). Its main features will be: > > * all calculations done using OpenGL 2.1 and GLSL > * support for many planar and packed video formats > * correct color space conversion taking into account chroma siting, > primaries, nominal range, etc. > * asynchronous high speed video uploading using pixel buffer objects > * high quality resampling (including chroma upsampling): bicubic, Lanczos > * pixel-based motion and edge adaptive de-interlacing with automatic > inverse > telecine supporting arbitrary cadences > * 3D noise filter > * color correction > * video mixing using alpha blending > This library would represent the first part of the project. The second part > would be a gstreamer plugin which would use this library. > What do you think about this project? I can give more details about it, > because I've made some good progress on the design. Indeed this is a very interesting project. I myself am investigating a similar one, based on the clutter framework (http://www.clutter-project.org) -- more animated video compositing oriented ; it can do offscreen rendering, supports FBOs and shaders already. Fluendo's Pigment API would be a very nice starting point as well (i'd be surprised if a pigment-based gst video mixing plugin isn't on a roadmap yet ;) Would you mind giving more details ? Nice proposal IMHO, gstreamer's existing video mixing elements are already very capable yet not hardware accelerated, nor 3D-capable. In my gstreamer applications, most of the CPU is eaten by decoding or encoding, so GPU power that is left is a huge benefit ! Cheers FLo ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/gstreamer-devel |
Re: Google Summer of Code: advanced OpenGL video mixer/renderer
|
|
> Indeed this is a very interesting project.
> > I myself am investigating a similar one, based on the clutter > framework (http://www.clutter-project.org) -- more animated video > compositing oriented ; it can do offscreen rendering, supports FBOs > and shaders already. Fluendo's Pigment API would be a very nice > starting point as well (i'd be surprised if a pigment-based gst video > mixing plugin isn't on a roadmap yet ;) > > Would you mind giving more details ? I would like to concentrate more on video processing, and less on mixing. Usually a simple 2D alpha blending mixer is sufficient, and this is what I intend to make. If you need a more complex (perhaps 3D with lots of special effects) mixer then you should implement that by yourself (or use a specialized library like Pigment or Clutter) and tell my library to render the video into a texture (similarly to VMR's renderless mode) without using its simple mixing capabilites. This way you can customize the mixer to your own needs. ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/gstreamer-devel |
Re: Google Summer of Code: advanced OpenGL video mixer/renderer
|
|
> I would like to concentrate more on video processing, and less on mixing.
In fact, pigment's gstreamer excellent videosink already does OpenGL-accelerated colorspace conversion, are you thinking of a more general-purpose (and portable) plugin + library ? https://code.fluendo.com/pigment/trac/browser/trunk/pigment/pgm/pgmimagesink.c On the same vein, you might be aware of the new gst-plugins-gl repository, which contains similar, independant elements (ex: implementing colorspace conversion in an independant manner). However, when multiple elements are used, unless implementing some form of context sharing, textures are uploaded back and forth between each processing step... Anyway, good luck :) FLo ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/gstreamer-devel |
Re: Google Summer of Code: advanced OpenGL video mixer/renderer
|
|
Sorry, wrong link
https://code.fluendo.com/pigment/trac/browser/trunk/pigment/plugins/opengl/pgmtexture.c ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/gstreamer-devel |
Re: Google Summer of Code: advanced OpenGL video mixer/renderer
|
|
In reply to this post by Attila Áfra
On Thu, 2008-03-20 at 19:47 +0200, Attila Áfra wrote: > > Indeed this is a very interesting project. > > > > I myself am investigating a similar one, based on the clutter > > framework (http://www.clutter-project.org) -- more animated video > > compositing oriented ; it can do offscreen rendering, supports FBOs > > and shaders already. Fluendo's Pigment API would be a very nice > > starting point as well (i'd be surprised if a pigment-based gst video > > mixing plugin isn't on a roadmap yet ;) > > > > Would you mind giving more details ? > > I would like to concentrate more on video processing, and less on mixing. > Usually a simple 2D alpha blending mixer is sufficient, and this is what I > intend to make. I'm more than willing to mentor that ! We need good gpu-accelerated plugins (and not sinks/applications) for generic use in all gstreamer-based applications. PiTiVi would take one hell of a boost with this. Maybe David (Schleef) has some more insight on how to best leverage the existing work from the newly created gst-gl module. > > If you need a more complex (perhaps 3D with lots of special effects) mixer > then you should implement that by yourself (or use a specialized library like > Pigment or Clutter) and tell my library to render the video into a > texture (similarly > to VMR's renderless mode) without using its simple mixing capabilites. This > way you can customize the mixer to your own needs. > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > gstreamer-devel mailing list > [hidden email] > https://lists.sourceforge.net/lists/listinfo/gstreamer-devel Edward Hervey <[hidden email]> Collabora Multimedia ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/gstreamer-devel |
Re: Google Summer of Code: advanced OpenGL video mixer/renderer
|
|
In reply to this post by Attila Áfra
On Thu, Mar 20, 2008 at 03:37:30PM +0200, Attila Áfra wrote:
> I'm planning to write an OpenGL accelerated video mixer/renderer library: > AXVR > (Accelerated Cross-platform Video Renderer). Its main features will be: > > * all calculations done using OpenGL 2.1 and GLSL > * support for many planar and packed video formats > * correct color space conversion taking into account chroma siting, > primaries, nominal range, etc. > * asynchronous high speed video uploading using pixel buffer objects > * high quality resampling (including chroma upsampling): bicubic, Lanczos > * pixel-based motion and edge adaptive de-interlacing with automatic > inverse > telecine supporting arbitrary cadences > * 3D noise filter > * color correction > * video mixing using alpha blending Implementing that list well would take much longer than a Google Summer of Code project. > What do you think about this project? I can give more details about it, > because I've made some good progress on the design. I think it would be wise to decrease the scope of the project. Pick a few video processes, implement them with a dirt-simple API, wrap it all in a GStreamer plugin, then go back and redesign the infrastructure, redesign the API, reassess the level of quality you expect from the algorithms, etc. At that point, you might have some time left in your summer to start reimplementing various stuff and adding more processes. There already exists a similar library hidden inside Schroedinger that does similar stuff with a C and CUDA backend. The signal processing is currently mostly Dirac and video compression oriented, though. Also, as others have mentioned, there is gst-plugins-gl. This has gotten bogged down in the craptastic API that is OpenGL. dave... ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/gstreamer-devel |
Re: Google Summer of Code: advanced OpenGL video mixer/renderer
|
|
In reply to this post by Edward Hervey-2
I would like to clarify a few things.
First of all, my library is designed for playback, not for general purpose video processing. It will be able to do a limited, but very important set of operations required for high quality playback: colorspace conversion, color correction, resampling, deinterlacing, inverse telecine, noise reduction, etc. In terms of features it will be quite similar to EVR which is part of Microsoft's Media Foundation (http://msdn2.microsoft.com/en-us/library/ms694197.aspx), but I don't want it to be dependent on any media framework. Because of this, I want to start the project from scratch. I want the library to have a monolithic design (one big module), which has several advantages, but disadvantages too. The biggest advantages are performance (thanks to special internal optimizations), API simplicity and better portability (OpenGL 3, CUDA, CTM, Direct3D, Larrabee-like many-core processors, etc). In my opinion the only disadvantage worth mentioning is that it will not be possible to easily extend it with new fancy filters/effects, but in the case of real-time video playback I believe it's not really important. So, the bridge between this library and GStreamer would be a video sink. ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/gstreamer-devel |
Re: Google Summer of Code: advanced OpenGL video mixer/renderer
|
|
In reply to this post by David Schleef
> Implementing that list well would take much longer than a Google Summer
> of Code project. This is why I will soon start developing. I'm aware that it's a big project, so I will drop a few features, if necessary (e.g. the noise filter, the advanced deinterlacer). > I think it would be wise to decrease the scope of the project. Pick > a few video processes, implement them with a dirt-simple API, wrap it > all in a GStreamer plugin, then go back and redesign the infrastructure, > redesign the API, reassess the level of quality you expect from the > algorithms, etc. At that point, you might have some time left in your > summer to start reimplementing various stuff and adding more processes. I don't think it's a good idea. In my opinion it's too much extra unnecessary work. > Also, as others have mentioned, there is gst-plugins-gl. This has > gotten bogged down in the craptastic API that is OpenGL. Fortunately OpenGL 3 looks really promising. :) Thanks for the advices! Attila ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/gstreamer-devel |
«
Return to GStreamer-devel
|
1 view|%1 views
| Free forum by Nabble | Edit this page |