Google Summer of Code: advanced OpenGL video mixer/renderer

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Google Summer of Code: advanced OpenGL video mixer/renderer

Attila Áfra
Hello!

My name is Attila Afra and I'm a 3rd year CS student. I would like to
participate in Google Summer of Code with the following project:

I'm planning to write an OpenGL accelerated video mixer/renderer library: AXVR
(Accelerated Cross-platform Video Renderer). Its main features will be:

  * all calculations done using OpenGL 2.1 and GLSL
  * support for many planar and packed video formats
  * correct color space conversion taking into account chroma siting,
    primaries, nominal range, etc.
  * asynchronous high speed video uploading using pixel buffer objects
  * high quality resampling (including chroma upsampling): bicubic, Lanczos
  * pixel-based motion and edge adaptive de-interlacing with automatic inverse
    telecine supporting arbitrary cadences
  * 3D noise filter
  * color correction
  * video mixing using alpha blending

I will write the library in C++, but of course I want to create a C binding too.
I want it to be as generic as possible in order to be usable in several
multimedia frameworks and players. Because of this, it will handle only the
rendering of individual frames and it will not be restricted to OpenGL (possible
other backends would be: CUDA, Direct3D, etc.).

This library would represent the first part of the project. The second part
would be a gstreamer plugin which would use this library.

This wouldn't be my first open source project. My first project was FilmShrink
(http://filmshrink.sourceforge.net/), a DVD-Video transcoding application, which
I wrote 3-4 years ago. The second one is quite recent (I've just uploaded it):
a small, cross-platform, OpenGL-based GPGPU library called GPUC
(http://code.google.com/p/gpuc/). I also did research on 3D graphics (real-time
atmospheric scattering using OpenGL/GLSL).

What do you think about this project? I can give more details about it, because
I've made some good progress on the design.

Regards,
Attila


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Google Summer of Code: advanced OpenGL video mixer/renderer

Florent THIERY-2
Hi Attila,

> I'm planning to write an OpenGL accelerated video mixer/renderer library: AXVR
>  (Accelerated Cross-platform Video Renderer). Its main features will be:
>
>   * all calculations done using OpenGL 2.1 and GLSL
>   * support for many planar and packed video formats
>   * correct color space conversion taking into account chroma siting,
>      primaries, nominal range, etc.
>   * asynchronous high speed video uploading using pixel buffer objects
>   * high quality resampling (including chroma upsampling): bicubic, Lanczos
>   * pixel-based motion and edge adaptive de-interlacing with automatic
> inverse
>      telecine supporting arbitrary cadences
>   * 3D noise filter
>   * color correction
>   * video mixing using alpha blending
> This library would represent the first part of the project. The second part
> would be a gstreamer plugin which would use this library.
> What do you think about this project? I can give more details about it,
> because I've made some good progress on the design.

Indeed this is a very interesting project.

I myself am investigating a similar one, based on the clutter
framework (http://www.clutter-project.org) -- more animated video
compositing oriented ; it can do offscreen rendering, supports FBOs
and shaders already. Fluendo's Pigment API would be a very nice
starting point as well (i'd be surprised if a pigment-based gst video
mixing plugin isn't on a roadmap yet ;)

Would you mind giving more details ?

Nice proposal IMHO, gstreamer's existing video mixing elements are
already very capable yet not hardware accelerated, nor 3D-capable. In
my gstreamer applications, most of the CPU is eaten by decoding or
encoding, so GPU power that is left is a huge benefit !

Cheers

FLo

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Google Summer of Code: advanced OpenGL video mixer/renderer

Attila Áfra
>  Indeed this is a very interesting project.
>
>  I myself am investigating a similar one, based on the clutter
>  framework (http://www.clutter-project.org) -- more animated video
>  compositing oriented ; it can do offscreen rendering, supports FBOs
>  and shaders already. Fluendo's Pigment API would be a very nice
>  starting point as well (i'd be surprised if a pigment-based gst video
>  mixing plugin isn't on a roadmap yet ;)
>
>  Would you mind giving more details ?

I would like to concentrate more on video processing, and less on mixing.
Usually a simple 2D alpha blending mixer is sufficient, and this is what I
intend to make.

If you need a more complex (perhaps 3D with lots of special effects) mixer
then you should implement that by yourself (or use a specialized library like
Pigment or Clutter) and tell my library to render the video into a
texture (similarly
to VMR's renderless mode) without using its simple mixing capabilites. This
way you can customize the mixer to your own needs.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Google Summer of Code: advanced OpenGL video mixer/renderer

Florent THIERY-2
>  I would like to concentrate more on video processing, and less on mixing.

In fact, pigment's gstreamer excellent videosink already does
OpenGL-accelerated colorspace conversion, are you thinking of a more
general-purpose (and portable) plugin + library ?
https://code.fluendo.com/pigment/trac/browser/trunk/pigment/pgm/pgmimagesink.c

On the same vein, you might be aware of the new gst-plugins-gl
repository, which contains similar, independant elements (ex:
implementing colorspace conversion in an independant manner). However,
when multiple elements are used, unless implementing some form of
context sharing, textures are uploaded back and forth between each
processing step...

Anyway, good luck :)

FLo

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Google Summer of Code: advanced OpenGL video mixer/renderer

Florent THIERY-2
Sorry, wrong link

https://code.fluendo.com/pigment/trac/browser/trunk/pigment/plugins/opengl/pgmtexture.c

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Google Summer of Code: advanced OpenGL video mixer/renderer

Edward Hervey-2
In reply to this post by Attila Áfra

On Thu, 2008-03-20 at 19:47 +0200, Attila Áfra wrote:

> >  Indeed this is a very interesting project.
> >
> >  I myself am investigating a similar one, based on the clutter
> >  framework (http://www.clutter-project.org) -- more animated video
> >  compositing oriented ; it can do offscreen rendering, supports FBOs
> >  and shaders already. Fluendo's Pigment API would be a very nice
> >  starting point as well (i'd be surprised if a pigment-based gst video
> >  mixing plugin isn't on a roadmap yet ;)
> >
> >  Would you mind giving more details ?
>
> I would like to concentrate more on video processing, and less on mixing.
> Usually a simple 2D alpha blending mixer is sufficient, and this is what I
> intend to make.

  I'm more than willing to mentor that ! We need good gpu-accelerated
plugins (and not sinks/applications) for generic use in all
gstreamer-based applications. PiTiVi would take one hell of a boost with
this.
  Maybe David (Schleef) has some more insight on how to best leverage
the existing work from the newly created gst-gl module.

>
> If you need a more complex (perhaps 3D with lots of special effects) mixer
> then you should implement that by yourself (or use a specialized library like
> Pigment or Clutter) and tell my library to render the video into a
> texture (similarly
> to VMR's renderless mode) without using its simple mixing capabilites. This
> way you can customize the mixer to your own needs.
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2008.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
--
Edward Hervey <[hidden email]>
Collabora Multimedia


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Google Summer of Code: advanced OpenGL video mixer/renderer

David Schleef
In reply to this post by Attila Áfra
On Thu, Mar 20, 2008 at 03:37:30PM +0200, Attila Áfra wrote:

> I'm planning to write an OpenGL accelerated video mixer/renderer library:
> AXVR
> (Accelerated Cross-platform Video Renderer). Its main features will be:
>
>   * all calculations done using OpenGL 2.1 and GLSL
>   * support for many planar and packed video formats
>   * correct color space conversion taking into account chroma siting,
>     primaries, nominal range, etc.
>   * asynchronous high speed video uploading using pixel buffer objects
>   * high quality resampling (including chroma upsampling): bicubic, Lanczos
>   * pixel-based motion and edge adaptive de-interlacing with automatic
> inverse
>     telecine supporting arbitrary cadences
>   * 3D noise filter
>   * color correction
>   * video mixing using alpha blending

Implementing that list well would take much longer than a Google Summer
of Code project.

> What do you think about this project? I can give more details about it,
> because I've made some good progress on the design.

I think it would be wise to decrease the scope of the project.  Pick
a few video processes, implement them with a dirt-simple API, wrap it
all in a GStreamer plugin, then go back and redesign the infrastructure,
redesign the API, reassess the level of quality you expect from the
algorithms, etc.  At that point, you might have some time left in your
summer to start reimplementing various stuff and adding more processes.

There already exists a similar library hidden inside Schroedinger that
does similar stuff with a C and CUDA backend.  The signal processing is
currently mostly Dirac and video compression oriented, though.

Also, as others have mentioned, there is gst-plugins-gl.  This has
gotten bogged down in the craptastic API that is OpenGL.



dave...


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Google Summer of Code: advanced OpenGL video mixer/renderer

Attila Áfra
In reply to this post by Edward Hervey-2
I would like to clarify a few things.

First of all, my library is designed for playback, not for general
purpose video processing. It will be able to do a limited, but very
important set of operations required for high quality playback:
colorspace conversion, color correction, resampling, deinterlacing,
inverse telecine, noise reduction, etc. In terms of features it will
be quite similar to EVR which is part of Microsoft's Media Foundation
(http://msdn2.microsoft.com/en-us/library/ms694197.aspx), but I don't
want it to be dependent on any media framework. Because of this, I
want to start the project from scratch.

I want the library to have a monolithic design (one big module), which
has several advantages, but disadvantages too. The biggest advantages
are performance (thanks to special internal optimizations), API
simplicity and better portability (OpenGL 3, CUDA, CTM, Direct3D,
Larrabee-like many-core processors, etc). In my opinion the only
disadvantage worth mentioning is that it will not be possible to
easily extend it with new fancy filters/effects, but in the case of
real-time video playback I believe it's not really important.

So, the bridge between this library and GStreamer would be a video sink.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Google Summer of Code: advanced OpenGL video mixer/renderer

Attila Áfra
In reply to this post by David Schleef
>  Implementing that list well would take much longer than a Google Summer
>  of Code project.

This is why I will soon start developing. I'm aware that it's a big
project, so I will drop a few features, if necessary (e.g. the noise
filter, the advanced deinterlacer).

>  I think it would be wise to decrease the scope of the project.  Pick
>  a few video processes, implement them with a dirt-simple API, wrap it
>  all in a GStreamer plugin, then go back and redesign the infrastructure,
>  redesign the API, reassess the level of quality you expect from the
>  algorithms, etc.  At that point, you might have some time left in your
>  summer to start reimplementing various stuff and adding more processes.

I don't think it's a good idea. In my opinion it's too much extra
unnecessary work.

>  Also, as others have mentioned, there is gst-plugins-gl.  This has
>  gotten bogged down in the craptastic API that is OpenGL.

Fortunately OpenGL 3 looks really promising. :)

Thanks for the advices!

Attila

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel