best way to internally represent an image

[Florin Andrei]

(apparently offtopic, but bear with me, you'll see it's not)

I'm writing a 3D raytracer to analyze relativistic phenomena. The
raytracer must be able to draw the 3D scene in real time, or to parse a
script and then generate a movie based on the scripted movements of the
camera. I'm in a very early stage, still planning the high level stuff.
 From a user's perspective, it will superficially resemble Celestia
(display a view of the "cosmos" that the user can navigate with the
joystick or keyboard):

http://www.shatters.net/celestia/gallery.html

I think I may use Gstreamer to generate the movies.

But anyway, here's the question:
I need to decide on the internal format to represent the image being
rendered. It will probably be some kind of bitmap, a matrix of pixels.
But I have a hard time deciding between RGB, YUV, and all that stuff.

I am tempted to use a matrix of pixels, each pixel being a C structure
like this:

struct pixel
{
        unsigned char red;
        unsigned char green;
        unsigned char blue;
};

That would be 24 bit RGB, I guess. I like it because it feels "natural",
and it's trivial to come up with algorithms that, e.g., give a blue tint
("blue shift"), etc.

But I need to decide on a format that will make it easiest to accomplish
the 2 main goals of the program's output:

1. Project an interactive 3D scene on the screen in real time using SDL
or Xv, or whatever fast and portable method I can find

2. Interface with a program or a library that can dump a movie on the
disc in some nice universal format (MPEG2, MPEG4, etc, 480p, 1080p, or
user-defined such as 1024x768, etc.) regardless of whether I'll actually
use Gstreamer for that purpose or not (but the likeliest candidate at
this moment is Gstreamer, it's really nice and powerful)

Speed is crucial, as the project has some pretty ambitious goals (HD
resolutions up to 1080p, real time display for as high resolutions as
possible - ideally for 1080p on very good hardware, or whatever can be
achieved with the best hardware today). So choosing the internal image
representation has to take that in consideration.

I'm not an expert in video programming. I'm not even a professional
programmer. I'm just a hobbyist. So I would appreciate any advice from
experienced people.

So - which internal format would you use for the image?

Thanks,

--
Florin Andrei

http://florin.myip.org/

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

[Florent THIERY-2]

Hi

>  2. Interface with a program or a library that can dump a movie on the
>  disc in some nice universal format (MPEG2, MPEG4, etc, 480p, 1080p, or
>  user-defined such as 1024x768, etc.) regardless of whether I'll actually
>  use Gstreamer for that purpose or not (but the likeliest candidate at
>  this moment is Gstreamer, it's really nice and powerful)

3D capture (from real time rendering) is tricky, especially for high
resolutions, and especially when you want to capture using GStreamer
(there's no such thing as a glReadPixels src element for gst yet).

As for a simple solution for dumping, check out libseom [1]
If you want to capture "by hand" into gstreamer, you might have to
check apprsc [2]

FLo

[1] http://dbservice.com/projects/seom/
[2] http://webcvs.freedesktop.org/gstreamer/gst-plugins-bad/gst-libs/gst/app/gstappsrc.c?revision=1.4&view=markup

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

[Florin Andrei]

Florent wrote:

> Hi
>
>>  2. Interface with a program or a library that can dump a movie on the
>>  disc in some nice universal format (MPEG2, MPEG4, etc, 480p, 1080p, or
>>  user-defined such as 1024x768, etc.) regardless of whether I'll actually
>>  use Gstreamer for that purpose or not (but the likeliest candidate at
>>  this moment is Gstreamer, it's really nice and powerful)
>
> 3D capture (from real time rendering) is tricky, especially for high
> resolutions, and especially when you want to capture using GStreamer
> (there's no such thing as a glReadPixels src element for gst yet).

Well, all I care about is the area being displayed, and that's 2D. The
entire 3D magic happens in the math engine, before the bitmap is
constructed.

> As for a simple solution for dumping, check out libseom [1]
> If you want to capture "by hand" into gstreamer, you might have to
> check apprsc [2]

I would be content to run the app in two separate modes: display (show
the scene on the screen, no movie file generated) and render (generate a
movie file, nothing is displayed). I guess that shouldn't be too
complicated, right?

--
Florin Andrei

http://florin.myip.org/

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

[Florent THIERY-2]

Sorry i thought you were planning real time rendering. My bad, real
time raytracing is not for tomorrow :p

You're talking software rendering then ?

If your software can output sequential frames, then the gstreamer
multifilesrc[1] should do the trick for generating the video, or
mencoder[2].

[1] http://gstreamer.freedesktop.org/wiki/MultiFileSrc
[2] http://www.mplayerhq.hu/DOCS/HTML/en/menc-feat-enc-images.html

FLo

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

[Gustavo Sverzut Barbieri]

On Sun, Feb 24, 2008 at 8:33 PM, Florin Andrei <[hidden email]> wrote:

> (apparently offtopic, but bear with me, you'll see it's not)
>
>  I'm writing a 3D raytracer to analyze relativistic phenomena. The
>  raytracer must be able to draw the 3D scene in real time, or to parse a
>  script and then generate a movie based on the scripted movements of the
>  camera. I'm in a very early stage, still planning the high level stuff.
>   From a user's perspective, it will superficially resemble Celestia
>  (display a view of the "cosmos" that the user can navigate with the
>  joystick or keyboard):
>
>  http://www.shatters.net/celestia/gallery.html
>
>  I think I may use Gstreamer to generate the movies.
>
>  But anyway, here's the question:
>  I need to decide on the internal format to represent the image being
>  rendered. It will probably be some kind of bitmap, a matrix of pixels.
>  But I have a hard time deciding between RGB, YUV, and all that stuff.
>
>  I am tempted to use a matrix of pixels, each pixel being a C structure
>  like this:
>
>  struct pixel
>  {
>         unsigned char red;
>         unsigned char green;
>         unsigned char blue;
>  };
>
>  That would be 24 bit RGB, I guess. I like it because it feels "natural",
>  and it's trivial to come up with algorithms that, e.g., give a blue tint
>  ("blue shift"), etc.
>
>  But I need to decide on a format that will make it easiest to accomplish
>  the 2 main goals of the program's output:
>
>  1. Project an interactive 3D scene on the screen in real time using SDL
>  or Xv, or whatever fast and portable method I can find
>
>  2. Interface with a program or a library that can dump a movie on the
>  disc in some nice universal format (MPEG2, MPEG4, etc, 480p, 1080p, or
>  user-defined such as 1024x768, etc.) regardless of whether I'll actually
>  use Gstreamer for that purpose or not (but the likeliest candidate at
>  this moment is Gstreamer, it's really nice and powerful)
>
>  Speed is crucial, as the project has some pretty ambitious goals (HD
>  resolutions up to 1080p, real time display for as high resolutions as
>  possible - ideally for 1080p on very good hardware, or whatever can be
>  achieved with the best hardware today). So choosing the internal image
>  representation has to take that in consideration.
>
>  I'm not an expert in video programming. I'm not even a professional
>  programmer. I'm just a hobbyist. So I would appreciate any advice from
>  experienced people.

Ok, so as not many people replied, I'll give my shot.

24bits is often not a good format if you need to access data much,
why? because we use 32bits (or multiple of) machines, using it will
mean more optimized access which is often faster. But going 32bpp
(bits per pixel) will waste more memory, so my advice is to do some
tests on your desired hardware and see for your use case which one
fits better.

Also, if going 32bpp maybe it's better to handle it as an int32_t
instead of a struct, but most compilers should optimize it anyway.

I'd also do YUV tests. Often used variations have half planes for
color channels, so you have less data and thus it tend to be faster.
Most video cards have accelerated interface to get this colorspace, so
you can get scale and other operations done by hardware.
    However YUV is not a good format if you want to mix it back into
your RGB color space. Say you want to do alpha blending of your video
on some RGB data (window or like), you'd have to convert YUV->RGB (or
the other way around) and it will be slow... also you loose Xv.
However I have to admit it's uncommon, but worth thinking about it.

As Florent already said, most systems do accept image buffers in most
colorspaces and do the conversion for you (often the video format is
YUV variant). If you choose a weird video encoding stack that does not
support it by default, you can use ffmpeg code, as in gstreamer's
ffmpegcolorspace.

--
Gustavo Sverzut Barbieri
http://profusion.mobi - Embedded and Mobile Software Development
--------------------------------------
MSN: [hidden email]
Skype: gsbarbieri
Mobile: [phone]

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

[Florin Andrei]

Gustavo Sverzut Barbieri wrote:
>     However YUV is not a good format if you want to mix it back into
> your RGB color space. Say you want to do alpha blending of your video
> on some RGB data (window or like), you'd have to convert YUV->RGB (or
> the other way around) and it will be slow... also you loose Xv.

Wait - I can't do Xv if I use YUV?

--
Florin Andrei

http://florin.myip.org/

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

[Gustavo Sverzut Barbieri]

On Tue, Feb 26, 2008 at 5:10 AM, Florin Andrei <[hidden email]> wrote:
> Gustavo Sverzut Barbieri wrote:
>  >     However YUV is not a good format if you want to mix it back into
>  > your RGB color space. Say you want to do alpha blending of your video
>  > on some RGB data (window or like), you'd have to convert YUV->RGB (or
>  > the other way around) and it will be slow... also you loose Xv.
>
>  Wait - I can't do Xv if I use YUV?

You misunderstood me (possible I was not clear enough). You CAN do YUV
with Xv, of course.    You can NOT do Xv if you want to get the YUV
mixed in some window RGB, for example make the YUV stream
semi-transparent on top of your application.  You'd have to rely on
other tricks like composite manager, ... But as I said, it's not
common to have such things.
    IOW: if you want to display a regular movie in some rectangular
window, YUV might be a best choice as you have less data do move and
less points to process for color planes.

--
Gustavo Sverzut Barbieri
http://profusion.mobi - Embedded and Mobile Software Development
--------------------------------------
MSN: [hidden email]
Skype: gsbarbieri
Mobile: [phone]

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel