jpegenc a theora playbin2 buffer?

[Erik Blankinship-2]

I am trying to convert a playbin2 ogv theora buffer to a jpeg encoded buffer through an appsrc ! ... ! appsink pipeline.

I am taking my cue from the approach described here [1].

The simple code below works... but the  appsrc ! appsink pipeline is obviously not right -- the resultant buffer is not jpegencoded.

If I try to expand my pipeline to  appsrc ! ffmpegcolorspace ! jpegenc ! appsink the app hangs.  I've tried inserting a theoradec in there too, but that doesn't seem necessary as the buffer from the playbin2 should be decoded.

    def get_last_frame_as_pixbuf( self ):

        last_frame = self.playbin2.get_property( "frame" )

        if last_frame is None:

            return None

        frame_grab_line = gst.parse_launch( "appsrc name=src ! appsink name=sink" )

                          # this hangs... --> "appsrc ! ffmpegcolorspace ! jpegenc ! appsink"

        frame_src = frame_grab_line.get_by_name( 'src' )

        frame_sink = frame_grab_line.get_by_name( 'sink' )

        frame_grab_line.set_state( gst.STATE_PAUSED )

        frame_src.emit( "push-buffer", last_frame )

        print( "about to grab" )

        buf = frame_sink.emit('pull-preroll')

        print( "grabbed(!):", buf )

        pic = gtk.gdk.pixbuf_loader_new_with_mime_type("image/jpeg")

        loader.write(buffer)

        loader.close()

        return loader.get_pixbuf()

[1] http://lists.freedesktop.org/archives/gstreamer-devel/2009-December/024770.html

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Erik Blankinship-2]

The simple code below works... 

Sorry, I meant to say the code works in getting a buffer out of the appsrc ! appsink pipeline, just not the right type of buffer. 

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Xabier Rodríguez Calvar]

O Lun, 19-09-2011 ás 09:38 -0400, Erik Blankinship escribiu:
> I am trying to convert a playbin2 ogv theora buffer to a jpeg encoded
> buffer through an appsrc ! ... ! appsink pipeline.

I'd use the convert-frame element action that you can find with
"gst-inspect playbin2":

"convert-frame" :  GstBuffer user_function (GstElement* object,
                                              GstCaps* arg0);

It gives you the current frame with the caps you need. For jpeg it would
be something like:

{
  GstBuffer *buffer = NULL;
  GstCaps *caps;

  caps = gst_caps_new_simple ("image/jpeg",
                              "bpp", G_TYPE_INT, 24, "depth",
G_TYPE_INT, 24,
                              "pixel-aspect-ratio", GST_TYPE_FRACTION,
1, 1,
                              "endianness", G_TYPE_INT, G_BIG_ENDIAN,
                              "red_mask", G_TYPE_INT, 0xff0000,
                              "green_mask", G_TYPE_INT, 0x00ff00,
                              "blue_mask", G_TYPE_INT, 0x0000ff,
                              NULL);

  g_signal_emit_by_name (pipeline, "convert-frame", caps, &buffer);
 
  gst_caps_unref (caps);

  if (buffer) {
    GstStructure *structure;
    gint width, height;

    caps = GST_BUFFER_CAPS (buffer);
    structure = gst_caps_get_structure (caps, 0);

    gst_structure_get_int (structure, "width", &width);
    gst_structure_get_int (structure, "height", &height);

    /* Load the pixbuf */

    gst_buffer_unref(buffer);
  }

Actually, you wouldn't need create it as jpeg. It would be enough if you
get video/x-raw-rgb and load that as pixbuf. This way you save the jpeg
encoding and decoding.

Best regards.



_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Erik Blankinship-2]

Actually, you wouldn't need create it as jpeg. It would be enough if you
get video/x-raw-rgb and load that as pixbuf. This way you save the jpeg
encoding and decoding.

Oh wow, this is a much better approach!  Thank you.

last_frame = self.playbin2.get_property( "frame" )

if last_frame is None:

    return

pix_buf = gtk.gdk.pixbuf_new_from_data( last_frame, gtk.gdk.COLORSPACE_RGB, True, 8, hardcoded_width, hardcoded_height, 4*hardcoded_width ) 

But two problems arise:

1. It appears red and blue have been swapped in my pixbuf.  Do I need to run the buffer through a pre-process before pixbuf_new_from_data?
2. What is the best way to get the values for the width and height of the buffer we're converting to a pixbuf?  Grab and cache them from the playbin2?

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Xabier Rodríguez Calvar]

O Lun, 19-09-2011 ás 12:08 -0400, Erik Blankinship escribiu:
> But two problems arise:
>      1. It appears red and blue have been swapped in my pixbuf.  Do I
>         need to run the buffer through a pre-process before
>         pixbuf_new_from_data?

        I had used this at an old piece of code in C:

                pixbuf = gdk_pixbuf_new_from_data(
                        GST_BUFFER_DATA(new_buffer), GDK_COLORSPACE_RGB,
                        FALSE, 8, width, height,
                        GST_ROUND_UP_4(3 * width), _destroy_pixbuf,
                        new_buffer);


>      1. What is the best way to get the values for the width and
>         height of the buffer we're converting to a pixbuf?  Grab and
>         cache them from the playbin2?

You have them in the code I pasted. They should be in the the buffer
caps.

Best regards

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Erik Blankinship-2]

On Mon, Sep 19, 2011 at 1:08 PM, Xabier Rodriguez Calvar <[hidden email]> wrote:

O Lun, 19-09-2011 ás 12:08 -0400, Erik Blankinship escribiu:
> But two problems arise:
>      1. It appears red and blue have been swapped in my pixbuf.  Do I

>         need to run the buffer through a pre-process before
>         pixbuf_new_from_data?

       I had used this at an old piece of code in C:

               pixbuf = gdk_pixbuf_new_from_data(
                       GST_BUFFER_DATA(new_buffer), GDK_COLORSPACE_RGB,
                       FALSE, 8, width, height,
                       GST_ROUND_UP_4(3 * width), _destroy_pixbuf,
                       new_buffer);

I am continuing to have trouble with this.  Most every combination I try ends up with a distorted image.  Further, there doesn't appear to be a pygst counterpart to GST_ROUND_UP_4, but it is easy enough to compute.

def f(num):

    return (num + 3) & ~3

If it is of help to anyone out there, my caps read as follows:

video/x-raw-rgb, bpp=(int)32, depth=(int)24, endianness=(int)4321, red_mask=(int)65280, green_mask=(int)16711680, blue_mask=(int)-16777216, width=(int)512, height=(int)277, framerate=(fraction)24/1, pixel-aspect-ratio=(fraction)1/1, color-matrix=(string)sdtv, chroma-site=(string)jpeg;

What's the thinking and solution behind mapping this buffer into a gdk pixbuf successfully?

 

>      1. What is the best way to get the values for the width and

>         height of the buffer we're converting to a pixbuf?  Grab and
>         cache them from the playbin2?

You have them in the code I pasted. They should be in the the buffer
caps.

Yes, of course.  Thanks again for that.

In pygst for future searchers:

caps = last_frame.get_caps()[0]

width = int(caps["width"])

height = int(caps["height"])

 

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Tim-Philipp Müller-2]

On Mon, 2011-09-19 at 14:46 -0400, Erik Blankinship wrote:

> I am continuing to have trouble with this.  Most every combination I
> try ends up with a distorted image.  

Distorted in what way? Too narrow/wide? Or every line shifted a bit to
the left/right of the previous line?

You might find this useful for inspiration:
http://git.gnome.org/browse/totem/tree/src/gst/totem-gst-helpers.c#n81

Alternatively, there's also a gdkpixbufsink that you could use.

(Though it seems to me that just asking for an image/jpeg in the desired
width or height via the convert-frame action signal would be the easiest
way to achieve your goal)

 Cheers
  -Tim


_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Conrad Cooke]

Mark,

I have it figured out - have programmed 2 - they connect to my Manta - please come up and verify

Conrad

On Sep 19, 2011, at 11:46 AM, Erik Blankinship wrote:

On Mon, Sep 19, 2011 at 1:08 PM, Xabier Rodriguez Calvar <[hidden email]> wrote:

O Lun, 19-09-2011 ás 12:08 -0400, Erik Blankinship escribiu:
> But two problems arise:
>      1. It appears red and blue have been swapped in my pixbuf.  Do I

>         need to run the buffer through a pre-process before
>         pixbuf_new_from_data?

       I had used this at an old piece of code in C:

               pixbuf = gdk_pixbuf_new_from_data(
                       GST_BUFFER_DATA(new_buffer), GDK_COLORSPACE_RGB,
                       FALSE, 8, width, height,
                       GST_ROUND_UP_4(3 * width), _destroy_pixbuf,
                       new_buffer);

I am continuing to have trouble with this.  Most every combination I try ends up with a distorted image.  Further, there doesn't appear to be a pygst counterpart to GST_ROUND_UP_4, but it is easy enough to compute.

def f(num):

    return (num + 3) & ~3

If it is of help to anyone out there, my caps read as follows:

video/x-raw-rgb, bpp=(int)32, depth=(int)24, endianness=(int)4321, red_mask=(int)65280, green_mask=(int)16711680, blue_mask=(int)-16777216, width=(int)512, height=(int)277, framerate=(fraction)24/1, pixel-aspect-ratio=(fraction)1/1, color-matrix=(string)sdtv, chroma-site=(string)jpeg;

What's the thinking and solution behind mapping this buffer into a gdk pixbuf successfully?

 

>      1. What is the best way to get the values for the width and

>         height of the buffer we're converting to a pixbuf?  Grab and
>         cache them from the playbin2?

You have them in the code I pasted. They should be in the the buffer
caps.

Yes, of course.  Thanks again for that.

In pygst for future searchers:

caps = last_frame.get_caps()[0]

width = int(caps["width"])

height = int(caps["height"])

 

<ATT00001..txt>

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Erik Blankinship-2]

> I am continuing to have trouble with this.  Most every combination I
> try ends up with a distorted image.

Distorted in what way? Too narrow/wide? Or every line shifted a bit to
the left/right of the previous line?

Here is the original video:

http://alumni.media.mit.edu/~erikb/tmp/city.ogv

Here is the code I use to extract a pixbuf:

        caps = last_frame.get_caps( )[0]

        width = int(caps["width"])

        height = int(caps["height"])

        r4 = width * 3

        r4 = (r4 + 3) & ~ 3

        pixBuf = gtk.gdk.pixbuf_new_from_data( last_frame.data, gtk.gdk.COLORSPACE_RGB, False, 8, width, height, r4 )

and here are the results with rowstride set to width * 2:

http://alumni.media.mit.edu/~erikb/tmp/city_width_x2.png

and with width * 3:

http://alumni.media.mit.edu/~erikb/tmp/city_width_x3.png

and with width * 4:

http://alumni.media.mit.edu/~erikb/tmp/city_width_x4.png

Hopefully some can suggest what the rowstride should be based on these images, or suggest how I would determine what it should be?

 

(Though it seems to me that just asking for an image/jpeg in the desired
width or height via the convert-frame action signal would be the easiest
way to achieve your goal)

Yes, this might be a good idea if I can't get the approach I am exploring above to work. 

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Erik Blankinship-2]

On Mon, Sep 19, 2011 at 9:06 PM, Erik Blankinship <[hidden email]> wrote:

> I am continuing to have trouble with this.  Most every combination I
> try ends up with a distorted image.

Distorted in what way? Too narrow/wide? Or every line shifted a bit to
the left/right of the previous line?

Here is the original video:

http://alumni.media.mit.edu/~erikb/tmp/city.ogv

Here is the code I use to extract a pixbuf:

        caps = last_frame.get_caps( )[0]

        width = int(caps["width"])

        height = int(caps["height"])

        r4 = width * 3

        r4 = (r4 + 3) & ~ 3

        pixBuf = gtk.gdk.pixbuf_new_from_data( last_frame.data, gtk.gdk.COLORSPACE_RGB, False, 8, width, height, r4 )

and here are the results with rowstride set to width * 2:

http://alumni.media.mit.edu/~erikb/tmp/city_width_x2.png

and with width * 3:

http://alumni.media.mit.edu/~erikb/tmp/city_width_x3.png

and with width * 4:

http://alumni.media.mit.edu/~erikb/tmp/city_width_x4.png

And, rowstride*4, with has_alpha set to True..

http://alumni.media.mit.edu/~erikb/tmp/city_width_x4_alpha.png 

which appears to have swapped the red and the blue channels, but at least looks right!

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Tim-Philipp Müller-2]

On Mon, 2011-09-19 at 21:06 -0400, Erik Blankinship wrote:

> Here is the code I use to extract a pixbuf:
>
>
>         caps = last_frame.get_caps( )[0]
>         width = int(caps["width"])
>         height = int(caps["height"])
>         r4 = width * 3
>         r4 = (r4 + 3) & ~ 3
>         pixBuf = gtk.gdk.pixbuf_new_from_data( last_frame.data,
> gtk.gdk.COLORSPACE_RGB, False, 8, width, height, r4 )
>
> and here are the results with rowstride set to width * 2: (..)
> and with width * 3: (...)
> and with width * 4: (...)
>
> Hopefully some can suggest what the rowstride should be based on these
> images, or suggest how I would determine what it should be?

Rowstrides are implicit, but there are helper function in libgstvideo
[1] which will tell you what the strides are.

*However*, what you're doing here isn't really going to work reliably.

If you get the last frame via the last-frame property, you have very
little control over what format it will be in. It may be RGB24, RGB32,
RGB15, BGR*, some YUV layout, AYUV, ARGB - pretty much anything (unless
you know exactly what sink is used and that sink only accepts exactly
one format).

It's best not to use the last-frame property at all.

If you want to get an RGB buffer, use the "convert-frame" action signal
to get an RGB buffer in the exact format you need for GdkPixbuf, like
the totem code I linked to does.

 Cheers
  -Tim


[1]
http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-libs/html/gst-plugins-base-libs-gstvideo.html

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel