OpenCv Mat to GstBuffer -> appsrc, possible buffer underflow?

[borisRobson]

Hi all

I have a project which aims to de-warp a fisheye camera, using two gstreamer pipelines with opencv processing in between:
|   input-pipe                              |   OpenCV processing           |  output-pipe
|  rtsprc -> decodebin -> appsink  |   gst-buf -> cvMat -> gstbuf |  appsrc -> videoparse -> videoconvert ->   xvimagesink

I can save the image from the cvMat and the result is as I expect, however the image shown in xvimagesink is grayscale and constantly shifting upwards and to the left. (Best I could do on description there)




The code where I suspect the problem is:
(conv is the Mat shown in the attached image)
//get outputize & allocate mem block
        gint outSize = conv.cols * conv.rows * inBpp;
        GstMemory *outMem = gst_allocator_alloc(NULL, outSize, NULL);

        //assign frame data to mem
        GstMapInfo outMap;
        if(!gst_memory_map(outMem, &outMap, GST_MAP_WRITE))
        {
                g_printerr("could not map outMem\n");
                return;
        }

        memcpy(outMap.data, conv.data, outMap.size);

        gst_memory_unmap(outMem, &outMap);

        GstBuffer *outbuf;
        outbuf = gst_buffer_new();

        gst_buffer_insert_memory(outbuf, -1,outMem);

        GstVideoMeta *outmeta = new GstVideoMeta();
        outmeta = gst_buffer_add_video_meta(outbuf,GST_VIDEO_FRAME_FLAG_NONE, GST_VIDEO_FORMAT_RGB, conv.cols, conv.rows );

        //set timestamp values
        GST_BUFFER_DURATION(outbuf) = (GstClockTime)40000000;
        GST_BUFFER_DTS(outbuf) = GST_CLOCK_TIME_NONE;
        GST_BUFFER_PTS(outbuf) = timestamp;
        timestamp += GST_BUFFER_DURATION(outbuf);

        GST_BUFFER_FLAG_SET(outbuf, GST_BUFFER_FLAG_LIVE);

        //ref buffer to give copy to appsrc
        gst_buffer_ref(outbuf);

        GstFlowReturn ret;
        ret = gst_app_src_push_buffer((GstAppSrc*)asrc, outbuf);
        if(ret != GST_FLOW_OK)
        {
                g_printerr("could not push buffer\n");
                g_printerr("ret enum: %i\n", ret);
        }

        //dec. ref count so that we can edit data on next run
        gst_buffer_unref(outbuf);


I have tried adding extra conversion stages in my out-pipe but to no effect.
Has anyone had experience with this before or is able to spot some rookie error I'm making?
Thanks
Brandon


                       

[Mailing List SVR]

Hi,

you can use the gstreamer dewarp plugin available in git master

https://cgit.freedesktop.org/gstreamer/gst-plugins-bad/tree/ext/opencv/gstdewarp.cpp

please report here if you have any issues with your fisheye image,

Nicola


Il 03/02/2017 14:06, borisRobson ha scritto:

> Hi all
>
> I have a project which aims to de-warp a fisheye camera, using two gstreamer
> pipelines with opencv processing in between:
> |   input-pipe                              |   OpenCV processing
> |  output-pipe
> |  rtsprc -> decodebin -> appsink  |   gst-buf -> cvMat -> gstbuf |  appsrc
> -> videoparse -> videoconvert ->
> xvimagesink
>
> I can save the image from the cvMat and the result is as I expect, however
> the image shown in xvimagesink is grayscale and constantly shifting upwards
> and to the left. (Best I could do on description there)
>
> <http://gstreamer-devel.966125.n4.nabble.com/file/n4681701/xvimagesink.png>
> <http://gstreamer-devel.966125.n4.nabble.com/file/n4681701/dewarped.png>
>
> The code where I suspect the problem is:
> (conv is the Mat shown in the attached image)
> //get outputize & allocate mem block
> gint outSize = conv.cols * conv.rows * inBpp;
> GstMemory *outMem = gst_allocator_alloc(NULL, outSize, NULL);
>
> //assign frame data to mem
> GstMapInfo outMap;
> if(!gst_memory_map(outMem, &outMap, GST_MAP_WRITE))
> {
> g_printerr("could not map outMem\n");
> return;
> }
>
> memcpy(outMap.data, conv.data, outMap.size);
>
> gst_memory_unmap(outMem, &outMap);
>
> GstBuffer *outbuf;
> outbuf = gst_buffer_new();
>
> gst_buffer_insert_memory(outbuf, -1,outMem);
>
> GstVideoMeta *outmeta = new GstVideoMeta();
> outmeta = gst_buffer_add_video_meta(outbuf,GST_VIDEO_FRAME_FLAG_NONE,
> GST_VIDEO_FORMAT_RGB, conv.cols, conv.rows );
>
> //set timestamp values
> GST_BUFFER_DURATION(outbuf) = (GstClockTime)40000000;
> GST_BUFFER_DTS(outbuf) = GST_CLOCK_TIME_NONE;
> GST_BUFFER_PTS(outbuf) = timestamp;
> timestamp += GST_BUFFER_DURATION(outbuf);
>
> GST_BUFFER_FLAG_SET(outbuf, GST_BUFFER_FLAG_LIVE);
>
> //ref buffer to give copy to appsrc
> gst_buffer_ref(outbuf);
>
> GstFlowReturn ret;
> ret = gst_app_src_push_buffer((GstAppSrc*)asrc, outbuf);
> if(ret != GST_FLOW_OK)
> {
> g_printerr("could not push buffer\n");
> g_printerr("ret enum: %i\n", ret);
> }
>
> //dec. ref count so that we can edit data on next run
> gst_buffer_unref(outbuf);
>
>
> I have tried adding extra conversion stages in my out-pipe but to no effect.
> Has anyone had experience with this before or is able to spot some rookie
> error I'm making?
> Thanks
> Brandon
>
>
>                          
>
>
>
>
> --
> View this message in context: http://gstreamer-devel.966125.n4.nabble.com/OpenCv-Mat-to-GstBuffer-appsrc-possible-buffer-underflow-tp4681701.html
> Sent from the GStreamer-devel mailing list archive at Nabble.com.
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[borisRobson]

Hi Nicola,

I'm currently limited to v1.2.4 unfortunately. I came across that plugin yesterday and thought I'd hit the jackpot until I tried to compile it.

Thanks for the quick reply though

[filnet]

Hi,

OpenCV will often convert images to grey scale before applying some algorithm on it.

The shifting is probablu due to a stride issue.

Did you look at the various opencv plugins in gst-plugins-bad ?

It has all the code samples you need to convert cvMat to GstBuffer.

I am working on a OpenCV based camera calibration/undistort plugin for gstreamer.

It is wip but can be found here : filnet / LibrePilot / source / ground / gcs / src / libs / gstreamer / plugins / cameracalibration — Bitbucket

While working on that plugin, I also struggled with gstreamer vs opencv image formats.

What I understood is captured here : https://cgit.freedesktop.org/gstreamer/gst-plugins-bad/tree/gst-libs/gst/opencv/gstopencvutils.cpp

Le Vendredi 3 février 2017 14h09, borisRobson <[hidden email]> a écrit :

Hi all

I have a project which aims to de-warp a fisheye camera, using two gstreamer
pipelines with opencv processing in between:
|  input-pipe                              |  OpenCV processing         
|  output-pipe
|  rtsprc -> decodebin -> appsink  |  gst-buf -> cvMat -> gstbuf |  appsrc
-> videoparse -> videoconvert ->                                                                                               
xvimagesink

I can save the image from the cvMat and the result is as I expect, however
the image shown in xvimagesink is grayscale and constantly shifting upwards
and to the left. (Best I could do on description there)

<http://gstreamer-devel.966125.n4.nabble.com/file/n4681701/xvimagesink.png>
<http://gstreamer-devel.966125.n4.nabble.com/file/n4681701/dewarped.png>

The code where I suspect the problem is:
(conv is the Mat shown in the attached image)
//get outputize & allocate mem block
    gint outSize = conv.cols * conv.rows * inBpp;
    GstMemory *outMem = gst_allocator_alloc(NULL, outSize, NULL);   

    //assign frame data to mem
    GstMapInfo outMap;
    if(!gst_memory_map(outMem, &outMap, GST_MAP_WRITE))
    {
        g_printerr("could not map outMem\n");
        return;
    }

    memcpy(outMap.data, conv.data, outMap.size);

    gst_memory_unmap(outMem, &outMap);

    GstBuffer *outbuf;
    outbuf = gst_buffer_new();

    gst_buffer_insert_memory(outbuf, -1,outMem);

    GstVideoMeta *outmeta = new GstVideoMeta();
    outmeta = gst_buffer_add_video_meta(outbuf,GST_VIDEO_FRAME_FLAG_NONE,
GST_VIDEO_FORMAT_RGB, conv.cols, conv.rows );

    //set timestamp values
    GST_BUFFER_DURATION(outbuf) = (GstClockTime)40000000;
    GST_BUFFER_DTS(outbuf) = GST_CLOCK_TIME_NONE;
    GST_BUFFER_PTS(outbuf) = timestamp;
    timestamp += GST_BUFFER_DURATION(outbuf);

    GST_BUFFER_FLAG_SET(outbuf, GST_BUFFER_FLAG_LIVE);

    //ref buffer to give copy to appsrc
    gst_buffer_ref(outbuf);

    GstFlowReturn ret;
    ret = gst_app_src_push_buffer((GstAppSrc*)asrc, outbuf);
    if(ret != GST_FLOW_OK)
    {
        g_printerr("could not push buffer\n");
        g_printerr("ret enum: %i\n", ret);
    }

    //dec. ref count so that we can edit data on next run
    gst_buffer_unref(outbuf);


I have tried adding extra conversion stages in my out-pipe but to no effect.
Has anyone had experience with this before or is able to spot some rookie
error I'm making?
Thanks
Brandon


                       




--
View this message in context: http://gstreamer-devel.966125.n4.nabble.com/OpenCv-Mat-to-GstBuffer-appsrc-possible-buffer-underflow-tp4681701.html
Sent from the GStreamer-devel mailing list archive at Nabble.com.
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[borisRobson]

Just had a look into stride (I'm new to this level of detail of videoframes) and managed to fix my app with one line of code!

Thanks

[Mikl]

Hello,

Can you give more information about your solution?

What line?

Where?

For future readers. :)

Mikl

---

From: gstreamer-devel <[hidden email]> on behalf of borisRobson <[hidden email]>
Sent: Monday, February 6, 2017 9:36:30 AM
To: [hidden email]
Subject: Re: OpenCv Mat to GstBuffer -> appsrc, possible buffer underflow?

 

Just had a look into stride (I'm new to this level of detail of videoframes)
and managed to fix my app with one line of code!

Thanks



--
View this message in context: http://gstreamer-devel.966125.n4.nabble.com/OpenCv-Mat-to-GstBuffer-appsrc-possible-buffer-underflow-tp4681701p4681740.html
Sent from the GStreamer-devel mailing list archive at Nabble.com.
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[borisRobson]

I can't stand when people don't post their solutions, didn't even realise I forgot to!

I create two arrays initialised to 0 :
        offset = new gsize[4] {0};
        stride = new gint[4] {0};

Then later when creating the buffer:
        GstVideoMeta *outmeta = new GstVideoMeta();

        ostrm->stride[0] = conv.cols * inBpp;
        outmeta = gst_buffer_add_video_meta_full(outbuf,GST_VIDEO_FRAME_FLAG_NONE, GST_VIDEO_FORMAT_RGB, conv.cols, conv.rows, 1, ostrm->offset, ostrm->stride );

Set the stride value to image width * bytes per plane.

More info on stride:
https://msdn.microsoft.com/en-us/library/windows/desktop/aa473780(v=vs.85).aspx

Hope this helps someone eventually