Possible buffering issue

[dayunbao]

Hi everyone,

Here's the pipeline I'm using:

v4l2src -> videorate -> queue -> identity -> jpegenc -> queue -> jifmux -> multifilesink

The pipeline is taking a picture once a second, and then does the following two things:

1) when the identity's handoff signal fires, it sends the data to another process (via a socket) for some processing of the image, and

2) also writes the images to disk as jpegs.

The images that are written to disk are fine, but there seems to be an issue with the data being sent over the socket.  I get images like the one below quite often.  It seems that I'm not always getting all the data from the buffer (even though I'm verifying that the proper amount of data is being sent and received on both ends of the socket), although I'm not totally sure that's the issue.  I'm using OpenCV to display the data sent over the socket as an image, by the way.  In case you're wondering, what you're seeing in the top of the picture are two comics tacked to a wall.  The colors are all out of wack, but I'm not too worried about that right now.

I realize there are a lot of places where things could go wrong in my program, but I'm trying to narrow them down.  Is there something I can do with my pipeline to make sure I'm getting all the data, and that it  isn't being overwritten as I'm trying to read it?  I recently added the first queue in the pipeline, thinking that maybe that would help, but it didn't.

Thanks!

[Nicolas Dufresne-5]

Le mardi 06 juin 2017 à 10:28 -0700, dayunbao a écrit :

> Here's the pipeline I'm using:
>
> v4l2src -> videorate -> queue -> identity -> jpegenc -> queue -> jifmux ->
> multifilesink
>
> The pipeline is taking a picture once a second, and then does the following
> two things:
>
> 1) when the identity's handoff signal fires, it sends the data to another
> process (via a socket) for some processing of the image, and 
>
> 2) also writes the images to disk as jpegs. 
>
> The images that are written to disk are fine, but there seems to be an issue
> with the data being sent over the socket.  I get images like the one below
> quite often.  It seems that I'm not always getting all the data from the
> buffer (even though I'm verifying that the proper amount of data is being
> sent and received on both ends of the socket), although I'm not totally sure
> that's the issue.  I'm using OpenCV to display the data sent over the socket
> as an image, by the way.  In case you're wondering, what you're seeing in
> the top of the picture are two comics tacked to a wall.  The colors are all
> out of wack, but I'm not too worried about that right now.
>
> I realize there are a lot of places where things could go wrong in my
> program, but I'm trying to narrow them down.  Is there something I can do
> with my pipeline to make sure I'm getting all the data, and that it  isn't
> being overwritten as I'm trying to read it?  I recently added the first

To be honest, drivers should never produce a frame like this, or at
least should mark it as corrupted. Unless of course it's a bug in
libv4l2 (that you can try by rebuilding without that).

As of today, you only last option would be to add a tee between v4l2src
and the queue. This will break the allocation query, and will ensure
v4l2src keep a certain amount of buffers in the driver and copy as
needed.

Nicolas
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[dayunbao]

Thanks for the prompt reply!  The jpegs that are saved to disk further down the pipeline are all fine.  Would that indicate a problem somewhere in my code outside of Gstreamer?

Here's what I'm doing inside the function that's called when the identity's handoff signal is fired (keep in mind this is Python):

memory = Gst.Buffer.get_all_memory(buf)

(result, info) = Gst.memory.map(memory, Gst.MapFlahs.READ)

if result:

   set up sockets...

   sentTotal = 0

   while sentTotal < info.size:

      sent = self.connection.send(info.data)

      sentTotal += sent

   Gst.Memory.unmap(memory, info)

   close socket...

The process that's receiving the data, which is written in C++, is displaying the data as images with the following code:

read data over socket into buffer called data...

Mat bayer(720, 1280, CV_16U, data);

Mat converted;

cvtColor(bayer, converted, COLOR_BayerRG2RGB);

imshow("Socket Image", converted);

waitKey(0);

On Tue, Jun 6, 2017 at 11:46 AM, Nicolas Dufresne <[hidden email]> wrote:

Le mardi 06 juin 2017 à 10:28 -0700, dayunbao a écrit :
> Here's the pipeline I'm using:
>
> v4l2src -> videorate -> queue -> identity -> jpegenc -> queue -> jifmux ->
> multifilesink
>
> The pipeline is taking a picture once a second, and then does the following
> two things:
>
> 1) when the identity's handoff signal fires, it sends the data to another
> process (via a socket) for some processing of the image, and 
>
> 2) also writes the images to disk as jpegs. 
>
> The images that are written to disk are fine, but there seems to be an issue
> with the data being sent over the socket.  I get images like the one below
> quite often.  It seems that I'm not always getting all the data from the
> buffer (even though I'm verifying that the proper amount of data is being
> sent and received on both ends of the socket), although I'm not totally sure
> that's the issue.  I'm using OpenCV to display the data sent over the socket
> as an image, by the way.  In case you're wondering, what you're seeing in
> the top of the picture are two comics tacked to a wall.  The colors are all
> out of wack, but I'm not too worried about that right now.
>
> I realize there are a lot of places where things could go wrong in my
> program, but I'm trying to narrow them down.  Is there something I can do
> with my pipeline to make sure I'm getting all the data, and that it  isn't
> being overwritten as I'm trying to read it?  I recently added the first

To be honest, drivers should never produce a frame like this, or at
least should mark it as corrupted. Unless of course it's a bug in
libv4l2 (that you can try by rebuilding without that).

As of today, you only last option would be to add a tee between v4l2src
and the queue. This will break the allocation query, and will ensure
v4l2src keep a certain amount of buffers in the driver and copy as
needed.

Nicolas
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[dayunbao]

I'm hoping someone has some thoughts on the code in my last post.  Is there a better way to get the data from the buffer, and then pass it along?  Inside the function called when the identity's handoff signal is fired I added the following lines of code to rule out the possibility that the data I'm reading is being partially overwritten while trying to read it:

newBuf = Gst.Buffer.copy_deep(buf)

memory = Gst.Buffer.get_all_memory(newBuf)

Then I send the data over a socket in the same way as in my last post.

Is it possible I'm reading data from memory I shouldn't?  By which I mean should I be using the identity plugin's drop-buffer-flags property, and dropping buffers with the CORRUPTED flags, for example?  Or would I be better off using a tee and then an appsink plugin, as was recommended as an alternative solution in a different post?  Would that give me markedly different results?

Thanks!

On Tue, Jun 6, 2017 at 12:48 PM, yun bao <[hidden email]> wrote:

Thanks for the prompt reply!  The jpegs that are saved to disk further down the pipeline are all fine.  Would that indicate a problem somewhere in my code outside of Gstreamer?

Here's what I'm doing inside the function that's called when the identity's handoff signal is fired (keep in mind this is Python):

memory = Gst.Buffer.get_all_memory(buf)

(result, info) = Gst.memory.map(memory, Gst.MapFlahs.READ)

if result:

   set up sockets...

   sentTotal = 0

   while sentTotal < info.size:

      sent = self.connection.send(info.data)

      sentTotal += sent

   Gst.Memory.unmap(memory, info)

   close socket...

The process that's receiving the data, which is written in C++, is displaying the data as images with the following code:

read data over socket into buffer called data...

Mat bayer(720, 1280, CV_16U, data);

Mat converted;

cvtColor(bayer, converted, COLOR_BayerRG2RGB);

imshow("Socket Image", converted);

waitKey(0);

On Tue, Jun 6, 2017 at 11:46 AM, Nicolas Dufresne <[hidden email]> wrote:

Le mardi 06 juin 2017 à 10:28 -0700, dayunbao a écrit :
> Here's the pipeline I'm using:
>
> v4l2src -> videorate -> queue -> identity -> jpegenc -> queue -> jifmux ->
> multifilesink
>
> The pipeline is taking a picture once a second, and then does the following
> two things:
>
> 1) when the identity's handoff signal fires, it sends the data to another
> process (via a socket) for some processing of the image, and 
>
> 2) also writes the images to disk as jpegs. 
>
> The images that are written to disk are fine, but there seems to be an issue
> with the data being sent over the socket.  I get images like the one below
> quite often.  It seems that I'm not always getting all the data from the
> buffer (even though I'm verifying that the proper amount of data is being
> sent and received on both ends of the socket), although I'm not totally sure
> that's the issue.  I'm using OpenCV to display the data sent over the socket
> as an image, by the way.  In case you're wondering, what you're seeing in
> the top of the picture are two comics tacked to a wall.  The colors are all
> out of wack, but I'm not too worried about that right now.
>
> I realize there are a lot of places where things could go wrong in my
> program, but I'm trying to narrow them down.  Is there something I can do
> with my pipeline to make sure I'm getting all the data, and that it  isn't
> being overwritten as I'm trying to read it?  I recently added the first

To be honest, drivers should never produce a frame like this, or at
least should mark it as corrupted. Unless of course it's a bug in
libv4l2 (that you can try by rebuilding without that).

As of today, you only last option would be to add a tee between v4l2src
and the queue. This will break the allocation query, and will ensure
v4l2src keep a certain amount of buffers in the driver and copy as
needed.

Nicolas

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel