Can the identity plugin cause frame drop?

[pisymbol .]

Is this true?

So if I have a source like this:

nvcamerasrc ! identity name=tap ! magicunicorns ! ...

Can tap's "handoff" callback cause frame drops in the pipe/source? Or is it that the "handoff" function won't see every frame no matter what I do? I'm talking fixed 30fps here which isn't THAT fast.

I thought "identity" was the safest way to tape the pipe (I need to get all the buffers to extra some metadata out of each one etc.).

If it can cause a drop, should I try to use tee/queue idiom to see all the buffers?

Thanks!

-aps

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Michael Gruner]

The handoff will be emitted for every buffer and won’t drop buffers. Recall, however, that signals block. I suspect that you have an indirect buffer drop on nvcamerasrc.

Nvcamera src has an internal (limited) buffer queue. If you block the processing on the signal callback for too long, the queue will start filling up and, eventually, drop buffers. I believe that you can increase this queue size a lot, but if your signal processing takes more than 33ms (assuming a 30fps capture), eventually you’ll end up losing buffers in the same way.

Make sure the processing your are doing on the callback actually is below your period deadline. If it’s below, but high, I recommend adding a queue after the identity to split processing in different threads. 

Michael

www.ridgerun.com

On Jul 8, 2019, at 3:40 PM, pisymbol . <[hidden email]> wrote:

Is this true?

So if I have a source like this:

nvcamerasrc ! identity name=tap ! magicunicorns ! ...

Can tap's "handoff" callback cause frame drops in the pipe/source? Or is it that the "handoff" function won't see every frame no matter what I do? I'm talking fixed 30fps here which isn't THAT fast.

I thought "identity" was the safest way to tape the pipe (I need to get all the buffers to extra some metadata out of each one etc.).

If it can cause a drop, should I try to use tee/queue idiom to see all the buffers?

Thanks!

-aps

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[pisymbol .]

On Mon, Jul 8, 2019 at 8:47 PM Michael Gruner <[hidden email]> wrote:

The handoff will be emitted for every buffer and won’t drop buffers. Recall, however, that signals block. I suspect that you have an indirect buffer drop on nvcamerasrc.

Nvcamera src has an internal (limited) buffer queue. If you block the processing on the signal callback for too long, the queue will start filling up and, eventually, drop buffers. I believe that you can increase this queue size a lot, but if your signal processing takes more than 33ms (assuming a 30fps capture), eventually you’ll end up losing buffers in the same way.

Make sure the processing your are doing on the callback actually is below your period deadline. If it’s below, but high, I recommend adding a queue after the identity to split processing in different threads. 

Michael, after the identity? Not before?

-aps

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[pisymbol .]

On Mon, Jul 8, 2019 at 9:07 PM pisymbol . <[hidden email]> wrote:

On Mon, Jul 8, 2019 at 8:47 PM Michael Gruner <[hidden email]> wrote:

The handoff will be emitted for every buffer and won’t drop buffers. Recall, however, that signals block. I suspect that you have an indirect buffer drop on nvcamerasrc.

Nvcamera src has an internal (limited) buffer queue. If you block the processing on the signal callback for too long, the queue will start filling up and, eventually, drop buffers. I believe that you can increase this queue size a lot, but if your signal processing takes more than 33ms (assuming a 30fps capture), eventually you’ll end up losing buffers in the same way.

Make sure the processing your are doing on the callback actually is below your period deadline. If it’s below, but high, I recommend adding a queue after the identity to split processing in different threads. 

Michael, after the identity? Not before?

Just to clarify: I'm seeing a drop I guess in nvcamerasrc since my identity signal callback ("handoff") is dumping the frame numbers per buf and that's where I see what appears to be drops/gaps between them.

-aps

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[pisymbol .]

On Mon, Jul 8, 2019 at 9:15 PM pisymbol . <[hidden email]> wrote:

On Mon, Jul 8, 2019 at 9:07 PM pisymbol . <[hidden email]> wrote:

On Mon, Jul 8, 2019 at 8:47 PM Michael Gruner <[hidden email]> wrote:

The handoff will be emitted for every buffer and won’t drop buffers. Recall, however, that signals block. I suspect that you have an indirect buffer drop on nvcamerasrc.

Nvcamera src has an internal (limited) buffer queue. If you block the processing on the signal callback for too long, the queue will start filling up and, eventually, drop buffers. I believe that you can increase this queue size a lot, but if your signal processing takes more than 33ms (assuming a 30fps capture), eventually you’ll end up losing buffers in the same way.

Make sure the processing your are doing on the callback actually is below your period deadline. If it’s below, but high, I recommend adding a queue after the identity to split processing in different threads. 

Michael, after the identity? Not before?

Just to clarify: I'm seeing a drop I guess in nvcamerasrc since my identity signal callback ("handoff") is dumping the frame numbers per buf and that's where I see what appears to be drops/gaps between them.

-aps

Note that:

nvcamerasrc queue-size=50 ... ! queue max-size-buffers=2000 ! identity name=tap ! ...

still results in drops! My avg runtime per signal callback is ~40-50ms right now (that includes the overhead of calling time.time() twice and appending it to a list).

-aps

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[pisymbol .]

On Mon, Jul 8, 2019 at 9:56 PM pisymbol . <[hidden email]> wrote:

On Mon, Jul 8, 2019 at 9:15 PM pisymbol . <[hidden email]> wrote:

On Mon, Jul 8, 2019 at 9:07 PM pisymbol . <[hidden email]> wrote:

On Mon, Jul 8, 2019 at 8:47 PM Michael Gruner <[hidden email]> wrote:

The handoff will be emitted for every buffer and won’t drop buffers. Recall, however, that signals block. I suspect that you have an indirect buffer drop on nvcamerasrc.

Nvcamera src has an internal (limited) buffer queue. If you block the processing on the signal callback for too long, the queue will start filling up and, eventually, drop buffers. I believe that you can increase this queue size a lot, but if your signal processing takes more than 33ms (assuming a 30fps capture), eventually you’ll end up losing buffers in the same way.

Make sure the processing your are doing on the callback actually is below your period deadline. If it’s below, but high, I recommend adding a queue after the identity to split processing in different threads. 

Michael, after the identity? Not before?

Just to clarify: I'm seeing a drop I guess in nvcamerasrc since my identity signal callback ("handoff") is dumping the frame numbers per buf and that's where I see what appears to be drops/gaps between them.

-aps

Note that:

nvcamerasrc queue-size=50 ... ! queue max-size-buffers=2000 ! identity name=tap ! ...

still results in drops! My avg runtime per signal callback is ~40-50ms right now (that includes the overhead of calling time.time() twice and appending it to a list).

Well, I kinda take that back. Here is the total number of frames I collect in my signal callback on a 10s basis (max-file-duration=10s):

259
300
302
299
299
300
298

That's the number of frames I see at 30fps at 10s intervals. Looks pretty good actually. What's throwing me off is that the nvcamerasrc metadata is not specifying the frame numbers as 0,1,2,3 order, i.e. sometimes it skips a frame_num which I interpret as a drop? Maybe it isn't?

But the number of frames looks fairly consistent and within range.

-aps

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[pisymbol .]

On Mon, Jul 8, 2019 at 8:47 PM Michael Gruner <[hidden email]> wrote:

The handoff will be emitted for every buffer and won’t drop buffers. Recall, however, that signals block. I suspect that you have an indirect buffer drop on nvcamerasrc.

Nvcamera src has an internal (limited) buffer queue. If you block the processing on the signal callback for too long, the queue will start filling up and, eventually, drop buffers. I believe that you can increase this queue size a lot, but if your signal processing takes more than 33ms (assuming a 30fps capture), eventually you’ll end up losing buffers in the same way.

Make sure the processing your are doing on the callback actually is below your period deadline. If it’s below, but high, I recommend adding a queue after the identity to split processing in different threads. 

Michael, I am seeing duplicate frames which I suspect is the in-drect frame drop issue you spoke about above. Here is my exact pipeline:

matroskamux name=muxer streamable=true ! multifilesink name=mfsink post-messages=true next-file=5 max-file-duration=\"{}\" location=\"{}\"
nvcamerasrc do-timestamp=true enable-meta=true fpsRange="30.0 30.0" auto-exposure=1 aeLock=true sensor_id=0 name=nvcam0 ! identity name=tap0
! video/x-raw(memory:NVMM), width=(int)4104, height=(int)3046, format=(string)I420, framerate=(fraction)30/1
! nvvidconv flip-method=2 ! video/x-raw(memory:NVMM), width=(int)3840, height=(int)2160, format=(string)I420
! omxh264enc bitrate=30000000 ! video/x-h264, stream-format=(string)byte-stream ! h264parse ! muxer.video_0
nvcamerasrc do-timestamp=true enable-meta=true fpsRange="30.0 30.0" auto-exposure=1 aeLock=true sensor_id=1 name=nvcam1 ! identity name=tap1
! video/x-raw(memory:NVMM), width=(int)4104, height=(int)3046, format=(string)I420, framerate=(fraction)30/1
! nvvidconv flip-method=2 ! video/x-raw(memory:NVMM), width=(int)3840, height=(int)2160, format=(string)I420
! omxh264enc bitrate=30000000 ! video/x-h264, stream-format=(string)byte-stream ! h264parse ! muxer.video_1

where max-file-duration is 60seconds and location is calculated at run-time.

Both tap0 and tap1's "handoff" callback take almost nothing in terms of time:

2019-07-29 15:22:53,224 INFO: 0: 0.011405869294668882s
2019-07-29 15:22:53,224 INFO: 1: 0.010900382847748986s

So that's ~11ms. That's WAY before the 33ms deadline. (this is expected since the callback just copies in some memory, pointer derefs essentially, and returns).

Yet I still see duplicate frames when I frame it out (a bit of chop during motion).

I tried adding a "queue" after "nvcamerasrc" and that did nothing which even has me more confuzzled?

-aps

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel