Can muxers/multifilesink drop frames?

[pisymbol .]

I have a pipeline that looks like this:

matroskamux name=muxer streamable=true ! multifilesink max-file-duration=60s  locaiton=capture_%08d.mkv

.. ! h264parse ! identity ! muxer.video_0

(I believe this is the only pertinent snippet needed for this discussion)

And each frame is timestamped in my identity handoff callback via essentially gettimeofday()).

But occasionally we seem to be seeing that the timestamps in the file I am writing out don't align with the frames in the final stream. Barring obviously flushing errors of the data, shouldn't the timestamps of each frame parsed by the h264parse correlate with my timestamp file. The frames are also numbered monotonically increasing (1, 2, 3, 4, ...) and I have been told that identity doesn't drop frames so this has me confuzzled.

Is there a fundamental assumption I am making that might lead to the problem I am describing?

-aps

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[pisymbol .]

On Mon, Sep 23, 2019 at 11:40 AM pisymbol . <[hidden email]> wrote:

I have a pipeline that looks like this:

matroskamux name=muxer streamable=true ! multifilesink max-file-duration=60s  locaiton=capture_%08d.mkv

.. ! h264parse ! identity ! muxer.video_0

(I believe this is the only pertinent snippet needed for this discussion)

And each frame is timestamped in my identity handoff callback via essentially gettimeofday()).

But occasionally we seem to be seeing that the timestamps in the file I am writing out don't align with the frames in the final stream. Barring obviously flushing errors of the data, shouldn't the timestamps of each frame parsed by the h264parse correlate with my timestamp file. The frames are also numbered monotonically increasing (1, 2, 3, 4, ...) and I have been told that identity doesn't drop frames so this has me confuzzled.

Is there a fundamental assumption I am making that might lead to the problem I am describing?

Let me re-ask the question given the overwhelming response:

Should I expect every single frame the identity handoff callback processes to wound up in the stream?

Again, I'm talking about a pipeline that ends with something like this:

matroskamux name=muxer streamable=true ! multifilesink max-file-duration=60s  locaiton=capture_%08d.mkv

... ! h264parse ! identity ! muxer.video_0

... ! h264parse ! identity ! muxer.video_1

So should I expect the total number of frames in the MKV for stream 0 and 1 equal to the number of times "handoff" is called? I think the answer is YES, right?

-aps

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Tim Müller]

On Wed, 2019-09-25 at 09:18 -0400, pisymbol . wrote:

Hi,

Should I expect every single frame the identity handoff callback processes to wound up in the stream?

Again, I'm talking about a pipeline that ends with something like this:

matroskamux name=muxer streamable=true ! multifilesink max-file-duration=60s  locaiton=capture_%08d.mkv

... ! h264parse ! identity ! muxer.video_0

... ! h264parse ! identity ! muxer.video_1

So should I expect the total number of frames in the MKV for stream 0 and 1 equal to the number of times "handoff" is called? I think the answer is YES, right?

Nothing in this pipeline should be dropping data (assuming data is well-formed and well-timestamped etc.).

However, you have limited control over how the files get split in this scenario with multifilesink.

Have you tried splitmuxsink instead? (git master supports multiple video streams).

It might behave slightly better (although with two video streams there are limits what it can do, since keyframes probably won't exactly align between the two video streams).

Cheers

Tim

-- 

Tim Müller, Centricular Ltd - http://www.centricular.com

Join us at the GStreamer Conference! 31 Oct-1 Nov 2019 in Lyon, France

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[pisymbol .]

On Wed, Sep 25, 2019 at 9:42 AM Tim Müller <[hidden email]> wrote:

On Wed, 2019-09-25 at 09:18 -0400, pisymbol . wrote:

Hi,

Should I expect every single frame the identity handoff callback processes to wound up in the stream?

Again, I'm talking about a pipeline that ends with something like this:

matroskamux name=muxer streamable=true ! multifilesink max-file-duration=60s  locaiton=capture_%08d.mkv

... ! h264parse ! identity ! muxer.video_0

... ! h264parse ! identity ! muxer.video_1

So should I expect the total number of frames in the MKV for stream 0 and 1 equal to the number of times "handoff" is called? I think the answer is YES, right?

Nothing in this pipeline should be dropping data (assuming data is well-formed and well-timestamped etc.).

However, you have limited control over how the files get split in this scenario with multifilesink.

Have you tried splitmuxsink instead? (git master supports multiple video streams).

It might behave slightly better (although with two video streams there are limits what it can do, since keyframes probably won't exactly align between the two video streams).

Hey Tim, thanks. Yeah, that was the problem with splitmuxsink. I wanted to use it for that EXACT reason you mentioned above. But I need master which complicates distribution.

I realize keyframes won't align between both streams, but I assume if I concatenate all the 60s timestamp files that should represent every frame in the stream? Right? :-)

-aps

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Tim Müller]

On Wed, 2019-09-25 at 10:08 -0400, pisymbol . wrote:

Yeah, that was the problem with splitmuxsink. I wanted to use it for that EXACT reason you mentioned above. But I need master which complicates distribution.

I realize keyframes won't align between both streams, but I assume if I concatenate all the 60s timestamp files that should represent every frame in the stream? Right? :-)

Mmh, not really/necessarily, you can't just concatenate Matroska files. It might work with splitmuxsrc however.

Cheers

Tim

-- 

Tim Müller, Centricular Ltd - http://www.centricular.com

Join us at the GStreamer Conference! 31 Oct-1 Nov 2019 in Lyon, France

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Tim Müller]

On Wed, 2019-09-25 at 15:28 +0100, Tim Müller wrote:

> Mmh, not really/necessarily, you can't just concatenate Matroska
> files. It might work with splitmuxsrc however.

Make that splitfilesrc.

--
Tim Müller, Centricular Ltd - http://www.centricular.com

Join us at the GStreamer Conference! 31 Oct-1 Nov 2019 in Lyon, France

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[pisymbol .]

On Wed, Sep 25, 2019 at 10:28 AM Tim Müller <[hidden email]> wrote:

On Wed, 2019-09-25 at 10:08 -0400, pisymbol . wrote:

Yeah, that was the problem with splitmuxsink. I wanted to use it for that EXACT reason you mentioned above. But I need master which complicates distribution.

I realize keyframes won't align between both streams, but I assume if I concatenate all the 60s timestamp files that should represent every frame in the stream? Right? :-)

Mmh, not really/necessarily, you can't just concatenate Matroska files. It might work with splitmuxsrc however.

No, what I mean is if I dump all the frames of every 60s file they should correlate to every single timestamp I wrote in a meta file for that frame (barring errors of course), i.e. the number of entries will be equal.

-aps

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[pisymbol .]

On Wed, Sep 25, 2019 at 10:29 AM Tim Müller <[hidden email]> wrote:

On Wed, 2019-09-25 at 15:28 +0100, Tim Müller wrote:

> Mmh, not really/necessarily, you can't just concatenate Matroska
> files. It might work with splitmuxsrc however.

Make that splitfilesrc.

So here is a concrete example of a 29+ minute recording:

$ for f in /mnt/storage/capture_*.mkv; do ffprobe -v fatal -count_frames -select_streams v:0 -show_entries stream=nb_read_frames -of default=nokey=1:noprint_wrappers=1 $f; done >> counts

$ cat counts | paste -sd+ - | bc

52375

That should be the total number of frames in all of the mkv captures of stream 0. The problem is if I count the number of times my handoff function is called for frames in stream 0, I get:

 

"total_frames_0": 52782

So I'm trying to figure out where did the extra 407 frames come from/go? Or am I counting the frames incorrectly above?

-aps

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[pisymbol .]

On Wed, Sep 25, 2019 at 11:31 AM pisymbol . <[hidden email]> wrote:

On Wed, Sep 25, 2019 at 10:29 AM Tim Müller <[hidden email]> wrote:

On Wed, 2019-09-25 at 15:28 +0100, Tim Müller wrote:

> Mmh, not really/necessarily, you can't just concatenate Matroska
> files. It might work with splitmuxsrc however.

Make that splitfilesrc.

So here is a concrete example of a 29+ minute recording:

$ for f in /mnt/storage/capture_*.mkv; do ffprobe -v fatal -count_frames -select_streams v:0 -show_entries stream=nb_read_frames -of default=nokey=1:noprint_wrappers=1 $f; done >> counts

$ cat counts | paste -sd+ - | bc

52375

That should be the total number of frames in all of the mkv captures of stream 0. The problem is if I count the number of times my handoff function is called for frames in stream 0, I get:

 

"total_frames_0": 52782

So I'm trying to figure out where did the extra 407 frames come from/go? Or am I counting the frames incorrectly above?

Question, is it possible that not all frames are being flushed out at the end by the muxer? Since I seem to always have more entries than frames in meta files, I am guessing my handoff function is being called but perhaps the frames aren't making it out to the filesystem via the muxer???

-aps

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[pisymbol .]

On Wed, Sep 25, 2019 at 12:06 PM pisymbol . <[hidden email]> wrote:

On Wed, Sep 25, 2019 at 11:31 AM pisymbol . <[hidden email]> wrote:

On Wed, Sep 25, 2019 at 10:29 AM Tim Müller <[hidden email]> wrote:

On Wed, 2019-09-25 at 15:28 +0100, Tim Müller wrote:

> Mmh, not really/necessarily, you can't just concatenate Matroska
> files. It might work with splitmuxsrc however.

Make that splitfilesrc.

So here is a concrete example of a 29+ minute recording:

$ for f in /mnt/storage/capture_*.mkv; do ffprobe -v fatal -count_frames -select_streams v:0 -show_entries stream=nb_read_frames -of default=nokey=1:noprint_wrappers=1 $f; done >> counts

$ cat counts | paste -sd+ - | bc

52375

That should be the total number of frames in all of the mkv captures of stream 0. The problem is if I count the number of times my handoff function is called for frames in stream 0, I get:

 

"total_frames_0": 52782

So I'm trying to figure out where did the extra 407 frames come from/go? Or am I counting the frames incorrectly above?

Question, is it possible that not all frames are being flushed out at the end by the muxer? Since I seem to always have more entries than frames in meta files, I am guessing my handoff function is being called but perhaps the frames aren't making it out to the filesystem via the muxer???

I can never get them to be exact and that is what has me concerned.

-aps

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[pisymbol .]

On Wed, Sep 25, 2019 at 1:16 PM pisymbol . <[hidden email]> wrote:

On Wed, Sep 25, 2019 at 12:06 PM pisymbol . <[hidden email]> wrote:

On Wed, Sep 25, 2019 at 11:31 AM pisymbol . <[hidden email]> wrote:

On Wed, Sep 25, 2019 at 10:29 AM Tim Müller <[hidden email]> wrote:

On Wed, 2019-09-25 at 15:28 +0100, Tim Müller wrote:

> Mmh, not really/necessarily, you can't just concatenate Matroska
> files. It might work with splitmuxsrc however.

Make that splitfilesrc.

So here is a concrete example of a 29+ minute recording:

$ for f in /mnt/storage/capture_*.mkv; do ffprobe -v fatal -count_frames -select_streams v:0 -show_entries stream=nb_read_frames -of default=nokey=1:noprint_wrappers=1 $f; done >> counts

$ cat counts | paste -sd+ - | bc

52375

That should be the total number of frames in all of the mkv captures of stream 0. The problem is if I count the number of times my handoff function is called for frames in stream 0, I get:

 

"total_frames_0": 52782

So I'm trying to figure out where did the extra 407 frames come from/go? Or am I counting the frames incorrectly above?

Question, is it possible that not all frames are being flushed out at the end by the muxer? Since I seem to always have more entries than frames in meta files, I am guessing my handoff function is being called but perhaps the frames aren't making it out to the filesystem via the muxer???

Another question: Is there a way to check the buffer's type? i.e. can I verify that each buffer passed to handoff is indeed an H.264 frame?

-aps

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel