GStreamer-devel

Decoded video frame timings

Classic

List

Threaded

3 messages Options

Darren Staples

Decoded video frame timings

Hi,

I'm decoding video frames (various file formats and codecs) and converting them into YUV420 format so that I can perform image processing on them.

And I have the commercial Fluendo codec pack installed.

The pipeline looks like this: uridecodebin -> queue -> ffmpegcolorspace -> tee -> queue -> appsink

YUV420 conversion is done in the ffmpegcolorspace component, and the other source pad of the tee is for a future feature.

This is built up dynamically in C, not on a command line.

My problem is that I need to extract the frame timings for each of the video frames being processed, and I'm seeing a large variance in this.

For example, one of my video files is a H.264 MP4, 82s long, 2057 frames and plays at 25 fps, so the average frame interval is 40ms.

When I recover a frame from the appsink component I use the gst_element_query_position() function to get the timing for that frame.

And what I'm seeing is that the timing is highly variable, even from one run to the next.

e.g. the time between frames for the first 32 frames for run 1:

0: 1020226us

1: 0us

2: 0us

3: 0us

4: 0us

5: 0us

6: 0us

7: 0us

8: 0us

9: 0us

10: 0us

11: 0us

12: 0us

13: 0us

14: 0us

15: 0us

16: 0us

17: 0us

18: 0us

19: 0us

20: 0us

21: 0us

22: 0us

23: 0us

24: 0us

25: 0us

26: 19774us

27: 40000us

28: 40000us

29: 1022947us

30: 46440us

31: 23220us

the first 32 frames for run 2:

0: 53151us

1: 0us

2: 26849us

3: 40000us

4: 40000us

5: 40000us

6: 40000us

7: 40000us

8: 40000us

9: 40000us

10: 40000us

11: 40000us

12: 40000us

13: 40000us

14: 40000us

15: 40000us

16: 40000us

17: 40000us

18: 40000us

19: 40000us

20: 1011428us

21: 46440us

22: 46440us

23: 23220us

24: 46440us

25: 46440us

26: 46440us

27: 23219us

28: 46440us

29: 46440us

30: 46440us

31: 23220us

Why do so many of the frames in the first run above have zero time delta between them ?

Is this a consequence of the file being in H.264/mp4 format ? And if, so is there an easy way to 'smooth' out the frame timings ?

Or would you suggest I do this 'manually' by approximating/averaging the timing based on the duration and no. of frames ?

Thanks !

Darren

------------------------------------------------------------------------------
Protect Your Site and Customers from Malware Attacks
Learn about various malware tactics and how to avoid them. Understand
malware threats, the impact they can have on your business, and how you
can protect your company and customers by using code signing.
http://p.sf.net/sfu/oracle-sfdevnl
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

praveen pandey-2

Re: Decoded video frame timings

There could be two possible reasons for this.
1) your stream might be a corrupted one
2) Any element in the pipeline is multifunctional (parser and/or decoder)

you can verify this using qtdemux and ffdec_h264

BR,
Praveen Pandey

On Wed, Jan 12, 2011 at 1:55 PM, Darren Staples <[hidden email]> wrote:

Hi,

I'm decoding video frames (various file formats and codecs) and converting them into YUV420 format so that I can perform image processing on them.
And I have the commercial Fluendo codec pack installed.

The pipeline looks like this: uridecodebin -> queue -> ffmpegcolorspace -> tee -> queue -> appsink
YUV420 conversion is done in the ffmpegcolorspace component, and the other source pad of the tee is for a future feature.

This is built up dynamically in C, not on a command line.

My problem is that I need to extract the frame timings for each of the video frames being processed, and I'm seeing a large variance in this.

For example, one of my video files is a H.264 MP4, 82s long, 2057 frames and plays at 25 fps, so the average frame interval is 40ms.
When I recover a frame from the appsink component I use the gst_element_query_position() function to get the timing for that frame.

And what I'm seeing is that the timing is highly variable, even from one run to the next.

e.g. the time between frames for the first 32 frames for run 1:
0: 1020226us

1: 0us
2: 0us
3: 0us
4: 0us
5: 0us
6: 0us
7: 0us
8: 0us
9: 0us
10: 0us
11: 0us
12: 0us
13: 0us

14: 0us
15: 0us
16: 0us
17: 0us
18: 0us
19: 0us
20: 0us
21: 0us
22: 0us
23: 0us
24: 0us
25: 0us
26: 19774us
27: 40000us
28: 40000us
29: 1022947us
30: 46440us
31: 23220us

the first 32 frames for run 2:
0: 53151us
1: 0us
2: 26849us
3: 40000us
4: 40000us
5: 40000us
6: 40000us
7: 40000us
8: 40000us
9: 40000us
10: 40000us
11: 40000us

12: 40000us
13: 40000us
14: 40000us
15: 40000us
16: 40000us
17: 40000us
18: 40000us
19: 40000us
20: 1011428us
21: 46440us

22: 46440us
23: 23220us
24: 46440us
25: 46440us
26: 46440us
27: 23219us
28: 46440us
29: 46440us
30: 46440us
31: 23220us

Why do so many of the frames in the first run above have zero time delta between them ?
Is this a consequence of the file being in H.264/mp4 format ? And if, so is there an easy way to 'smooth' out the frame timings ?

Or would you suggest I do this 'manually' by approximating/averaging the timing based on the duration and no. of frames ?

Thanks !
Darren

------------------------------------------------------------------------------
Protect Your Site and Customers from Malware Attacks
Learn about various malware tactics and how to avoid them. Understand
malware threats, the impact they can have on your business, and how you
can protect your company and customers by using code signing.
http://p.sf.net/sfu/oracle-sfdevnl
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

Tim-Philipp Müller-2

Re: Decoded video frame timings

In reply to this post by Darren Staples

On Wed, 2011-01-12 at 08:25 +0000, Darren Staples wrote:

Hi,

(is this a double-post?)

> I'm decoding video frames (various file formats and codecs) and
> converting them into YUV420 format so that I can perform image
> processing on them. (..)
> My problem is that I need to extract the frame timings (..),
> and I'm seeing a large variance in this. (..)
> When I recover a frame from the appsink component I use the
> gst_element_query_position() function to get the timing for that
> frame.

Try looking at the GST_BUFFER_TIMESTAMP (buffer) instead.

Cheers
-Tim

------------------------------------------------------------------------------
Protect Your Site and Customers from Malware Attacks
Learn about various malware tactics and how to avoid them. Understand
malware threats, the impact they can have on your business, and how you
can protect your company and customers by using code signing.
http://p.sf.net/sfu/oracle-sfdevnl
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel