GStreamer-devel

How to control pipeline frame rate when my camera needs some time to get one frame data

Classic

List

Threaded

8 messages Options

Zhao, Halley

How to control pipeline frame rate when my camera needs some time to get one frame data

Hi all:

My camera needs ~20ms to get one frame of data, and a video sink in the pipeline needs ~4ms to render it.

But when I set pipeline fps to 35, I only got 20 fps (calculate by GST CLOCK). CPU is not busy at this time, only 60%.

I guess the reason is following:

When we got nth frame at time 0, gst framework schedule to get next frame after 30ms (1000/35), and when the camera src element got scheduled to create one frame, it needs 20ms to get the data.

Then the time between n frame and n+1 frame is 30+20=50ms, so I just got 20 fps.

Is my assumption right?

If yes, is there any way to adjust it?

I had though to create another thread inside camera source, and prepare the frame data in a buffer before it is schedule.

However, I think it is hack, and make timestamp differ. I think it’s better settled inside gst framework.

ZHAO, Halley (Aihua)

Email: halley.zhao<a href="BLOCKED::mailto:aihua.zhao@intel.com" title="mailto:aihua.zhao@intel.com
mailto:hui.xue@intel.com">@intel.com

Tel: +86(21)61166476

iNet: 8821-6476

SSG/OTC/Moblin 3W038 Pole: F4

Thiago Sousa Santos-2

Re: How to control pipeline frame rate when my camera needs some time to get one frame data

On Tue, Nov 17, 2009 at 8:11 PM, Zhao, Halley <[hidden email]> wrote:

Hi all:

My camera needs ~20ms to get one frame of data, and a video sink in the pipeline needs ~4ms to render it.

But when I set pipeline fps to 35, I only got 20 fps (calculate by GST CLOCK). CPU is not busy at this time, only 60%.

I guess the reason is following:

When we got nth frame at time 0, gst framework schedule to get next frame after 30ms (1000/35), and when the camera src element got scheduled to create one frame, it needs 20ms to get the data.

Then the time between n frame and n+1 frame is 30+20=50ms, so I just got 20 fps.

Is my assumption right?

If yes, is there any way to adjust it?

You can speed it up by adding a queue right after your source element. It will create another thread to push data after it, the thread used to request a frame from the camera will quickly return to the source and it will request another one.

I had though to create another thread inside camera source, and prepare the frame data in a buffer before it is schedule.

However, I think it is hack, and make timestamp differ. I think it’s better settled inside gst framework.

ZHAO, Halley (Aihua)

Email: halley.zhao@intel.com

Tel: +86(21)61166476

iNet: 8821-6476

SSG/OTC/Moblin 3W038 Pole: F4

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

--
Thiago Sousa Santos

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

Zhao, Halley

Re: How to control pipeline frame rate when my camera needs some time to get one frame data

But the timestamp will be mess.

You could get some frames in a short time, then idle some time, then another some frames.

From: [hidden email] [mailto:[hidden email]]
Sent: 2009年11月18日 7:19
To: Discussion of the development of GStreamer
Subject: Re: [gst-devel] How to control pipeline frame rate when my camera needs some time to get one frame data

On Tue, Nov 17, 2009 at 8:11 PM, Zhao, Halley <[hidden email]> wrote:

Hi all:

My camera needs ~20ms to get one frame of data, and a video sink in the pipeline needs ~4ms to render it.

But when I set pipeline fps to 35, I only got 20 fps (calculate by GST CLOCK). CPU is not busy at this time, only 60%.

I guess the reason is following:

When we got nth frame at time 0, gst framework schedule to get next frame after 30ms (1000/35), and when the camera src element got scheduled to create one frame, it needs 20ms to get the data.

Then the time between n frame and n+1 frame is 30+20=50ms, so I just got 20 fps.

Is my assumption right?

If yes, is there any way to adjust it?

I had though to create another thread inside camera source, and prepare the frame data in a buffer before it is schedule.

However, I think it is hack, and make timestamp differ. I think it’s better settled inside gst framework.

ZHAO, Halley (Aihua)

Email: [hidden email]

Tel: +86(21)61166476

iNet: 8821-6476

SSG/OTC/Moblin 3W038 Pole: F4

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

--
Thiago Sousa Santos

Edward Hervey

Re: How to control pipeline frame rate when my camera needs some time to get one frame data

Administrator

On Wed, 2009-11-18 at 16:24 +0800, Zhao, Halley wrote:
> But the timestamp will be mess.
>
> You could get some frames in a short time, then idle some time, then
> another some frames.

What thiagoss recomended is correct (you basically end up with one
thread dedicated to capture the image and the other threads doing the
extra processing).
Timestamping is done at the source element (i.e. the timestamp will be
the moment at which the image is captured).

If you were to give the gst-launch pipeline you're testing with we
could give you more info.

Edward

>
>
>
>
>
>
> ______________________________________________________________________
> From:[hidden email] [mailto:[hidden email]]
> Sent: 2009年11月18日7:19
> To: Discussion of the development of GStreamer
> Subject: Re: [gst-devel] How to control pipeline frame rate when my
> camera needs some time to get one frame data
>
>
>
>
>
>
> On Tue, Nov 17, 2009 at 8:11 PM, Zhao, Halley <[hidden email]>
> wrote:
>
> Hi all:
>
> My camera needs ~20ms to get one frame of data, and a video sink in
> the pipeline needs ~4ms to render it.
>
> But when I set pipeline fps to 35, I only got 20 fps (calculate by GST
> CLOCK). CPU is not busy at this time, only 60%.
>
>
>
> I guess the reason is following:
>
> When we got nth frame at time 0, gst framework schedule to get next
> frame after 30ms (1000/35), and when the camera src element got
> scheduled to create one frame, it needs 20ms to get the data.
>
> Then the time between n frame and n+1 frame is 30+20=50ms, so I just
> got 20 fps.
>
> Is my assumption right?
>
> If yes, is there any way to adjust it?
>
>
> You can speed it up by adding a queue right after your source element.
> It will create another thread to push data after it, the thread used
> to request a frame from the camera will quickly return to the source
> and it will request another one.
>
>
>
>
>
> I had though to create another thread inside camera source,
> and prepare the frame data in a buffer before it is schedule.
>
> However, I think it is hack, and make timestamp differ. I
> think it’s better settled inside gst framework.
>
>
>
>
>
> ZHAO, Halley (Aihua)
>
> Email: [hidden email]
>
> Tel: +86(21)61166476
>
> iNet: 8821-6476
>
> SSG/OTC/Moblin 3W038 Pole: F4
>
>
>
>
>
>
> ------------------------------------------------------------------------------
> Let Crystal Reports handle the reporting - Free Crystal
> Reports 2008 30-Day
> trial. Simplify your report design, integration and deployment
> - and focus on
> what you do best, core application coding. Discover what's new
> with
> Crystal Reports now. http://p.sf.net/sfu/bobj-july
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>
>
>
>
> --
> Thiago Sousa Santos
>
>
> ------------------------------------------------------------------------------
> Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
> trial. Simplify your report design, integration and deployment - and focus on
> what you do best, core application coding. Discover what's new with
> Crystal Reports now. http://p.sf.net/sfu/bobj-july
> _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

Zhao, Halley

Re: How to control pipeline frame rate when my camera needs some time to get one frame data

Thanks all.

My camera src source code is at git://git.moblin.org/gst_mrst_camera_src.git
A little sorry that I haven't merge timestamp code into this git yet. :)

The gst-launch pipeline is like:
gst-launch -v mycamsrc! video/x-raw-yuv , framerate=35/1 ! queue ! xvimagesink

the frame I got with timestamp is something like:
#1 50ms
#2 53ms
#3 56ms
#4 140ms
#5 142ms
#6 144ms
#7 230ms
#8 233ms
#9 236

The above is some data derive from memory, since I tried queue some days ago.
And I will try to get some real data when I come back office.

mycamsrc is not a push source since hw don't have hard timing to create a frame of data.

-----Original Message-----
From: Edward Hervey [mailto:[hidden email]]
Sent: 2009年11月18日 16:42
To: Discussion of the development of GStreamer
Subject: Re: [gst-devel] How to control pipeline frame rate when my camera needs some time to get one frame data

On Wed, 2009-11-18 at 16:24 +0800, Zhao, Halley wrote:
> But the timestamp will be mess.
>
> You could get some frames in a short time, then idle some time, then
> another some frames.

What thiagoss recomended is correct (you basically end up with one
thread dedicated to capture the image and the other threads doing the
extra processing).
Timestamping is done at the source element (i.e. the timestamp will be
the moment at which the image is captured).

If you were to give the gst-launch pipeline you're testing with we
could give you more info.

Edward

Stefan Sauer

Re: How to control pipeline frame rate when my camera needs some time to get one frame data

In reply to this post by Edward Hervey

Edward Hervey schrieb:

> On Wed, 2009-11-18 at 16:24 +0800, Zhao, Halley wrote:
>> But the timestamp will be mess.
>>
>> You could get some frames in a short time, then idle some time, then
>> another some frames.
>
> What thiagoss recomended is correct (you basically end up with one
> thread dedicated to capture the image and the other threads doing the
> extra processing).
> Timestamping is done at the source element (i.e. the timestamp will be
> the moment at which the image is captured).

Unfortunately not entierly correct. The timestamp is the time when the capture
thread actualy gets scheduled :/ Sometimes using the timestampo provided by v4l2
itself would be better, but this would need skew slaving ...

Running the capture thread with SCHED_RR helps a lot.

Stefan

>
> If you were to give the gst-launch pipeline you're testing with we
> could give you more info.
>
>
> Edward
>
>>
>>
>>
>>
>>
>> ______________________________________________________________________
>> From:[hidden email] [mailto:[hidden email]]
>> Sent: 2009年11月18日7:19
>> To: Discussion of the development of GStreamer
>> Subject: Re: [gst-devel] How to control pipeline frame rate when my
>> camera needs some time to get one frame data
>>
>>
>>
>>
>>
>>
>> On Tue, Nov 17, 2009 at 8:11 PM, Zhao, Halley <[hidden email]>
>> wrote:
>>
>> Hi all:
>>
>> My camera needs ~20ms to get one frame of data, and a video sink in
>> the pipeline needs ~4ms to render it.
>>
>> But when I set pipeline fps to 35, I only got 20 fps (calculate by GST
>> CLOCK). CPU is not busy at this time, only 60%.
>>
>>
>>
>> I guess the reason is following:
>>
>> When we got nth frame at time 0, gst framework schedule to get next
>> frame after 30ms (1000/35), and when the camera src element got
>> scheduled to create one frame, it needs 20ms to get the data.
>>
>> Then the time between n frame and n+1 frame is 30+20=50ms, so I just
>> got 20 fps.
>>
>> Is my assumption right?
>>
>> If yes, is there any way to adjust it?
>>
>>
>> You can speed it up by adding a queue right after your source element.
>> It will create another thread to push data after it, the thread used
>> to request a frame from the camera will quickly return to the source
>> and it will request another one.
>>
>>
>>
>>
>>
>> I had though to create another thread inside camera source,
>> and prepare the frame data in a buffer before it is schedule.
>>
>> However, I think it is hack, and make timestamp differ. I
>> think it’s better settled inside gst framework.
>>
>>
>>
>>
>>
>> ZHAO, Halley (Aihua)
>>
>> Email: [hidden email]
>>
>> Tel: +86(21)61166476
>>
>> iNet: 8821-6476
>>
>> SSG/OTC/Moblin 3W038 Pole: F4
>>
>>
>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Let Crystal Reports handle the reporting - Free Crystal
>> Reports 2008 30-Day
>> trial. Simplify your report design, integration and deployment
>> - and focus on
>> what you do best, core application coding. Discover what's new
>> with
>> Crystal Reports now. http://p.sf.net/sfu/bobj-july
>> _______________________________________________
>> gstreamer-devel mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>>
>>
>>
>>
>> --
>> Thiago Sousa Santos
>>
>>
>> ------------------------------------------------------------------------------
>> Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
>> trial. Simplify your report design, integration and deployment - and focus on
>> what you do best, core application coding. Discover what's new with
>> Crystal Reports now. http://p.sf.net/sfu/bobj-july
>> _______________________________________________ gstreamer-devel mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>
>
>
> ------------------------------------------------------------------------------
> Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
> trial. Simplify your report design, integration and deployment - and focus on
> what you do best, core application coding. Discover what's new with
> Crystal Reports now. http://p.sf.net/sfu/bobj-july
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

Zhao, Halley

Re: How to control pipeline frame rate when my camera needs some time to get one frame data

Some issues after adding the queue:

1. the pipeline runs as fast as possible, while not controlled by framerate set by a caps_filter.
As the following data show, I set pipeline framerate to 5, but got 27 in reality.
2. the timestamp fluctuate.
See the following 'dur' data, it ranges from 23 to 43 ms.

0:00:01.627193170 21145 0x81add98 DEBUG capsfilter gstcapsfilter.c:164:gst_capsfilter_set_property:<capsfilter0> set new caps video/x-raw-va, width=(int)640, height=(int)480, framerate=(fraction)5/1
0:00:03.966024212 21145 0x81add98 DEBUG GST_CAPS gstpad.c:2553:gst_pad_set_caps:<mrstcamsrc0:src> caps video/x-raw-va, width=(int)640, height=(int)480, framerate=(fraction)5/1
current buffer time(ms): 1256, last buffer time: 1224, dur: 32
current buffer time(ms): 1280, last buffer time: 1256, dur: 24
current buffer time(ms): 1304, last buffer time: 1280, dur: 24
current buffer time(ms): 1330, last buffer time: 1304, dur: 26
current buffer time(ms): 1367, last buffer time: 1330, dur: 37
current buffer time(ms): 1390, last buffer time: 1367, dur: 23
current buffer time(ms): 1416, last buffer time: 1390, dur: 26
current buffer time(ms): 1439, last buffer time: 1416, dur: 23
current buffer time(ms): 1463, last buffer time: 1439, dur: 24
current buffer time(ms): 1489, last buffer time: 1463, dur: 26
current buffer time(ms): 1512, last buffer time: 1489, dur: 23
current buffer time(ms): 1537, last buffer time: 1512, dur: 25
current buffer time(ms): 1562, last buffer time: 1537, dur: 25
current buffer time(ms): 1586, last buffer time: 1562, dur: 24
current buffer time(ms): 1610, last buffer time: 1586, dur: 24
current buffer time(ms): 1650, last buffer time: 1610, dur: 40
current buffer time(ms): 1670, last buffer time: 1650, dur: 20
current buffer time(ms): 1694, last buffer time: 1670, dur: 24
current buffer time(ms): 1720, last buffer time: 1694, dur: 26
current buffer time(ms): 1756, last buffer time: 1720, dur: 36
current buffer time(ms): 1799, last buffer time: 1756, dur: 43
current buffer time(ms): 1830, last buffer time: 1799, dur: 31
current buffer time(ms): 1867, last buffer time: 1830, dur: 37 func: gst_mrstcamsrc_create, current frame #: 50
avg fps of recent 50 frames is:27.73

-----Original Message-----
From: Stefan Kost [mailto:[hidden email]]
Sent: 2009年11月20日 5:17
To: Discussion of the development of GStreamer
Subject: Re: [gst-devel] How to control pipeline frame rate when my camera needs some time to get one frame data

Edward Hervey schrieb:

Zhao, Halley

Re: How to control pipeline frame rate when my camera needs some time to get one frame data

I had tried to limit the size of queue to reduce the impact, but no effect. The queue isn't full in fact.

0:00:06.255354897 21145 0x82618f0 LOG queue_dataflow gstqueue.c:727:gst_queue_locked_dequeue:<queue0> (queue0:src) signal DEL: 6 of 0-10 buffers, 24 of 0-10485760 bytes, 158273457 of 0-1000000000 ns, 6 items

-----Original Message-----
From: Zhao, Halley [mailto:[hidden email]]
Sent: 2009年11月20日 10:21
To: 'Discussion of the development of GStreamer'
Subject: Re: [gst-devel] How to control pipeline frame rate when my camera needs some time to get one frame data

Some issues after adding the queue:

1. the pipeline runs as fast as possible, while not controlled by framerate set by a caps_filter.
As the following data show, I set pipeline framerate to 5, but got 27 in reality.
2. the timestamp fluctuate.
See the following 'dur' data, it ranges from 23 to 43 ms.

0:00:01.627193170 21145 0x81add98 DEBUG capsfilter gstcapsfilter.c:164:gst_capsfilter_set_property:<capsfilter0> set new caps video/x-raw-va, width=(int)640, height=(int)480, framerate=(fraction)5/1
0:00:03.966024212 21145 0x81add98 DEBUG GST_CAPS gstpad.c:2553:gst_pad_set_caps:<mrstcamsrc0:src> caps video/x-raw-va, width=(int)640, height=(int)480, framerate=(fraction)5/1
current buffer time(ms): 1256, last buffer time: 1224, dur: 32
current buffer time(ms): 1280, last buffer time: 1256, dur: 24
current buffer time(ms): 1304, last buffer time: 1280, dur: 24
current buffer time(ms): 1330, last buffer time: 1304, dur: 26
current buffer time(ms): 1367, last buffer time: 1330, dur: 37
current buffer time(ms): 1390, last buffer time: 1367, dur: 23
current buffer time(ms): 1416, last buffer time: 1390, dur: 26
current buffer time(ms): 1439, last buffer time: 1416, dur: 23
current buffer time(ms): 1463, last buffer time: 1439, dur: 24
current buffer time(ms): 1489, last buffer time: 1463, dur: 26
current buffer time(ms): 1512, last buffer time: 1489, dur: 23
current buffer time(ms): 1537, last buffer time: 1512, dur: 25
current buffer time(ms): 1562, last buffer time: 1537, dur: 25
current buffer time(ms): 1586, last buffer time: 1562, dur: 24
current buffer time(ms): 1610, last buffer time: 1586, dur: 24
current buffer time(ms): 1650, last buffer time: 1610, dur: 40
current buffer time(ms): 1670, last buffer time: 1650, dur: 20
current buffer time(ms): 1694, last buffer time: 1670, dur: 24
current buffer time(ms): 1720, last buffer time: 1694, dur: 26
current buffer time(ms): 1756, last buffer time: 1720, dur: 36
current buffer time(ms): 1799, last buffer time: 1756, dur: 43
current buffer time(ms): 1830, last buffer time: 1799, dur: 31
current buffer time(ms): 1867, last buffer time: 1830, dur: 37 func: gst_mrstcamsrc_create, current frame #: 50
avg fps of recent 50 frames is:27.73

-----Original Message-----
From: Stefan Kost [mailto:[hidden email]]
Sent: 2009年11月20日 5:17
To: Discussion of the development of GStreamer
Subject: Re: [gst-devel] How to control pipeline frame rate when my camera needs some time to get one frame data

Edward Hervey schrieb:

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel