Variable framerate/fps input video encoding questions

[jles]

Hello,

I'm new with gstreamer and h264 encoding and I'm trying to understand the
capabilities and limitations of this.

My main goal is to be able to record an input live video stream from a
camera that changes the framerate dynamically  depending of the exposure and
light conditions. This camera is capable to work a high framerates for low
resolutions like 700fps on VGA(640x480) mode, so frame rate can change from
10fps to 700fps.

My first question is: would it be possible to encode video to h264 for a
variable frame rate as described above?

The pipeline I'm using looks similar to this:

appsrc -> videorate -> videoconvert -> caps -> queue -> omxh264enc -> queue
-> h264parse -> mpegtsmux -> filesink

These are the things I tried and the issues I found:

- If I set the appsrc caps to a fix framerate of 15fps (regardless camera
fps) the resulting video plays a "normal" speed but during fast movements I
can see frame tearing issues.

-For any fps different to 15fps the resulting encoded video plays faster
than is should be.

Any idea what could it be happening?

I did spend long time searching but I could not find any similar
application.

Any light on this would it be appreciated.
Thanks in advance.









--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le jeudi 18 juillet 2019 à 04:28 -0500, jles a écrit :

> Hello,
>
> I'm new with gstreamer and h264 encoding and I'm trying to understand the
> capabilities and limitations of this.
>
> My main goal is to be able to record an input live video stream from a
> camera that changes the framerate dynamically  depending of the exposure and
> light conditions. This camera is capable to work a high framerates for low
> resolutions like 700fps on VGA(640x480) mode, so frame rate can change from
> 10fps to 700fps.
>
> My first question is: would it be possible to encode video to h264 for a
> variable frame rate as described above?

Yes, assuming you have a capable CPU or HW Accelerator, 700 fps will
trigger few bottleneck in GStreamer (even if the resolution is low), so
you really need a powerful machine.

What is not possible which this, is for GStreamer to evaluate the
latency requires if you would like to do live streaming out of it. Best
is to pretend you are doing 10fps, and the latency calculation will be
made from that, then having early frames should not be a problem.

>
> The pipeline I'm using looks similar to this:
>
> appsrc -> videorate -> videoconvert -> caps -> queue -> omxh264enc -> queue
> -> h264parse -> mpegtsmux -> filesink

Note that videorate will likely drop or add frames to you stream, is
this really what you are looking for ?

>
> These are the things I tried and the issues I found:
>
> - If I set the appsrc caps to a fix framerate of 15fps (regardless camera
> fps) the resulting video plays a "normal" speed but during fast movements I
> can see frame tearing issues.

What version of GStreamer are you using ? Do you reuse you buffers too
soon ? There was also some issues in omxh264enc in older versions.

>
> -For any fps different to 15fps the resulting encoded video plays faster
> than is should be.
>
> Any idea what could it be happening?

That can only be a miss-calculation of timestamp on your end. Using
appsrc requires good care on crafting correct timestamps.

>
> I did spend long time searching but I could not find any similar
> application.
>
> Any light on this would it be appreciated.
> Thanks in advance.
>
>
>
>
>
>
>
>
>
> --
> Sent from: http://gstreamer-devel.966125.n4.nabble.com/
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[jles]

Hi Nicolas,

Thanks for your reply.


Nicolas Dufresne-5 wrote
> What is not possible which this, is for GStreamer to evaluate the
> latency requires if you would like to do live streaming out of it. Best
> is to pretend you are doing 10fps, and the latency calculation will be
> made from that, then having early frames should not be a problem.

Do you mean that for a live source like this case it would be convenient to
set the appsrc caps framerate to the lowest expected fps?

If I'm doing this the resulting encoded and container file after mpegtsmux
will have that let's say 10fps low framerate timestamps....so what would
happen if the framerate is increased during that time let's say to 400fps
would all frames be discarded?

 

Nicolas Dufresne-5 wrote
> Note that videorate will likely drop or add frames to you stream, is
> this really what you are looking for ?

You are right it might be no ideal here, I used it initially because I tried
to set a variable framerate in the appsrc caps and it was the only way to
don't get a "no enough resources available" error from the encoder. But now
I'm setting a appsrc caps constant framerate  I'll  delete it from the
pipeline...



Nicolas Dufresne-5 wrote
> That can only be a miss-calculation of timestamp on your end. Using
> appsrc requires good care on crafting correct timestamps.

Agree this could it be the key of everything but I'm confuse with this, what
I've done is:
I do the calculation getting the appsrc clock (gst_element_get_clock) and
base time (gst_element_get_base_time) after I get the time of the computed
clock (gst_clock_get_time)  and then I calculate the difference:

                clock = gst_element_get_clock (GST_ELEMENT_CAST (appsrc));
                if (clock) {
                 base_time =
                 gst_element_get_base_time (GST_ELEMENT_CAST (appsrc));
                 now = gst_clock_get_time (clock);
                 if (now > base_time)
                         now -= base_time;
                 else
                         now = 0;

                 gst_object_unref (clock);
                 GST_BUFFER_PTS (buffer) = now;
                 GST_BUFFER_DTS (buffer) = now;
                }
I compared with the actual camera framerate (fps given by the camera) and
matches so I assume that the calculation is correct...
I'm a bit lost here, I looked around and I couldn't find any good
tutorial/examples on how to calculate properly timestamps with appsrc,
what would it be the best way/idea to compute properly timestamps?

On the other hand I tried  to change the encoder target bit rate and this
also affects to the resulting playback speed...so could it be instead a
timestamp issue a bandwidth one?

Thanks.



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le jeudi 18 juillet 2019 à 08:19 -0500, jles a écrit :

> Hi Nicolas,
>
> Thanks for your reply.
>
>
> Nicolas Dufresne-5 wrote
> > What is not possible which this, is for GStreamer to evaluate the
> > latency requires if you would like to do live streaming out of it. Best
> > is to pretend you are doing 10fps, and the latency calculation will be
> > made from that, then having early frames should not be a problem.
>
> Do you mean that for a live source like this case it would be convenient to
> set the appsrc caps framerate to the lowest expected fps?

If you are writing to filesrc, it does not matter, but if you re-
transmit let's say over RTP, then yes, picking the lowest fps will
setup the worst possible latency on your pipeline and ensure packets
aren't late even at the lowest.

>
> If I'm doing this the resulting encoded and container file after mpegtsmux
> will have that let's say 10fps low framerate timestamps....so what would
> happen if the framerate is increased during that time let's say to 400fps
> would all frames be discarded?

What about setting the appsrc caps property to lowest fps, and then use
videorate drop-only=1 with a caps filter afert that match the maximum
you'd like to record ?

>
>  
>
> Nicolas Dufresne-5 wrote
> > Note that videorate will likely drop or add frames to you stream, is
> > this really what you are looking for ?
>
> You are right it might be no ideal here, I used it initially because I tried
> to set a variable framerate in the appsrc caps and it was the only way to
> don't get a "no enough resources available" error from the encoder. But now
> I'm setting a appsrc caps constant framerate  I'll  delete it from the
> pipeline...
>
>
>
> Nicolas Dufresne-5 wrote
> > That can only be a miss-calculation of timestamp on your end. Using
> > appsrc requires good care on crafting correct timestamps.
>
> Agree this could it be the key of everything but I'm confuse with this, what
> I've done is:
> I do the calculation getting the appsrc clock (gst_element_get_clock) and
> base time (gst_element_get_base_time) after I get the time of the computed
> clock (gst_clock_get_time)  and then I calculate the difference:
>
> clock = gst_element_get_clock (GST_ELEMENT_CAST (appsrc));
> if (clock) {
> base_time =
> gst_element_get_base_time (GST_ELEMENT_CAST (appsrc));
> now = gst_clock_get_time (clock);
> if (now > base_time)
> now -= base_time;
> else
> now = 0;
>
> gst_object_unref (clock);
> GST_BUFFER_PTS (buffer) = now;
> GST_BUFFER_DTS (buffer) = now;
> }
> I compared with the actual camera framerate (fps given by the camera) and
> matches so I assume that the calculation is correct...
> I'm a bit lost here, I looked around and I couldn't find any good
> tutorial/examples on how to calculate properly timestamps with appsrc,
> what would it be the best way/idea to compute properly timestamps?
>
> On the other hand I tried  to change the encoder target bit rate and this
> also affects to the resulting playback speed...so could it be instead a
> timestamp issue a bandwidth one?
>
> Thanks.
>
>
>
> --
> Sent from: http://gstreamer-devel.966125.n4.nabble.com/
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le jeudi 18 juillet 2019 à 08:19 -0500, jles a écrit :

> Nicolas Dufresne-5 wrote
> > That can only be a miss-calculation of timestamp on your end. Using
> > appsrc requires good care on crafting correct timestamps.
>
> Agree this could it be the key of everything but I'm confuse with this, what
> I've done is:
> I do the calculation getting the appsrc clock (gst_element_get_clock) and
> base time (gst_element_get_base_time) after I get the time of the computed
> clock (gst_clock_get_time)  and then I calculate the difference:
>
> clock = gst_element_get_clock (GST_ELEMENT_CAST (appsrc));
> if (clock) {
> base_time =
> gst_element_get_base_time (GST_ELEMENT_CAST (appsrc));
> now = gst_clock_get_time (clock);
> if (now > base_time)
> now -= base_time;
> else
> now = 0;
>
> gst_object_unref (clock);
> GST_BUFFER_PTS (buffer) = now;
> GST_BUFFER_DTS (buffer) = now;
> }
> I compared with the actual camera framerate (fps given by the camera) and
> matches so I assume that the calculation is correct...
> I'm a bit lost here, I looked around and I couldn't find any good
> tutorial/examples on how to calculate properly timestamps with appsrc,
> what would it be the best way/idea to compute properly timestamps?
>
> On the other hand I tried  to change the encoder target bit rate and this
> also affects to the resulting playback speed...so could it be instead a
> timestamp issue a bandwidth one?

Any jitter in your application will cause jitter in your timestamp with
this method. Though, I don't see any specific errors right now. This is
similar to the what the do-timestamp property on appsrc would do. This
follow the formulat:

  (now - base_time)

Which is the running time, which match what appsrc will have by
default. Ideally the timestamp would come from your camera and would be
translated from one clock to another. This is what we try and do with
v4l2src as an example (but it still a bit jittery I was told).

>
> Thanks.
>
>
>
> --
> Sent from: http://gstreamer-devel.966125.n4.nabble.com/
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[jles]

Hi thanks again for you suggestions and info,

I did a test recording two files, one at 15fps (plays normal speed) and
other at 30fps (plays faster) and I printed the generate file timestamps
running:

ffprobe output.ts -hide_banner -show_entries
frame=frame_no,best_effort_timestamp_time -of json

For an appsrc frame rate of ~15fps (the one which plays at "normal speed")
shows:

.......
            "best_effort_timestamp_time": "3617.935644"
        },
        {
            "best_effort_timestamp_time": "3618.010522"
        },
        {
            "best_effort_timestamp_time": "3618.076589"

..............

If I count the number of frames for each time stamp second there're 15 which
matches the appsrc 'forced' frame rate, for instance calculating time
between "3618.010522" and "3618.076589" there's ~15fps, but weirdly there's
approximately a second between the frame time groups "3617.935644" and
"3618.010522" ...
any idea what is going on? is there any time of latency or delay I have to
set?

Now for frame rates > than 15 for instance I tried 30fps (which the
resulting file plays faster) I've got similar results as with 15fps but with
a significant difference:

.........

            "best_effort_timestamp_time": "3607.971956"
        },
        {
            "best_effort_timestamp_time": "3608.026967"
        },
        {
            "best_effort_timestamp_time": "3608.054644"

...........

Here the problem of the second between group of frames is ~ the same (as
with the 15fps example) but if we calculate frame rate, it's now ~36fps
instead 30fps, the number of frames for instance in the "3607.xxxx" time
group are 30.

So we've got groups of 30 frames with a time stamp of ~36fps, which I think
explains why the resulting video plays faster....

Any idea why this happening? is it related to the mentioned jittery issue?
it looks quite important difference (30 <-> 36) to be jitter? what could be
in a simple application like this be causing this?

Just as additional info, in my application I'm only running appsrc callback
for "feed/need data", should I also mange/add the "enough data"? could it be
this the reason?

Thanks.







 





--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le vendredi 19 juillet 2019 à 07:05 -0500, jles a écrit :

> Hi thanks again for you suggestions and info,
>
> I did a test recording two files, one at 15fps (plays normal speed) and
> other at 30fps (plays faster) and I printed the generate file timestamps
> running:
>
> ffprobe output.ts -hide_banner -show_entries
> frame=frame_no,best_effort_timestamp_time -of json
>
> For an appsrc frame rate of ~15fps (the one which plays at "normal speed")
> shows:
>
> .......
>             "best_effort_timestamp_time": "3617.935644"
>         },
>         {
>             "best_effort_timestamp_time": "3618.010522"
>         },
>         {
>             "best_effort_timestamp_time": "3618.076589"
>
> ..............
>
> If I count the number of frames for each time stamp second there're 15 which
> matches the appsrc 'forced' frame rate, for instance calculating time
> between "3618.010522" and "3618.076589" there's ~15fps, but weirdly there's
> approximately a second between the frame time groups "3617.935644" and
> "3618.010522" ...
> any idea what is going on? is there any time of latency or delay I have to
> set?
>
> Now for frame rates > than 15 for instance I tried 30fps (which the
> resulting file plays faster) I've got similar results as with 15fps but with
> a significant difference:
>
> .........
>
>             "best_effort_timestamp_time": "3607.971956"
>         },
>         {
>             "best_effort_timestamp_time": "3608.026967"
>         },
>         {
>             "best_effort_timestamp_time": "3608.054644"
>
> ...........
>
> Here the problem of the second between group of frames is ~ the same (as
> with the 15fps example) but if we calculate frame rate, it's now ~36fps
> instead 30fps, the number of frames for instance in the "3607.xxxx" time
> group are 30.
>
> So we've got groups of 30 frames with a time stamp of ~36fps, which I think
> explains why the resulting video plays faster....
>
> Any idea why this happening? is it related to the mentioned jittery issue?
> it looks quite important difference (30 <-> 36) to be jitter? what could be
> in a simple application like this be causing this?

This looks like buffering effect, just like cars on the high way. They
just bump into each other running slower and out of a sudden everyone
tries to catch up the lost time.

>
> Just as additional info, in my application I'm only running appsrc callback
> for "feed/need data", should I also mange/add the "enough data"? could it be
> this the reason?
>
> Thanks.
>
>
>
>
>
>
>
>  
>
>
>
>
>
> --
> Sent from: http://gstreamer-devel.966125.n4.nabble.com/
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le ven. 19 juill. 2019 09 h 55, jles <[hidden email]> a écrit :

Nicolas Dufresne-5 wrote
> This looks like buffering effect, just like cars on the high way. They
> just bump into each other running slower and out of a sudden everyone
> tries to catch up the lost time.

I see, do you have any idea about why could this be happening? have you seen
this before with gstreamer live stream pipelines?

The code is really simple, grab a frame from camera and push it to the
pipeline each time, gstreamer it is quite high level of abstraction to see
clearly where could it be the issue...could you please point me to right
direction?

I saw the debugging options using the gstreamer plugin (exporting env.
variables as export GST_DEBUG="*:6") but no sure  how to do it with code?

GST_SCHEDULING:5 might be handy. It will trace all buffers flowing and their timestamps. Sources like v4l2src make use of the timestamps provided by the kernel, that usually prevents jitter like this.

--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[jles]

Hi,

I finally got it working with proper playing speeds, using the code down
below to calculate the buffer timestamps.
The mistake I did is that apart of this code I had the appsrc  go-timestamp
set to true (both at the same time).
Setting this to false and using the code below instead to calculate
timestamp, plays at normal speed and the resulting file gets proper file
duration and timestamps:

>> clock = gst_element_get_clock (GST_ELEMENT_CAST (appsrc));
>> if (clock) {
>> base_time =
>> gst_element_get_base_time (GST_ELEMENT_CAST (appsrc));
>> now = gst_clock_get_time (clock);
>> if (now > base_time)
>> now -= base_time;
>> else
>> now = 0;
>>
>> gst_object_unref (clock);
>> GST_BUFFER_PTS (buffer) = now;
>> GST_BUFFER_DTS (buffer) = now;
>> }

Nicolas Dufresne-5 wrote

> Any jitter in your application will cause jitter in your timestamp with
> this method. Though, I don't see any specific errors right now. This is
> similar to the what the do-timestamp property on appsrc would do. This
> follow the formulat:
>
>   (now - base_time)
>
> Which is the running time, which match what appsrc will have by
> default. Ideally the timestamp would come from your camera and would be
> translated from one clock to another. This is what we try and do with
> v4l2src as an example (but it still a bit jittery I was told).

Yes, it is similar to do-timestamp would do but not quite, I test it with
only do-timestamp set to true to calculate time stamps, and it gets wrong
video frame duration, resulting in a fast played video (in my case). For
some reason the do-timestamp property doesn't work properly for recording
live streams...




--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel