videorate doesn't drop or duplicate frames in live stream?

[gotsring]

I'm using gstreamer 1.16.2 and C++ to process a IP cam video feed. I use the
videorate element to even out the live stream which averages ~30 FPS. I need
a smooth 20 FPS, so videorate should duplicate frames during the occasional
dropouts and drop the excess frames, so I have a pipeline as follows:

souphttpsrc location=http://192.168.1.145/mjpeg is-live=true ! queue !
jpegdec ! videorate ! 'video/x-raw, framerate=20/1' ! d3dvideosink

All this seems to do is re-timestamp all frames with a duration of 50ms, so
nothing is ever dropped or duplicated, messing up anything I do downstream.
I'm guessing this is probably because the incoming frames have no
timestamps.

Adding the do-timestamp option in souphttpsrc causes jpegdec to fail (messes
with jpeg data header? still looking into this)

Inserting a timecodestamper element in front of videorate spams a bunch of
gst_time_code_is_valid assertion fails. Desite the messages, the pipeline
still runs, but the behavior is the same as the original pipeline, i.e. the
timestamps are altered, but no frames are duplicated or dropped.

Is there a property in videorate that I'm missing that can perform
rate-smoothing based on an internal clock with no reliance on frame
timestamps, or some other way to add timestamps to frames before they enter
videorate?



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le vendredi 24 juillet 2020 à 20:15 -0500, gotsring a écrit :

> I'm using gstreamer 1.16.2 and C++ to process a IP cam video feed. I use the
> videorate element to even out the live stream which averages ~30 FPS. I need
> a smooth 20 FPS, so videorate should duplicate frames during the occasional
> dropouts and drop the excess frames, so I have a pipeline as follows:
>
> souphttpsrc location=http://192.168.1.145/mjpeg is-live=true ! queue !
> jpegdec ! videorate ! 'video/x-raw, framerate=20/1' ! d3dvideosink
>
> All this seems to do is re-timestamp all frames with a duration of 50ms, so
> nothing is ever dropped or duplicated, messing up anything I do downstream.
> I'm guessing this is probably because the incoming frames have no
> timestamps.
>
> Adding the do-timestamp option in souphttpsrc causes jpegdec to fail (messes
> with jpeg data header? still looking into this)
>
> Inserting a timecodestamper element in front of videorate spams a bunch of
> gst_time_code_is_valid assertion fails. Desite the messages, the pipeline
> still runs, but the behavior is the same as the original pipeline, i.e. the
> timestamps are altered, but no frames are duplicated or dropped.
>
> Is there a property in videorate that I'm missing that can perform
> rate-smoothing based on an internal clock with no reliance on frame
> timestamps, or some other way to add timestamps to frames before they enter
> videorate?

videorate does not have a live mode implemented atm. In fact, it does
not have the needed thread to allow this. So even if you where usinging
a proper container with timestamp, it would bursts frames on the next
buffer after the gap.

I've heard some folks have managed to workaround using compositor
element, with a single pad. In live mode, it produces data on the
clock. The only issue I see is how it behaves on gaps (it renders the
background rather then repeating last image). I think it's wrong
default, I would need to discuss this.

gst-launch-1.0 videotestsrc is-live=1 ! video/x-raw,framerate=10/1 ! compositor ! video/x-raw,framerate=30/1 ! fpsdisplaysink video-sink=glimagesink

>
>
>
> --
> Sent from: http://gstreamer-devel.966125.n4.nabble.com/
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[gotsring]

Thanks for the suggestion. I messed around with compositor a bit, but I ended
up just adding a probe to the sink pad of the videorate element that
manually added a timestamp to the incoming buffers. Videorate could then
perform as expected because the time information was added (callback code
below).

// Adds a timestamp as frames come in so videorate behaves normally.
GstPadProbeReturn do_timestamp_cb(GstPad * pad, GstPadProbeInfo * info,
gpointer data) {
    GstBuffer* buffer = GST_PAD_PROBE_INFO_BUFFER(info);
    buffer = gst_buffer_make_writeable(buffer);
    buffer->pts = gst_clock_get_time(pipeline_clock); // pipeline_clock from
gst_pipeline_get_pipeline_clock
    GST_PAD_PROBE_INFO_DATA(info) = buffer;
    return GST_PAD_PROBE_OK;
}



Nicolas Dufresne-5 wrote
> videorate does not have a live mode implemented atm. In fact, it does
> not have the needed thread to allow this. So even if you where usinging
> a proper container with timestamp, it would bursts frames on the next
> buffer after the gap.

My main concern with my method is that it assumes the frames come in
synchronously, by which I mean the frames are received and decoded, then
passed to videorate (and thus my timestamp function) with a near-constant
latency. I'm not sure if is necessarily true for most or all GStreamer
pipelines. Is this the thread issue you are describing, or am I missing
something else?



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel