Timestamping frame buffers correctly

[jonmccormack]

Hello all,
I've been trying to learn how to use Gstreamer for the past few weeks, using
the appsrc element in a c++ application stream video to an RTMP server.

I have raw frame data coming in from another part of my application, which
has the timestamp the frame was created in milliseconds since the unix
epoch.

Being a total beginner with regards to working with video streaming, it
looks like timestamping the frame buffers that are piped into the gstreamer
pipeline is essential, especially when creating a networks stream. It looks
like the buffers are timestamped in nanoseconds since the moment the
gstreamer pipeline was created.

I thought I would be able to save a record of when the pipeline was created,
then use the following formula to timestamp the frames appropriately:

(frame timestamp (ms) - pipeline start time (ms)) * 1000000

However, the avenc_flv element fails to encode the buffers as the pts
(presentation time stamp (I believe)) is invalid. The output suggests the
timestamp of the frame buffer is invalid (pts invalid (0) <= last (0)).

My pipeline is defined as follows:

m_pipeline = gst_parse_launch (" appsrc name=stream-transmitter do-timestamp=true is-live=true format=3 ! video/x-raw,width=1280,height=720,framerate=1/15,format=BGR ! videoconvert ! avenc_flv ! flvmux streamable=true ! rtmpsink location='rtmp://localhost/live/jon live=1'", NULL);

And my function to pipe data into the pipeline is as follows:

////////////////////

ProcessedFrameData newestFrame = streamManager->getFrameManager ()->getNewestFrame ();

GstBuffer *buffer;
gint size;
size = newestFrame.size;

buffer = gst_buffer_new_and_alloc (size);

GST_BUFFER_PTS (buffer) = (newestFrame.timestamp - streamManager->getPipelineStartTime ()->count ()) * 1000000;
GST_BUFFER_DURATION (buffer) = 1 / newestFrame.framerate;  

GstMapInfo map;
GstFlowReturn ret;

gst_buffer_map (buffer, &map, GST_MAP_WRITE);
memcpy (map.data, newestFrame.frame.data, size);
gst_buffer_unmap (buffer, &map);
g_signal_emit_by_name (streamManager->getAppsrc (), "push-buffer", buffer, &ret);

gst_buffer_unref (buffer);

if (ret != GST_FLOW_OK)
{
  std::cout << "GST_FLOW_OK FALSE" << std::endl;
  return FALSE;
}
return TRUE;

////////////////////

Could anyone offer some advice for fixing my problem?

Thanks in advance
Jon

P.S. I appriciate there may be some fine tuning required to improve
streaming performance etc, but I was hoping to get a basic RTMP stream
working first before I began tweaking. When outputting to the autovideosink
sink element, the stream works fine, though I do believe that is more
forgiving regards frame timestamps :)




--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[e-mail]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le samedi 08 février 2020 à 16:23 -0600, jonmccormack a écrit :

> Hello all,
> I've been trying to learn how to use Gstreamer for the past few weeks, using
> the appsrc element in a c++ application stream video to an RTMP server.
>
> I have raw frame data coming in from another part of my application, which
> has the timestamp the frame was created in milliseconds since the unix
> epoch.
>
> Being a total beginner with regards to working with video streaming, it
> looks like timestamping the frame buffers that are piped into the gstreamer
> pipeline is essential, especially when creating a networks stream. It looks
> like the buffers are timestamped in nanoseconds since the moment the
> gstreamer pipeline was created.

Not exacty, for live sources (with a GstSegment of 0 to infinity),
which is the case with appsrc, the timestamp should match the running
time this buffer has been created. As you want to use this to tell the
pipeline when to render, you will have to set how much time it took
before these buffers actually reach GStreamer and configure this as the
min-latency.

>
> I thought I would be able to save a record of when the pipeline was created,
> then use the following formula to timestamp the frames appropriately:
>
> (frame timestamp (ms) - pipeline start time (ms)) * 1000000

As you have two clocks, you need to translate from one clock to
another. So you'll need to get the block from GStreamer. And do
something in these lines (assuming everything is nanosecond for
simplicity):

  unix_now = (GstCLockTime)g_get_real_time() * (GstClockTime)1000
  gst_now = gst_clock_get_time();
 
  /* We need find gst_ts in: unix_now - unix_ts == gst_now - gst_ts */
  gst_ts = gst_now - (unix_now - unix_ts)

So now we have a clock timestamp, the buffer PTS has to be running
time, so that will be:

  buf_ts = gst_ts - gst_element_get_base_time(element);

This isn't perfect, prone to scheduling delays, but (unless I made a
mistake) should give you a proper timestamp. If you don't know the
latency, you can always estimate it with:

  latency = (unix_now - unix_ts)

Like on first buffer. But it could also be very variable, in which case
you'd need something a little smarter, and maybe more observation.
There might be a way with a GstClock implementation, and slaving, not
sure if it's simpler or worst though.

>
> However, the avenc_flv element fails to encode the buffers as the pts
> (presentation time stamp (I believe)) is invalid. The output suggests the
> timestamp of the frame buffer is invalid (pts invalid (0) <= last (0)).
>
> My pipeline is defined as follows:
>
>
> And my function to pipe data into the pipeline is as follows:
>
>
> Could anyone offer some advice for fixing my problem?
>
> Thanks in advance
> Jon
>
> P.S. I appriciate there may be some fine tuning required to improve
> streaming performance etc, but I was hoping to get a basic RTMP stream
> working first before I began tweaking. When outputting to the autovideosink
> sink element, the stream works fine, though I do believe that is more
> forgiving regards frame timestamps :)
>
>
>
>
> --
> Sent from: http://gstreamer-devel.966125.n4.nabble.com/
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[jonmccormack]

Thank you for the quick and detailed reply :)

I will update the thread on Monday, with my updated data piping function
(should it work, fingers crossed :))





--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel