Measure element latency

[clubberlang]

Hi,

I am working a WebrRTC RTP streaming application which has a low latency requirements using GStreamer 1.14. I would like to calculate latency, and emit it as a metrics on an element basis. I have several ideas, but I'm not sure if they are valid.

1. GStreamer Tracer -- This has too much burden on the CPU as far as I tested.
2. Latency query -- Latency query calculates latency for the pipeline/element, and I think it only updates when elements are added or removed.
3. Buffer probe with PTS -- I thought PTS will be a unique ID and will not be changed, and we can calculate the running time (absolute time - base time) before the buffer comes into the element, and calculate the running time after the buffer comes out of the element -- However, it seems that buffers can either be dropped or changed -- I wonder in this case, will the buffer PTS be changed?
4. Buffer probe/new plugin with GstMeta -- If the PTS is not reliable, I thought about adding the running time timestamp into buffers using GstMeta before coming into the element, and when the buffers come out of the element, we can check the added timestamp with the current running time which is the latency measurement -- However, again, will the metadata that I added be changed in the element?

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le mer. 5 juin 2019 17 h 55, Taoming Huang <[hidden email]> a écrit :

Hi,

I am working a WebrRTC RTP streaming application which has a low latency requirements using GStreamer 1.14. I would like to calculate latency, and emit it as a metrics on an element basis. I have several ideas, but I'm not sure if they are valid.

1. GStreamer Tracer -- This has too much burden on the CPU as far as I tested.

Since 1.16 the latency tracer supports per element latency, and also trace the latency being reported. The per element latency trace adds a bit of load, but I've never seen anything close to 1ms overhead (and is constant), so nothing close to a problem for the quality of the measurements. The tracing of the reported latency is near to free. All algorithm in this tracer are 0(1).

I'm sure you can backport this tracer to 1.14 without issues if needed.

1. Latency query -- Latency query calculates latency for the pipeline/element, and I think it only updates when elements are added or removed.

This is tracked by the new latency tracer. When an element has a latency change, a latency message is posted. It's up to the application to decide when to update.

1. Buffer probe with PTS -- I thought PTS will be a unique ID and will not be changed, and we can calculate the running time (absolute time - base time) before the buffer comes into the element, and calculate the running time after the buffer comes out of the element -- However, it seems that buffers can either be dropped or changed -- I wonder in this case, will the buffer PTS be changed?

Timestamp on buffers don't really help with measuring latency. 

1. Buffer probe/new plugin with GstMeta -- If the PTS is not reliable, I thought about adding the running time timestamp into buffers using GstMeta before coming into the element, and when the buffers come out of the element, we can check the added timestamp with the current running time which is the latency measurement -- However, again, will the metadata that I added be changed in the element?

Same.

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Thornton, Keith]

Hi,

I check the pipeline latency using a method similar to your last suggestion. If you define your own metadata, nobody is going to change it.

 

Von: gstreamer-devel <[hidden email]> Im Auftrag von Taoming Huang
Gesendet: Mittwoch, 5. Juni 2019 23:48
An: [hidden email]
Betreff: Measure element latency

 

Hi,

 

I am working a WebrRTC RTP streaming application which has a low latency requirements using GStreamer 1.14. I would like to calculate latency, and emit it as a metrics on an element basis. I have several ideas, but I'm not sure if they are valid.

1. GStreamer Tracer -- This has too much burden on the CPU as far as I tested.
2. Latency query -- Latency query calculates latency for the pipeline/element, and I think it only updates when elements are added or removed.
3. Buffer probe with PTS -- I thought PTS will be a unique ID and will not be changed, and we can calculate the running time (absolute time - base time) before the buffer comes into the element, and calculate the running time after the buffer comes out of the element -- However, it seems that buffers can either be dropped or changed -- I wonder in this case, will the buffer PTS be changed?
4. Buffer probe/new plugin with GstMeta -- If the PTS is not reliable, I thought about adding the running time timestamp into buffers using GstMeta before coming into the element, and when the buffers come out of the element, we can check the added timestamp with the current running time which is the latency measurement -- However, again, will the metadata that I added be changed in the element?

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[clubberlang]

Hey,

Thanks for sharing. May I ask do you find the latency you calculated
useful/accurate enough?



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[clubberlang]

Hey,

Thanks for sharing!

For GStreamer Tracer, I will definitely checkout 1.16 version. But another
problem is that, what if I only want latency metrics for a few elements (not
for every element). Is there a way to specify?

For the buffer PTS, I didn't intend to use it as a timestamp, but more for
identifying. But with the GstMeta approach, I don't really need PTS. I would
insert a running time timestamp in the buffer before coming in to the
element. And calculate a new one coming out of the buffer. The new timestamp
- the inserted one would be the latency.



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le jeudi 06 juin 2019 à 13:09 -0500, clubberlang a écrit :
> Hey,
>
> Thanks for sharing!
>
> For GStreamer Tracer, I will definitely checkout 1.16 version. But another
> problem is that, what if I only want latency metrics for a few elements (not
> for every element). Is there a way to specify?

You can pass parameters to tracers. It's not currently implemented, but
using the leaks tracer as inspiration, it could be:

  GST_TRACERS="latency(flags=element,filter=mycustomelement0+myotherlement0"

Now that question is how to do that in a speedy manner. Another way is
just to use "grep" or gst-debug-log-viewer to filter. I also have used
LibreOffice Calc importation (using the raw log) and then use a pivot
table.

>
> For the buffer PTS, I didn't intend to use it as a timestamp, but more for
> identifying. But with the GstMeta approach, I don't really need PTS. I would
> insert a running time timestamp in the buffer before coming in to the
> element. And calculate a new one coming out of the buffer. The new timestamp
> - the inserted one would be the latency.
>
>
>
> --
> Sent from: http://gstreamer-devel.966125.n4.nabble.com/
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[clubberlang]

Hey Nicolas,

Thanks for the information! I am wondering if it's possible to propagate
this message to the application level? i.e. Is it possible to post this
information to the BUS maybe or somewhere similar so that the application
level can get this information?



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Michael Gruner]

You can set the GST_DEBUG_FILE env variable to that the output get’s written to a file, instead of stderr. Then your app could consume that. 

Another rather complex alternative is to create your own log handler. Take a look at:

https://developer.gnome.org/gstreamer/stable/gstreamer-GstInfo.html#gst-debug-add-log-function

https://developer.gnome.org/gstreamer/stable/gstreamer-GstInfo.html#GstLogFunction

This way you app could install a callback and filter out every tracer logs received.

On Jun 11, 2019, at 11:19 AM, clubberlang <[hidden email]> wrote:

Hey Nicolas,

Thanks for the information! I am wondering if it's possible to propagate
this message to the application level? i.e. Is it possible to post this
information to the BUS maybe or somewhere similar so that the application
level can get this information?



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le mar. 11 juin 2019 16 h 40, Michael Gruner <[hidden email]> a écrit :

You can set the GST_DEBUG_FILE env variable to that the output get’s written to a file, instead of stderr. Then your app could consume that. 

Another rather complex alternative is to create your own log handler. Take a look at:

https://developer.gnome.org/gstreamer/stable/gstreamer-GstInfo.html#gst-debug-add-log-function

https://developer.gnome.org/gstreamer/stable/gstreamer-GstInfo.html#GstLogFunction

This way you app could install a callback and filter out every tracer logs received.

I also think that long term we need tracer specific hook, so we could avoid parsing log strings and could implement sysprof and other trace formats.

On Jun 11, 2019, at 11:19 AM, clubberlang <[hidden email]> wrote:

Hey Nicolas,

Thanks for the information! I am wondering if it's possible to propagate
this message to the application level? i.e. Is it possible to post this
information to the BUS maybe or somewhere similar so that the application
level can get this information?



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[clubberlang]

Is there a way to sample the tracers -- I would like the latency tracers to
not write to disk as on a per-buffer basis. Also, is it the right approach
to build our own custom version of latency on top of GstTracer?



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le mercredi 12 juin 2019 à 15:21 -0500, clubberlang a écrit :
> Is there a way to sample the tracers -- I would like the latency tracers to
> not write to disk as on a per-buffer basis. Also, is it the right approach
> to build our own custom version of latency on top of GstTracer?

There is already a fork at RidgeRun. My opinion is that it is best to
help improve the current upstream latency tracer instead of forking.
The forks tends to carry bugs that inaccuracies that we have worked
hard to fix upstream.

Now about pacing the output of the latency, this is best done outside
of the tracer. Trace points are not bound to a large data structure, so
to page the output of the latency tracer. So best is if we get these
tracer output hooks upstream, and then you can implement something on
top, rather then inside of it.

>
>
>
> --
> Sent from: http://gstreamer-devel.966125.n4.nabble.com/
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[clubberlang]

>  So best is if we get these tracer output hooks upstream, and then you can
implement something on top, rather then inside of it.

Is this what Michael suggested to hook a log handler? The reason I would
like to emit those latency measurements is because I think writing to disks
increases the CPU usage drastically. We want these measurements to be
emitted as application metrics and in a cost-effective way -- not eat up too
much CPU cycles for writing to file.



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le jeu. 13 juin 2019 11 h 10, clubberlang <[hidden email]> a écrit :

>  So best is if we get these tracer output hooks upstream, and then you can
implement something on top, rather then inside of it.

Is this what Michael suggested to hook a log handler? The reason I would
like to emit those latency measurements is because I think writing to disks
increases the CPU usage drastically. We want these measurements to be
emitted as application metrics and in a cost-effective way -- not eat up too
much CPU cycles for writing to file.

What I'm proposing is to add hooks so that application can gain access to the GVariant containing the trace. Right now, we only support turning this to string and logging through gstlog facility. Making metrics out of the raw trace is left to application (like sysprof, gst-stats etc.)

--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[chakra]

Not fully verified, but recently came across the tool
https://github.com/gdesmott/gst-log-parser/ which seems to generated few
metrics from logs.

May be its not complete but certainly can be extended or updated in near
future

Controlling the log level to TRACER when on request basis as mentioned at
http://gstreamer-devel.966125.n4.nabble.com/Dynamic-Gst-Debug-Level-td4678504.html
might help to generate logs

Not sure if this is an appropriate approach but as I am looking into same
requirement as yours, came across this possible solution

Thanks



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel