textoverlay stability issue

[GStreamer-devel mailing list]

Hi all,

We are using textoverlay in our pipeline to dynamically update text on a camera feed. It is not the optimal way to do this since it introduces a very high CPU overhead (without the text overlay it uses ~20% with the overlay its ~60% in 1080p at 30fps) but there were no other big CPU users so it was not crucial. 

After 6-8 hours of operation we see that the stream is stopped and the CPU usage of the pipeline drops to 1% of CPU.

Our pipeline:

nvarguscamerasrc sensor-id=0 sensor-mode=0 gainrange="1 16" ispdigitalgainrange="1 1"

! video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)NV12, framerate=(fraction)30/1

! nvvidconv ! textoverlay name=text_overlay ! video/x-raw,format=I420

! nvvidconv ! nvv4l2vp8enc bitrate=8000000 control-rate=1 ! rtpvp8pay mtu=1400 ! udpsink auto-multicast=true clients=<DCL_UDP_SINK_CLIENTS>

We tried to debug it in:

• info log we dont have any printouts (unfortunately we are not able to set the debug level higher since logs would fill up the disk)
• we added the 'silent=false' option to 'nvarguscamerasrc' to verify that frames still arriving from the camera but they do
• did some exploratory testing to remove elements from the pipeline and if we remove the textoverlay the issue is not happening

This is how I get a reference to the overlay and set it up:

textOverlay = gst_bin_get_by_name(GST_BIN(pipelineElement), getName().c_str());

g_assert_nonnull(textOverlay);

g_object_set(textOverlay,

"font-desc", FONT_TYPE.c_str(),

"shaded-background", true,

"shading-value", 40,

"xpos", 0.0,

"ypos", 0.0,

"wrap-mode", -1,

"halignment", /* position */ 0,

"valignment", /* position */ 3,

NULL);

and this is how I update the text on it from a thread:

g_object_set(textOverlay, "text", firstLine.c_str(), NULL);

After each cycle I call 'clear' on the string to empty the previous content.

Also after each update we sleep the thread for 0.3 seconds, we tried to play around with the refresh rate but it does not seem to have any effect.

Do you have any idea what can cause this? What I am doing wrong here? Or any suggestions how could we debug it further?

Thanks for your help!

Bests,

Peter

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[GStreamer-devel mailing list]

Hi Peter,

> We tried to debug it in:
>  * info log we dont have any printouts (unfortunately we are not able
> to set the debug level higher since logs would fill up the disk)

You can remove the default log function with

  gst_debug_remove_log_function(NULL)

and do

  gst_debug_add_ring_buffer_logger(..)

and then grab the last X MB of logs using

  gst_debug_ring_buffer_logger_get_logs()

when you detect that things have gone wrong.

Cheers
 Tim

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[GStreamer-devel mailing list]

Hi Tim,

Thanks! Such thing would would be great way to debug. Is there any environment variable which I can use to configure this ring buffer logs?

I mean I can set this ringbuffer up with gst_debug_remove_log_function(NULL) and gst_debug_add_ring_buffer_logger but we were not able to detect the problem from within the application so far. We only notice that the CPU load drops and the stream is stopped and as I guess I should call the gst_debug_ring_buffer_logger_get_logs from within the application and I dont really have a trigger point.

I tried to add a callback on the pipeline to print status messages / state changes with:

bus = gst_element_get_bus (pipeline);

gst_bus_add_signal_watch (bus);

g_signal_connect (bus, "message", G_CALLBACK (cb_message), pipeline);

and then:

static void

cb_message (GstBus * bus, GstMessage * msg, GstElement* pipeline)

{

INFO("bus message: " + std::string(gst_message_type_get_name(GST_MESSAGE_TYPE (msg))));

}

But this is remain silent as well.

So is there a way to configure this behaviour / trigger the log  flushing externally? Or setup a log rotation? (all I saw so far is the GST_DEBUG_FILE)

Thanks!

Bests,

Peter

On 2021. May 12., at 11:38, Tim Müller <[hidden email]> wrote:

Hi Peter,

We tried to debug it in:
 * info log we dont have any printouts (unfortunately we are not able
to set the debug level higher since logs would fill up the disk)

You can remove the default log function with

 gst_debug_remove_log_function(NULL)

and do

 gst_debug_add_ring_buffer_logger(..)

and then grab the last X MB of logs using

 gst_debug_ring_buffer_logger_get_logs()

when you detect that things have gone wrong.

Cheers
Tim

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[GStreamer-devel mailing list]

Hi Peter,

There's no way to set it up externally, it requires code in the
application.


> We only notice that the CPU load drops and the stream is stopped and
> as I guess I should call the gst_debug_ring_buffer_logger_get_logs
> from within the application and I dont really have a trigger point.

You could monitor cpu load in the application, or buffer flows.

But if streaming stops entirely there should usually be an error
message on the pipeline bus.

There's also a watchdog element that can post a message if it hasn't
seen data for a while, but I guess for that you need your bus handler
to work.


> I tried to add a callback on the pipeline to print status messages /
> state changes with:
>
>
>  bus = gst_element_get_bus (pipeline);
>  gst_bus_add_signal_watch (bus);
>  g_signal_connect (bus, "message", G_CALLBACK (cb_message),
> pipeline);

That will only work if you run a glib/gtk main loop (or Qt main loop
also on Linux).

Alternatives are:

 - set_sync_handler(), but then your callback gets called from random
gstreamer streaming threads, possibly in parallel. Can also be used to
wake up other event loops

 - just pop messages every now and then from whatever event loop you're
using.

Cheers
 Tim


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel