Gracefully Handle (or ignore) GstFlowReturns

[gotsring]

*TL;DR - How do I intercept and handle a bad GstFlowReturn (GST_FLOW_ERROR)
to prevent stopping the pipeline, without modifying the plugin source code?
I am using C++ on MSYS2 using GStreamer 1.16.2.*


I'm trying to process an MJPEG stream from an IP camera starting with the
following GStreamer pipeline:

/souphttpsrc location="http://192.168.18.101/mjpeg" is-live=true retries=-1
! queue leaky=2 max-size-buffers=10 ! multipartdemux boundary=fbdr !
multiqueue ! jpegparse ! videorate ! 'image/jpeg, framerate=20/1' ! jpegdec
! d3dvideosink sync=false async=false/

Whether using gst-launch or the c++ program I'm writing, I am able to get a
smooth live view of the camera, and this works until multipartdemux receives
a bad header. When this occurs, it returns a GST_FLOW_ERROR which travels to
the bus event listener and eventually kills the pipeline. The pipeline may
run flawlessly for minutes or hours before this occurs, making debugging
difficult.

How can I intercept and handle GstFlowReturns? I expect bad frames every now
and then since the cameras stream images over a network connection, so
errors are expected and shouldn't be fatal. Ideally, I would drop the bad
buffer and allow videorate to replace the missing frame. The closest thing I
could find is here:
http://gstreamer-devel.966125.n4.nabble.com/Ignoring-errors-of-fdsink-td972387.html
<http://gstreamer-devel.966125.n4.nabble.com/Ignoring-errors-of-fdsink-td972387.html>
. Most internet searches return results for finding the cause of a
GST_FLOW_ERROR, not handling one.

I tried adding a probe to the multipartdemux sink pad to listen for events
(GST_PAD_PROBE_TYPE_EVENT_BOTH ), but the errors aren't caught by this. I've
also tried removing the multipartdemux and multiqueue from the pipeline, but
that just pushes the error downstream to jpegdemux.

Environment is Windows 10. Pipeline is launched either using gst-launch-1.0
or C++ code compiled in MSYS2. GStreamer version is 1.16.2 (x64) installed
using MSYS2's pacman. Application log is attached, output from gst-launch
(--gst-debug=3) is below.


*Background for the C++ program log*
The program creates 4 identical pipelines to connect to 4 cameras. There are
extra tees and branches built in to allow linking of processing/recording
branches as needed, but these were not enabled when the error occured. GLib
is used in general and for logging, and d3dvideosink windows are captured
into a GTK+3 GUI. This particular log shows it running for almost 16 hours
without error. 2 of the 4 cameras failed when I came in, but this was
probably by chance, not due to interaction (I was on my laptop on a separate
network, no one else was in the office). I doubt this is a bandwidth problem
since the error occurs when streaming 1 camera (gst-launch, around 50Mbps)
or 4 cameras (around 200 Mbps) with no appreciable change in error
frequency.


*Logs*
$ gst-launch-1.0.exe --gst-debug=3 souphttpsrc
location="http://192.168.18.101/mjpeg?res=half" is-live=true retries=-1 !
queue leaky=2 max-size-buffers=2 ! multipartdemux boundary=fbdr ! multiqueue
! jpegparse ! videorate ! 'image/jpeg, framerate=20/1' ! jpegdec ! queue !
d3dvideosink

0:00:00.020476900  2124        34454b0 WARN      GST_PLUGIN_LOADING
gstplugin.c:792:_priv_gst_plugin_load_file_for_registry: module_open failed:
'C:\msys64\mingw64\lib\gstreamer-1.0\libgstdvdread.dll': The specified
module could not be found.

(gst-launch-1.0:2124): GStreamer-WARNING **: 09:25:59.204: Failed to load
plugin 'C:\msys64\mingw64\lib\gstreamer-1.0\libgstdvdread.dll':
'C:\msys64\mingw64\lib\gstreamer-1.0\libgstdvdread.dll': The specified
module could not be found.
0:00:00.023569700  2124        34454b0 WARN      GST_PLUGIN_LOADING
gstplugin.c:792:_priv_gst_plugin_load_file_for_registry: module_open failed:
'C:\msys64\mingw64\lib\gstreamer-1.0\libgstresindvd.dll': The specified
module could not be found.

(gst-launch-1.0:2124): GStreamer-WARNING **: 09:25:59.207: Failed to load
plugin 'C:\msys64\mingw64\lib\gstreamer-1.0\libgstresindvd.dll':
'C:\msys64\mingw64\lib\gstreamer-1.0\libgstresindvd.dll': The specified
module could not be found.
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
0:00:00.108345400  2124        34454b0 WARN               structure
gststructure.c:1861:priv_gst_structure_append_to_gstring: No value transform
to serialize field 'session' of type 'SoupSession'
Got context from element 'souphttpsrc0': gst.soup.session=context,
session=(SoupSession)NULL, force=(boolean)false;
Setting pipeline to PLAYING ...
New clock: GstSystemClock
0:00:02.440240200  2124        2521a80 FIXME           videodecoder
gstvideodecoder.c:944:gst_video_decoder_drain_out:<jpegdec0> Sub-class
should implement drain()
0:00:04.402283600  2124        2521900 WARN                basesink
gstbasesink.c:3003:gst_base_sink_is_too_late:<d3dvideosink0> warning: A lot
of buffers are being dropped.
0:00:04.402307600  2124        2521900 WARN                basesink
gstbasesink.c:3003:gst_base_sink_is_too_late:<d3dvideosink0> warning: There
may be a timestamping problem, or this computer is too slow.
WARNING: from element /GstPipeline:pipeline0/GstD3DVideoSink:d3dvideosink0:
A lot of buffers are being dropped.
Additional debug info:
../gstreamer-1.16.2/libs/gst/base/gstbasesink.c(3003):
gst_base_sink_is_too_late ():
/GstPipeline:pipeline0/GstD3DVideoSink:d3dvideosink0:
There may be a timestamping problem, or this computer is too slow.
0:00:06.173914500  2124        2521960 WARN          multipartdemux
multipartdemux.c:505:multipart_parse_header:<multipartdemux0> error:
Boundary not found in the multipart header
0:00:06.174099900  2124        25219c0 WARN                 basesrc
gstbasesrc.c:3072:gst_base_src_loop:<souphttpsrc0> error: Internal data
stream error.
0:00:06.174137000  2124        25219c0 WARN                 basesrc
gstbasesrc.c:3072:gst_base_src_loop:<souphttpsrc0> error: streaming stopped,
reason error (-5)
0:00:06.174165400  2124        25219c0 WARN                   queue
gstqueue.c:988:gst_queue_handle_sink_event:<queue0> error: Internal data
stream error.
0:00:06.174192700  2124        25219c0 WARN                   queue
gstqueue.c:988:gst_queue_handle_sink_event:<queue0> error: streaming
stopped, reason error (-5)
ERROR: from element
/GstPipeline:pipeline0/GstMultipartDemux:multipartdemux0: Could not
demultiplex stream.
Additional debug info:
../gst-plugins-good-1.16.2/gst/multipart/multipartdemux.c(505):
multipart_parse_header ():
/GstPipeline:pipeline0/GstMultipartDemux:multipartdemux0:
Boundary not found in the multipart header
Execution ended after 0:00:06.069975900
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

CapProc_2020-07-07_16-46-54.log
<http://gstreamer-devel.966125.n4.nabble.com/file/t379531/CapProc_2020-07-07_16-46-54.log>  




--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Tim Müller]

On Wed, 2020-07-08 at 11:34 -0500, gotsring wrote:

Hi,

> *TL;DR - How do I intercept and handle a bad GstFlowReturn
> (GST_FLOW_ERROR) to prevent stopping the pipeline, without modifying
> the plugin source code?

There's an errorignore element in gst-plugins-bad (we should really
move that to core in the next cycle).

https://gstreamer.freedesktop.org/documentation/debugutilsbad/errorignore.html

Cheers
 Tim

--
Tim Müller, Centricular Ltd - http://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[gotsring]

Tim Müller wrote
> There's an errorignore element in gst-plugins-bad (we should really
> move that to core in the next cycle).

I have added errorignore to create the following pipeline:

souphttpsrc location="http://192.168.18.101/mjpeg" is-live=true retries=-1 !
queue leaky=2 max-size-buffers=10 ! errorignore ignore-error=true
convert-to=0 ! multipartdemux boundary=fbdr ! multiqueue ! jpegparse !
videorate ! 'image/jpeg, framerate=20/1' ! jpegdec ! d3dvideosink sync=false
async=false

This indeed appears to prevent the GST_FLOW_ERROR from travelling upstream
to the souphttpsrc. In order to restart the errorignore element after an
error (allowing it to drop the bad frame, then continue operation), I have a
section in the bus event listener that tests if the posted error is from
multipartdemux. If so, it sends a GST_EVENT_RECONFIGURE to the errorignore
sink pad using:

GstPad* errorignore_sinkpad = gst_element_get_static_pad(data->errorignore,
"sink");
if (errorignore_sinkpad != NULL) {
    gst_pad_push_event(errorignore_sinkpad, gst_event_new_reconfigure());
    gst_object_unref(errorignore_sinkpad);
}


This does reset the errorignore element, but I have the following questions:

1. The error continues/repeatedly occurs
By this, I mean that as soon as multipartdemux receives a bad buffer, it
immediately throws a GST_FLOW_ERROR. Errorignore catches it to prevent it
going to the source, then the bus event listener catches the error and
restarts errorignore. Buffers start flowing out of errorignore into
multipartdemux again, immediately triggering the same error, ad infinitum.
Documentation says errorignore unrefs the buffers after catching an error,
but do I also have to send a FLUSH signal or something to clean out
something? I'm running tests now, as well as verifying the error isn't
originating upstream, but I figured I'd ask if I was missing something.

2. The method to restart errorignore looks sloppy.
Catching an error in the bus seems a bit awkward. Is there a cleaner way
send a signal that restarts errorignore? Should I be triggering the
RECONFIGURE event with a probe or a different event callback attached to the
errorignore element itself? I may end up tinkering with the errorignore
source to add an option to silently continue after an error without the need
to send a signal.



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel