[RFC] provide a bin to GstRtspMediaFactory

[Eloi Bail]

Hi all,

We were thinking about the ability to set / get a bin to a GstRtspMediaFactory.

The use case is to be able to modify an element of the pipeline during a live RTSP session without renegotiating a RTSP stream.

The modification needs to be compatible with the format provided by the SDP.

In our case it is a video-flip which does not modify the video resolution.

The patches in attachment adds the ability to get/set a bin. They are based on a old work of Tristan Matthews [1]

We currently see one limitation:

* After all RTSP client disconnected from the RTSP session, we see a critical issue:

(lt-test-bin:2344): GStreamer-CRITICAL **: gst_ghost_pad_new: assertion '!gst_pad_is_linked (target)' failed

(lt-test-bin:2344): GStreamer-CRITICAL **: gst_pad_set_active: assertion 'GST_IS_PAD (pad)' failed

(lt-test-bin:2344): GStreamer-CRITICAL **: gst_element_add_pad: assertion 'GST_IS_PAD (pad)' failed

** (lt-test-bin:2344): CRITICAL **: gst_rtsp_stream_new: assertion 'GST_IS_PAD (pad)' failed

** (lt-test-bin:2344): CRITICAL **: gst_rtsp_stream_set_multicast_iface: assertion 'GST_IS_RTSP_STREAM (stream)' failed

** (lt-test-bin:2344): CRITICAL **: gst_rtsp_stream_set_profiles: assertion 'GST_IS_RTSP_STREAM (stream)' failed

** (lt-test-bin:2344): CRITICAL **: gst_rtsp_stream_set_protocols: assertion 'GST_IS_RTSP_STREAM (stream)' failed

This issue does not occur with test-launch.

Any comments or questions are welcomed !

Eloi

[1]: https://github.com/tmatth/gst-prototypes/tree/master/gstrtspserver

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le mardi 05 décembre 2017 à 18:00 +0100, Eloi Bail a écrit :
>
> The patches in attachment adds the ability to get/set a bin. They are
> based on a old work of Tristan Matthews [1]

GStreamer project accept and review patches over Bugzilla. Please open
a ticket on bugs.gnome.org, Project GStreamer. Finally, attach you
patches, the format is correct and check the "patch" checkbox.

regards,
Nicolas
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le mardi 05 décembre 2017 à 18:00 +0100, Eloi Bail a écrit :
> We were thinking about the ability to set / get a bin to a
> GstRtspMediaFactory.
> The use case is to be able to modify an element of the pipeline
> during a live RTSP session without renegotiating a RTSP stream.

This does not work conceptually. The factory will be unable to clone
the provided GstBin if there is a second connection and the media is
not shared. You can get the same functionality by connecting the media-
configured signal and then using gst_rtsp_media_get_element() to get
the GstBin create from the template provided to the factory. From there
you can extract the flip element using gst_bin_get_by_name() or the
_recurse_up() variant if you have multiple level of bins. For an
example how to interact with RTSP elements see:

https://github.com/GStreamer/gst-rtsp-server/blob/master/examples/test-appsrc.cs

Nicolas
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Eloi Bail]

On Tue, Dec 5, 2017 at 8:38 PM, Nicolas Dufresne <[hidden email]> wrote:

Le mardi 05 décembre 2017 à 18:00 +0100, Eloi Bail a écrit :
> We were thinking about the ability to set / get a bin to a
> GstRtspMediaFactory.
> The use case is to be able to modify an element of the pipeline
> during a live RTSP session without renegotiating a RTSP stream.

This does not work conceptually. The factory will be unable to clone
the provided GstBin if there is a second connection and the media is
not shared. You can get the same functionality by connecting the media-
configured signal and then using gst_rtsp_media_get_element() to get
the GstBin create from the template provided to the factory. From there
you can extract the flip element using gst_bin_get_by_name() or the
_recurse_up() variant if you have multiple level of bins. For an
example how to interact with RTSP elements see:

https://github.com/GStreamer/gst-rtsp-server/blob/master/examples/test-appsrc.cs

Perfect. It works like a charm.

Thanks !

Eloi

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Eloi Bail]

Perfect. It works like a charm.

Thanks !

Actually we are facing a new challenge.

We wanted to have a  bus message watch for the running pipeline to set/get custom message between pipeline elements. As gst-rtsp-server already set a bus message watch, it was not possible to have 2 message watchs.

That's why we wanted to provide our subclass GstBin with overwritten message handler.

A solution could be to subclass GstRtspMedia, set create_rtpbin function and set a subclass of rtpbin with our custom message handler.

But it looks like I can not subclass rtpbin.

Does my solution makes sense ?

Thanks,

Eloi

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On Wed, 2017-12-06 at 14:28 +0100, Eloi Bail wrote:

>
>
> > Perfect. It works like a charm.
> > Thanks !
> >
>
> Actually we are facing a new challenge.
> We wanted to have a  bus message watch for the running pipeline to
> set/get custom message between pipeline elements. As gst-rtsp-server
> already set a bus message watch, it was not possible to have 2
> message watchs.
>
> That's why we wanted to provide our subclass GstBin with overwritten
> message handler.

You can provide a custom bin for the part of the pipeline that has the
sources up to the payloader, and catch messages of those elements.

All the other elements in the RTSP server pipeline are not under your
control but something the RTSP server manages. Do you need to catch any
messages of those elements, and if so, what for?

--
Sebastian Dröge, Centricular Ltd · https://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Eloi Bail]

You can provide a custom bin for the part of the pipeline that has the
sources up to the payloader, and catch messages of those elements.

All the other elements in the RTSP server pipeline are not under your
control but something the RTSP server manages. Do you need to catch any
messages of those elements, and if so, what for?

Something like this:

videotestsrc -> tee -> queue -> annotationElement -> x264enc -> rtph264pay name=pay0
                             |
                             |
                             |
                             |
                             |
                              -> queue -> motionDetector -> fakesink

motionDetector would send messages to annotationElement.

Otherwise, could I subclass GstRTSPMedia, overwrite handle_message, check my custom message and call the parent message handler of GstRTSPMedia ?

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On Wed, 2017-12-06 at 16:41 +0100, Eloi Bail wrote:

>
>
> > You can provide a custom bin for the part of the pipeline that has the
> > sources up to the payloader, and catch messages of those elements.
> >
> > All the other elements in the RTSP server pipeline are not under your
> > control but something the RTSP server manages. Do you need to catch any
> > messages of those elements, and if so, what for?
> >
> >
> >
>
> Something like this:
>
> videotestsrc -> tee -> queue -> annotationElement -> x264enc -> rtph264pay name=pay0
>                              |
>                              |
>                              |
>                              |
>                              |
>                               -> queue -> motionDetector -> fakesink
>
>
> motionDetector would send messages to annotationElement.
>
> Otherwise, could I subclass GstRTSPMedia, overwrite handle_message,
> check my custom message and call the parent message handler of
> GstRTSPMedia ?

As both elements are at the same level of the bin hierarchy, annotation
element would never see any messages from motion detector. Only the
surrounding bin would see messages.

And the surrounding bin is exactly the "element" that is created by the
RTSP media factory, and you can return your bin subclass there that is
catching messages of its child elements (like the motion detector).

A better approach might be to place the motion detector in front of the
annotation element, so that it can be to information directly on its
output buffers (GstMeta e.g.) and let the annotation element directly
read it from there instead of having yet another way of communication.

Alternatively you could also let the motion detection element know
about the annotation element and have them communicate via direct
function calls.

--
Sebastian Dröge, Centricular Ltd · https://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Eloi Bail]

On Wed, Dec 6, 2017 at 5:17 PM, Sebastian Dröge <[hidden email]> wrote:

As both elements are at the same level of the bin hierarchy, annotation
element would never see any messages from motion detector. Only the
surrounding bin would see messages.

 

yes.

And the surrounding bin is exactly the "element" that is created by the
RTSP media factory, and you can return your bin subclass there that is
catching messages of its child elements (like the motion detector).

This element is created by gst_parse_launch_full with GST_PARSE_FLAG_PLACE_IN_BIN.

Should I then modify grammar.tab.c to add my custom flag GST_PARSE_FLAG_PLACE_IN_MY_BIN which would return a "MyBin" instead of "GstBin" ?

 

A better approach might be to place the motion detector in front of the
annotation element, so that it can be to information directly on its
output buffers (GstMeta e.g.) and let the annotation element directly
read it from there instead of having yet another way of communication.

So then the motion detector would act as a passthrough with my custom metadata ?

 

Alternatively you could also let the motion detection element know
about the annotation element and have them communicate via direct
function calls.

Yes but I would need to implement my own inter-thread communication ?

-

Thanks for your help.

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On Wed, 2017-12-06 at 17:52 +0100, Eloi Bail wrote:
>
> This element is created by gst_parse_launch_full with
> GST_PARSE_FLAG_PLACE_IN_BIN.
> Should I then modify grammar.tab.c to add my custom flag
> GST_PARSE_FLAG_PLACE_IN_MY_BIN which would return a "MyBin" instead
> of "GstBin" ?

Don't use gst_parse_launch() but create the bin and other elements
normally. There's no need to change any GStreamer code.

> > A better approach might be to place the motion detector in front of the
> > annotation element, so that it can be to information directly on its
> > output buffers (GstMeta e.g.) and let the annotation element directly
> > read it from there instead of having yet another way of communication.
>
> So then the motion detector would act as a passthrough with my custom
> metadata ?

Yes

> > Alternatively you could also let the motion detection element know
> > about the annotation element and have them communicate via direct
> > function calls.
>
> Yes but I would need to implement my own inter-thread communication

No, you can just do normal function calls. And as usual with threads,
you have to make sure your code is thread-safe. The messages wouldn't
help with that either.

--
Sebastian Dröge, Centricular Ltd · https://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Eloi Bail]

>
> This element is created by gst_parse_launch_full with
> GST_PARSE_FLAG_PLACE_IN_BIN.
> Should I then modify grammar.tab.c to add my custom flag
> GST_PARSE_FLAG_PLACE_IN_MY_BIN which would return a "MyBin" instead
> of "GstBin" ?

Don't use gst_parse_launch() but create the bin and other elements
normally. There's no need to change any GStreamer code.

Fine. But rtsp-media-factory does not provide a way to get/set the bin except with a "gst-launch" string.

So how do I create the bin and other elements ? Should I subclass the  rtsp-media-factory to create the bin ?

 

> > A better approach might be to place the motion detector in front of the
> > annotation element, so that it can be to information directly on its
> > output buffers (GstMeta e.g.) and let the annotation element directly
> > read it from there instead of having yet another way of communication.
>
> So then the motion detector would act as a passthrough with my custom
> metadata ?

Yes

> > Alternatively you could also let the motion detection element know
> > about the annotation element and have them communicate via direct
> > function calls.
>
> Yes but I would need to implement my own inter-thread communication

No, you can just do normal function calls. And as usual with threads,
you have to make sure your code is thread-safe. The messages wouldn't
help with that either.

ok.

 

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

--

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On Thu, 2017-12-07 at 14:33 +0100, Eloi Bail wrote:

> > >
> > > This element is created by gst_parse_launch_full with
> > > GST_PARSE_FLAG_PLACE_IN_BIN.
> > > Should I then modify grammar.tab.c to add my custom flag
> > > GST_PARSE_FLAG_PLACE_IN_MY_BIN which would return a "MyBin"
> > instead
> > > of "GstBin" ?
> >
> > Don't use gst_parse_launch() but create the bin and other elements
> > normally. There's no need to change any GStreamer code.
>
> Fine. But rtsp-media-factory does not provide a way to get/set the
> bin except with a "gst-launch" string.
> So how do I create the bin and other elements ? Should I subclass
> the  rtsp-media-factory to create the bin ?

Yes

--
Sebastian Dröge, Centricular Ltd · https://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le jeudi 07 décembre 2017 à 14:33 +0100, Eloi Bail a écrit :

> > >
> > > This element is created by gst_parse_launch_full with
> > > GST_PARSE_FLAG_PLACE_IN_BIN.
> > > Should I then modify grammar.tab.c to add my custom flag
> > > GST_PARSE_FLAG_PLACE_IN_MY_BIN which would return a "MyBin"
> > instead
> > > of "GstBin" ?
> >
> > Don't use gst_parse_launch() but create the bin and other elements
> > normally. There's no need to change any GStreamer code.
> >
> >
>
> Fine. But rtsp-media-factory does not provide a way to get/set the
> bin except with a "gst-launch" string.
> So how do I create the bin and other elements ? Should I subclass
> the  rtsp-media-factory to create the bin ?

When it gets more complicated like this, I do subclass the factory and
the media. I then get much better control, but more boiler plate code
of course.

regards,
Nicolas
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Eloi Bail]

When it gets more complicated like this, I do subclass the factory and
the media. I then get much better control, but more boiler plate code
of course.

regards,
Nicolas

Yes, I am a lazy guy but am now a subclass master, so let's subclass !

Thanks

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel