[rust] waylandsink with video overlay

[Joel Winarske-2]

I am trying to add a video overlay to a wayland surface.

I'm seeing this runtime error:

Error! Received error from /GstPipeline:pipeline0/GstWaylandSink:waylandsink0: Application did not provide a wayland display handle (debug: Some("../ext/wayland/gstwaylandsink.c(946): gst_wayland_sink_set_window_handle (): /GstPipeline:pipeline0/GstWaylandSink:waylandsink0:\nwaylandsink cannot use an externally-supplied surface without an externally-supplied display handle. Consider providing a display handle from your application with GstContext"))

What is the pattern for adding display handle with GstContext in Rust?  There doesn't appear to be a reference to this in any of the samples.

I believe I have set the video_overlay window handle (pointer to wl_surface) correctly:
https://github.com/jwinarske/waylandsink-with-video-overlay-rs/blob/main/src/main.rs#L170

Thanks,

Joel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On Wed, 2021-01-27 at 15:54 -0800, Joel Winarske wrote:

> I am trying to add a video overlay to a wayland surface.
>
> I'm seeing this runtime error:
> Error! Received error from /GstPipeline:pipeline0/GstWaylandSink:waylandsink0: Application did not provide a wayland display handle (debug: Some("../ext/wayland/gstwaylandsink.c(946): gst_wayland_sink_set_window_handle (): /GstPipeline:pipeline0/GstWaylandSink:waylandsink0:\nwaylandsink cannot use an externally-supplied surface without an externally-supplied display handle. Consider providing a display handle from your application with GstContext"))
>
> What is the pattern for adding display handle with GstContext in Rust?
>
> There doesn't appear to be a reference to this in any of the samples.
>
> I believe I have set the video_overlay window handle (pointer to wl_surface) correctly:
> https://github.com/jwinarske/waylandsink-with-video-overlay-rs/blob/main/src/main.rs#L170

Wayland is a bit special compared to all other platforms in this regard
unfortunately.

You can find the relevant code in C here:
https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/blob/66eed8a61d14d08daff2d37b5ae22e8e7c8c05a7/tests/examples/waylandsink/main.c#L84-132

Basically in addition to setting the wl_surface as window handle, you
also need to set a GstContext on the pipeline (or at least the sink)
with the wl_display.

This requires using the libgstwayland library, which unfortunately is
not included in the bindings yet.

You could call those functions via Rust FFI directly after making sure
that your application links to the library, or you could replicate that
function in Rust too:

https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/blob/66eed8a61d14d08daff2d37b5ae22e8e7c8c05a7/gst-libs/gst/wayland/wayland.c#L43-51

Due to the raw pointer type used there you'll still have to make use of
some unsafe code though.

I see that you're trying exactly that here already:
https://github.com/jwinarske/waylandsink-with-video-overlay-rs/blob/main/src/main.rs#L57-L64

The missing bit would be to create a GValue from the pointer

  let mut value = glib::Value::from_type(glib::Type::Pointer);
  unsafe {
    use glib::translate::ToGlibPtrMut;
    glib_sys::g_value_set_pointer(value.to_glib_none_mut().0, handle);
  }

And then setting that in the structure of the context.

--
Sebastian Dröge, Centricular Ltd · https://www.centricular.com


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Joel Winarske-2]

Hi Sebastian,

Thanks for your response, and all your great work on GST Rust.

I'm down to https://github.com/jwinarske/waylandsink-with-video-overlay-rs/blob/main/src/main.rs#L59-L72

error[E0277]: the trait bound `Value: SetValue` is not satisfied
  --> src/main.rs:70:25
   |
70 |         s.set("handle", &value);
   |                         ^^^^^^ the trait `SetValue` is not implemented for `Value`
   |
   = note: required because of the requirements on the impl of `ToSendValue` for `Value`

Thanks,

Joel

On Thu, Jan 28, 2021 at 12:09 AM Sebastian Dröge <[hidden email]> wrote:

On Wed, 2021-01-27 at 15:54 -0800, Joel Winarske wrote:
> I am trying to add a video overlay to a wayland surface.
>
> I'm seeing this runtime error:
> Error! Received error from /GstPipeline:pipeline0/GstWaylandSink:waylandsink0: Application did not provide a wayland display handle (debug: Some("../ext/wayland/gstwaylandsink.c(946): gst_wayland_sink_set_window_handle (): /GstPipeline:pipeline0/GstWaylandSink:waylandsink0:\nwaylandsink cannot use an externally-supplied surface without an externally-supplied display handle. Consider providing a display handle from your application with GstContext"))
>
> What is the pattern for adding display handle with GstContext in Rust?
>
> There doesn't appear to be a reference to this in any of the samples.
>
> I believe I have set the video_overlay window handle (pointer to wl_surface) correctly:
> https://github.com/jwinarske/waylandsink-with-video-overlay-rs/blob/main/src/main.rs#L170

Wayland is a bit special compared to all other platforms in this regard
unfortunately.

You can find the relevant code in C here:
https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/blob/66eed8a61d14d08daff2d37b5ae22e8e7c8c05a7/tests/examples/waylandsink/main.c#L84-132

Basically in addition to setting the wl_surface as window handle, you
also need to set a GstContext on the pipeline (or at least the sink)
with the wl_display.

This requires using the libgstwayland library, which unfortunately is
not included in the bindings yet.

You could call those functions via Rust FFI directly after making sure
that your application links to the library, or you could replicate that
function in Rust too:

https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/blob/66eed8a61d14d08daff2d37b5ae22e8e7c8c05a7/gst-libs/gst/wayland/wayland.c#L43-51

Due to the raw pointer type used there you'll still have to make use of
some unsafe code though.

I see that you're trying exactly that here already:
https://github.com/jwinarske/waylandsink-with-video-overlay-rs/blob/main/src/main.rs#L57-L64

The missing bit would be to create a GValue from the pointer

  let mut value = glib::Value::from_type(glib::Type::Pointer);
  unsafe {
    use glib::translate::ToGlibPtrMut;
    glib_sys::g_value_set_pointer(value.to_glib_none_mut().0, handle);
  }

And then setting that in the structure of the context.

--
Sebastian Dröge, Centricular Ltd · https://www.centricular.com


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On Thu, 2021-01-28 at 08:57 -0800, Joel Winarske wrote:

>
> I'm down to https://github.com/jwinarske/waylandsink-with-video-overlay-rs/blob/main/src/main.rs#L59-L72
>
> error[E0277]: the trait bound `Value: SetValue` is not satisfied
>   --> src/main.rs:70:25
>    |
> 70 |         s.set("handle", &value);
>    |                         ^^^^^^ the trait `SetValue` is not implemented for `Value`
>    |
>    = note: required because of the requirements on the impl of `ToSendValue` for `Value`

It indeed needs a bit more massaging. The following code should do it.

    {
        let context = context.get_mut().unwrap();
        let s = context.get_mut_structure();
        let value = unsafe {
            use gst::glib::translate::*;
            use std::mem;

            let handle = display.c_ptr();
            let mut value = mem::MaybeUninit::zeroed();
            gst::gobject_sys::g_value_init(value.as_mut_ptr(), gst::gobject_sys::G_TYPE_POINTER);
            gst::gobject_sys::g_value_set_pointer(value.as_mut_ptr(), handle as *mut c_void);
            gst::glib::SendValue::from_glib_none(&value.assume_init() as *const _)
        };
        s.set_value("handle", value);
    }

Apart from everything else there's also the problem that you need to
tell the type system that your value is actually thread-safe. So the
easiest/shortest here is actually to just create the whole GValue like
we would be doing in C.

--
Sebastian Dröge, Centricular Ltd · https://www.centricular.com


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Joel Winarske-2]

Works great, thank you!

On Thu, Jan 28, 2021 at 9:26 AM Sebastian Dröge <[hidden email]> wrote:

On Thu, 2021-01-28 at 08:57 -0800, Joel Winarske wrote:
>
> I'm down to https://github.com/jwinarske/waylandsink-with-video-overlay-rs/blob/main/src/main.rs#L59-L72
>
> error[E0277]: the trait bound `Value: SetValue` is not satisfied
>   --> src/main.rs:70:25
>    |
> 70 |         s.set("handle", &value);
>    |                         ^^^^^^ the trait `SetValue` is not implemented for `Value`
>    |
>    = note: required because of the requirements on the impl of `ToSendValue` for `Value`

It indeed needs a bit more massaging. The following code should do it.

    {
        let context = context.get_mut().unwrap();
        let s = context.get_mut_structure();
        let value = unsafe {
            use gst::glib::translate::*;
            use std::mem;

            let handle = display.c_ptr();
            let mut value = mem::MaybeUninit::zeroed();
            gst::gobject_sys::g_value_init(value.as_mut_ptr(), gst::gobject_sys::G_TYPE_POINTER);
            gst::gobject_sys::g_value_set_pointer(value.as_mut_ptr(), handle as *mut c_void);
            gst::glib::SendValue::from_glib_none(&value.assume_init() as *const _)
        };
        s.set_value("handle", value);
    }

Apart from everything else there's also the problem that you need to
tell the type system that your value is actually thread-safe. So the
easiest/shortest here is actually to just create the whole GValue like
we would be doing in C.

--
Sebastian Dröge, Centricular Ltd · https://www.centricular.com


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel