New, also help on dealing with multiple audio streams using gstreamer-rs

[Michael Johnson]

To whom it may concern,

I was invited to join here by Sebastian for discussion on problems.

Looking through the src and the docs it's very hard to find an instance where you can extract information about the stream info so you can switch audio streams (for example from french to english) from element::Element when using parse_launch.

I feel it needs some flags, but they are not documented in the rust version. Can anyone point me in the right direction?

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On Wed, 2020-12-16 at 10:00 +0000, Michael Johnson wrote:

To whom it may concern,

I was invited to join here by Sebastian for discussion on problems.

Looking through the src and the docs it's very hard to find an instance where you can extract information about the stream info so you can switch audio streams (for example from french to english) from element::Element when using parse_launch.

You would do that the same way as in C by using the GObject properties, signals and action signals on the playbin element.

The playbin example should help here: https://gitlab.freedesktop.org/gstreamer/gstreamer-rs/blob/0.16/examples/src/bin/playbin.rs

I feel it needs some flags, but they are not documented in the rust version. Can anyone point me in the right direction?

See the commented flags code in the example.

Let me know if something's still unclear after reading that code.

-- 

Sebastian Dröge, Centricular Ltd · https://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Michael Johnson]

I remember reading through this. And running the commented code resulted in an error due to not having a FlagsClass which doesn't exist in the docs nor the file mentioned, which is used to get the video and audio

Do you know where this class is?

Many thanks

On Wed, 16 Dec 2020, 13:27 Sebastian Dröge, <[hidden email]> wrote:

On Wed, 2020-12-16 at 10:00 +0000, Michael Johnson wrote:

To whom it may concern,

I was invited to join here by Sebastian for discussion on problems.

Looking through the src and the docs it's very hard to find an instance where you can extract information about the stream info so you can switch audio streams (for example from french to english) from element::Element when using parse_launch.

You would do that the same way as in C by using the GObject properties, signals and action signals on the playbin element.

The playbin example should help here: https://gitlab.freedesktop.org/gstreamer/gstreamer-rs/blob/0.16/examples/src/bin/playbin.rs

I feel it needs some flags, but they are not documented in the rust version. Can anyone point me in the right direction?

See the commented flags code in the example.

Let me know if something's still unclear after reading that code.

-- 

Sebastian Dröge, Centricular Ltd · https://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On Wed, 2020-12-16 at 13:39 +0000, Michael Johnson wrote:

I remember reading through this. And running the commented code resulted in an error due to not having a FlagsClass which doesn't exist in the docs nor the file mentioned, which is used to get the video and audio

Yes, it's part of the GLib bindings: https://gstreamer.pages.freedesktop.org/gstreamer-rs/glib/struct.FlagsClass.html

-- 

Sebastian Dröge, Centricular Ltd · https://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Michael Johnson]

Ah I see, this implements a toggle on the video audio and subs streams. Is there a convenience flag that allows for switching stream / counting the amount of audio streams?

I'm very glad you shown me glib, I feel I've been trying to brute force gstreamer-rs without this

On Wed, 16 Dec 2020, 13:43 Sebastian Dröge, <[hidden email]> wrote:

On Wed, 2020-12-16 at 13:39 +0000, Michael Johnson wrote:

I remember reading through this. And running the commented code resulted in an error due to not having a FlagsClass which doesn't exist in the docs nor the file mentioned, which is used to get the video and audio

Yes, it's part of the GLib bindings: https://gstreamer.pages.freedesktop.org/gstreamer-rs/glib/struct.FlagsClass.html

-- 

Sebastian Dröge, Centricular Ltd · https://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On Wed, 2020-12-16 at 13:56 +0000, Michael Johnson wrote:

Ah I see, this implements a toggle on the video audio and subs streams. Is there a convenience flag that allows for switching stream / counting the amount of audio streams?

Check the playbin documentation: https://gstreamer.freedesktop.org/documentation/playback/playbin.html

There is the "n-audio" property for the number of audio streams, for example.

There also is the "audio-changed" signal that is emitted whenever the number or order of audio streams changed.

-- 

Sebastian Dröge, Centricular Ltd · https://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Michael Johnson]

Then this has no relation to flags surely? (In terms of switching streams)

Yes I read through that page too and again found no counterpart in gstreamer-rs. Is it in glib then?

On Wed, 16 Dec 2020, 14:14 Sebastian Dröge, <[hidden email]> wrote:

On Wed, 2020-12-16 at 13:56 +0000, Michael Johnson wrote:

Ah I see, this implements a toggle on the video audio and subs streams. Is there a convenience flag that allows for switching stream / counting the amount of audio streams?

Check the playbin documentation: https://gstreamer.freedesktop.org/documentation/playback/playbin.html

There is the "n-audio" property for the number of audio streams, for example.

There also is the "audio-changed" signal that is emitted whenever the number or order of audio streams changed.

-- 

Sebastian Dröge, Centricular Ltd · https://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On Wed, 2020-12-16 at 14:25 +0000, Michael Johnson wrote:

Then this has no relation to flags surely? (In terms of switching streams)

Check the playbin docs for details what the flags are doing. The flags are for completely disabling audio, for example. Or for enabling/disabling other features of playbin.

Yes I read through that page too and again found no counterpart in gstreamer-rs. Is it in glib then?

For doing what exactly? You can use the playbin API (GObject properties, signals, action signals) like any others. That would be via the GLib bindings.

The playbin.rs example I linked earlier makes use of various of such APIs.

-- 

Sebastian Dröge, Centricular Ltd · https://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Michael Johnson]

Ah yes. Properties is highly useful. However, n-audio is by default 0, which is strange because I have 5 audio tracks

Also setting 

pipeline.set_property("current-audio",&2) doesn't change the track. I'm pretty sure it's because I'm giving the wrong second argument

How can I get n_audio to give the correct amount and set audio? Is it because I'm doing it from the pipeline and not through an audio source?

File is mkv

Many thanks

On Wed, 16 Dec 2020, 14:29 Sebastian Dröge, <[hidden email]> wrote:

On Wed, 2020-12-16 at 14:25 +0000, Michael Johnson wrote:

Then this has no relation to flags surely? (In terms of switching streams)

Check the playbin docs for details what the flags are doing. The flags are for completely disabling audio, for example. Or for enabling/disabling other features of playbin.

Yes I read through that page too and again found no counterpart in gstreamer-rs. Is it in glib then?

For doing what exactly? You can use the playbin API (GObject properties, signals, action signals) like any others. That would be via the GLib bindings.

The playbin.rs example I linked earlier makes use of various of such APIs.

-- 

Sebastian Dröge, Centricular Ltd · https://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On Wed, 2020-12-16 at 20:25 +0000, Michael Johnson wrote:

Ah yes. Properties is highly useful. However, n-audio is by default 0, which is strange because I have 5 audio tracks

Connect to the "audio-changed" signal and whenever that is emitted, check the value.

Before the playbin is in PAUSED state, you will always get 0 audio/video/text tracks.

Also setting 

pipeline.set_property("current-audio",&2) doesn't change the track. I'm pretty sure it's because I'm giving the wrong second argument

Same as above, you have to set it after the playbin is PLAYING.

-- 

Sebastian Dröge, Centricular Ltd · https://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Michael Johnson]

Ah I haven't got anything to do with the properties connected to a signal? Do you reckon that's the issue then

I'm setting and getting the properties all from playing. So I can't get n-audio from playing state? 

On Thu, 17 Dec 2020, 08:19 Sebastian Dröge, <[hidden email]> wrote:

On Wed, 2020-12-16 at 20:25 +0000, Michael Johnson wrote:

Ah yes. Properties is highly useful. However, n-audio is by default 0, which is strange because I have 5 audio tracks

Connect to the "audio-changed" signal and whenever that is emitted, check the value.

Before the playbin is in PAUSED state, you will always get 0 audio/video/text tracks.

Also setting 

pipeline.set_property("current-audio",&2) doesn't change the track. I'm pretty sure it's because I'm giving the wrong second argument

Same as above, you have to set it after the playbin is PLAYING.

-- 

Sebastian Dröge, Centricular Ltd · https://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Michael Johnson]

Another point here, connecting to the audio-stream signal and printing the current-audio yields -1 until I physically close the window when it gets the correct audio stream here

Also, I imagine when I want to bind changing stream to a key press I need to emit this signal to be caught. But I can't access any data structures from the callback, so it'll always chose a constant stream. Am I missing something? here 

On Thu, 17 Dec 2020, 09:46 Michael Johnson, <[hidden email]> wrote:

Ah I haven't got anything to do with the properties connected to a signal? Do you reckon that's the issue then

I'm setting and getting the properties all from playing. So I can't get n-audio from playing state? 

On Thu, 17 Dec 2020, 08:19 Sebastian Dröge, <[hidden email]> wrote:

On Wed, 2020-12-16 at 20:25 +0000, Michael Johnson wrote:

Ah yes. Properties is highly useful. However, n-audio is by default 0, which is strange because I have 5 audio tracks

Connect to the "audio-changed" signal and whenever that is emitted, check the value.

Before the playbin is in PAUSED state, you will always get 0 audio/video/text tracks.

Also setting 

pipeline.set_property("current-audio",&2) doesn't change the track. I'm pretty sure it's because I'm giving the wrong second argument

Same as above, you have to set it after the playbin is PLAYING.

-- 

Sebastian Dröge, Centricular Ltd · https://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On Thu, 2020-12-17 at 09:46 +0000, Michael Johnson wrote:

Ah I haven't got anything to do with the properties connected to a signal? Do you reckon that's the issue then

I don't understand your question.

I'm setting and getting the properties all from playing. So I can't get n-audio from playing state? 

Paused or playing state.

Maybe you can provide a testcase for your problem? That might make it clearer what goes wrong.

-- 

Sebastian Dröge, Centricular Ltd · https://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On Thu, 2020-12-17 at 10:46 +0000, Michael Johnson wrote:

Another point here, connecting to the audio-stream signal and printing the current-audio yields -1 until I physically close the window when it gets the correct audio stream here

That seems wrong, please provide a testcase for that.

Also, I imagine when I want to bind changing stream to a key press I need to emit this signal to be caught. But I can't access any data structures from the callback, so it'll always chose a constant stream. Am I missing something? here 

You can pass any heap allocated data without references to stack memory into the signal handler closure.

A common pattern would be to store your state in a std::sync::Arc to give it reference counting, and use a std::sync::Mutex for getting interior mutability of whatever state you need to change.

-- 

Sebastian Dröge, Centricular Ltd · https://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Michael Johnson]

I meant to say that I was getting and setting properties outside of the callback used in signal.connect originally ie

playbin.set_property(..) with no association to connecting signals. 

I'll construct a minimal working example as a reply to this email shortly

On Thu, 17 Dec 2020, 11:00 Sebastian Dröge, <[hidden email]> wrote:

On Thu, 2020-12-17 at 09:46 +0000, Michael Johnson wrote:

Ah I haven't got anything to do with the properties connected to a signal? Do you reckon that's the issue then

I don't understand your question.

I'm setting and getting the properties all from playing. So I can't get n-audio from playing state? 

Paused or playing state.

Maybe you can provide a testcase for your problem? That might make it clearer what goes wrong.

-- 

Sebastian Dröge, Centricular Ltd · https://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Michael Johnson]

Do you have a link to a video that has multiple audio streams for the test case, I'm only finding single audio stream files 

On Thu, 17 Dec 2020, 11:02 Sebastian Dröge, <[hidden email]> wrote:

On Thu, 2020-12-17 at 10:46 +0000, Michael Johnson wrote:

Another point here, connecting to the audio-stream signal and printing the current-audio yields -1 until I physically close the window when it gets the correct audio stream here

That seems wrong, please provide a testcase for that.

Also, I imagine when I want to bind changing stream to a key press I need to emit this signal to be caught. But I can't access any data structures from the callback, so it'll always chose a constant stream. Am I missing something? here 

You can pass any heap allocated data without references to stack memory into the signal handler closure.

A common pattern would be to store your state in a std::sync::Arc to give it reference counting, and use a std::sync::Mutex for getting interior mutability of whatever state you need to change.

-- 

Sebastian Dröge, Centricular Ltd · https://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Michael Johnson]

As requested a toy example,

Here we see that the audio-changed emit never activates the connect, maybe because it emits before the video has began playing?

I'm probably doing something very wrong 

On Thu, 17 Dec 2020, 11:24 Michael Johnson, <[hidden email]> wrote:

Do you have a link to a video that has multiple audio streams for the test case, I'm only finding single audio stream files 

On Thu, 17 Dec 2020, 11:02 Sebastian Dröge, <[hidden email]> wrote:

On Thu, 2020-12-17 at 10:46 +0000, Michael Johnson wrote:

Another point here, connecting to the audio-stream signal and printing the current-audio yields -1 until I physically close the window when it gets the correct audio stream here

That seems wrong, please provide a testcase for that.

Also, I imagine when I want to bind changing stream to a key press I need to emit this signal to be caught. But I can't access any data structures from the callback, so it'll always chose a constant stream. Am I missing something? here 

You can pass any heap allocated data without references to stack memory into the signal handler closure.

A common pattern would be to store your state in a std::sync::Arc to give it reference counting, and use a std::sync::Mutex for getting interior mutability of whatever state you need to change.

-- 

Sebastian Dröge, Centricular Ltd · https://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On Thu, 2020-12-17 at 12:28 +0000, Michael Johnson wrote:

As requested a toy example,

Here we see that the audio-changed emit never activates the connect, maybe because it emits before the video has began playing?

I'm probably doing something very wrong 

Yeah, you're ignoring compiler warnings here that tell you that you're ignoring potential errors :)

Make sure to at least `unwrap()` `Result`s instead of ignoring them, then you immediately know if you did something wrong there.

In this case the emitting of the "audio-changed" signal fails because of the wrong number of arguments.

But that also doesn't matter here: that signal is supposed to be emitted by playbin whenever something changes, not by you.

After removing that line what I get is

current audio track is -1

number of audio tracks is 1

current audio track is -1

number of audio tracks is 2

current audio track is -1

number of audio tracks is 3

That looks as expected. Every signal emission you're setting the current audio track to 2, but only the last time there actually is a 3rd (0, 1, 2) audio track available.

So in the end it should use the 3rd audio track.

If you change it for example to set it to 1 then you'll see that during the last signal emission the current audio track is 1.

For getting state into the signal handler closure, take a look at the overlay-composition.rs example and what it does with the DrawingContext inside the Arc<Mutex<_>> there.

-- 

Sebastian Dröge, Centricular Ltd · https://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Michael Johnson]

All those warnings are related to not handling the None type, I didn't think that would have any bearing. 

Wait. You're not supposed to use emit? Then why does it exist? Surely you emit a signal to then be caught by the connect. By the logic of setting the value of 2 every them then current audio should go -1,-1,2 not -1,-1,-1. It yields 2 on exit. Which was my point

What is the alternative to using emit then, and why does the alternative still relate to connect? 

On Thu, 17 Dec 2020, 13:36 Sebastian Dröge, <[hidden email]> wrote:

On Thu, 2020-12-17 at 12:28 +0000, Michael Johnson wrote:

As requested a toy example,

Here we see that the audio-changed emit never activates the connect, maybe because it emits before the video has began playing?

I'm probably doing something very wrong 

Yeah, you're ignoring compiler warnings here that tell you that you're ignoring potential errors :)

Make sure to at least `unwrap()` `Result`s instead of ignoring them, then you immediately know if you did something wrong there.

In this case the emitting of the "audio-changed" signal fails because of the wrong number of arguments.

But that also doesn't matter here: that signal is supposed to be emitted by playbin whenever something changes, not by you.

After removing that line what I get is

current audio track is -1

number of audio tracks is 1

current audio track is -1

number of audio tracks is 2

current audio track is -1

number of audio tracks is 3

That looks as expected. Every signal emission you're setting the current audio track to 2, but only the last time there actually is a 3rd (0, 1, 2) audio track available.

So in the end it should use the 3rd audio track.

If you change it for example to set it to 1 then you'll see that during the last signal emission the current audio track is 1.

For getting state into the signal handler closure, take a look at the overlay-composition.rs example and what it does with the DrawingContext inside the Arc<Mutex<_>> there.

-- 

Sebastian Dröge, Centricular Ltd · https://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Michael Johnson]

Oh wait I see what you mean about the first point. My eyes can't see a minus sign apparently :P

On Thu, 17 Dec 2020, 13:49 Michael Johnson, <[hidden email]> wrote:

All those warnings are related to not handling the None type, I didn't think that would have any bearing. 

Wait. You're not supposed to use emit? Then why does it exist? Surely you emit a signal to then be caught by the connect. By the logic of setting the value of 2 every them then current audio should go -1,-1,2 not -1,-1,-1. It yields 2 on exit. Which was my point

What is the alternative to using emit then, and why does the alternative still relate to connect? 

On Thu, 17 Dec 2020, 13:36 Sebastian Dröge, <[hidden email]> wrote:

On Thu, 2020-12-17 at 12:28 +0000, Michael Johnson wrote:

As requested a toy example,

Here we see that the audio-changed emit never activates the connect, maybe because it emits before the video has began playing?

I'm probably doing something very wrong 

Yeah, you're ignoring compiler warnings here that tell you that you're ignoring potential errors :)

Make sure to at least `unwrap()` `Result`s instead of ignoring them, then you immediately know if you did something wrong there.

In this case the emitting of the "audio-changed" signal fails because of the wrong number of arguments.

But that also doesn't matter here: that signal is supposed to be emitted by playbin whenever something changes, not by you.

After removing that line what I get is

current audio track is -1

number of audio tracks is 1

current audio track is -1

number of audio tracks is 2

current audio track is -1

number of audio tracks is 3

That looks as expected. Every signal emission you're setting the current audio track to 2, but only the last time there actually is a 3rd (0, 1, 2) audio track available.

So in the end it should use the 3rd audio track.

If you change it for example to set it to 1 then you'll see that during the last signal emission the current audio track is 1.

For getting state into the signal handler closure, take a look at the overlay-composition.rs example and what it does with the DrawingContext inside the Arc<Mutex<_>> there.

-- 

Sebastian Dröge, Centricular Ltd · https://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel