request for comments: plan for audio/video policy integration in gstreamer

[Alexander Kanavin]

Hello,

me and a few colleagues at Intel Open Source Technology Center are
working on adding multimedia policy mechanisms to the open source stack
so that it could be used by the automotive industry in IVI (in-vehicle
infotainment) systems. In particular, I've been looking at how to
integrate policy into gstreamer, and the following is the direction we'd
like to take:

1. On IVI systems there is a policy daemon which keeps track of the
system state, has a database of policy rules, and makes decisions about
any ongoing multimedia playback:

a) whether any particular stream should be paused or can continue to
play (think of a phone call interrupting music playback)
b) where any particular stream should be routed (for example front seat
or back seat)

2. All of the above should be hidden from applications; they should not
be concerned with policy or routing and can use gstreamer as they always
did for playback.

3. Policy integration in gstreamer is handled by a policy element. All
implementations of that element subclass from GstPolicy (which is a
subclass of GstElement and brings together things common to all policy
elements). One such common thing to all policy implementations is a
stream 'role' property that applications can set if they want to, for
example 'phonecall' or 'music' (this property affects policy decisions).
The property would be picked up from the policy element and exposed to
applications by playbin.

4.  There is also an autopolicy element that wraps a
distribution-specific policy element if it can find one, or otherwise a
default policy element that does nothing. Autopolicy element could be
located for example in playsink bin just after stream synchronizer
element. Since policy needs to be enforced, there should be no way to
replace it by the applications.

5. The policy element that we plan to implement for IVI use cases would:

a) handle all communication with the policy daemon
b) receive ongoing information from policy daemon about routing and
whether the stream should be paused or resumed, based on that 'role'
property and whether there is audio, video or both in the stream.
c) routing information is passed downstream as GstMeta. Sinks can
cherry-pick and propagate this information to audio and video subsystems.
d) policy element is using gst_element_get_parent() to find out the
top-level pipeline element, and will issue state change requests
(pause/resume) on it as instructed by policy daemon.

We plan to make this work fully open source and hope to have it
upstream, but first of course we'd like to know what you think about it.
Please let us know.


Thanks,
Alex
_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Wim Taymans]

On 10/12/2012 03:44 PM, Alexander Kanavin wrote:

> Hello,
>
> me and a few colleagues at Intel Open Source Technology Center are
> working on adding multimedia policy mechanisms to the open source stack
> so that it could be used by the automotive industry in IVI (in-vehicle
> infotainment) systems. In particular, I've been looking at how to
> integrate policy into gstreamer, and the following is the direction we'd
> like to take:
>
> 1. On IVI systems there is a policy daemon which keeps track of the
> system state, has a database of policy rules, and makes decisions about
> any ongoing multimedia playback:
>
> a) whether any particular stream should be paused or can continue to
> play (think of a phone call interrupting music playback)
> b) where any particular stream should be routed (for example front seat
> or back seat)
>
> 2. All of the above should be hidden from applications; they should not
> be concerned with policy or routing and can use gstreamer as they always
> did for playback.
>
> 3. Policy integration in gstreamer is handled by a policy element. All
> implementations of that element subclass from GstPolicy (which is a
> subclass of GstElement and brings together things common to all policy
> elements). One such common thing to all policy implementations is a
> stream 'role' property that applications can set if they want to, for
> example 'phonecall' or 'music' (this property affects policy decisions).
> The property would be picked up from the policy element and exposed to
> applications by playbin.
>
> 4.  There is also an autopolicy element that wraps a
> distribution-specific policy element if it can find one, or otherwise a
> default policy element that does nothing. Autopolicy element could be
> located for example in playsink bin just after stream synchronizer
> element. Since policy needs to be enforced, there should be no way to
> replace it by the applications.
>
> 5. The policy element that we plan to implement for IVI use cases would:
>
> a) handle all communication with the policy daemon
> b) receive ongoing information from policy daemon about routing and
> whether the stream should be paused or resumed, based on that 'role'
> property and whether there is audio, video or both in the stream.
> c) routing information is passed downstream as GstMeta. Sinks can
> cherry-pick and propagate this information to audio and video subsystems.
> d) policy element is using gst_element_get_parent() to find out the
> top-level pipeline element, and will issue state change requests
> (pause/resume) on it as instructed by policy daemon.
>
> We plan to make this work fully open source and hope to have it
> upstream, but first of course we'd like to know what you think about it.
> Please let us know.

Why no use pulseaudio? I believe it can do all this already. It doesn't
sound like something that GStreamer should do.

Also, you can't pause the pipeline from an element. We have a REQUEST_STATE
message for this purpose (which pulseaudio sends to the pipeline when a
policy requires you to pause a stream).

Wim

>
>
> Thanks,
> Alex
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Cedric Hombourger]

Agreed.

There are indeed different ways of achieving the goals. We had similar requirements in our IVI system and we're having an Audio Manager component pausing the active source when a phone call comes in. For our MediaPlayer, pausing means saving the LastMode data and stopping the pipeline. When the phone call ends, the Audio Manager re-enables the previous source which for the MediaPlayer means recreating the pipeline and playing from where it left of using the LastMode data. The MediaPlayer application we have is as such not concerned with routing or policy but is simply responding to simple source commands. The Audio Manager is a separate component to which all sources of the IVI system (Tuner, MediaPlayer, Phone, etc...). We did not use PulseAudio since the MOST INIC we have on-board is doing all the mixing for us so we are having the Audio Manager tell the various sources which ALSA device to stream to.

Having said that, are there other use-cases that GstPolicy could possibly fulfil?

Cedric

On Oct 12, 2012, at 3:54 PM, Wim Taymans <[hidden email]> wrote:

> On 10/12/2012 03:44 PM, Alexander Kanavin wrote:
>> Hello,
>>
>> me and a few colleagues at Intel Open Source Technology Center are
>> working on adding multimedia policy mechanisms to the open source stack
>> so that it could be used by the automotive industry in IVI (in-vehicle
>> infotainment) systems. In particular, I've been looking at how to
>> integrate policy into gstreamer, and the following is the direction we'd
>> like to take:
>>
>> 1. On IVI systems there is a policy daemon which keeps track of the
>> system state, has a database of policy rules, and makes decisions about
>> any ongoing multimedia playback:
>>
>> a) whether any particular stream should be paused or can continue to
>> play (think of a phone call interrupting music playback)
>> b) where any particular stream should be routed (for example front seat
>> or back seat)
>>
>> 2. All of the above should be hidden from applications; they should not
>> be concerned with policy or routing and can use gstreamer as they always
>> did for playback.
>>
>> 3. Policy integration in gstreamer is handled by a policy element. All
>> implementations of that element subclass from GstPolicy (which is a
>> subclass of GstElement and brings together things common to all policy
>> elements). One such common thing to all policy implementations is a
>> stream 'role' property that applications can set if they want to, for
>> example 'phonecall' or 'music' (this property affects policy decisions).
>> The property would be picked up from the policy element and exposed to
>> applications by playbin.
>>
>> 4.  There is also an autopolicy element that wraps a
>> distribution-specific policy element if it can find one, or otherwise a
>> default policy element that does nothing. Autopolicy element could be
>> located for example in playsink bin just after stream synchronizer
>> element. Since policy needs to be enforced, there should be no way to
>> replace it by the applications.
>>
>> 5. The policy element that we plan to implement for IVI use cases would:
>>
>> a) handle all communication with the policy daemon
>> b) receive ongoing information from policy daemon about routing and
>> whether the stream should be paused or resumed, based on that 'role'
>> property and whether there is audio, video or both in the stream.
>> c) routing information is passed downstream as GstMeta. Sinks can
>> cherry-pick and propagate this information to audio and video subsystems.
>> d) policy element is using gst_element_get_parent() to find out the
>> top-level pipeline element, and will issue state change requests
>> (pause/resume) on it as instructed by policy daemon.
>>
>> We plan to make this work fully open source and hope to have it
>> upstream, but first of course we'd like to know what you think about it.
>> Please let us know.
>
> Why no use pulseaudio? I believe it can do all this already. It doesn't
> sound like something that GStreamer should do.
>
> Also, you can't pause the pipeline from an element. We have a REQUEST_STATE
> message for this purpose (which pulseaudio sends to the pipeline when a
> policy requires you to pause a stream).
>
> Wim
>
>>
>>
>> Thanks,
>> Alex
>> _______________________________________________
>> gstreamer-devel mailing list
>> [hidden email]
>> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Alexander Kanavin]

On 10/12/2012 04:54 PM, Wim Taymans wrote:

> Why no use pulseaudio? I believe it can do all this already. It doesn't
> sound like something that GStreamer should do.
>
> Also, you can't pause the pipeline from an element. We have a REQUEST_STATE
> message for this purpose (which pulseaudio sends to the pipeline when a
> policy requires you to pause a stream).

We're using pulseaudio already :) and we've hit limits with it that can
be addressed by moving policy support to gstreamer:

1) It's not just about audio policy - we also need to have a policy for
video playback. Doing that in pulseaudio would be rather odd, while
gstreamer pipeline is a natural, common spot for it. There's also a
possibility of playing videos that don't have audio.

2) REQUEST_STATE message is supposed to be handled by applications, not
by pipeline. I couldn't find a single application that's actually acting
on it - except gst-launch. It's not realistic to go out and fix them
all, or to track all new applications that are written. Also, we need to
enforce the policy, not to leave it up to applications.

3) While it's possible to use 'pactl suspend-sink 0 1' to pause the
playback, applications that use gstreamer get no notification about it,
or a reason why it has happened. The pipeline just silently stops.
That's not a good user experience.


As for pausing the pipeline from an element, it could be done by
obtaining the top-level pipeline object via gst_element_get_parent() in
a loop and issuing a state change request on that. Is there a technical
reason why this wouldn't work?

Regards,
Alex
_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel