GStreamer-devel

Pipeline freeze when audio source is ended

Classic

List

Threaded

7 messages Options

stic

Pipeline freeze when audio source is ended

Hello everyone,

I have a simple pipeline running with an audio source and video source, muxed together into a tcpserversink.
The problem I have is that the pipeline freeze once the audio source is ended, but I would like it to continue even is there is no audio anymore in the pipeline.
What should I do in such case ?
I tried to drop the EOS signal arriving on the src pad of the audio source bin but it doesn't seem to change anything.
Could someone describe what should be done in such case ?
That would really help me, thank you very much in advance !
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

Michael MacIntosh

Re: Pipeline freeze when audio source is ended

Hey,

If you have an audio and a video source being fed into a muxer, and
assuming you can't get around the source producing the EOS, then you
probably want to drop the EOS signal, and use a blocking probe to
disconnect the old source and replace it with a source that would
produce "silence", because most video file formats expect both streams
to be the same length.

But the problem you are running into is the muxer will block your video
pipeline because it is waiting for audio data, you need to keep the
audio data going if you want it to be unblocked. The muxer is too smart
to assume no data is silence.

Cheers,
Michael.

On 8/9/2017 7:44 AM, [hidden email] wrote:

> Hello everyone,
>
> I have a simple pipeline running with an audio source and video source, muxed together into a tcpserversink.
> The problem I have is that the pipeline freeze once the audio source is ended, but I would like it to continue even is there is no audio anymore in the pipeline.
> What should I do in such case ?
> I tried to drop the EOS signal arriving on the src pad of the audio source bin but it doesn't seem to change anything.
> Could someone describe what should be done in such case ?
> That would really help me, thank you very much in advance !
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

stic

Re: Pipeline freeze when audio source is ended

Thanks a lot Michael for your answer.
This is also something I tried but I failed to get it working.
As you already described I tried to use a blocking probe to replace the audio source (a uridecodebin) by a fakesrc in order to keep having audio buffers in the pipeline.
I also had to post a GstMessage on the bus to make the removal of the audio source and replacement by fakesrc on the main thread (message callback).
After your message I tried again hard but I still fail...

Do you think I can use fakesrc to send empty audio buffer on the pipeline to replace the audio source, or should I use something else ?

I am also wondering when the blocking pad should be unblocked ? Should it be unblocked from the message callback (the callback function receiving the GstMessage) ?
If so how could this be done ?

Here is the code I am using for blocking pad and message callback function:

/*** callback function receiving the GstMessage to replace the audio source ***/
/* Retrieve application custom messages from the bus */
static void application_cb (GstBus *bus, GstMessage *msg, CustomData *data) {
// look for object coming from the audio source only
if (GST_MESSAGE_SRC (msg) == GST_OBJECT (data->asource) && gst_message_has_name (msg, "asourceReplace")) {
// application message from audio source, the message means the elements received EOS and needs to be replaced

// stop element to remove
gst_element_set_state (data->asource, GST_STATE_NULL);
// remove unlinks automatically
GST_DEBUG_OBJECT (data->pipeline, "removing %" GST_PTR_FORMAT, data->asource);
gst_bin_remove (GST_BIN (data->pipeline), data->asource);
// add a new fakesink
data->asource = gst_element_factory_make ("fakesrc", "asource");
if (G_IS_OBJECT(data->asource)) g_object_set (G_OBJECT (data->asource), "is-live", TRUE, NULL);
gst_bin_add (GST_BIN (data->pipeline), data->asource);
// link fakesink
gst_element_link (data->asource, data->audioSourceQueue);

gst_element_sync_state_with_parent (data->asource);
}
}

/*** Callback function from blocking probe ***/
static GstPadProbeReturn
audio_source_event_probe_cb (GstPad * pad, GstPadProbeInfo * info, gpointer user_data)
{
CustomData *data = (CustomData *)user_data;

if (GST_EVENT_TYPE (GST_PAD_PROBE_INFO_DATA (info)) != GST_EVENT_EOS)
return GST_PAD_PROBE_PASS;

gst_pad_remove_probe (pad, GST_PAD_PROBE_INFO_ID (info));

// we need to remove asource and replace it, this can only be done asynchronously, so we post a message on the bus for it
GstBus *bus = gst_element_get_bus (data->pipeline);
if (GST_IS_BUS(bus)) {
gst_bus_post(bus, gst_message_new_application(GST_OBJECT(data->asource), gst_structure_new_empty("asourceReplace")));
gst_object_unref(bus);
}

return GST_PAD_PROBE_DROP; // unblock from here or can it be done from application_cb ?
}

----- Mail original -----
De: "Michael MacIntosh" <[hidden email]>
À: [hidden email]
Envoyé: Mercredi 9 Août 2017 21:19:24
Objet: Re: Pipeline freeze when audio source is ended

Hey,

If you have an audio and a video source being fed into a muxer, and
assuming you can't get around the source producing the EOS, then you
probably want to drop the EOS signal, and use a blocking probe to
disconnect the old source and replace it with a source that would
produce "silence", because most video file formats expect both streams
to be the same length.

But the problem you are running into is the muxer will block your video
pipeline because it is waiting for audio data, you need to keep the
audio data going if you want it to be unblocked. The muxer is too smart
to assume no data is silence.

Cheers,
Michael.

On 8/9/2017 7:44 AM, [hidden email] wrote:

Thornton, Keith

AW: Pipeline freeze when audio source is ended

Hi, I have personally no expierience with this but have you tried using an inputSelector with both your source and faesrc connected and then switching from one to the other?

-----Ursprüngliche Nachricht-----
Von: gstreamer-devel [mailto:[hidden email]] Im Auftrag von [hidden email]
Gesendet: Donnerstag, 10. August 2017 13:53
An: Discussion of the development of and with GStreamer <[hidden email]>
Betreff: Re: Pipeline freeze when audio source is ended

Thanks a lot Michael for your answer.
This is also something I tried but I failed to get it working.
As you already described I tried to use a blocking probe to replace the audio source (a uridecodebin) by a fakesrc in order to keep having audio buffers in the pipeline.
I also had to post a GstMessage on the bus to make the removal of the audio source and replacement by fakesrc on the main thread (message callback).
After your message I tried again hard but I still fail...

Do you think I can use fakesrc to send empty audio buffer on the pipeline to replace the audio source, or should I use something else ?

I am also wondering when the blocking pad should be unblocked ? Should it be unblocked from the message callback (the callback function receiving the GstMessage) ?
If so how could this be done ?

Here is the code I am using for blocking pad and message callback function:

/*** callback function receiving the GstMessage to replace the audio source ***/
/* Retrieve application custom messages from the bus */ static void application_cb (GstBus *bus, GstMessage *msg, CustomData *data) {
// look for object coming from the audio source only
if (GST_MESSAGE_SRC (msg) == GST_OBJECT (data->asource) && gst_message_has_name (msg, "asourceReplace")) {
// application message from audio source, the message means the elements received EOS and needs to be replaced

// stop element to remove
gst_element_set_state (data->asource, GST_STATE_NULL);
// remove unlinks automatically
GST_DEBUG_OBJECT (data->pipeline, "removing %" GST_PTR_FORMAT, data->asource);
gst_bin_remove (GST_BIN (data->pipeline), data->asource);
// add a new fakesink
data->asource = gst_element_factory_make ("fakesrc", "asource");
if (G_IS_OBJECT(data->asource)) g_object_set (G_OBJECT (data->asource), "is-live", TRUE, NULL);
gst_bin_add (GST_BIN (data->pipeline), data->asource);
// link fakesink
gst_element_link (data->asource, data->audioSourceQueue);

gst_element_sync_state_with_parent (data->asource);
}
}

/*** Callback function from blocking probe ***/ static GstPadProbeReturn audio_source_event_probe_cb (GstPad * pad, GstPadProbeInfo * info, gpointer user_data) {
CustomData *data = (CustomData *)user_data;

if (GST_EVENT_TYPE (GST_PAD_PROBE_INFO_DATA (info)) != GST_EVENT_EOS)
return GST_PAD_PROBE_PASS;

gst_pad_remove_probe (pad, GST_PAD_PROBE_INFO_ID (info));

// we need to remove asource and replace it, this can only be done asynchronously, so we post a message on the bus for it
GstBus *bus = gst_element_get_bus (data->pipeline);
if (GST_IS_BUS(bus)) {
gst_bus_post(bus, gst_message_new_application(GST_OBJECT(data->asource), gst_structure_new_empty("asourceReplace")));
gst_object_unref(bus);
}

return GST_PAD_PROBE_DROP; // unblock from here or can it be done from application_cb ?
}

----- Mail original -----
De: "Michael MacIntosh" <[hidden email]>
À: [hidden email]
Envoyé: Mercredi 9 Août 2017 21:19:24
Objet: Re: Pipeline freeze when audio source is ended

Hey,

If you have an audio and a video source being fed into a muxer, and assuming you can't get around the source producing the EOS, then you probably want to drop the EOS signal, and use a blocking probe to disconnect the old source and replace it with a source that would produce "silence", because most video file formats expect both streams to be the same length.

But the problem you are running into is the muxer will block your video pipeline because it is waiting for audio data, you need to keep the audio data going if you want it to be unblocked. The muxer is too smart to assume no data is silence.

Cheers,
Michael.

On 8/9/2017 7:44 AM, [hidden email] wrote:

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

stic

Re: AW: Pipeline freeze when audio source is ended

Thanks,
I am now trying to use the "concat" element which seems to be perfect for what I would like to do (the idea is quite similar to use an inputSelector) but I am now facing the following problem:
I need to use it with a uridecodebin element and a secondary audio source that could be a fakesrc.
But the uridecodebin needs to be the 1st source, so I think I need to link the 2nd source from the pad-added signal from uridecodebin, only after uridecodebin has been linked.
But if I do so I have errors from the 2nd source saying "pad not activated yet" and "streaming stopped, reason not-linked (-1)"

What should I do to be able to link concat with uridecodebin in 1st position ?

----- Mail original -----
De: "Keith Thornton" <[hidden email]>
À: "Discussion of the development of and with GStreamer" <[hidden email]>
Envoyé: Jeudi 10 Août 2017 14:52:59
Objet: AW: Pipeline freeze when audio source is ended

Hi, I have personally no expierience with this but have you tried using an inputSelector with both your source and faesrc connected and then switching from one to the other?

-----Ursprüngliche Nachricht-----
Von: gstreamer-devel [mailto:[hidden email]] Im Auftrag von [hidden email]
Gesendet: Donnerstag, 10. August 2017 13:53
An: Discussion of the development of and with GStreamer <[hidden email]>
Betreff: Re: Pipeline freeze when audio source is ended

Thanks a lot Michael for your answer.
This is also something I tried but I failed to get it working.
As you already described I tried to use a blocking probe to replace the audio source (a uridecodebin) by a fakesrc in order to keep having audio buffers in the pipeline.
I also had to post a GstMessage on the bus to make the removal of the audio source and replacement by fakesrc on the main thread (message callback).
After your message I tried again hard but I still fail...

Do you think I can use fakesrc to send empty audio buffer on the pipeline to replace the audio source, or should I use something else ?

I am also wondering when the blocking pad should be unblocked ? Should it be unblocked from the message callback (the callback function receiving the GstMessage) ?
If so how could this be done ?

Here is the code I am using for blocking pad and message callback function:

/*** callback function receiving the GstMessage to replace the audio source ***/
/* Retrieve application custom messages from the bus */ static void application_cb (GstBus *bus, GstMessage *msg, CustomData *data) {
// look for object coming from the audio source only
if (GST_MESSAGE_SRC (msg) == GST_OBJECT (data->asource) && gst_message_has_name (msg, "asourceReplace")) {
// application message from audio source, the message means the elements received EOS and needs to be replaced

// stop element to remove
gst_element_set_state (data->asource, GST_STATE_NULL);
// remove unlinks automatically
GST_DEBUG_OBJECT (data->pipeline, "removing %" GST_PTR_FORMAT, data->asource);
gst_bin_remove (GST_BIN (data->pipeline), data->asource);
// add a new fakesink
data->asource = gst_element_factory_make ("fakesrc", "asource");
if (G_IS_OBJECT(data->asource)) g_object_set (G_OBJECT (data->asource), "is-live", TRUE, NULL);
gst_bin_add (GST_BIN (data->pipeline), data->asource);
// link fakesink
gst_element_link (data->asource, data->audioSourceQueue);

gst_element_sync_state_with_parent (data->asource);
}
}

/*** Callback function from blocking probe ***/ static GstPadProbeReturn audio_source_event_probe_cb (GstPad * pad, GstPadProbeInfo * info, gpointer user_data) {
CustomData *data = (CustomData *)user_data;

if (GST_EVENT_TYPE (GST_PAD_PROBE_INFO_DATA (info)) != GST_EVENT_EOS)
return GST_PAD_PROBE_PASS;

gst_pad_remove_probe (pad, GST_PAD_PROBE_INFO_ID (info));

// we need to remove asource and replace it, this can only be done asynchronously, so we post a message on the bus for it
GstBus *bus = gst_element_get_bus (data->pipeline);
if (GST_IS_BUS(bus)) {
gst_bus_post(bus, gst_message_new_application(GST_OBJECT(data->asource), gst_structure_new_empty("asourceReplace")));
gst_object_unref(bus);
}

return GST_PAD_PROBE_DROP; // unblock from here or can it be done from application_cb ?
}

----- Mail original -----
De: "Michael MacIntosh" <[hidden email]>
À: [hidden email]
Envoyé: Mercredi 9 Août 2017 21:19:24
Objet: Re: Pipeline freeze when audio source is ended

Hey,

If you have an audio and a video source being fed into a muxer, and assuming you can't get around the source producing the EOS, then you probably want to drop the EOS signal, and use a blocking probe to disconnect the old source and replace it with a source that would produce "silence", because most video file formats expect both streams to be the same length.

But the problem you are running into is the muxer will block your video pipeline because it is waiting for audio data, you need to keep the audio data going if you want it to be unblocked. The muxer is too smart to assume no data is silence.

Cheers,
Michael.

On 8/9/2017 7:44 AM, [hidden email] wrote:

Michael MacIntosh

Re: AW: Pipeline freeze when audio source is ended

Out of curiosity, if you replace the fakesrc with an appsrc does it at
least link? Also I am assuming you are using
gst_element_get_request_pad to get the pads on the sink side of the
concat element. Also make sure that whatever source you use has the
same caps as the audio source you were using before to avoid
complications. Otherwise I am not sure, but it sounds like you are on
the right track. Connecting the silence source in the pad_added
callback should be the way to go.

Cheers,
Michael.

On 8/11/2017 3:53 AM, [hidden email] wrote:

> Thanks,
> I am now trying to use the "concat" element which seems to be perfect for what I would like to do (the idea is quite similar to use an inputSelector) but I am now facing the following problem:
> I need to use it with a uridecodebin element and a secondary audio source that could be a fakesrc.
> But the uridecodebin needs to be the 1st source, so I think I need to link the 2nd source from the pad-added signal from uridecodebin, only after uridecodebin has been linked.
> But if I do so I have errors from the 2nd source saying "pad not activated yet" and "streaming stopped, reason not-linked (-1)"
>
> What should I do to be able to link concat with uridecodebin in 1st position ?
>
> ----- Mail original -----
> De: "Keith Thornton" <[hidden email]>
> À: "Discussion of the development of and with GStreamer" <[hidden email]>
> Envoyé: Jeudi 10 Août 2017 14:52:59
> Objet: AW: Pipeline freeze when audio source is ended
>
> Hi, I have personally no expierience with this but have you tried using an inputSelector with both your source and faesrc connected and then switching from one to the other?
>
> -----Ursprüngliche Nachricht-----
> Von: gstreamer-devel [mailto:[hidden email]] Im Auftrag von [hidden email]
> Gesendet: Donnerstag, 10. August 2017 13:53
> An: Discussion of the development of and with GStreamer <[hidden email]>
> Betreff: Re: Pipeline freeze when audio source is ended
>
> Thanks a lot Michael for your answer.
> This is also something I tried but I failed to get it working.
> As you already described I tried to use a blocking probe to replace the audio source (a uridecodebin) by a fakesrc in order to keep having audio buffers in the pipeline.
> I also had to post a GstMessage on the bus to make the removal of the audio source and replacement by fakesrc on the main thread (message callback).
> After your message I tried again hard but I still fail...
>
> Do you think I can use fakesrc to send empty audio buffer on the pipeline to replace the audio source, or should I use something else ?
>
> I am also wondering when the blocking pad should be unblocked ? Should it be unblocked from the message callback (the callback function receiving the GstMessage) ?
> If so how could this be done ?
>
> Here is the code I am using for blocking pad and message callback function:
>
> /*** callback function receiving the GstMessage to replace the audio source ***/
> /* Retrieve application custom messages from the bus */ static void application_cb (GstBus *bus, GstMessage *msg, CustomData *data) {
> // look for object coming from the audio source only
> if (GST_MESSAGE_SRC (msg) == GST_OBJECT (data->asource) && gst_message_has_name (msg, "asourceReplace")) {
> // application message from audio source, the message means the elements received EOS and needs to be replaced
>
> // stop element to remove
> gst_element_set_state (data->asource, GST_STATE_NULL);
> // remove unlinks automatically
> GST_DEBUG_OBJECT (data->pipeline, "removing %" GST_PTR_FORMAT, data->asource);
> gst_bin_remove (GST_BIN (data->pipeline), data->asource);
> // add a new fakesink
> data->asource = gst_element_factory_make ("fakesrc", "asource");
> if (G_IS_OBJECT(data->asource)) g_object_set (G_OBJECT (data->asource), "is-live", TRUE, NULL);
> gst_bin_add (GST_BIN (data->pipeline), data->asource);
> // link fakesink
> gst_element_link (data->asource, data->audioSourceQueue);
>
> gst_element_sync_state_with_parent (data->asource);
> }
> }
>
> /*** Callback function from blocking probe ***/ static GstPadProbeReturn audio_source_event_probe_cb (GstPad * pad, GstPadProbeInfo * info, gpointer user_data) {
> CustomData *data = (CustomData *)user_data;
>
> if (GST_EVENT_TYPE (GST_PAD_PROBE_INFO_DATA (info)) != GST_EVENT_EOS)
> return GST_PAD_PROBE_PASS;
>
> gst_pad_remove_probe (pad, GST_PAD_PROBE_INFO_ID (info));
>
> // we need to remove asource and replace it, this can only be done asynchronously, so we post a message on the bus for it
> GstBus *bus = gst_element_get_bus (data->pipeline);
> if (GST_IS_BUS(bus)) {
> gst_bus_post(bus, gst_message_new_application(GST_OBJECT(data->asource), gst_structure_new_empty("asourceReplace")));
> gst_object_unref(bus);
> }
>
> return GST_PAD_PROBE_DROP; // unblock from here or can it be done from application_cb ?
> }
>
>
> ----- Mail original -----
> De: "Michael MacIntosh" <[hidden email]>
> À: [hidden email]
> Envoyé: Mercredi 9 Août 2017 21:19:24
> Objet: Re: Pipeline freeze when audio source is ended
>
> Hey,
>
> If you have an audio and a video source being fed into a muxer, and assuming you can't get around the source producing the EOS, then you probably want to drop the EOS signal, and use a blocking probe to disconnect the old source and replace it with a source that would produce "silence", because most video file formats expect both streams to be the same length.
>
> But the problem you are running into is the muxer will block your video pipeline because it is waiting for audio data, you need to keep the audio data going if you want it to be unblocked. The muxer is too smart to assume no data is silence.
>
> Cheers,
> Michael.
>
>
> On 8/9/2017 7:44 AM, [hidden email] wrote:
>> Hello everyone,
>>
>> I have a simple pipeline running with an audio source and video source, muxed together into a tcpserversink.
>> The problem I have is that the pipeline freeze once the audio source is ended, but I would like it to continue even is there is no audio anymore in the pipeline.
>> What should I do in such case ?
>> I tried to drop the EOS signal arriving on the src pad of the audio source bin but it doesn't seem to change anything.
>> Could someone describe what should be done in such case ?
>> That would really help me, thank you very much in advance !
>> _______________________________________________
>> gstreamer-devel mailing list
>> [hidden email]
>> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

Michael MacIntosh

Re: AW: Pipeline freeze when audio source is ended

In reply to this post by stic

One tactic that might be worth trying is have a queue ready for the
audio and use that to connect to the concat element in the first
position, then use your fakesrc element and link it to the second
position, then connect uridecodebin to the queue.

So something like:

uridecodebin queue -> concat
fakesrc->

then in your callback link the uridecodebin to the queue:
uridecodebin -> queue -> concat
fakesrc->

That way you should be able to link the concat element properly before
hitting ready.

Also, if the fakesrc is giving you issues, it might be worth it to use
appsrc instead and just make your own silence. Because if I'm not
mistaken fakesrc is in the debug plugin, so your mileage may vary.

Cheers,
Michael.

On 8/11/2017 3:53 AM, [hidden email] wrote:

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel