Push source scheduling and the create function

[Martin Kelly]

Hi,

I apologize in advance if some documentation makes this question clear;
I searched but have not found an answer. I have an issue with a
Gstreamer application that I cannot figure out. Specifically, I'm trying
two different pipeline configurations that I would expect to yield
identical results, but they are not. One of these cases is breaking
while the other appears to be working. Below I describe the setup:

Case 1:
- create one pipeline
- use gst_bin_add_many to add many bins to it
- the first bin is of the form "some-video-element ! [encode and many
other elements] udpsink"
- all other bins are in the form "my-push-src ! fakesink"
- set the pipeline to playing

Case 2:
- create one pipeline per bin via gst_pipeline_new + gst_bin_add
- the first pipeline is of the form "some-video-element ! [encode and
many other elements] udpsink"
- set each pipeline to playing

In both cases, I see the video element playing correctly. In case 1, I'm
seeing that the create function in "my-push-src" is being called once
for each push source and never being called again. In case 2, create
continually gets called, and everything seems OK. It seems that somehow
the scheduling is different between cases 1 and 2, and I'm guessing the
video element's scheduling is somehow changing that of the push source
elements.

I don't understand the interplay between push sources and other sources,
so I'm not sure how to debug this. I'm wondering if someone could give
me a pointer as to what may be wrong or where I should look.

In case it matters, I'm using gstreamer 1.6.3 on Android, and
"my-push-src" is my own custom element, so it's entirely possible it has
some bug that I haven't caught.

Thanks in advance,
Martin
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Tim Müller]

On Fri, 2016-04-29 at 08:53 -0700, Martin Kelly wrote:

Hi Martin,

> Case 1:
> - create one pipeline
> - use gst_bin_add_many to add many bins to it
> - the first bin is of the form "some-video-element ! [encode and
> many 
> other elements] udpsink"
> - all other bins are in the form "my-push-src ! fakesink"
> - set the pipeline to playing
>
> Case 2:
> - create one pipeline per bin via gst_pipeline_new + gst_bin_add
> - the first pipeline is of the form "some-video-element ! [encode
> and many other elements] udpsink"
> - set each pipeline to playing
>
> In both cases, I see the video element playing correctly. In case 1,
> I'm  seeing that the create function in "my-push-src" is being called
> once  for each push source and never being called again. In case 2,
> create  continually gets called, and everything seems OK. It seems
> that somehow  the scheduling is different between cases 1 and 2, and
> I'm guessing the  video element's scheduling is somehow changing that
> of the push source elements.

What kind of source is "video-element" ? Is it a live capture source
such as v4l2src or such?

Is it possible there is something missing in your description of Case
2? I assume it's the same as Case 1 just everything in separate
pipelines instead of separate bins in a single pipeline?

Possibly the difference is related to the "preroll" mechanism we have
in GStreamer. Basically, at least in a non-live pipeline sinks will
block when they receive the first buffer, and only when *all* sinks in
a pipeline have received a first buffer. If just one sink does not
receive a buffer, all the other sub-pipelines will be blocked/waiting.
I wonder if that's what you're seeing here. If you use separate
pipelines they all operate 100% independently of course.

Otherwise it's hard to see what's going on without more info. Perhaps
you could make a snapshot of the pipeline and dump it into a "dot
graph" to see which sinks are prerolled and which are not.

You can also set async=false on sinks if you know they may not get a
buffer immediately initially, but that should be rarely needed.

Cheers
 -Tim

--
Tim Müller, Centricular Ltd - http://www.centricular.com


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Martin Kelly]

(friendly ping)

On 04/29/2016 08:53 AM, Martin Kelly wrote:

> Hi,
>
> I apologize in advance if some documentation makes this question clear;
> I searched but have not found an answer. I have an issue with a
> Gstreamer application that I cannot figure out. Specifically, I'm trying
> two different pipeline configurations that I would expect to yield
> identical results, but they are not. One of these cases is breaking
> while the other appears to be working. Below I describe the setup:
>
> Case 1:
> - create one pipeline
> - use gst_bin_add_many to add many bins to it
> - the first bin is of the form "some-video-element ! [encode and many
> other elements] udpsink"
> - all other bins are in the form "my-push-src ! fakesink"
> - set the pipeline to playing
>
> Case 2:
> - create one pipeline per bin via gst_pipeline_new + gst_bin_add
> - the first pipeline is of the form "some-video-element ! [encode and
> many other elements] udpsink"
> - set each pipeline to playing
>
> In both cases, I see the video element playing correctly. In case 1, I'm
> seeing that the create function in "my-push-src" is being called once
> for each push source and never being called again. In case 2, create
> continually gets called, and everything seems OK. It seems that somehow
> the scheduling is different between cases 1 and 2, and I'm guessing the
> video element's scheduling is somehow changing that of the push source
> elements.
>
> I don't understand the interplay between push sources and other sources,
> so I'm not sure how to debug this. I'm wondering if someone could give
> me a pointer as to what may be wrong or where I should look.
>
> In case it matters, I'm using gstreamer 1.6.3 on Android, and
> "my-push-src" is my own custom element, so it's entirely possible it has
> some bug that I haven't caught.
>
> Thanks in advance,
> Martin

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Tim Müller]

On Thu, 2016-05-05 at 09:58 -0700, Martin Kelly wrote:

> (friendly ping)

Did you see my response last Friday?

Cheers
 -Tim

--
Tim Müller, Centricular Ltd - http://www.centricular.com


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Martin Kelly]

Sorry, I missed it! Let me tweak my mail filtering :).

On 05/05/2016 10:12 AM, Tim Müller wrote:
> On Thu, 2016-05-05 at 09:58 -0700, Martin Kelly wrote:
>
>> (friendly ping)
>
> Did you see my response last Friday?
>
> Cheers
>   -Tim
>
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Martin Kelly]

On 04/29/2016 12:19 PM, Tim Müller wrote:
> On Fri, 2016-04-29 at 08:53 -0700, Martin Kelly wrote:
>
> Hi Martin,

Thanks for your reply; sorry for missing it last week! I need to fix my
mail filter settings.

>
> What kind of source is "video-element" ? Is it a live capture source
> such as v4l2src or such?

"video-element" is ahcsrc on Android; sorry for the vaguery. Based on
the code, it's a live source.

>
> Is it possible there is something missing in your description of Case
> 2? I assume it's the same as Case 1 just everything in separate
> pipelines instead of separate bins in a single pipeline?

Unless I'm missing something, they are the same pipelines, just in
separate bins in a single pipeline vs one pipeline per bin.

>
> Possibly the difference is related to the "preroll" mechanism we have
> in GStreamer. Basically, at least in a non-live pipeline sinks will
> block when they receive the first buffer, and only when *all* sinks in
> a pipeline have received a first buffer. If just one sink does not
> receive a buffer, all the other sub-pipelines will be blocked/waiting.
> I wonder if that's what you're seeing here. If you use separate
> pipelines they all operate 100% independently of course.

That explanation would make sense to me, but in this case ahcsrc is
live, and so is my custom video element. I will try setting sync=false
on the udpsink and see if it makes a difference.

>
> Otherwise it's hard to see what's going on without more info. Perhaps
> you could make a snapshot of the pipeline and dump it into a "dot
> graph" to see which sinks are prerolled and which are not.
>

Thanks, I will look into this.

One follow-up question I have, and something that confuses me about the
code of ahcsrc: exactly when does the create function get called for a
push source? In the case of gstahcsrc.c, this is gst_ahc_src_create, and
it's the function that pushes out a buffer. I can't find any logic
inside the ahcsrc plugin to indicate scheduling, which tells me the
surround pipeline must somehow decide this. How does that scheduling work?
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Martin Kelly]

On 05/06/2016 10:40 AM, Martin Kelly wrote:

> On 04/29/2016 12:19 PM, Tim Müller wrote:
>> On Fri, 2016-04-29 at 08:53 -0700, Martin Kelly wrote:
>>
>> Possibly the difference is related to the "preroll" mechanism we have
>> in GStreamer. Basically, at least in a non-live pipeline sinks will
>> block when they receive the first buffer, and only when *all* sinks in
>> a pipeline have received a first buffer. If just one sink does not
>> receive a buffer, all the other sub-pipelines will be blocked/waiting.
>> I wonder if that's what you're seeing here. If you use separate
>> pipelines they all operate 100% independently of course.
>
> That explanation would make sense to me, but in this case ahcsrc is
> live, and so is my custom video element. I will try setting sync=false
> on the udpsink and see if it makes a difference.
>

Looks like setting async=false fixes my issue; thanks! I think the issue
is that each my bins generates data at drastically varying rate, from
many buffers per second to very rare buffers every few minutes. Although
adding the "async" property fixes the problem, is there a cleaner way I
should be handling this?

Another followup: I see both "sync" and "async" properties on fakesink.
Could you clarify the distinction between them and when each should be used?

Thanks again,
Martin
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Tim Müller]

On Fri, 2016-05-06 at 10:40 -0700, Martin Kelly wrote:

Hi,

> One follow-up question I have, and something that confuses me about
> the  code of ahcsrc: exactly when does the create function get called
> for a  push source? In the case of gstahcsrc.c, this is
> gst_ahc_src_create, and  it's the function that pushes out a buffer.
> I can't find any logic  inside the ahcsrc plugin to indicate
> scheduling, which tells me the  surround pipeline must somehow decide
> this. How does that scheduling work?

The source base class will call the create function over and over again
in a loop. The function is expected to block until it can produce a
buffer (or is told to shut down/unblock). In case of video capture
sources it will depend on settings, sensor and lighting and such how
quickly a new frame will be produced. The throttling/scheduling is
usually inherent in the capture API and the source simply blocks until
it gets a new frame.

Cheers
 -Tim

--
Tim Müller, Centricular Ltd - http://www.centricular.com


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Tim Müller]

On Fri, 2016-05-06 at 12:19 -0700, Martin Kelly wrote:

> Looks like setting async=false fixes my issue; thanks! I think the
> issue  is that each my bins generates data at drastically varying
> rate, from  many buffers per second to very rare buffers every few
> minutes.

Yes, rare buffers every few minutes sounds like a problem (without
async=false preroll could be delayed until such a rare buffer is
produced)

> Although adding the "async" property fixes the problem, is there a
> cleaner way I should be handling this?

It sounds like a reasonable/clean solution. You could make the encoding
bin send a gap event instead to make it preroll, but it's not clear to
me that that'd be cleaner.

> Another followup: I see both "sync" and "async" properties on
> fakesink. 
> Could you clarify the distinction between them and when each should
> be used?

"async" is about state changes (preroll mechanism)

"sync" determines whether the sink should pay attention to timestamps
on the buffers and schedule them according to the clock and such. If
you disable it buffers will be rendered as soon as possible without
paying any attention to timestamps. 

Cheers
-Tim

--
Tim Müller, Centricular Ltd - http://www.centricular.com


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Martin Kelly]

On 05/08/2016 07:33 AM, Tim Müller wrote:
>
> The source base class will call the create function over and over again
> in a loop. The function is expected to block until it can produce a
> buffer (or is told to shut down/unblock). In case of video capture
> sources it will depend on settings, sensor and lighting and such how
> quickly a new frame will be produced. The throttling/scheduling is
> usually inherent in the capture API and the source simply blocks until
> it gets a new frame.
>

Got it, thanks!
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Martin Kelly]

On 05/08/2016 07:38 AM, Tim Müller wrote:

> On Fri, 2016-05-06 at 12:19 -0700, Martin Kelly wrote:
>
>> Another followup: I see both "sync" and "async" properties on
>> fakesink.
>> Could you clarify the distinction between them and when each should
>> be used?
>
> "async" is about state changes (preroll mechanism)
>
> "sync" determines whether the sink should pay attention to timestamps
> on the buffers and schedule them according to the clock and such. If
> you disable it buffers will be rendered as soon as possible without
> paying any attention to timestamps.
>
> Cheers
> -Tim
>

Thanks for the clarification.
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel