gstreamer threading internal

[mirtexxan]

Hello everybody,
I'm studying Gstreamer threading internal and I have a question.
From the documentation it seems that every source-sink pad connection generates a different thread through the GstTask API (from which chain functions and the like are called). Reading the design document on scheduling however, it seems that a new thread is created only in specific conditions.
Specifically there's an example that bugs me:

 +---------+    +----------+    +------------+    +----------+
 | filesrc |    | identity |    | avidemuxer |    | fakesink |
 |        src--sink       src--sink         src--sink        |
 +---------+    +----------+    +------------+    +----------+
         (l-g) (c-l)      (g)  (l)          ()   (c)

   * fakesink has a chain function and the peer pad has no
     loop function, no scheduling is done.
   * avidemuxer and identity expose an (g) - (l) connection,
     a thread is created to call the sinkpad loop function.
   * identity knows the srcpad is getrange based and uses the
     thread from avidemux to getrange data from filesrc.

So here my two questions:
1) what exactly means that the fakesink-avidemuxer connection is not scheduled? Is a new thread generated or not?
2) In regards to the identity-filesrc connection, what exacly happens? is a new thread generated? If not, why? What are the general rules for thread generation?

Finally It would be very appreciated if someone could explain to me how can gstreamer scale when so many threads are employed... what about the synchronization overhead? My guess is that in a streaming framework synchronization overhead is unavoidable due to the need to synch the streaming data itself, so the overhead for synching thread is somehow "absorbed"...

Thank you very much,
Mirto

[Tim-Philipp Müller-2]

On Tue, 2012-02-28 at 08:14 -0800, mirtexxan wrote:

> Specifically there's an example that bugs me:
>
>  +---------+    +----------+    +------------+    +----------+
>  | filesrc |    | identity |    | avidemuxer |    | fakesink |
>  |        src--sink       src--sink         src--sink        |
>  +---------+    +----------+    +------------+    +----------+
>          (l-g) (c-l)      (g)  (l)          ()   (c)
>
>    * fakesink has a chain function and the peer pad has no
>      loop function, *no scheduling is done*.
>    * avidemuxer and identity expose an (g) - (l) connection,
>      a thread is created to call the sinkpad loop function.
>    * identity knows the srcpad is getrange based and *uses the
>      thread from avidemux to getrange data from filesrc*.
>
> So here my two questions:
> 1) what exactly means that the fakesink-avidemuxer connection is not
> scheduled? Is a new thread generated or not?
> 2) In regards to the identity-filesrc connection, what exacly happens? is a
> new thread generated? If not, why? What are the general rules for thread
> generation?

A thread/GstTask is created for avidemux's sink pad which iterates
avidemux's loop function. avidemux pulls data from filesrc and pushes it
our towards fakesink. There's one streaming thread only, namely
avidemux's loop thread.

You basically have two main scenarios:

 (Source in pull mode) <-- pull <-- (demuxer/parser) --> push --> ...

 (Source in push mode) --> push --> ...

In the first case, the demuxer/parser are driving that particular
streaming thread, in the second case there's a thread operating for the
source, which just creates data and pushes it in a loop.

In addition to that, a queue element (required usually when branches
split, e.g. after demuxers or tee) will create a thread boundary, so
there'll be a new thread for every queue.

Some elements also have other internal threads, e.g. audio elements may
have a thread for the internal ring buffer.

 Cheers
  -Tim

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[mirtexxan]

Thanks a lot Tim!
So If I understand correctly, in this case we'd have a totale of three threads: the application thread, the streaming thread (where elements live) and the "scheduling" thread that "moves" data and call chain, loop functions and the like?

Is there a situation when (apart from using queue) the gstreamer infrastracture generates more than this number of threads?
Note that I'm ignoring threads generated internally by the elements.

It appears that i was wrong - I thought that each pad-to-pad connection in gstreamer lived in his own thread... so it appears that gstreamer DOES NOT SCALE AUTOMATICALLY to multicore machine with a high number of cores...

Could you please explain the threading internal in more detail? I'd be very grateful! :)

ps a note on queue: what happens if I don't use queues after a demuxer? There will be only one thread in every case?

[Tim-Philipp Müller-2]

On Tue, 2012-02-28 at 10:52 -0800, mirtexxan wrote:

> So If I understand correctly, in this case we'd have a totale of three
> threads: the application thread, the streaming thread (where elements live)
> and the "scheduling" thread that "moves" data and call chain, loop functions
> and the like?

Two threads. The "streaming threads" are "the scheduling threads" (I
don't think we use that terminology anywhere, do we?). Streaming threads
are the threads where the data processing happens, and data gets pushed
around.

> Is there a situation when (apart from using queue) the gstreamer
> infrastracture generates more than this number of threads?
> *Note that I'm ignoring threads generated internally by the elements.*

Yes, there may be additional threads, e.g. a thread for the system
clock, or one for audio ring buffers, or an RTP jitterbuffer.


> It appears that I was wrong - I thought that each pad-to-pad connection in
> gstreamer lived in his own thread...

That is indeed not the case.

> so it appears that gstreamer DOES NOT SCALE AUTOMATICALLY to multicore
> machine with a high number of cores...

Well, no, and even if it did that, it's not clear that that would
automatically improve anything. In many cases, it will just add
unnecessary overhead. In practice, it's likely to take advantage of
multiple cores, if the kernel allows it to do so. Since you have audio
and video decoding typically in different streaming threads (remember
those queues after the demuxer), the kernel is likely to schedule audio
and video decoding to be done on multiple cores, for example.

Then it also depends on your codec implementations of course. Decoders
or encoders might automatically make use of multiple threads/cores
internally (without GStreamer necessarily knowing about it). e.g. x264,
libvp8, libav for decoding.


> ps a note on queue: what happens if I don't use queues after a demuxer?
> There will be only one thread in every case?

You need to use multiple queues because of the "preroll" mechanism in
GStreamer. Sinks will block when they receive a first buffer, and all
sinks need a buffer before the pipeline prerolls. That means you need
queues after demuxers and tees.

Cheers
 -Tim

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[mirtexxan]

Again, thanks a lot.
The scheduling thread is my own invention, sorry.
Maybe my confusion originates from the various points in the documentation where there are reference to streaming threadS - now I think that they refers to the elements internal threads.

Just to be sure, I'd like to have your opinion on this matter. When I launch a simple gst-launch-0.10 audiotestsrc ! identity ! alsasink, I see 4 threads when using "top". One of course is the gst-launch application thread. Another (called threaded-ml) I guess is the scheduling thread and the other 2 (called audiotesrc0:s) I guess are the source element internal threads. Am I correct?

Another question: could please be a bit more specific on the "accessory" threads generated by the gstreamer infrastracture? (the system clock thread, etc) Are they always generated? Why aren't they visible by the "top" application (if that's the case)? I'm particularly interested in the clock thread.

And finally a more advanced question: what about setting core affinity in gstreamer? Thread priority? It is possible to create special purpose element to do the trick?

Regards,
Mirto

[Tim-Philipp Müller-2]

On Wed, 2012-02-29 at 02:22 -0800, mirtexxan wrote:

> When I launch
> a simple gst-launch-0.10 audiotestsrc ! identity ! alsasink, I see 4 threads
> when using "top". One of course is the gst-launch application thread.
> Another (called threaded-ml) I guess is the scheduling thread and the other
> 2 (called audiotesrc0:s) I guess are the source element internal threads. Am
> I correct?

There is

 - the main application thread

 - an audiotestsrc thread generating
   data and pushing it downstream

 - an audio ring buffer thread inside
   the alsasink element

 - threaded-ml, which belongs to pulseaudio
   (which is where your alsa output goes I guess)

 - (no system clock thread in this case)


> And finally a more advanced question: what about setting core affinity in
> gstreamer? Thread priority? It is possible to create special purpose element
> to do the trick?

You can do things like that. STREAM_STATUS messages get posted on the
bus when threads are started, you can catch those synchronously (from
the thread context that emitted them) and configure stuff. You can also
use your own thread pools, check out the examples under
tests/examples/streams/ in the gstreamer core source code.

Cheers
 -Tim

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel