GStreamer - pipeline design - multi-threading

[Bogdan Pavkovic]

Dear all,

I am a newby to GStremer Community
but I managed to thoroughly go through ALL offered tutorials.

GOAL: I need to design a performance-critical pipeline and create a couple of new filter plugins.
Any help will be appreciated.

General pipeline is divided in 3 bins: input, core, and output.
For the input part we have already completed 4 options: RTP client (existing plugin), MJPEG, V2L, and FILE (existing).
For the output part we use Wayland.

Core part is processing which roughly is done in 3 stages:
pre-processing, heavy processing, and some sort of Kalman filtering.
Q1: My idea is to make 3 plugins that will form a larger bin. Is this the right way to go?
Pre-processing plug-in need to have on-request sink pads since number of input video streams can vary between 1 and 4.
Source (output) pads (on request) will be created and linked to consecutive elements on the fly as they appear.
Q2: Do all consecutive elements in Core bin need to have on request pads due to non-deterministic number of input videos?

Heavy processing plug-in needs to do heavy lifting and we intend to use multi-threading on 4 available cores.
Q2: Do we need to use 2 queue elements before each of the pads of this element to separate processing of each video stream?
Q3: I understood there are tasks that can do further multi-threading. Can we use them behind a queue element?
Q4: If we use 4 tasks, is there a way to signal and sync all of them? i.e. Do we get a signal  when all 4 task outputs are ready?
Q5: How can we configure a task to target a specific core?

Finaly, Kalman plugin is not calculation intensive and can handle both streams.
Q6: Can we somehow undo the queue element? Does it make sense?

I can provide a diagram of the proposed pipeline if needed to clarify things.

Many thanks in advance.

Best regards,
Bogdan

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Bogdan Pavkovic]

Dear all,

Here is the diagram for your convenience.

I hope there is someone that can answer at least some subset of my questions.

Thanks again in advance.

Best,

Bogdan
​

On Fri, May 26, 2017 at 1:15 PM, Bogdan Pavkovic <[hidden email]> wrote:

Dear all,

I am a newby to GStremer Community
but I managed to thoroughly go through ALL offered tutorials.

GOAL: I need to design a performance-critical pipeline and create a couple of new filter plugins.
Any help will be appreciated.

General pipeline is divided in 3 bins: input, core, and output.
For the input part we have already completed 4 options: RTP client (existing plugin), MJPEG, V2L, and FILE (existing).
For the output part we use Wayland.

Core part is processing which roughly is done in 3 stages:
pre-processing, heavy processing, and some sort of Kalman filtering.
Q1: My idea is to make 3 plugins that will form a larger bin. Is this the right way to go?
Pre-processing plug-in need to have on-request sink pads since number of input video streams can vary between 1 and 4.
Source (output) pads (on request) will be created and linked to consecutive elements on the fly as they appear.
Q2: Do all consecutive elements in Core bin need to have on request pads due to non-deterministic number of input videos?

Heavy processing plug-in needs to do heavy lifting and we intend to use multi-threading on 4 available cores.
Q2: Do we need to use 2 queue elements before each of the pads of this element to separate processing of each video stream?
Q3: I understood there are tasks that can do further multi-threading. Can we use them behind a queue element?
Q4: If we use 4 tasks, is there a way to signal and sync all of them? i.e. Do we get a signal  when all 4 task outputs are ready?
Q5: How can we configure a task to target a specific core?

Finaly, Kalman plugin is not calculation intensive and can handle both streams.
Q6: Can we somehow undo the queue element? Does it make sense?

I can provide a diagram of the proposed pipeline if needed to clarify things.

Many thanks in advance.

Best regards,
Bogdan

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Bogdan Pavkovic]

Hi all,

I realized that an attachment cannot be seen from the archives so here is the link just in case...

https://mega.nz/#!h8wFVarA!leTOvjLg4VhBxX-wNPY3_w8jcFE-W-vspXw7FOT_SAU

Best,

B

On Tue, May 30, 2017 at 10:49 AM, Bogdan Pavkovic <[hidden email]> wrote:

Dear all,

Here is the diagram for your convenience.

I hope there is someone that can answer at least some subset of my questions.

Thanks again in advance.

Best,

Bogdan
​

On Fri, May 26, 2017 at 1:15 PM, Bogdan Pavkovic <[hidden email]> wrote:

Dear all,

I am a newby to GStremer Community
but I managed to thoroughly go through ALL offered tutorials.

GOAL: I need to design a performance-critical pipeline and create a couple of new filter plugins.
Any help will be appreciated.

General pipeline is divided in 3 bins: input, core, and output.
For the input part we have already completed 4 options: RTP client (existing plugin), MJPEG, V2L, and FILE (existing).
For the output part we use Wayland.

Core part is processing which roughly is done in 3 stages:
pre-processing, heavy processing, and some sort of Kalman filtering.
Q1: My idea is to make 3 plugins that will form a larger bin. Is this the right way to go?
Pre-processing plug-in need to have on-request sink pads since number of input video streams can vary between 1 and 4.
Source (output) pads (on request) will be created and linked to consecutive elements on the fly as they appear.
Q2: Do all consecutive elements in Core bin need to have on request pads due to non-deterministic number of input videos?

Heavy processing plug-in needs to do heavy lifting and we intend to use multi-threading on 4 available cores.
Q2: Do we need to use 2 queue elements before each of the pads of this element to separate processing of each video stream?
Q3: I understood there are tasks that can do further multi-threading. Can we use them behind a queue element?
Q4: If we use 4 tasks, is there a way to signal and sync all of them? i.e. Do we get a signal  when all 4 task outputs are ready?
Q5: How can we configure a task to target a specific core?

Finaly, Kalman plugin is not calculation intensive and can handle both streams.
Q6: Can we somehow undo the queue element? Does it make sense?

I can provide a diagram of the proposed pipeline if needed to clarify things.

Many thanks in advance.

Best regards,
Bogdan

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Nicolas Dufresne-5]

Le mardi 30 mai 2017 à 11:02 +0200, Bogdan Pavkovic a écrit :

> Hi all,
>
> I realized that an attachment cannot be seen from the archives so here is the link just in case...
>
> https://mega.nz/#!h8wFVarA!leTOvjLg4VhBxX-wNPY3_w8jcFE-W-vspXw7FOT_SAU
>
> Best,
> B
>
> > On Tue, May 30, 2017 at 10:49 AM, Bogdan Pavkovic <[hidden email]> wrote:
> > Dear all,
> >
> > Here is the diagram for your convenience.
> > I hope there is someone that can answer at least some subset of my questions.
> >
> > Thanks again in advance.
> >
> > Best,
> > Bogdan
> >
> >
> > > > On Fri, May 26, 2017 at 1:15 PM, Bogdan Pavkovic <[hidden email]> wrote:
> > > Dear all,
> > >
> > > I am a newby to GStremer Community 
> > > but I managed to thoroughly go through ALL offered tutorials.
> > >
> > > GOAL: I need to design a performance-critical pipeline and create a couple of new filter plugins.
> > > Any help will be appreciated.
> > >
> > > General pipeline is divided in 3 bins: input, core, and output.
> > > For the input part we have already completed 4 options: RTP client (existing plugin), MJPEG, V2L, and FILE (existing).
> > > For the output part we use Wayland.
> > >
> > > Core part is processing which roughly is done in 3 stages: 
> > > pre-processing, heavy processing, and some sort of Kalman filtering.
> > > Q1: My idea is to make 3 plugins that will form a larger bin. Is this the right way to go?
> > > Pre-processing plug-in need to have on-request sink pads since number of input video streams can vary between 1 and 4.
> > > Source (output) pads (on request) will be created and linked to consecutive elements on the fly as they appear.

Is there a relationship between the input streams ? Since having a
single element in the context were there is not link between each
input/output pair will just make your pipeline more complex to handle.
You will endup having N+1 thread making call into your element. You
could though use a bin just to group and manage the internal filters.

> > > Q2: Do all consecutive elements in Core bin need to have on request pads due to non-deterministic number of input videos?

If I understand correctly, you are asking if the element inside the bin
need to have request pads. If that is correct, the answer is no. You
can have a bin with request sink pads, that will use ghost pad to
interface with internal filters static sink pads. The bin can then
notify sometimes src pads by ghosting a filter static src pad.

> > >
> > > Heavy processing plug-in needs to do heavy lifting and we intend to use multi-threading on 4 available cores.
> > > Q2: Do we need to use 2 queue elements before each of the pads of this element to separate processing of each video stream?

It depends of how you implement the parallel processing. If you can do
parallel processing on a single GstBuffer, then you don't need any
queuing. This method is usually used to achieve low latency processing
as queuing multiple buffers to feed multiple threads will introduce
latency.

> > > Q3: I understood there are tasks that can do further multi-threading. Can we use them behind a queue element?

I'm not sure I understand this question. What do you mean by tasks ?
You can of course create and manage threads in your element.

> > > Q4: If we use 4 tasks, is there a way to signal and sync all of them? i.e. Do we get a signal  when all 4 task outputs are ready?

I'm not sure this question correlate with anything in GStreamer.
GstTask is a wrapper around a thread, with proper state that let you
start/stop/pause the task and link it to a thread pool. Signalling that
data is ready has to be done by your code, could be using GCond or some
advance queue data structure.

> > > Q5: How can we configure a task to target a specific core?

This is doable in a portable way, hence not expose through the GThread
abstraction. In general, you should trust your scheduler not to move
the thread around in a way that it would degrade the performance. If
you are implementing this only for unix system, you can of course
implement your own, and use pthread API directly (see
pthread_setaffinity_np).

> > >
> > > Finaly, Kalman plugin is not calculation intensive and can handle both streams.
> > > Q6: Can we somehow undo the queue element? Does it make sense?

What does "undoing" an element means ? If by chance you are referring
to the requires queues after demuxer elements, be aware that this is
needed because demuxers receives buffer from a single streaming thread
and does not create any new threads for the output. The queues are
needed so that each branch of the demuxer can work independently. If
you have input/output pairs in your bins, you'll have 1 streaming
thread per pair, hence each output will already work independently from
each other.

> > >
> > > I can provide a diagram of the proposed pipeline if needed to clarify things.
> > >
> > > Many thanks in advance.
> > >
> > > Best regards,
> > > Bogdan
> > >
> >
> >
>
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Bogdan Pavkovic]

On Tue, May 30, 2017 at 4:25 PM, Nicolas Dufresne <[hidden email]> wrote:

Le mardi 30 mai 2017 à 11:02 +0200, Bogdan Pavkovic a écrit :
> Hi all,
>
> I realized that an attachment cannot be seen from the archives so here is the link just in case...
>
> https://mega.nz/#!h8wFVarA!leTOvjLg4VhBxX-wNPY3_w8jcFE-W-vspXw7FOT_SAU
>
> Best,
> B
>
> > On Tue, May 30, 2017 at 10:49 AM, Bogdan Pavkovic <[hidden email]> wrote:
> > Dear all,
> >
> > Here is the diagram for your convenience.
> > I hope there is someone that can answer at least some subset of my questions.
> >
> > Thanks again in advance.
> >
> > Best,
> > Bogdan
> >
> >
> > > > On Fri, May 26, 2017 at 1:15 PM, Bogdan Pavkovic <[hidden email]> wrote:
> > > Dear all,
> > >
> > > I am a newby to GStremer Community 
> > > but I managed to thoroughly go through ALL offered tutorials.
> > >
> > > GOAL: I need to design a performance-critical pipeline and create a couple of new filter plugins.
> > > Any help will be appreciated.
> > >
> > > General pipeline is divided in 3 bins: input, core, and output.
> > > For the input part we have already completed 4 options: RTP client (existing plugin), MJPEG, V2L, and FILE (existing).
> > > For the output part we use Wayland.
> > >
> > > Core part is processing which roughly is done in 3 stages: 
> > > pre-processing, heavy processing, and some sort of Kalman filtering.
> > > Q1: My idea is to make 3 plugins that will form a larger bin. Is this the right way to go?
> > > Pre-processing plug-in need to have on-request sink pads since number of input video streams can vary between 1 and 4.
> > > Source (output) pads (on request) will be created and linked to consecutive elements on the fly as they appear.

Bonjour Nicolas. Merci pour ta réponse.

Find my answers below.

 

Is there a relationship between the input streams ? Since having a
single element in the context were there is not link between each
input/output pair will just make your pipeline more complex to handle.
You will endup having N+1 thread making call into your element. You
could though use a bin just to group and manage the internal filters.

Input streams are not in any relationship. Each comes from separate camera and after same processing stages (3 in total as described) will be displayed in 2 different windows.

If I understand correctly your answer, for each PAD in an element, there will be a separate thread called to handle the _chain function every time a buffer arrives.

 

> > > Q2: Do all consecutive elements in Core bin need to have on request pads due to non-deterministic number of input videos?

If I understand correctly, you are asking if the element inside the bin
need to have request pads. If that is correct, the answer is no. You
can have a bin with request sink pads, that will use ghost pad to
interface with internal filters static sink pads. The bin can then
notify sometimes src pads by ghosting a filter static src pad.

Understood. We will then implement 3 filters with N static sink pads and map them with ghost pads to a bin.

What about src pads of an element inside bin? Do they need to be created on request or as a simple static pads?

 

> > >
> > > Heavy processing plug-in needs to do heavy lifting and we intend to use multi-threading on 4 available cores.
> > > Q2: Do we need to use 2 queue elements before each of the pads of this element to separate processing of each video stream?

It depends of how you implement the parallel processing. If you can do
parallel processing on a single GstBuffer, then you don't need any
queuing. This method is usually used to achieve low latency processing
as queuing multiple buffers to feed multiple threads will introduce
latency.

Clear. Since we will do parallel processing on a single GstBuffer then we will not use a queue element.
 

> > > Q3: I understood there are tasks that can do further multi-threading. Can we use them behind a queue element?

I'm not sure I understand this question. What do you mean by tasks ?
You can of course create and manage threads in your element.

I meant GTask but you've already explained below.

 

> > > Q4: If we use 4 tasks, is there a way to signal and sync all of them? i.e. Do we get a signal  when all 4 task outputs are ready?

I'm not sure this question correlate with anything in GStreamer.
GstTask is a wrapper around a thread, with proper state that let you
start/stop/pause the task and link it to a thread pool. Signalling that
data is ready has to be done by your code, could be using GCond or some
advance queue data structure.

Understood. GCond seems like an elegant way.

 

> > > Q5: How can we configure a task to target a specific core?

This is doable in a portable way, hence not expose through the GThread
abstraction. In general, you should trust your scheduler not to move
the thread around in a way that it would degrade the performance. If
you are implementing this only for unix system, you can of course
implement your own, and use pthread API directly (see
pthread_setaffinity_np).

Great! Looks exactly like what we need.

 

> > >
> > > Finaly, Kalman plugin is not calculation intensive and can handle both streams.
> > > Q6: Can we somehow undo the queue element? Does it make sense?

What does "undoing" an element means ? If by chance you are referring
to the requires queues after demuxer elements, be aware that this is
needed because demuxers receives buffer from a single streaming thread
and does not create any new threads for the output. The queues are
needed so that each branch of the demuxer can work independently. If
you have input/output pairs in your bins, you'll have 1 streaming
thread per pair, hence each output will already work independently from
each other.

Actually, this question became redundant with your previous answers.

Thanks for giving details about demuxers.

Best,

Bogdan

 

> > >
> > > I can provide a diagram of the proposed pipeline if needed to clarify things.
> > >
> > > Many thanks in advance.
> > >
> > > Best regards,
> > > Bogdan
> > >
> >
> >
>
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel