Memory usage is increasing Gstreamer testing

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Memory usage is increasing Gstreamer testing

Parveen Kumar Jain
Hi All,
  I am using Gstreamer for one of my bulk call video calling application, using sip as my signalling part.I tried to make around 60 simulative video calls without much of the problem on a IBM server(4 GB RAM,Intel QuadCore,Linux 64 bit).
  But somehow, after around 15-17 hours of testing, I see that "free memory" left in the system is only 300 -400 MB, which initially was around 2.5 GB. From "process id details" I can see that there is not much increase of "RESIDENT" and "VIRTUAL" memory usage in my "application process".
Can any one please comment/help on this behavior of system ? why overall system memory is increasing  ?
At this kind of video OBD load, I had seen that Gstreamer creates around 600-700 threads.Can it be a source of this problem? 

Best Regards,
Parveen Jain


------------------------------------------------------------------------------
Nokia and AT&T present the 2010 Calling All Innovators-North America contest
Create new apps & games for the Nokia N8 for consumers in  U.S. and Canada
$10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing
Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store
http://p.sf.net/sfu/nokia-dev2dev
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Memory usage is increasing Gstreamer testing

Stefan Sauer
hi,

Am 29.10.2010 15:16, schrieb Parveen Kumar Jain:

> Hi All,
>   I am using Gstreamer for one of my bulk call video calling application, using
> sip as my signalling part.I tried to make around 60 simulative video calls
> without much of the problem on a IBM server(4 GB RAM,Intel QuadCore,Linux 64 bit).
>   But somehow, after around 15-17 hours of testing, I see that "free memory"
> left in the system is only 300 -400 MB, which initially was around 2.5 GB. From
> "process id details" I can see that there is not much increase of "RESIDENT" and
> "VIRTUAL" memory usage in my "application process".
> Can any one please comment/help on this behavior of system ? why overall system
> memory is increasing  ?
> At this kind of video OBD load, I had seen that Gstreamer creates around 600-700
> threads.Can it be a source of this problem?

If the threads are 'leaked' (never join) it could. Can you use valgrind
--tool=memcehck ... to check?

Stefan

>
> Best Regards,
> Parveen Jain
>
>
>
> ------------------------------------------------------------------------------
> Nokia and AT&T present the 2010 Calling All Innovators-North America contest
> Create new apps & games for the Nokia N8 for consumers in  U.S. and Canada
> $10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing
> Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store
> http://p.sf.net/sfu/nokia-dev2dev
>
>
>
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel


------------------------------------------------------------------------------
Nokia and AT&T present the 2010 Calling All Innovators-North America contest
Create new apps & games for the Nokia N8 for consumers in  U.S. and Canada
$10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing
Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store
http://p.sf.net/sfu/nokia-dev2dev
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Memory usage is increasing Gstreamer testing

Parveen Kumar Jain
Hi Stefan,
  Just to tell you that I don't create all these thread from own application.For every call I create two threads so in total I create a thread pool of 80 at the start of the application, and don't increase/decrease it any further.But I see all these threads(around 600 more threads apart from what I created) automatically created by Gstreamer itself.
  Infact, I am also looking for ways where I can control these no. of threads creation in my own way.As I understand creating so many threads in a Quadcore system just increases the load of scheduling without adding anything significant efficiency to system.

Thanks & Regards,
Parveen Jain

  

On Fri, Oct 29, 2010 at 9:11 PM, Stefan Kost <[hidden email]> wrote:
hi,

Am 29.10.2010 15:16, schrieb Pearveen Kumar Jain:
> Hi All,
>   I am using Gstreamer for one of my bulk call video calling application, using
> sip as my signalling part.I tried to make around 60 simulative video calls
> without much of the problem on a IBM server(4 GB RAM,Intel QuadCore,Linux 64 bit).
>   But somehow, after around 15-17 hours of testing, I see that "free memory"
> left in the system is only 300 -400 MB, which initially was around 2.5 GB. From
> "process id details" I can see that there is not much increase of "RESIDENT" and
> "VIRTUAL" memory usage in my "application process".
> Can any one please comment/help on this behavior of system ? why overall system
> memory is increasing  ?
> At this kind of video OBD load, I had seen that Gstreamer creates around 600-700
> threads.Can it be a source of this problem?

If the threads are 'leaked' (never join) it could. Can you use valgrind
--tool=memcehck ... to check?

Stefan

>
> Best Regards,
> Parveen Jain
>
>
>
> ------------------------------------------------------------------------------
> Nokia and AT&T present the 2010 Calling All Innovators-North America contest
> Create new apps & games for the Nokia N8 for consumers in  U.S. and Canada
> $10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing
> Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store
> http://p.sf.net/sfu/nokia-dev2dev
>
>
>
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel




--
Parveen Jain
Technical Lead – Network Engineering        
One97 Communications (P) Ltd                       
B121, Sector 5, Noida, UP 201301                                                      
P:  + 91 120 4770770      Extn:352
M: + 91   9212708203
W: www.one97world.com


------------------------------------------------------------------------------
Nokia and AT&T present the 2010 Calling All Innovators-North America contest
Create new apps & games for the Nokia N8 for consumers in  U.S. and Canada
$10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing
Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store
http://p.sf.net/sfu/nokia-dev2dev
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Memory usage is increasing Gstreamer testing

Marco Ballesio
Hi,

the first idea coming to my mind is that the pipeline(s) you're using are not set to NULL state after terminating the communication and/or you're re-instantiating anew pipeline for each call.. is it possible?

Regards

On Fri, Oct 29, 2010 at 7:10 PM, Parveen Kumar Jain <[hidden email]> wrote:
Hi Stefan,
  Just to tell you that I don't create all these thread from own application.For every call I create two threads so in total I create a thread pool of 80 at the start of the application, and don't increase/decrease it any further.But I see all these threads(around 600 more threads apart from what I created) automatically created by Gstreamer itself.
  Infact, I am also looking for ways where I can control these no. of threads creation in my own way.As I understand creating so many threads in a Quadcore system just increases the load of scheduling without adding anything significant efficiency to system.

Thanks & Regards,
Parveen Jain

  

On Fri, Oct 29, 2010 at 9:11 PM, Stefan Kost <[hidden email]> wrote:
hi,

Am 29.10.2010 15:16, schrieb Pearveen Kumar Jain:

> Hi All,
>   I am using Gstreamer for one of my bulk call video calling application, using
> sip as my signalling part.I tried to make around 60 simulative video calls
> without much of the problem on a IBM server(4 GB RAM,Intel QuadCore,Linux 64 bit).
>   But somehow, after around 15-17 hours of testing, I see that "free memory"
> left in the system is only 300 -400 MB, which initially was around 2.5 GB. From
> "process id details" I can see that there is not much increase of "RESIDENT" and
> "VIRTUAL" memory usage in my "application process".
> Can any one please comment/help on this behavior of system ? why overall system
> memory is increasing  ?
> At this kind of video OBD load, I had seen that Gstreamer creates around 600-700
> threads.Can it be a source of this problem?

If the threads are 'leaked' (never join) it could. Can you use valgrind
--tool=memcehck ... to check?

Stefan

>
> Best Regards,
> Parveen Jain
>
>
>
> ------------------------------------------------------------------------------
> Nokia and AT&T present the 2010 Calling All Innovators-North America contest
> Create new apps & games for the Nokia N8 for consumers in  U.S. and Canada
> $10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing
> Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store
> http://p.sf.net/sfu/nokia-dev2dev
>
>
>
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel




--
Parveen Jain
Technical Lead – Network Engineering        
One97 Communications (P) Ltd                       
B121, Sector 5, Noida, UP 201301                                                      
P:  + 91 120 4770770      Extn:352
M: + 91   9212708203
W: www.one97world.com


------------------------------------------------------------------------------
Nokia and AT&T present the 2010 Calling All Innovators-North America contest
Create new apps & games for the Nokia N8 for consumers in  U.S. and Canada
$10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing
Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store
http://p.sf.net/sfu/nokia-dev2dev
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel



------------------------------------------------------------------------------
Nokia and AT&T present the 2010 Calling All Innovators-North America contest
Create new apps & games for the Nokia N8 for consumers in  U.S. and Canada
$10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing
Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store
http://p.sf.net/sfu/nokia-dev2dev
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Memory usage is increasing Gstreamer testing

Parveen Kumar Jain
Hi Marco/All,
  Actually I create fixed no. of pipelines(around 60 pipelines) initially and don't create any pipeline after this and reuse only those in my successive calls.Similarly I do it with my own threads(as stated in my previous mail).
  So I see no reason for increasing the no. of threads or no. of video pipelines because of my application.

Just a quick question,Has anybody tried to use Gsteramer for bulk video calls(only H263 and G711A as codec).

Best Regards,
Parveen Jain

On Fri, Oct 29, 2010 at 11:50 PM, Marco Ballesio <[hidden email]> wrote:
Hi,

the first idea coming to my mind is that the pipeline(s) you're using are not set to NULL state after terminating the communication and/or you're re-instantiating anew pipeline for each call.. is it possible?

Regards


On Fri, Oct 29, 2010 at 7:10 PM, Parveen Kumar Jain <[hidden email]> wrote:
Hi Stefan,
  Just to tell you that I don't create all these thread from own application.For every call I create two threads so in total I create a thread pool of 80 at the start of the application, and don't increase/decrease it any further.But I see all these threads(around 600 more threads apart from what I created) automatically created by Gstreamer itself.
  Infact, I am also looking for ways where I can control these no. of threads creation in my own way.As I understand creating so many threads in a Quadcore system just increases the load of scheduling without adding anything significant efficiency to system.

Thanks & Regards,
Parveen Jain

  

On Fri, Oct 29, 2010 at 9:11 PM, Stefan Kost <[hidden email]> wrote:
hi,

Am 29.10.2010 15:16, schrieb Pearveen Kumar Jain:

> Hi All,
>   I am using Gstreamer for one of my bulk call video calling application, using
> sip as my signalling part.I tried to make around 60 simulative video calls
> without much of the problem on a IBM server(4 GB RAM,Intel QuadCore,Linux 64 bit).
>   But somehow, after around 15-17 hours of testing, I see that "free memory"
> left in the system is only 300 -400 MB, which initially was around 2.5 GB. From
> "process id details" I can see that there is not much increase of "RESIDENT" and
> "VIRTUAL" memory usage in my "application process".
> Can any one please comment/help on this behavior of system ? why overall system
> memory is increasing  ?
> At this kind of video OBD load, I had seen that Gstreamer creates around 600-700
> threads.Can it be a source of this problem?

If the threads are 'leaked' (never join) it could. Can you use valgrind
--tool=memcehck ... to check?

Stefan

>
> Best Regards,
> Parveen Jain
>
>
>
> ------------------------------------------------------------------------------
> Nokia and AT&T present the 2010 Calling All Innovators-North America contest
> Create new apps & games for the Nokia N8 for consumers in  U.S. and Canada
> $10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing
> Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store
> http://p.sf.net/sfu/nokia-dev2dev
>
>
>
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel




--
Parveen Jain
Technical Lead – Network Engineering        
One97 Communications (P) Ltd                       
B121, Sector 5, Noida, UP 201301                                                      
P:  + 91 120 4770770      Extn:352
M: + 91   9212708203
W: www.one97world.com


------------------------------------------------------------------------------
Nokia and AT&T present the 2010 Calling All Innovators-North America contest
Create new apps & games for the Nokia N8 for consumers in  U.S. and Canada
$10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing
Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store
http://p.sf.net/sfu/nokia-dev2dev
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel



------------------------------------------------------------------------------
Nokia and AT&T present the 2010 Calling All Innovators-North America contest
Create new apps & games for the Nokia N8 for consumers in  U.S. and Canada
$10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing
Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store
http://p.sf.net/sfu/nokia-dev2dev
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel




--
Parveen Jain
Technical Lead – Network Engineering        
One97 Communications (P) Ltd                       
B121, Sector 5, Noida, UP 201301                                                      
P:  + 91 120 4770770      Extn:352
M: + 91   9212708203
W: www.one97world.com


------------------------------------------------------------------------------
Nokia and AT&T present the 2010 Calling All Innovators-North America contest
Create new apps & games for the Nokia N8 for consumers in  U.S. and Canada
$10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing
Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store
http://p.sf.net/sfu/nokia-dev2dev
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Memory usage is increasing Gstreamer testing

Marco Ballesio
Hi,

On Mon, Nov 1, 2010 at 8:18 AM, Parveen Kumar Jain <[hidden email]> wrote:
Hi Marco/All,
  Actually I create fixed no. of pipelines(around 60 pipelines) initially and don't create any pipeline after this and reuse only those in my successive calls.Similarly I do it with my own threads(as stated in my previous mail).
  So I see no reason for increasing the no. of threads or no. of video pipelines because of my application.

It appears really strange to me as well.. I've been testing cases similar to yours for a while on embedded devices not seeing such a behaviour. Would it be possible for you to check with valgrind?
 

Just a quick question,Has anybody tried to use Gsteramer for bulk video calls(only H263 and G711A as codec).

Yes it definitely works, just grab an N900 ;) (to be honest, you need to modify a couple of files to get it working with H263/G711 by default).

I should have a very simple test app somewhere working with gtk and on any environments with a sane gstreamer installation and make it available if needed, but you could just use the example pipelines reported in the rtpbin documentation (just google "gstreamer rtpbin").

Regards
 

Best Regards,
Parveen Jain


On Fri, Oct 29, 2010 at 11:50 PM, Marco Ballesio <[hidden email]> wrote:
Hi,

the first idea coming to my mind is that the pipeline(s) you're using are not set to NULL state after terminating the communication and/or you're re-instantiating anew pipeline for each call.. is it possible?

Regards


On Fri, Oct 29, 2010 at 7:10 PM, Parveen Kumar Jain <[hidden email]> wrote:
Hi Stefan,
  Just to tell you that I don't create all these thread from own application.For every call I create two threads so in total I create a thread pool of 80 at the start of the application, and don't increase/decrease it any further.But I see all these threads(around 600 more threads apart from what I created) automatically created by Gstreamer itself.
  Infact, I am also looking for ways where I can control these no. of threads creation in my own way.As I understand creating so many threads in a Quadcore system just increases the load of scheduling without adding anything significant efficiency to system.

Thanks & Regards,
Parveen Jain

  

On Fri, Oct 29, 2010 at 9:11 PM, Stefan Kost <[hidden email]> wrote:
hi,

Am 29.10.2010 15:16, schrieb Pearveen Kumar Jain:

> Hi All,
>   I am using Gstreamer for one of my bulk call video calling application, using
> sip as my signalling part.I tried to make around 60 simulative video calls
> without much of the problem on a IBM server(4 GB RAM,Intel QuadCore,Linux 64 bit).
>   But somehow, after around 15-17 hours of testing, I see that "free memory"
> left in the system is only 300 -400 MB, which initially was around 2.5 GB. From
> "process id details" I can see that there is not much increase of "RESIDENT" and
> "VIRTUAL" memory usage in my "application process".
> Can any one please comment/help on this behavior of system ? why overall system
> memory is increasing  ?
> At this kind of video OBD load, I had seen that Gstreamer creates around 600-700
> threads.Can it be a source of this problem?

If the threads are 'leaked' (never join) it could. Can you use valgrind
--tool=memcehck ... to check?

Stefan

>
> Best Regards,
> Parveen Jain
>
>
>
> ------------------------------------------------------------------------------
> Nokia and AT&T present the 2010 Calling All Innovators-North America contest
> Create new apps & games for the Nokia N8 for consumers in  U.S. and Canada
> $10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing
> Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store
> http://p.sf.net/sfu/nokia-dev2dev
>
>
>
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel




--
Parveen Jain
Technical Lead – Network Engineering        
One97 Communications (P) Ltd                       
B121, Sector 5, Noida, UP 201301                                                      
P:  + 91 120 4770770      Extn:352
M: + 91   9212708203
W: www.one97world.com


------------------------------------------------------------------------------
Nokia and AT&T present the 2010 Calling All Innovators-North America contest
Create new apps & games for the Nokia N8 for consumers in  U.S. and Canada
$10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing
Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store
http://p.sf.net/sfu/nokia-dev2dev
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel



------------------------------------------------------------------------------
Nokia and AT&T present the 2010 Calling All Innovators-North America contest
Create new apps & games for the Nokia N8 for consumers in  U.S. and Canada
$10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing
Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store
http://p.sf.net/sfu/nokia-dev2dev
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel




--
Parveen Jain
Technical Lead – Network Engineering        
One97 Communications (P) Ltd                       
B121, Sector 5, Noida, UP 201301                                                      
P:  + 91 120 4770770      Extn:352
M: + 91   9212708203
W: www.one97world.com


------------------------------------------------------------------------------
Nokia and AT&T present the 2010 Calling All Innovators-North America contest
Create new apps & games for the Nokia N8 for consumers in  U.S. and Canada
$10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing
Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store
http://p.sf.net/sfu/nokia-dev2dev
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel



------------------------------------------------------------------------------
Nokia and AT&T present the 2010 Calling All Innovators-North America contest
Create new apps & games for the Nokia N8 for consumers in  U.S. and Canada
$10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing
Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store
http://p.sf.net/sfu/nokia-dev2dev
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
Reply | Threaded
Open this post in threaded view
|

Re: Memory usage is increasing Gstreamer testing

David Hoyt
In reply to this post by Parveen Kumar Jain
> At this kind of video OBD load, I had seen that Gstreamer creates around
600-700 threads.

I noticed something similar in my own app, but didn't have time to track it
down. What I was able to discover was that gstreamer tasks weren't being
eliminated when I transitioned from NULL -> PLAYING -> READY -> PLAYING ->
READY, etc. My thread count would gradually increase for every transition to
PLAYING, but would then go back to normal after moving to READY. However,
then there was a memory leak. When I did NULL -> PLAYING -> NULL -> PLAYING
-> NULL -> PLAYING, etc., the tasks were successfully removed and the thread
leak went away. I have no idea why that was happening. It was working fine
before...(I don't recall which release, though)



------------------------------------------------------------------------------
Nokia and AT&T present the 2010 Calling All Innovators-North America contest
Create new apps & games for the Nokia N8 for consumers in  U.S. and Canada
$10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing
Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store
http://p.sf.net/sfu/nokia-dev2dev
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel