GStreamer-devel

Gst-python object cleanup

Classic

List

Threaded

4 messages Options

Ron McOuat

Gst-python object cleanup

Hello Python gstreamer devs, users,

I have a long running application written in Python recording from a
live source which is an IP camera such as what Axis produces. The video
source is connected to a tee sink pad after processing and then from one
of the tee src pads I dynamically connect a queue, Arnout's resettime,
matroska mux and filesink to store the mjpeg stream. When I am done with
one file recording I remove the recording bin by disconnecting from the
tee and removing the bin from the pipeline. When I need to record again
I build a new recording bin and attach to the tee. The next file could
start recording on the next frame from the source so I use pad blocking
during the record bin swap.

This is all working properly for me at this point except for one thing.
As part of the stress testing phase I set the program to stop the
recording bin and create a new one once per minute. What I find is the
bin number and bus numbers in the recording bin detected by Python print
bus or print bin are increasing with each cycle and the program slowly
grows. After 12 hours of this the program has gone from 10 MBytes to
about 40 MBytes. This is enough of a memory leak to eventually cause me
problems. Most of the example programs I see are short lived and cleanup
by exiting.

When I destroy a record bin I remove it from the pipeline and set the
state to gst.STATE_NULL and have added code to unlink all the elements
inside the bin and then remove the elements from the bin. Finally I set
the Python variable holding the recording bin to None. Is there
something I missed in the cleanup process possibly equivalent to the C
unref on a Gobject? I am presuming the Python elements, bins and
pipeline etc are proxy objects that are wrappers to the C objects. I
have also considered that maybe I shouldn't be doing this in this manner
but I ran into lots of time issues trying to reuse the recording bin.

Thanks for any suggestion.

Ron McOuat

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

Edward Hervey

Re: Gst-python object cleanup

Administrator

Hi,

On Thu, 2009-10-29 at 02:08 -0700, Ron McOuat wrote:

> Hello Python gstreamer devs, users,
>
> I have a long running application written in Python recording from a
> live source which is an IP camera such as what Axis produces. The video
> source is connected to a tee sink pad after processing and then from one
> of the tee src pads I dynamically connect a queue, Arnout's resettime,
> matroska mux and filesink to store the mjpeg stream. When I am done with
> one file recording I remove the recording bin by disconnecting from the
> tee and removing the bin from the pipeline. When I need to record again
> I build a new recording bin and attach to the tee. The next file could
> start recording on the next frame from the source so I use pad blocking
> during the record bin swap.
>
> This is all working properly for me at this point except for one thing.
> As part of the stress testing phase I set the program to stop the
> recording bin and create a new one once per minute. What I find is the
> bin number and bus numbers in the recording bin detected by Python print
> bus or print bin are increasing with each cycle and the program slowly
> grows. After 12 hours of this the program has gone from 10 MBytes to
> about 40 MBytes. This is enough of a memory leak to eventually cause me
> problems. Most of the example programs I see are short lived and cleanup
> by exiting.
>
> When I destroy a record bin I remove it from the pipeline and set the
> state to gst.STATE_NULL and have added code to unlink all the elements
> inside the bin and then remove the elements from the bin. Finally I set
> the Python variable holding the recording bin to None. Is there
> something I missed in the cleanup process possibly equivalent to the C
> unref on a Gobject? I am presuming the Python elements, bins and
> pipeline etc are proxy objects that are wrappers to the C objects. I
> have also considered that maybe I shouldn't be doing this in this manner
> but I ran into lots of time issues trying to reuse the recording bin.
>

The description of how you get rid of your pipelines seems correct.

There's only one other thing that *could* go wrong, is if you're using
a signal watch on the bus, you should remove it (i.e.
gst.Bus.remove_signal_watch()).

If you've made sure that's correct, I'd then suggest to start tracing
the leaks at the python level. There's a couple of good articles/tools
to do that:
http://www.lshift.net/blog/2008/11/14/tracing-python-memory-leaks
http://mg.pov.lt/blog/hunting-python-memleaks
And you can also have a look at how debugging memory leaks is done in
the gst-python testsuite (gst-python/tests/common.py)

Edward

> Thanks for any suggestion.
>
> Ron McOuat
>
>
>
> ------------------------------------------------------------------------------
> Come build with us! The BlackBerry(R) Developer Conference in SF, CA
> is the only developer event you need to attend this year. Jumpstart your
> developing skills, take BlackBerry mobile applications to market and stay
> ahead of the curve. Join us from November 9 - 12, 2009. Register now!
> http://p.sf.net/sfu/devconference
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gstreamer-devel

Ron McOuat

Re: Gst-python object cleanup

Thanks Edward, the links helped a lot. I do have a signal watch and was
removing it in the message callback once the EOS arrived on the filesink
element bus which was inserted in order to detect EOS. I was able to
determine the Python object count is not increasing using some gc calls
from the article referred to by the second link you gave me. Moving the
application to Karmic from Jaunty and re-arranging some of the callback
code has improved the memory use rate. The old record bin cleanup was in
the EOS message callback and the new record bin was allocated right
after injecting the EOS into the old record bin so there was a race
between the two. The old and new bins had different class variable
references bound to keep them apart so I am reasonably sure I wasn't
stomping on the wrong things with a shared reference. I now create the
new record bin in the EOS callback after the old one is cleaned up
making the operations serial. There is a queue in front to accept frames
while this is taking place so the front end doesn't lose data or block.

Watching the python process with gnome-system-monitor it seems to jump
up about a MByte once in a while and not in a monotonic once per cycle
way. It is like the Python arena gets fragmented somehow and so the
virtual machine grabs a new chunk to work with - highly speculative but
pymalloc front ends system malloc allocating large chucks. To put it in
perspective, the stress testing cycles the application through about 2
months of work every 24 hours of test run time, with the latest changes
memory consumption will probably be acceptable.

BTW I would like to say thanks for gstreamer, it is a great multimedia
processing library and is improving steadily. The Python bindings make
it even easier to work with - I am relatively new to the language, it is
so clean and concise - I come from Java since 2000, C, C++ since 1983
and Fortran/Assembler since 1972.

Ron

Edward Hervey wrote:

> The description of how you get rid of your pipelines seems correct.
>
> There's only one other thing that *could* go wrong, is if you're using
> a signal watch on the bus, you should remove it (i.e.
> gst.Bus.remove_signal_watch()).
>
> If you've made sure that's correct, I'd then suggest to start tracing
> the leaks at the python level. There's a couple of good articles/tools
> to do that:
> http://www.lshift.net/blog/2008/11/14/tracing-python-memory-leaks
> http://mg.pov.lt/blog/hunting-python-memleaks
> And you can also have a look at how debugging memory leaks is done in
> the gst-python testsuite (gst-python/tests/common.py)
>
> Edward
>
>
>> Thanks for any suggestion.
>>
>> Ron McOuat
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Come build with us! The BlackBerry(R) Developer Conference in SF, CA
>> is the only developer event you need to attend this year. Jumpstart your
>> developing skills, take BlackBerry mobile applications to market and stay
>> ahead of the curve. Join us from November 9 - 12, 2009. Register now!
>> http://p.sf.net/sfu/devconference
>> _______________________________________________
>> gstreamer-devel mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>>
>
>
> ------------------------------------------------------------------------------
> Come build with us! The BlackBerry(R) Developer Conference in SF, CA
> is the only developer event you need to attend this year. Jumpstart your
> developing skills, take BlackBerry mobile applications to market and stay
> ahead of the curve. Join us from November 9 - 12, 2009. Register now!
> http://p.sf.net/sfu/devconference
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>
>

Edward Hervey

Re: Gst-python object cleanup

Administrator

On Fri, 2009-10-30 at 00:39 -0700, Ron McOuat wrote:

> Thanks Edward, the links helped a lot. I do have a signal watch and was
> removing it in the message callback once the EOS arrived on the filesink
> element bus which was inserted in order to detect EOS. I was able to
> determine the Python object count is not increasing using some gc calls
> from the article referred to by the second link you gave me. Moving the
> application to Karmic from Jaunty and re-arranging some of the callback
> code has improved the memory use rate. The old record bin cleanup was in
> the EOS message callback and the new record bin was allocated right
> after injecting the EOS into the old record bin so there was a race
> between the two. The old and new bins had different class variable
> references bound to keep them apart so I am reasonably sure I wasn't
> stomping on the wrong things with a shared reference. I now create the
> new record bin in the EOS callback after the old one is cleaned up
> making the operations serial. There is a queue in front to accept frames
> while this is taking place so the front end doesn't lose data or block.
>
> Watching the python process with gnome-system-monitor it seems to jump
> up about a MByte once in a while and not in a monotonic once per cycle
> way. It is like the Python arena gets fragmented somehow and so the
> virtual machine grabs a new chunk to work with - highly speculative but
> pymalloc front ends system malloc allocating large chucks. To put it in
> perspective, the stress testing cycles the application through about 2
> months of work every 24 hours of test run time, with the latest changes
> memory consumption will probably be acceptable.

If you've exhausted your investigations with the gc module (and the
related tools/scripts), you could go down to the C layer and use tools
like massif (from valgrind) to figure out where the memory consumption
happens over time. Unfortunately they removed the graphical visualiser
in some recent versions.

>
> BTW I would like to say thanks for gstreamer, it is a great multimedia
> processing library and is improving steadily.

yay \o/

> The Python bindings make
> it even easier to work with - I am relatively new to the language, it is
> so clean and concise - I come from Java since 2000, C, C++ since 1983
> and Fortran/Assembler since 1972.

yay again :) You've come a long way.

Edward

>
> Ron
>
> Edward Hervey wrote:
> > The description of how you get rid of your pipelines seems correct.
> >
> > There's only one other thing that *could* go wrong, is if you're using
> > a signal watch on the bus, you should remove it (i.e.
> > gst.Bus.remove_signal_watch()).
> >
> > If you've made sure that's correct, I'd then suggest to start tracing
> > the leaks at the python level. There's a couple of good articles/tools
> > to do that:
> > http://www.lshift.net/blog/2008/11/14/tracing-python-memory-leaks
> > http://mg.pov.lt/blog/hunting-python-memleaks
> > And you can also have a look at how debugging memory leaks is done in
> > the gst-python testsuite (gst-python/tests/common.py)
> >
> > Edward
> >
> >
> >> Thanks for any suggestion.
> >>
> >> Ron McOuat
> >>
> >>
> >>
> >> ------------------------------------------------------------------------------
> >> Come build with us! The BlackBerry(R) Developer Conference in SF, CA
> >> is the only developer event you need to attend this year. Jumpstart your
> >> developing skills, take BlackBerry mobile applications to market and stay
> >> ahead of the curve. Join us from November 9 - 12, 2009. Register now!
> >> http://p.sf.net/sfu/devconference
> >> _______________________________________________
> >> gstreamer-devel mailing list
> >> [hidden email]
> >> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
> >>
> >
> >
> > ------------------------------------------------------------------------------
> > Come build with us! The BlackBerry(R) Developer Conference in SF, CA
> > is the only developer event you need to attend this year. Jumpstart your
> > developing skills, take BlackBerry mobile applications to market and stay
> > ahead of the curve. Join us from November 9 - 12, 2009. Register now!
> > http://p.sf.net/sfu/devconference
> > _______________________________________________
> > gstreamer-devel mailing list
> > [hidden email]
> > https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
> >
> >
>
> ------------------------------------------------------------------------------
> Come build with us! The BlackBerry(R) Developer Conference in SF, CA
> is the only developer event you need to attend this year. Jumpstart your
> developing skills, take BlackBerry mobile applications to market and stay
> ahead of the curve. Join us from November 9 - 12, 2009. Register now!
> http://p.sf.net/sfu/devconference
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel