python memory leak

[Bram Van Steenlandt]

Hi,

I have an application (rtsp camera viewing and recording) in python
using gstreamer 1.0.

It all works but I think one (or more) of the pipelines are leaking
memory, at first I thought the problem was related to the restarting of
the stream but this doesn't seem to be the case.
It also doesn't happen (less camera and a more complicated pipeline with
an extra tcpclient) at all on one machine and on another machine (where
the load is higher) it happens "all the time" I think an external factor
causes the leak (corrupt jpeg stream, packet loss etc.)
On the machine where it happens "all the time", I think the maximum time
without leaking is 6 hours I think, it also not every 6 hours, sometimes
it's less.

I now have a temporary fix where the application restarts when memory is
running out, this works but it's not ideal.
I have 9 pipelines running, for another project I'm hoping to run 32 or 64.

Is there a way I can:
-Find out how much memory one pipeline is taking up from python ? sizeof
didn't work
-Tell a pipeline to limit it's memory consumption to a maximum amount
and "destroy" itself when it fails to do so.

What memory consumption is to be considered normal for streaming 9
cameras (720p/15fps) ? I saw values of 4GB and up on the problem
machine, the other machine seems stable at 200MB (only 2 streams).

The pipeline I'm using is along these lines:
rtspsrc name=src location=rtsp://x.x.x.x latency=100 ! rtpjpegdepay !  
tee name=a !  jpegdec ! queue  ! xvimagesink  a. ! queue ! videorate !
image/jpeg,framerate=5/1 !  matroskamux ! multifilesink
aggregate-gops=true next-file=max-size max-file-size=500000000
location=recordings/tmp/cam9.%02d.mkv

When looking at this, is there any element (matroskamux ?) that isn't
suppose to be used in this way and may cause problems ?
I thought about replacing matroskamux with multipartmux (I can convert
them afterwards) but as one installation is running fine with
matroskamux maybe this isn't the issue
I also checked the trsp connections with netstat and there is only one
connection to every camera.

rebooting the source (camera's) and thus launching a new stream does not
free the memory.

Is it at all possible this is infact a hardware issue ? (bad memory?)

thx
Bram
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On Di, 2016-03-15 at 18:54 +0100, Bram Van Steenlandt wrote:

> Hi,
>
> I have an application (rtsp camera viewing and recording) in python 
> using gstreamer 1.0.
>
> It all works but I think one (or more) of the pipelines are leaking 
> memory, at first I thought the problem was related to the restarting of 
> the stream but this doesn't seem to be the case.
> It also doesn't happen (less camera and a more complicated pipeline with 
> an extra tcpclient) at all on one machine and on another machine (where 
> the load is higher) it happens "all the time" I think an external factor 
> causes the leak (corrupt jpeg stream, packet loss etc.)
> On the machine where it happens "all the time", I think the maximum time 
> without leaking is 6 hours I think, it also not every 6 hours, sometimes 
> it's less.
>
> I now have a temporary fix where the application restarts when memory is 
> running out, this works but it's not ideal.
> I have 9 pipelines running, for another project I'm hoping to run 32 or 64.
>
> Is there a way I can:
> -Find out how much memory one pipeline is taking up from python ? sizeof 
> didn't work
> -Tell a pipeline to limit it's memory consumption to a maximum amount 
> and "destroy" itself when it fails to do so.

Your pipeline looks relatively simple, so best would be to recreate a
testcase in C and then a) check if it still has increasing memory usage
and b) check with valgrind's massif where that memory is mostly
allocated.

If I understand you correctly the problem is that memory usage grows
over time and restarting the pipeline alone does not help? Unreffing
the pipeline neither? In that case, valgrind's memcheck might also
point you to memory leaks when exiting the application after a run.

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Bram Van Steenlandt]

Hi,

An update on the situation:
-I replaced the matroskamux with multipartmux, it now has been running
for 20 hours and seems stable (at 380 MB for 9 streams).
-I also posted to this mailinglist with another problem concerning
tcpserversink, this problem was completely solved by using tcpclientsink
and implementing a kind of tcpserversink in python (where gstreamer
connects to).

Thx for your help and input.
Bram
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On Do, 2016-03-17 at 20:37 +0100, Bram Van Steenlandt wrote:
> Hi,
>
> An update on the situation:
> -I replaced the matroskamux with multipartmux, it now has been
> running for 20 hours and seems stable (at 380 MB for 9 streams).

For matroskamux, did you set streamable=false? If it's not set to
false, matroskamux will create a seek table and that's going to consume
more and more memory over time. And in the end it would try writing
this seek table out, which is a bit useless when streaming the media.

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Bram Van Steenlandt]

Hi,

Op 18-03-16 om 09:55 schreef Sebastian Dröge:

> On Do, 2016-03-17 at 20:37 +0100, Bram Van Steenlandt wrote:
>> Hi,
>>
>> An update on the situation:
>> -I replaced the matroskamux with multipartmux, it now has been
>> running for 20 hours and seems stable (at 380 MB for 9 streams).
> For matroskamux, did you set streamable=false? If it's not set to
> false, matroskamux will create a seek table and that's going to consume
> more and more memory over time. And in the end it would try writing
> this seek table out, which is a bit useless when streaming the media.

No, I did not, I should have.
This night I had another memory leak (after running ok for over 12
hours), not sure if it's the same issue or a new one.
With the matroskamux if it started it was like this:
cam 2 -> 2016/3/16    3:53:26    Playing started (ASYNC DONE)
cam MEM -> 2016/3/16    3:53:29    Memory usage jumped to 220.85 MB
cam MEM -> 2016/3/16    4:14:4    Memory usage jumped to 256.22 MB
cam MEM -> 2016/3/16    5:57:39    Memory usage jumped to 294.7 MB
cam MEM -> 2016/3/16    6:19:4    Memory usage jumped to 352.4 MB
cam MEM -> 2016/3/16    6:20:9    Memory usage jumped to 413.34 MB
cam MEM -> 2016/3/16    6:20:14    Memory usage jumped to 459.5 MB
cam MEM -> 2016/3/16    6:20:24    Memory usage jumped to 493.61 MB
cam MEM -> 2016/3/16    6:20:29    Memory usage jumped to 531.62 MB
cam MEM -> 2016/3/16    6:20:44    Memory usage jumped to 578.13 MB
cam MEM -> 2016/3/16    6:20:54    Memory usage jumped to 610.66 MB
cam MEM -> 2016/3/16    6:21:19    Memory usage jumped to 648.69 MB
cam MEM -> 2016/3/16    6:21:29    Memory usage jumped to 684.39 MB
cam MEM -> 2016/3/16    6:21:39    Memory usage jumped to 720.0 MB
cam MEM -> 2016/3/16    6:21:49    Memory usage jumped to 757.09 MB
cam MEM -> 2016/3/16    6:21:59    Memory usage jumped to 792.17 MB
thus, I lost 571MB in 2 hours and 28 minutes which is 3.85MB every minute.
This is more or less the amount of data being streamed by one camera.
I could see this being because of the streamable=false, I do find it
strange that the other machine didn't have any issues but here it are
different and less cameras and a more solid network...

Now, I had this (with the multipart mux):
cam 9 -> 2016/3/17    21:16:49    Playing started (ASYNC DONE)
cam MEM -> 2016/3/17    21:16:53    Memory usage jumped to 136.95 MB
cam MEM -> 2016/3/17    21:28:18    Memory usage jumped to 168.96 MB
cam MEM -> 2016/3/17    21:36:13    Memory usage jumped to 220.84 MB
cam MEM -> 2016/3/18    1:51:29    Memory usage jumped to 272.02 MB
cam MEM -> 2016/3/18    1:51:33    Memory usage jumped to 327.98 MB
cam MEM -> 2016/3/18    1:51:38    Memory usage jumped to 399.15 MB
cam MEM -> 2016/3/18    1:51:43    Memory usage jumped to 472.88 MB
cam MEM -> 2016/3/18    1:51:48    Memory usage jumped to 548.75 MB
cam MEM -> 2016/3/18    1:51:53    Memory usage jumped to 623.47 MB
cam MEM -> 2016/3/18    1:51:58    Memory usage jumped to 698.37 MB
cam MEM -> 2016/3/18    1:52:3    Memory usage jumped to 774.9 MB
cam MEM -> 2016/3/18    1:52:9    Memory usage jumped to 850.0 MB
cam MEM -> 2016/3/18    1:52:13    Memory usage jumped to 923.92 MB
cam MEM -> 2016/3/18    1:52:18    Memory usage jumped to 999.24 MB
cam MEM -> 2016/3/18    1:52:23    Memory usage jumped to 1074.46 MB
here, I lost 802MB in one minute, this feels like a different memory
leak than the one above.
I have now added some code to try and stop all streams (set to null) to
see if this frees up the memory in this case.

Bram



_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel