Gstreamer memory leak: how do I hunt it down?

[Graham Leggett]

Hi all,

I have a gstreamer that slowly increases in size over time, until oomkiller steps in after a few hours and blows it away.

Using GST_REFCOUNTING set to 9, I am finding what looks like the leak, but I need help interpreting this. What do the addresses 0x740f7580 and 0x74204818 refer to? Am I looking at one object with thousands of references to it, or many objects?

Is this maybe an array of some kind that keeps increasing in size?

0:01:01.779640164  1232 0x740f7580 TRACE        GST_REFCOUNTING gstobject.c:275:gst_object_unref:<allocatorsysmem0> 0x74204818 unref 5958->5957
0:01:01.779719382  1232 0x7408cc60 TRACE        GST_REFCOUNTING gstobject.c:275:gst_object_unref:<allocatorsysmem0> 0x74204818 unref 5958->5957
0:01:01.779825319  1232 0x7408cc60 TRACE        GST_REFCOUNTING gstobject.c:275:gst_object_unref:<allocatorsysmem0> 0x74204818 unref 5956->5955
0:01:01.779882089  1232 0x7407e0f0 TRACE        GST_REFCOUNTING gstobject.c:275:gst_object_unref:<allocatorsysmem0> 0x74204818 unref 5956->5955
0:01:01.779953182  1232 0x740f7580 TRACE        GST_REFCOUNTING gstobject.c:275:gst_object_unref:<allocatorsysmem0> 0x74204818 unref 5955->5954
0:01:01.780327084  1232 0x740f7580 TRACE        GST_REFCOUNTING gstobject.c:275:gst_object_unref:<allocatorsysmem0> 0x74204818 unref 5953->5952
0:01:01.780604269  1232 0x7408cc60 TRACE        GST_REFCOUNTING gstobject.c:249:gst_object_ref:<allocatorsysmem0> 0x74204818 ref 5952->5953
0:01:01.780793173  1232 0x7407e0f0 TRACE        GST_REFCOUNTING gstobject.c:275:gst_object_unref:<allocatorsysmem0> 0x74204818 unref 5953->5952
0:01:01.781219106  1232 0x7407e0f0 TRACE        GST_REFCOUNTING gstobject.c:249:gst_object_ref:<allocatorsysmem0> 0x74204818 ref 5952->5953
0:01:01.781276397  1232 0x7408cc60 TRACE        GST_REFCOUNTING gstobject.c:249:gst_object_ref:<allocatorsysmem0> 0x74204818 ref 5952->5953
0:01:01.781873943  1232 0x740f7580 TRACE        GST_REFCOUNTING gstobject.c:275:gst_object_unref:<allocatorsysmem0> 0x74204818 unref 5954->5953
0:01:01.782029150  1232 0x7407e0f0 TRACE        GST_REFCOUNTING gstobject.c:249:gst_object_ref:<allocatorsysmem0> 0x74204818 ref 5953->5954
0:01:01.782796017  1232 0x7407e0f0 TRACE        GST_REFCOUNTING gstobject.c:275:gst_object_unref:<allocatorsysmem0> 0x74204818 unref 5954->5953
0:01:01.783040181  1232 0x740f7580 TRACE        GST_REFCOUNTING gstobject.c:249:gst_object_ref:<allocatorsysmem0> 0x74204818 ref 5953->5954
0:01:01.783131638  1232 0x7408cc60 TRACE        GST_REFCOUNTING gstobject.c:249:gst_object_ref:<allocatorsysmem0> 0x74204818 ref 5953->5954

Regards,
Graham
—


_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On Sun, 2016-12-18 at 02:41 +0200, Graham Leggett wrote:

> Hi all,
>
> I have a gstreamer that slowly increases in size over time, until
> oomkiller steps in after a few hours and blows it away.
>
> Using GST_REFCOUNTING set to 9, I am finding what looks like the
> leak, but I need help interpreting this. What do the addresses
> 0x740f7580 and 0x74204818 refer to? Am I looking at one object with
> thousands of references to it, or many objects?
>
> Is this maybe an array of some kind that keeps increasing in size?
>
> 0:01:01.779640164  1232 0x740f7580 TRACE        GST_REFCOUNTING
> gstobject.c:275:gst_object_unref:<allocatorsysmem0> 0x74204818 unref
> 5958->5957

This suggests that something somewhere is leaking system memory
GstMemory. Each of them has a reference to its allocator, so you seem
to have about 6000 of them around currently.
It's not the allocator that is leaked here though.


Use valgrind, valgrind's massif or the GStreamer leak tracer for
tracking this down. Also before doing anything like that, make your
pipeline as minimal as possible.

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Graham Leggett]

On 18 Dec 2016, at 10:57 AM, Sebastian Dröge <[hidden email]> wrote:

>> 0:01:01.779640164  1232 0x740f7580 TRACE        GST_REFCOUNTING
>> gstobject.c:275:gst_object_unref:<allocatorsysmem0> 0x74204818 unref
>> 5958->5957
>
> This suggests that something somewhere is leaking system memory
> GstMemory. Each of them has a reference to its allocator, so you seem
> to have about 6000 of them around currently.
> It's not the allocator that is leaked here though.

This makes more sense now.

> Use valgrind, valgrind's massif or the GStreamer leak tracer for
> tracking this down. Also before doing anything like that, make your
> pipeline as minimal as possible.

Narrowed it down to the tsparse element.

Valgrind doesn’t run on a pi, and neither does LeakAnalyzer, so I’m stuck with the gstreamer leak tracer. Is this documented anywhere?

All I could find was https://www.collabora.com/news-and-blog/blog/2016/06/20/gstreamer-leaks-tracer/, and following the instructions leads to just this, which appears to show the leak tracer is present, but it isn’t telling me anything I understand:

minfrin@towerofpi9:/var/www/html/stream$ cat /mnt/stream/stream22.out | grep TRACER
0:00:00.417222937  5468 0x73f75fa0 DEBUG             GST_TRACER gsttracer.c:163:gst_tracer_register:<registry0> update existing feature 0x74104888 (latency)
0:00:00.417918242  5468 0x73f75fa0 DEBUG             GST_TRACER gsttracer.c:163:gst_tracer_register:<registry0> update existing feature 0x74104900 (log)
0:00:00.418445632  5468 0x73f75fa0 DEBUG             GST_TRACER gsttracer.c:163:gst_tracer_register:<registry0> update existing feature 0x74104978 (rusage)
0:00:00.418818024  5468 0x73f75fa0 DEBUG             GST_TRACER gsttracer.c:163:gst_tracer_register:<registry0> update existing feature 0x741049f0 (stats)
0:00:00.419100260  5468 0x73f75fa0 DEBUG             GST_TRACER gsttracer.c:163:gst_tracer_register:<registry0> update existing feature 0x74104a68 (leaks)
0:00:00.420038063  5468 0x73f75fa0 TRACE             GST_TRACER gsttracerrecord.c:110:gst_tracer_record_build_format: object-alive.class, type-name=(structure)"value\,\ type\=\(GType\)NULL\,\ related-to\=\(GstTracerValueScope\)GST_TRACER_VALUE_SCOPE_PROCESS\;", address=(structure)"value\,\ type\=\(GType\)NULL\,\ related-to\=\(GstTracerValueScope\)GST_TRACER_VALUE_SCOPE_PROCESS\;", description=(structure)"value\,\ type\=\(GType\)NULL\,\ related-to\=\(GstTracerValueScope\)GST_TRACER_VALUE_SCOPE_PROCESS\;", ref-count=(structure)"value\,\ type\=\(GType\)NULL\,\ related-to\=\(GstTracerValueScope\)GST_TRACER_VALUE_SCOPE_PROCESS\;", trace=(structure)"value\,\ type\=\(GType\)NULL\,\ related-to\=\(GstTracerValueScope\)GST_TRACER_VALUE_SCOPE_PROCESS\;";
0:00:00.421316747  5468 0x73f75fa0 DEBUG             GST_TRACER gsttracerrecord.c:124:gst_tracer_record_build_format: new format string: object-alive, type-name=(string)%s, address=(gpointer)%p, description=(string)%s, ref-count=(uint)%u, trace=(string)%s;

Regards,
Graham
—





_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge-3]

On Sun, 2016-12-18 at 17:58 +0200, Graham Leggett wrote:

> On 18 Dec 2016, at 10:57 AM, Sebastian Dröge <[hidden email]
> om> wrote:
>
> > > 0:01:01.779640164  1232 0x740f7580 TRACE        GST_REFCOUNTING
> > > gstobject.c:275:gst_object_unref:<allocatorsysmem0> 0x74204818
> > > unref
> > > 5958->5957
> >
> > This suggests that something somewhere is leaking system memory
> > GstMemory. Each of them has a reference to its allocator, so you
> > seem
> > to have about 6000 of them around currently.
> > It's not the allocator that is leaked here though.
>
> This makes more sense now.
>
> > Use valgrind, valgrind's massif or the GStreamer leak tracer for
> > tracking this down. Also before doing anything like that, make your
> > pipeline as minimal as possible.
>
> Narrowed it down to the tsparse element. 
>
> Valgrind doesn’t run on a pi, and neither does LeakAnalyzer, so I’m
> stuck with the gstreamer leak tracer. Is this documented anywhere?

Do you have a way to reproduce it? Getting the stream and reproducing
it with that stream on a machine that can run valgrind would seem like
the best option then.

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com
_______________________________________________
gstreamer-devel mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel