gst-plugin-scanner hangs, eating CPU time

[Rüdiger Kupper]

Dear gstreamer developers,

we run Linux Terminal Servers (LTSP) at a German school, for approx. 800
users. The servers run standard Edubuntu, but are located behind a
firewall that requires authentication for outgoing traffic.

I frequently see a lot of "gst-plugin-scanner" processes running on our
servers, without ever terminating, and eating up CPU time. Since many
users log into the same server, this easily pulls up the load to 10 or
above, basically killing the servers and frustrating the users (and
admin ;-) ).

I have not been able to determine, what "gst-plugin-scanner" actually
does. I can reproduce that it starts when users try to open sound files
using banshee or totem.

From its name I suspect that "gst-plugin-scanner" may try to contact an
external web site in order to check for plugins, but this is just my
wild guess. If so, it certainly gets rejected by our proxy.

Could you clarify for me what "gst-plugin-scanner" actually does? If it
does web traffic, which site is it trying to reach? (If I knew the
address I could configure our proxy to let it pass.)

Or is there any other reason why "gst-plugin-scanner" should fail so
badly in an LTSP-environment?

I'd strongly appreciate your help, since this is really a critical
problem in our setup.

As a workaround, I have removed execute permissions on the
"gst-plugin-scanner" binary, but I do not feel well with that, not
knowing what it does.

Thank you in advance for any comments,
regards,
Rüdiger

--
Dr. Rüdiger Kupper <[hidden email]>
Kepler-Gymnasium Freundenstadt


_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Tim-Philipp Müller-2]

On Fri, 2011-11-11 at 01:57 +0100, Rüdiger Kupper wrote:

Hi Rüdiger,

> we run Linux Terminal Servers (LTSP) at a German school, for approx. 800
> users. The servers run standard Edubuntu, but are located behind a
> firewall that requires authentication for outgoing traffic.
>
> I frequently see a lot of "gst-plugin-scanner" processes running on our
> servers, without ever terminating, and eating up CPU time. Since many
> users log into the same server, this easily pulls up the load to 10 or
> above, basically killing the servers and frustrating the users (and
> admin ;-) ).
>
> I have not been able to determine, what "gst-plugin-scanner" actually
> does. I can reproduce that it starts when users try to open sound files
> using banshee or totem.

GStreamer maintains a cache file ("registry") of available plugins. When
GStreamer is initialised by an application, it loads the existing
registry cache if there is one and then checks the available plugins. If
any plugins were were added or have changed since the last time they
were introspected, GStreamer will load those plugins to update the
cached information.

gst-plugin-scanner does exactly this, but out of process, so if a plugin
crashes while loading it doesn't take down the entire application and
can instead just be blacklisted.

> From its name I suspect that "gst-plugin-scanner" may try to contact an
> external web site in order to check for plugins, but this is just my
> wild guess. If so, it certainly gets rejected by our proxy.

The scanner itself does not establish any network connections (unless
the plugins are on a network-mounted part of the file system of course).
It just loads (filesystem-)local plugins and sends the information it
found in the plugins back to the application that spawned it.

It is possible of course that some plugin does that kind of thing in its
plugin init function, but that seems rather unlikely and I'm not aware
of any such plugins.

> Could you clarify for me what "gst-plugin-scanner" actually does? If it
> does web traffic, which site is it trying to reach? (If I knew the
> address I could configure our proxy to let it pass.)
>
> Or is there any other reason why "gst-plugin-scanner" should fail so
> badly in an LTSP-environment?
>
> I'd strongly appreciate your help, since this is really a critical
> problem in our setup.
>
> As a workaround, I have removed execute permissions on the
> "gst-plugin-scanner" binary, but I do not feel well with that, not
> knowing what it does.

That should not be a problem. In this case GStreamer should just fall
back to loading the plugin in-process. You can achieve the same thing by
setting the GST_REGISTRY_FORK environment variable to "no".

What version of GStreamer core are you using on those machines? There
have been a few fixes for corner-cases in gstpoll if I remember
correctly, but I am not sure those ever caused problems on Linux
systems.

Have you tried attaching to one of those runaway scanner instances with
strace or gdb to see what they're doing?

Did you check what plugins those runaway scanners have open?
(cat /proc/$PID/maps | grep gst) Maybe there's one that all those
runaway scanners have in common..

 Cheers
  -Tim


_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Rüdiger Kupper]

Dear Tim-Philipp,

thank you very much for your answer! I do now better understand what
gst-plugin-scanner does.

My effort to work around the problem by removing execute permissions
from gst-plugin-scanner did not succeed. As you explained in your mail,
the applications then fall back to testing the plugins in-process. This
causes the applications (e.g. totem) to hang.

Interestingly, when I kill the hung totem and after that invoke it again
(on any sound file), it works.

I attach the relevant part of an strace of the hung totem: I am not well
versed in reading straces, but obviously the process tries to read from
some resource that is not available and receives signal EAGAIN, telling
it to try again, which it does, over and over. I am afraid I cannot
deduce what resource it is. Perhaps you can make more of it.

I also attach the output of "cat /proc/$PID/maps | grep gst" for the
hung totem. As you can see I have gstreamer-0.10 installed.
To be specific, the following packages:

$ dpkg -l gstreamer*plugins* | grep ^ii
ii  gstreamer0.10-plugins-bad              0.10.22-2ubuntu4
ii  gstreamer0.10-plugins-bad-multiverse   0.10.21-1
ii  gstreamer0.10-plugins-base             0.10.35-1
ii  gstreamer0.10-plugins-base-apps        0.10.35-1
ii  gstreamer0.10-plugins-good             0.10.30-1ubuntu7
ii  gstreamer0.10-plugins-ugly             0.10.18-3ubuntu1

I have found these bug reports, do you think they are related?

- https://bugs.launchpad.net/gst-inspector/+bug/489408
- https://bugzilla.gnome.org/show_bug.cgi?id=615357
  (also referenced here:
http://lists.freedesktop.org/archives/gstreamer-commits/2011-April/048641.html
)

Thanks again, your expertise is highly valued,
regards,

Rüdiger


Am 11.11.2011 10:46, schrieb Tim-Philipp Müller:

> On Fri, 2011-11-11 at 01:57 +0100, Rüdiger Kupper wrote:
>
> Hi Rüdiger,
>
>> we run Linux Terminal Servers (LTSP) at a German school, for approx. 800
>> users. The servers run standard Edubuntu, but are located behind a
>> firewall that requires authentication for outgoing traffic.
>>
>> I frequently see a lot of "gst-plugin-scanner" processes running on our
>> servers, without ever terminating, and eating up CPU time. Since many
>> users log into the same server, this easily pulls up the load to 10 or
>> above, basically killing the servers and frustrating the users (and
>> admin ;-) ).
>>
>> I have not been able to determine, what "gst-plugin-scanner" actually
>> does. I can reproduce that it starts when users try to open sound files
>> using banshee or totem.
>
> GStreamer maintains a cache file ("registry") of available plugins. When
> GStreamer is initialised by an application, it loads the existing
> registry cache if there is one and then checks the available plugins. If
> any plugins were were added or have changed since the last time they
> were introspected, GStreamer will load those plugins to update the
> cached information.
>
> gst-plugin-scanner does exactly this, but out of process, so if a plugin
> crashes while loading it doesn't take down the entire application and
> can instead just be blacklisted.
>
>>  From its name I suspect that "gst-plugin-scanner" may try to contact an
>> external web site in order to check for plugins, but this is just my
>> wild guess. If so, it certainly gets rejected by our proxy.
>
> The scanner itself does not establish any network connections (unless
> the plugins are on a network-mounted part of the file system of course).
> It just loads (filesystem-)local plugins and sends the information it
> found in the plugins back to the application that spawned it.
>
> It is possible of course that some plugin does that kind of thing in its
> plugin init function, but that seems rather unlikely and I'm not aware
> of any such plugins.
>
>> Could you clarify for me what "gst-plugin-scanner" actually does? If it
>> does web traffic, which site is it trying to reach? (If I knew the
>> address I could configure our proxy to let it pass.)
>>
>> Or is there any other reason why "gst-plugin-scanner" should fail so
>> badly in an LTSP-environment?
>>
>> I'd strongly appreciate your help, since this is really a critical
>> problem in our setup.
>>
>> As a workaround, I have removed execute permissions on the
>> "gst-plugin-scanner" binary, but I do not feel well with that, not
>> knowing what it does.
>
> That should not be a problem. In this case GStreamer should just fall
> back to loading the plugin in-process. You can achieve the same thing by
> setting the GST_REGISTRY_FORK environment variable to "no".
>
> What version of GStreamer core are you using on those machines? There
> have been a few fixes for corner-cases in gstpoll if I remember
> correctly, but I am not sure those ever caused problems on Linux
> systems.
>
> Have you tried attaching to one of those runaway scanner instances with
> strace or gdb to see what they're doing?
>
> Did you check what plugins those runaway scanners have open?
> (cat /proc/$PID/maps | grep gst) Maybe there's one that all those
> runaway scanners have in common..
>
>   Cheers
>    -Tim
>
>
> _______________________________________________
> gstreamer-devel mailing list
> [hidden email]
> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

--
Dr. Rüdiger Kupper <[hidden email]>
Kepler-Gymnasium Freudenstadt

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Rüdiger Kupper]

Dear Tim,

after reactivating execute permissions of gst-plugin-scanner I was able
to do an strace of one of the runaway gst-plugin-scanner processes.

The EAGAIN signals a saw in the strace of totem do not appear, so this
was not the cause.

I attach strace and proc/maps of the runaway gst-plugin-scanner process.

Removing gstreamer-plugins-bad and gstreamer-plugins-ugly did not help,
so the plugin causing the hang is not in there.

Any ideas will be welcome. Thank you very much,

regards,
Rüdiger


--
Dr. Rüdiger Kupper <[hidden email]>
Kepler-Gymnasium Freudenstadt

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Tim-Philipp Müller-2]

On Mon, 2011-11-14 at 01:26 +0100, Rüdiger Kupper wrote:

Hi,

> after reactivating execute permissions of gst-plugin-scanner I was able
> to do an strace of one of the runaway gst-plugin-scanner processes.
>
> The EAGAIN signals a saw in the strace of totem do not appear, so this
> was not the cause.
>
> I attach strace and proc/maps of the runaway gst-plugin-scanner process.
>
> Removing gstreamer-plugins-bad and gstreamer-plugins-ugly did not help,
> so the plugin causing the hang is not in there.
>
> Any ideas will be welcome. Thank you very much,

Could you also collect a GST_DEBUG=*:5 log please?

 $ GST_DEBUG=*:5 totem 2>dbg.log
 .. control-C after a while if it hangs ...

Also, it would be good if you could open a bug report about this in
bugzilla and attach all the logs there (gzipped or bzip2ed if too
large).

 Cheers
  -Tim


_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Rüdiger Kupper]

On 11/14/2011 01:42 AM, Tim-Philipp Müller wrote:
> Could you also collect a GST_DEBUG=*:5 log please?
>
>  $ GST_DEBUG=*:5 totem 2>dbg.log
>  .. control-C after a while if it hangs ...

Yes, see attachment.

> Also, it would be good if you could open a bug report about this in
> bugzilla and attach all the logs there (gzipped or bzip2ed if too
> large).

I just did: https://bugzilla.gnome.org/show_bug.cgi?id=664073.

Can you learn something from the debug log?

Best,
Rüdiger

--
Dr. Rüdiger Kupper <[hidden email]>
Kepler-Gymnasium Freudenstadt

_______________________________________________
gstreamer-devel mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel