Dear colleagues,
I've heard that Golem bot, which is the heart of the connectivity project, stopped to function due to the recent toolserver reconfiguration.
Is it possible to adjust configuration specifically for Golem or to do something else to make it function again?
It is especially a pity that the connectivity project has problems now, just two days after the project was reported at Konferencija Wikimedia Polska and received much of interest from the listeners.
What could be done to fix the situation? Thanks!
Hello Vladimir
The problem is that Golem uses a very large amount of memory, about 4GB. That's 1/8 of the total capacity, and it's memory that can not be used for normal database operations if it's set aside for the memory tables golem uses (even if they are not in use). This by far exceeds the fair share of resources for each toolserver user.
It was only recently discovered that golem does use so much memory (because it does it on the database server, not the normal user server), but it is suspected that this is at least one of the causes that triggered system failured in the past. We do not currently see the possibility of allowing individual users to use that much memory, especially not on the database server. Basically, as it is implemented now, Golem is unfit for the toolserver, because it consumes far too much resources.
Earlier today, I ask Mashiah to consider alternative ways to implement the network analysis. I think it would be possible to reduce the memory use by at least a factor of 8. Should this not be possible, golem would have to run on a dedicated system.
If there are good reasons and sufficient funding, setting aside a VM or even a full server for a special project can be considered. How individual projects and chapters can participate more in the governance (and funding) of the toolserver is one of the topics that will be discussed at the upcoming chapters' conference in april in berlin. I recommend you bring up the topic of golem there.
Regards, Daniel
PS: Below I quote my reply to Mashiah.
Hello Mashiah
Connectivity is a property of a graph as a whole, there is no way to analyze it having just a part of all nodes and edges. Use of original tables in language database or use of MyISAM tables makes the analysis far too slow. Good thing with memory tables is not only in being located in memory (which is not always true of course), the engine is optimized for speed itself and the format is designed to allow that.
If your project requires more resources than are available as your fair share on the toolserver, then either the need for resources needs to be reduced, or the project has to run elsewhere. If there are good reasons and sufficient funding, setting aside a VM or even a full server for a special project can be considered. How individual projects and chapters can participate more in the givernance (and funding) of the toolserver is one of the topics that will be discussed at the upcoming chapter's conference in april in berlin. I suggest you contact someone who will attend the meeting, and discuss the issue with them.
Anyway, if using MySQL's memory tables consumes too much resources, perhaps consider alternatives? Have you looked at network analysis frameworks like JUNG (Java) or SNAP (C++)? Relational databases are not good at managing linked structures like trees and graphs anyway.
The memory requirements shouldn't be that huge anyway: two IDs per edge = 8 byte. The German language Wikipedia for instance has about 13 million links in the main namespace, 8*|E| would need about 1GB even for a naive implementation. With a little more effort, it can be nearly halved to 4*|E|+4*|V|.
I have used the trivial edge store for analyzing the category structure before, and Neil Harris is currently working on an nice standalone implementation of this for Wikimedia Germany. This should allow recursive category lookup in microseconds.
In any case, something needs to change. You can't expect to be frequently using 1/8 of the toolserver's RAM. Even more so since this amount of memory can't be used by MySQL for caching while you are not using it (because of the way the innodb cache pool works).
Regards, Daniel
Vladimir Medeyko schrieb:
Dear colleagues,
I've heard that Golem bot, which is the heart of the connectivity project, stopped to function due to the recent toolserver reconfiguration.
Is it possible to adjust configuration specifically for Golem or to do something else to make it function again?
It is especially a pity that the connectivity project has problems now, just two days after the project was reported at Konferencija Wikimedia Polska and received much of interest from the listeners.
What could be done to fix the situation? Thanks!
-- Медейко Владимир Владимирович НП "Викимедиа РУ" Директор тел. +7-921-940-39-79
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
toolserver-l@lists.wikimedia.org