Date: Mar 17, 2006 10:24 AM Subject: [Wikitech-l] Distributed process pool To: wikitech-l@wikimedia.org
Hello,
Is there a way analyze the wikipedia logs to figure out what processes takes the most time? There is no immediate need, but I wanted to shoot off an idea to consider. If we were able to capture the processes that do take a heavy server load and push them onto a distributed process pool, would it help wikipedia or mediawiki in general? I'd imagine there would be a difference with process time over the speed of network traffic. Let say we determined that the code that creates a diff for two pages is a hog and can be put into the pool. We could use something like BOINC, http://boinc.berkeley.edu/, to standardize the pool. We can add the diff process to the pool as the server load gets heavy. The use of BOINC is more specific to research tasks, and it would need to be different for mediawiki. I just used the idea to keep this message short to get your feedback.
Jonathan,
Jared and I are actually working on a project to research and build a peer-based distributed hosting framework for large free-content sites like Wiki{m,p}edia. In a practical sense, we're using Wikipedia as a starting point for the inquiry.
If we get accurate statistics on what kinds of processes these are and what quantity of resources they consume, we might be able to integrate these considerations into the simulation environment we'll be using to evaluate various architechtures for distributed hosting.
I'm finally subscribing to the wikitech list, and I'll be interested in learning about anything related that doesn't come up there.
-Erik
Hello,
If we get accurate statistics on what kinds of processes these are and what quantity of resources they consume, we might be able to integrate these considerations into the simulation environment we'll be using to evaluate various architechtures for distributed hosting.
We're doing some of this action for links processing, where delays can be tolerated, but that's more for serializing pagelinks load. Though nature of wikis does not allow lags for most operations - if edits were async (I'd love if they could be..), multiple people would be editing same pages and of course, getting conflicts. Async communication for conflict resolution would be somewhat difficult. And other actions, such as diffs, views, whatever, all need to be synchronized. Of course, distributed caching helps a lot, and our aim should be better efficiency of it, but still, we've got to be in sync.
As for distribution, I'd like to see some of application synchronization primitives (like message cache state, db server lag, etc) being pushed to app servers rather than pulled from db/memcache. But that's already fighting miliseconds :)
As for architecture, there're not that many low hanging fruits at the moment that would be easy to solve by distribution. If anyone sees it otherwise, please tell :)
Domas
wikitech-l@lists.wikimedia.org