On Sun, Dec 13, 2009 at 10:30 AM, Nikola Smolenski <smolensk(a)eunet.rs> wrote: But that is a one off expense. That is why we programmers can earn a living, because we can work on many projects. Also we drink coffee while playing UrbanTerror as well. 1. Php is very hard to optimize. 2. The mediawiki has a pretty nonstandard syntax. The best that I have seen is the python implementation of the wikibook parser. But given that each plugin can change the syntax as it will, it will get more complex. 3. Even python is easier to optimize than php. 4. The other questions are, does it make sense to have such a centralized client server architecture? We have been talking about using a distributed vcs for mediawiki. 5. Well, now even if the mediawiki is fully distributed, it will cost CPU, but that will be distributed. Each edit that has to be copied will cause work to be done. In a distributed system even more work in total. 6. Now, I have been wondering anyway who is the benefactor of all these millions spend on bandwidth, where do they go to anyway? What about making a wikipedia network and have the people who want to access it pay instead of having us pay to give it away? With these millions you can buy a lot of routers and cables. 7. Now, back to the optimization. Lets say you were able to optimize the program. We would identify the major cpu burners and optimize them out. That does not solve the problem. Because I would think that the php program is only a small part of the entire issue. The fact that the data is flowing in a certain wasteful way is the cause of the waste, not the program itself. Even if it would be much more efficient and moving around data that is not needed, the data is not needed. This would eventually lead, in an optimal world to updates not even being distributed at all. Not all changes have to be centralized. Lets say that there is one editor who would pull the changes from others and make a public version. That would mean that only they would need to have all data for that one topic. I think that you could optimize the wikipedia along the lines of data travelling only to the people who need it (editors versus viewers) and you would optimize first a way to route edits into special interest groups and create smaller virtual subnetworks of the editors CPUs working together in a peer to peer direct network. So if you have 10 people collaborating on a topic, only the results of that work will be checked into the central server. the decentralized communication would be between fewer parties and reduce the resources used. see also : http://strategy.wikimedia.org/wiki/Proposal:A_MediaWiki_Parser_in_C mike

Let me sum this up, The basic optimization is this : You don't need to transfer that new article in every revision to all users at all times. The central server could just say : this is the last revision that has been released by the editors responsible for it, there are 100 edits in process and you can get involved by going to this page here (hosted on a server someplace else). There is no need to transfer those 100 edits to all the users on the web and they are not interesting to everyone. On Sun, Dec 13, 2009 at 12:10 PM, Domas Mituzas <midom.lists(a)gmail.com> wrote: I hope you are serious here, Lets take a look at what the engine does, it allows editing of text. It renders the text. It serves the text. The wiki from ward cunningham is a perl script of the most basic form. There is not much magic involved. Of course you need search tools, version histories and such. There are places for optimizing all of those processes. It is not lunacy, it is a fact that such work can be done, and is done without a central server in many places. Just look at for example how people edit code in an open source software project using git. It is distributed, and it works. There are already wikis based on git available. There are other peer to peer networks such as TOR or freenet that would be possible to use. If you were to split up the editing of wikipedia articles into a network of git servers across the globe and the rendering and distribution of the resulting data would be the job of the WMF. Now the issue of resolving conflicts is pretty simple in the issue of git, everyone has a copy and can do what they want with it. If you like the version from someone else, you pull it. In terms of wikipedia as having only one viewpoint, the NPOV that is reflected by the current revision at any one point in time, that version would be one pushed from its editors repositories. It is imaginable that you would have one senior editor for each topic who has their own repository of of pages who pull in versions from many people. Please lets be serious here! I am talking about the fact that not all people need all the centralised services at all times. A tracker to manage what server is used for what group of editors can be pretty efficient. Essentially it is a form of DNS. A tracker need only show you the current repositories that are registered for a certain topic. Resolving conflicts is important, but you only need so many people for that. The entire community does not get involved in all the conflicts. There are only a certain number of people that are deeply involved in any one section of the wikipedia at any given time. Imagine that you had, lets say 1000 conference rooms available for discussion and working together spread around the world and the results of those rooms would be fed back into the Wikipedia. These rooms or servers would be for processing the edits and conflicts any given set of pages. My idea is that you don't need to have a huge server to resolve conflicts. many pages don't have many conflicts, there are certain areas which need constant arbitration of course. Even if you split up the groups into different viewpoints where the arbitration team only deals with the output of two teams (pro and contra). Even if you look at the number of editors in a highly contested page, they are not unlimited. editors are collaborating (enhancing each other) and conflicting (overwriting each other). If you split them up into different rooms when they should be collaborating and reduce the conflicts, then you will win alot. Even in Germany, most edits do not show up immediately. They have some person to check the commits. Now that would also mean that those edits before they are commited do not need to go a single data center. People interested in getting all the versions available would need to be able to find them. But for stuff like that people would be prepared to wait a bit longer to collect the data from many servers if needed. You should be able to just pull the versions you want in the depth that you want. That selection of versions and depth would be a large optimization in its self. So there are different ways to reduce the load on a single server and create pockets of processing for different topics. The only really important thing is that people who are working on the same topic are working on the same server or have a path of communication. To sum it up, if conflicts are the major problem in the wikipedia, the major cost in terms of review and coordination, then you should rethink the workflow to push the processing time back to the editor causing the conflict. Right now the revisions are stored in whole, but not in part. If you only add in new information then the you need less storage. That would be one big optimization for the wikipedia to transfer only the changes across the net and not full revisions. For course even a new section could be a conflict if the new text is garbage or in need of editing. If you want to replace a single word or a sentence then lets say would create a conflict branch in one of external conference rooms that would be the host of the page until the work is finished there. The main server would just have a pointer to the workgroup and the load would be pushed away. That also means that any local server would be able to process the data and host the branch until it is pushed back to the main server. OK, well I think this is enough for now. I do ask you to remain serious, and we can have a serious discussion on the topic of optimisation. thanks, mike

W dniu 12.12.2009 22:36, David Gerard pisze: but we need to cool not to heat our servers :) masti DataCenter Manager :)

2009/12/12, Geoffrey Plourde <geo.plrd(a)yahoo.com>om>: I am not sure the laws of thermodynamics (1) would allow to use that heat to cool a building. You would need a cold source like a river to convert heat back into electricity. But it might be more cost efficient to have the water from the river circulate directly into the building, so that your extra heat is still remaining unused. This is why I think it is more difficult to find solutions in a hot country like Florida than in a cold country (as long as you don't question the very existence of heated homes in cold countries, leaving aside the possibility of moving people and their homes from cold to warm countries). (1) http://en.wikipedia.org/wiki/Laws_of_thermodynamics#Second_law

2009/12/13, Andre Engels <andreengels(a)gmail.com>om>: In cold countries, energy can have two lives : a first life making calculations in a computer, or transforming matter (ore into metal, trees into books), and a second life heating homes. But the best is to use no energy at all : see the OLPC project in Afghanistan (A computer with pedals, like the sewing machines of our great-great-great-grand-mothers) (1) (1) http://www.olpcnews.com/countries/afghanistan/updates_from_olpc_afghanistan…

On Sun, Dec 13, 2009 at 1:22 PM, David Gerard <dgerard(a)gmail.com> wrote: And you automatically become an admin after 5MW! Magnus

Aryeh Gregor wrote: While the major program spending that Wikimedia performs should be defined by its mission, I think small spending decisions, relating to day-to-day operations, can be made without recourse to our mission. For instance, the office staff should be able use recycled paper without there being a Board resolution to put it in the mission statement. In terms of the ethics, there's a big difference between inaction on an issue, say poverty in Africa, and taking direct action in order to make things worse. Wikimedia is not paying people to take food from children's mouths, but it is paying people to burn coal for electricity. I don't think we can claim to be mere bystanders. -- Tim Starling

I personally support any initiative that would reduce energy consumption. I wonder though (in the pure sense of the term, i.e. I have no idea) if the biggest consumption of energy for the Wikimedia Foundation isn't actually travel. Cars consume huge amounts of fossil fuels, and don't get me started on airplanes (I do seem to recall reading somewhere though that the next Wikimania aims to have near zero impact, which is a Good ThingTM). Александр Дмитрий Alexandr Dmitri This message and any attachments (the "message") are intended solely for the addressees and is confidential. If you have received this message by mistake, please delete it and immediately inform me by replying to this email address. Any use not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except after formal approval. The internet can not guarantee the integrity of this message. I can not therefore be held liable for the message if it is modified. Ce message et toutes les pièces-jointes (ci-après le « message » ou « courrier email ») sont établis a l'intention exclusive de ses destinataires et sont confidentiels. Si vous recevez ce message par erreur, merci de le détruire et de m'en avertir immédiatement en répondant à cette adresse email. Toute utilisation de ce message non conforme à sa destination, toute diffusion ou toute publication, totale ou partielle, est interdite, sauf autorisation expresse. L'internet ne permettant pas d'assurer l'intégrité de ce message, je décline toute responsabilité au titre de ce message, dans l'hypothèse où il aurait été modifié.

On Mon, Dec 14, 2009 at 8:50 AM, Tim Starling <tstarling(a)wikimedia.org> wrote: If the sums we're discussing are so small that they can be reasonably compared to the difference between using recycled and regular paper, I don't think they're worth spending much time or effort on either way. How much money would we be talking about to offset Wikimania alone? By that logic, Wikimedia is actively supporting war (or whatever other government policy you dislike) by withholding income tax from its employees' paychecks to give to the US government. Sure, it has no real choice about paying taxes; but it has no real choice about using electricity, either. If using electricity makes you personally responsible for funding renewable energy, why doesn't paying taxes make you personally responsible for funding antiwar organizations? Of course, paying taxes funds war in a very direct way. The money goes to the feds and then straight to the military, where a large fraction is immediately spent on guns and bombs, which are possibly used to kill people within a year or two. In contrast, by emitting carbon dioxide, you're contributing to an effect that won't be a big deal for at least a few more decades. And that will probably become no big deal again a few decades after that when everyone's adapted to it. And that won't directly kill anyone in any event, mainly just cause economic harm. And that might not happen anyway if some clever soul comes up with a good enough fossil fuel replacement at any point in the next thirty years. Or if it becomes economical to pump greenhouse gases out of the atmosphere. Or if some cheap scheme is devised to reduce warming some other way, like releasing particles to block sunlight. Or if some unforeseen negative feedback causes warming to not get too bad after all. And of course maybe we've already hit a critical threshold and cutting emissions is pointless by now. Plus you can add the fact that Wikimedia's contribution to the affair isn't likely to be even measurable, especially if the major damage is from catastrophic changes (e.g., ice caps melting) rather than incremental ones. How much money do you owe for increasing mean global temperature by a billionth of a degree fifty years from now? All in all, I'd say Wikimedia has a lot more culpability for people being shot.

2009/12/15 Tim Starling <tstarling(a)wikimedia.org>rg>: I commend to you all John Birmingham's take: http://www.theage.com.au/opinion/blogs/blunt-instrument/the-sky-is-falling-… (huh, so Blunt Instrument runs nationally? Good stuff!) - d.

Might I suggest that we're getting a bit off-track here with these broad debates on climate change issues? I think if we're considering spending $20k/yr on environmental initiatives, then the most effective way for us and the path most in line with Wikimedia's core mission would be to spend that money directly on special efforts to increase high-quality free content about environmental topics on Wikipedia and the other projects. Thanks, Pharos On Tue, Dec 15, 2009 at 11:04 AM, Aryeh Gregor <Simetrical+wikilist(a)gmail.com> wrote:

I strongly encourage those who are interested in this to create a proposal for strategic planning consideration... Http://strategy.Wikimedia.org . The strategic planning initiative is thinking about the wmf's next five years... This type of conversation is very welcome there. ---------------------------- Philippe Beaudette philippe(a)wikimedia.org On Dec 14, 2009, at 12:50 AM, Tim Starling <tstarling(a)wikimedia.org> wrote: