I was talking with Tim Starling about the automation of the wiki creation, renaming (locking, removing). In theory, this should be the first and the most important step.
I am in favor of giving the button to stewards, as LangCom should stay clearly a non-executive body in relation to the Wikimedia web projects. However, it shouldn't be a big deal if we have the button, as well.
I would like to know what do you think, as well as what do stewards think (so, MF-Warburg, may you talk with stewards about that and give us their opinion). After we articulate our own opinions, this should be put on Meta for wider community discussion and decision.
At this moment we don't have ETA for the application, but I suppose something between the next couple of months and the end of the year should be reasonable to expect, as Tim has taken that to do.
---------- Forwarded message ---------- From: tstarling no-reply@phabricator.wikimedia.org Date: Wed, Feb 22, 2017 at 4:51 AM Subject: [Maniphest] [Created] T158730: Automate WMF wiki creation To: millosh@gmail.com
tstarling created this task. tstarling added projects: Services, Release-Engineering-Team, MediaWiki-Configuration. Herald added a subscriber: Aklapper. *TASK DESCRIPTION*
Wiki creation is quite an involved process, documented on wikitech https://wikitech.wikimedia.org/wiki/Add_a_wiki. I think, at least for certain common cases, the task could be almost completely automated.
For uncomplicated creation of new language editions under existing projects, with default configuration, the following tasks need to be done, none of which require complex human decision-making:
- Reconfigure many services by pushing configuration changes to Gerrit, and deploy those commits - mediawiki-config: wikiversions, *.dblist - WikimediaMessages - DNS - RESTBase - Parsoid - Analytics refinery - cxserver - Labs dnsrecursor - Run addWiki.php. This script aims to automate all tasks which can be executed with the privileges of a MW maintenance script. - Run Wikidata's populateSitesTable.php. It should probably be incorporated into addWiki.php. - Run labsdb maintain-views - Update wikistats labs
So at a minimum, you need to write and deploy commits to 8 different projects, run three scripts, and manually insert some rows into a DB in a labs instance.
Despite there being no human decision making in this process, the documentation requires that you involve people from approximately four different teams (services, ops, wikidata, analytics).
In my opinion, something is going wrong here in terms of development policy. The problem is getting progressively worse. In July 2004, I fully automated wiki creation and provided a web interface allowing people to create wikis. Now, it is unthinkable.
Obviously services are the main culprits. Is it possible for in-house services to follow pybal's example, by polling a central HTTP configuration service for their wiki lists? As with pybal, the service could just be a collection of static files on a webserver. Even MediaWiki could profitably use such a central service for its dblists, with APC caching.
So let's suppose we could get the procedure down to:
1. Commit/review/deploy the DNS update 2. Commit/review/deploy a configuration change to the new central config service. 3. Run addWiki.php
Labs instances needing to know about the change would either poll the config service, or be notified by addWiki.php. WikimediaMessages could be updated in advance via translatewiki.net.
(Thanks to Milos Rancic for raising this issue with me.)
*TASK DETAIL* https://phabricator.wikimedia.org/T158730
*EMAIL PREFERENCES* https://phabricator.wikimedia.org/settings/panel/emailpreferences/
*To: *tstarling *Cc: *mobrovac, Reedy, demon, millosh, tstarling, Aklapper, Liudvikas, Luke081515, Eevans, Hardikj, zeljkofilipin, Jay8g, Krenair, Legoktm, greg
To be honest, I don't know what you're talking about. Deploy commits? Run scripts? Pybal? If you think that it's not a big deal for LangCom to "have that button" ... I don't know.
Fwiw, Oliver
On 22-Feb-17 07:48, Milos Rancic wrote:
I was talking with Tim Starling about the automation of the wiki creation, renaming (locking, removing). In theory, this should be the first and the most important step.
I am in favor of giving the button to stewards, as LangCom should stay clearly a non-executive body in relation to the Wikimedia web projects. However, it shouldn't be a big deal if we have the button, as well.
I would like to know what do you think, as well as what do stewards think (so, MF-Warburg, may you talk with stewards about that and give us their opinion). After we articulate our own opinions, this should be put on Meta for wider community discussion and decision.
At this moment we don't have ETA for the application, but I suppose something between the next couple of months and the end of the year should be reasonable to expect, as Tim has taken that to do.
---------- Forwarded message ---------- From: *tstarling* <no-reply@phabricator.wikimedia.org mailto:no-reply@phabricator.wikimedia.org> Date: Wed, Feb 22, 2017 at 4:51 AM Subject: [Maniphest] [Created] T158730: Automate WMF wiki creation To: millosh@gmail.com mailto:millosh@gmail.com
tstarling created this task. tstarling added projects: Services, Release-Engineering-Team, MediaWiki-Configuration. Herald added a subscriber: Aklapper.
*TASK DESCRIPTION*
Wiki creation is quite an involved process, documented on wikitech https://wikitech.wikimedia.org/wiki/Add_a_wiki. I think, at least for certain common cases, the task could be almost completely automated.
For uncomplicated creation of new language editions under existing projects, with default configuration, the following tasks need to be done, none of which require complex human decision-making:
- Reconfigure many services by pushing configuration changes to Gerrit, and deploy those commits o mediawiki-config: wikiversions, *.dblist o WikimediaMessages o DNS o RESTBase o Parsoid o Analytics refinery o cxserver o Labs dnsrecursor
- Run addWiki.php. This script aims to automate all tasks which can be executed with the privileges of a MW maintenance script.
- Run Wikidata's populateSitesTable.php. It should probably be incorporated into addWiki.php.
- Run labsdb maintain-views
- Update wikistats labs
So at a minimum, you need to write and deploy commits to 8 different projects, run three scripts, and manually insert some rows into a DB in a labs instance.
Despite there being no human decision making in this process, the documentation requires that you involve people from approximately four different teams (services, ops, wikidata, analytics).
In my opinion, something is going wrong here in terms of development policy. The problem is getting progressively worse. In July 2004, I fully automated wiki creation and provided a web interface allowing people to create wikis. Now, it is unthinkable.
Obviously services are the main culprits. Is it possible for in-house services to follow pybal's example, by polling a central HTTP configuration service for their wiki lists? As with pybal, the service could just be a collection of static files on a webserver. Even MediaWiki could profitably use such a central service for its dblists, with APC caching.
So let's suppose we could get the procedure down to:
- Commit/review/deploy the DNS update
- Commit/review/deploy a configuration change to the new central config service.
- Run addWiki.php
Labs instances needing to know about the change would either poll the config service, or be notified by addWiki.php. WikimediaMessages could be updated in advance via translatewiki.net http://translatewiki.net.
(Thanks to Milos Rancic for raising this issue with me.)
*TASK DETAIL* https://phabricator.wikimedia.org/T158730 https://phabricator.wikimedia.org/T158730
*EMAIL PREFERENCES* https://phabricator.wikimedia.org/settings/panel/emailpreferences/ https://phabricator.wikimedia.org/settings/panel/emailpreferences/
*To: *tstarling *Cc: *mobrovac, Reedy, demon, millosh, tstarling, Aklapper, Liudvikas, Luke081515, Eevans, Hardikj, zeljkofilipin, Jay8g, Krenair, Legoktm, greg
Langcom mailing list Langcom@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/langcom
Basically, currently the process for creating a new wiki involves logging in to a server, running a lot of commands, and changin. It takes at least several days and can be done by only a handful of people who have the permissions and the required knowledge. (I recently heard a brilliant talk by WMF engineer Yuvi Panda about this: https://www.youtube.com/watch?v=rEyqVXzCJJc ; he calls this "command line tax".)
Because it's so manual, it's pretty error-prone—despite the best intentions, and the people's professionalism, I often find things to fix after the wiki is already created. So even reducing the number of manual steps that the tech operations people have to run would already an improvement, because it will save time (often it's volunteers' time), and reduce mistakes.
Making it so easy to do that it doesn't require any coding or running commands, but just going to a page like meta.wikipedia.org/wiki/Special:CreateWiki (don't click it, I just made it up), typing in the language code and pushing a button would be better yet, but it's totally OK if it doesn't happen next week. Who would have the permission to actually run it? LangCom members, stewards, somebody else? I don't care very much to be honest, as long as it's actually done with as few bugs as possible, and no silly unnecessary projects are created. Miloš is better at political thought than me :)
-- Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי http://aharoni.wordpress.com “We're living in pieces, I want to live in peace.” – T. Moore
2017-02-22 10:48 GMT+02:00 Oliver Stegen oliver_stegen@sil.org:
To be honest, I don't know what you're talking about. Deploy commits? Run scripts? Pybal? If you think that it's not a big deal for LangCom to "have that button" ... I don't know.
Fwiw, Oliver
On 22-Feb-17 07:48, Milos Rancic wrote:
I was talking with Tim Starling about the automation of the wiki creation, renaming (locking, removing). In theory, this should be the first and the most important step.
I am in favor of giving the button to stewards, as LangCom should stay clearly a non-executive body in relation to the Wikimedia web projects. However, it shouldn't be a big deal if we have the button, as well.
I would like to know what do you think, as well as what do stewards think (so, MF-Warburg, may you talk with stewards about that and give us their opinion). After we articulate our own opinions, this should be put on Meta for wider community discussion and decision.
At this moment we don't have ETA for the application, but I suppose something between the next couple of months and the end of the year should be reasonable to expect, as Tim has taken that to do.
---------- Forwarded message ---------- From: tstarling no-reply@phabricator.wikimedia.org Date: Wed, Feb 22, 2017 at 4:51 AM Subject: [Maniphest] [Created] T158730: Automate WMF wiki creation To: millosh@gmail.com
tstarling created this task. tstarling added projects: Services, Release-Engineering-Team, MediaWiki-Configuration. Herald added a subscriber: Aklapper. *TASK DESCRIPTION*
Wiki creation is quite an involved process, documented on wikitech https://wikitech.wikimedia.org/wiki/Add_a_wiki. I think, at least for certain common cases, the task could be almost completely automated.
For uncomplicated creation of new language editions under existing projects, with default configuration, the following tasks need to be done, none of which require complex human decision-making:
- Reconfigure many services by pushing configuration changes to
Gerrit, and deploy those commits - mediawiki-config: wikiversions, *.dblist - WikimediaMessages - DNS - RESTBase - Parsoid - Analytics refinery - cxserver - Labs dnsrecursor
- Run addWiki.php. This script aims to automate all tasks which can be
executed with the privileges of a MW maintenance script.
- Run Wikidata's populateSitesTable.php. It should probably be
incorporated into addWiki.php.
- Run labsdb maintain-views
- Update wikistats labs
So at a minimum, you need to write and deploy commits to 8 different projects, run three scripts, and manually insert some rows into a DB in a labs instance.
Despite there being no human decision making in this process, the documentation requires that you involve people from approximately four different teams (services, ops, wikidata, analytics).
In my opinion, something is going wrong here in terms of development policy. The problem is getting progressively worse. In July 2004, I fully automated wiki creation and provided a web interface allowing people to create wikis. Now, it is unthinkable.
Obviously services are the main culprits. Is it possible for in-house services to follow pybal's example, by polling a central HTTP configuration service for their wiki lists? As with pybal, the service could just be a collection of static files on a webserver. Even MediaWiki could profitably use such a central service for its dblists, with APC caching.
So let's suppose we could get the procedure down to:
- Commit/review/deploy the DNS update
- Commit/review/deploy a configuration change to the new central
config service. 3. Run addWiki.php
Labs instances needing to know about the change would either poll the config service, or be notified by addWiki.php. WikimediaMessages could be updated in advance via translatewiki.net.
(Thanks to Milos Rancic for raising this issue with me.)
*TASK DETAIL* https://phabricator.wikimedia.org/T158730
*EMAIL PREFERENCES* https://phabricator.wikimedia.org/settings/panel/emailpreferences/
*To: *tstarling *Cc: *mobrovac, Reedy, demon, millosh, tstarling, Aklapper, Liudvikas, Luke081515, Eevans, Hardikj, zeljkofilipin, Jay8g, Krenair, Legoktm, greg
Langcom mailing listLangcom@lists.wikimedia.orghttps://lists.wikimedia.org/mailman/listinfo/langcom
Langcom mailing list Langcom@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/langcom
According to someone who is more closely involved with wiki creation than me, having a web interface for stewards would be pretty scary. Wiki creation breaks often enough that you want to have someone competent on hand, with shell access, to deal with any problems.
If we had a decent semi-automated system in place for a while, like the one I describe, and no problems occur, then that would build confidence for granting access to a wider user group. But for now, it will require shell access.
I think part of the problem is that wiki creations are not done often enough. If we created a test wiki once a week, with every release cycle, then we would know at the time if a software deployment caused wiki creation to break.
Note that the backlog is quite small, only two wikis are awaiting creation at the moment (another two tickets are stalled for good reasons).
As I said in my private email, renames are much more complicated. The task I wrote today would help with renames, in that it provides a way to notify services when wikis are renamed, but there are a lot of other known issues with renames to fix.
I'm not sure of the ETA at the moment. I can't personally implement the whole thing, since it touches a lot of services that have different owners.
-- Tim Starling
On 22/02/17 17:48, Milos Rancic wrote:
I was talking with Tim Starling about the automation of the wiki creation, renaming (locking, removing). In theory, this should be the first and the most important step.
I am in favor of giving the button to stewards, as LangCom should stay clearly a non-executive body in relation to the Wikimedia web projects. However, it shouldn't be a big deal if we have the button, as well.
I would like to know what do you think, as well as what do stewards think (so, MF-Warburg, may you talk with stewards about that and give us their opinion). After we articulate our own opinions, this should be put on Meta for wider community discussion and decision.
At this moment we don't have ETA for the application, but I suppose something between the next couple of months and the end of the year should be reasonable to expect, as Tim has taken that to do.
---------- Forwarded message ---------- From: *tstarling* <no-reply@phabricator.wikimedia.org mailto:no-reply@phabricator.wikimedia.org> Date: Wed, Feb 22, 2017 at 4:51 AM Subject: [Maniphest] [Created] T158730: Automate WMF wiki creation To: millosh@gmail.com mailto:millosh@gmail.com
tstarling created this task. tstarling added projects: Services, Release-Engineering-Team, MediaWiki-Configuration. Herald added a subscriber: Aklapper.
*TASK DESCRIPTION*
Wiki creation is quite an involved process, documented on wikitech https://wikitech.wikimedia.org/wiki/Add_a_wiki. I think, at least for certain common cases, the task could be almost completely automated.
For uncomplicated creation of new language editions under existing projects, with default configuration, the following tasks need to be done, none of which require complex human decision-making:
- Reconfigure many services by pushing configuration changes to Gerrit, and deploy those commits o mediawiki-config: wikiversions, *.dblist o WikimediaMessages o DNS o RESTBase o Parsoid o Analytics refinery o cxserver o Labs dnsrecursor
- Run addWiki.php. This script aims to automate all tasks which can be executed with the privileges of a MW maintenance script.
- Run Wikidata's populateSitesTable.php. It should probably be incorporated into addWiki.php.
- Run labsdb maintain-views
- Update wikistats labs
So at a minimum, you need to write and deploy commits to 8 different projects, run three scripts, and manually insert some rows into a DB in a labs instance.
Despite there being no human decision making in this process, the documentation requires that you involve people from approximately four different teams (services, ops, wikidata, analytics).
In my opinion, something is going wrong here in terms of development policy. The problem is getting progressively worse. In July 2004, I fully automated wiki creation and provided a web interface allowing people to create wikis. Now, it is unthinkable.
Obviously services are the main culprits. Is it possible for in-house services to follow pybal's example, by polling a central HTTP configuration service for their wiki lists? As with pybal, the service could just be a collection of static files on a webserver. Even MediaWiki could profitably use such a central service for its dblists, with APC caching.
So let's suppose we could get the procedure down to:
- Commit/review/deploy the DNS update
- Commit/review/deploy a configuration change to the new central config service.
- Run addWiki.php
Labs instances needing to know about the change would either poll the config service, or be notified by addWiki.php. WikimediaMessages could be updated in advance via translatewiki.net http://translatewiki.net.
(Thanks to Milos Rancic for raising this issue with me.)
*TASK DETAIL* https://phabricator.wikimedia.org/T158730 https://phabricator.wikimedia.org/T158730
*EMAIL PREFERENCES* https://phabricator.wikimedia.org/settings/panel/emailpreferences/ https://phabricator.wikimedia.org/settings/panel/emailpreferences/
*To: *tstarling *Cc: *mobrovac, Reedy, demon, millosh, tstarling, Aklapper, Liudvikas, Luke081515, Eevans, Hardikj, zeljkofilipin, Jay8g, Krenair, Legoktm, greg
-- Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי http://aharoni.wordpress.com “We're living in pieces, I want to live in peace.” – T. Moore
2017-02-22 11:21 GMT+02:00 Tim Starling tstarling@wikimedia.org:
According to someone who is more closely involved with wiki creation than me, having a web interface for stewards would be pretty scary. Wiki creation breaks often enough that you want to have someone competent on hand, with shell access, to deal with any problems.
Yeah, that's OK. No rush. A better system for people with shell access would be an excellent first step.
As I said in my private email, renames are much more complicated. The task I wrote today would help with renames, in that it provides a way to notify services when wikis are renamed, but there are a lot of other known issues with renames to fix.
Yes... A rename was done once for be-x-old -> be-tarask, and there are still unresolved issues that came up as a result of that, like https://phabricator.wikimedia.org/T112426 and the issues around it. These issues block further renames. I ping relevant people occasionally to check whether it's possible to progress with renaming simple.wikipedia.org to en-simple.wikipedia.org , als to gsw, nrm to nrf, etc., but there's no progress yet.