Wikitech-l September 2016

wikitech-l@lists.wikimedia.org

105 participants
111 discussions

WikiDev'17 main topics -- more feedback needed
by Quim Gil 22 Sep '16

22 Sep '16

Hi, we need more voices and perspectives in the definition of the main topics of the Wikimedia developer Summit 2017. Your participation is very important! RobLa went through the suggestions made and has proposed an organization around main topics, and the discussion has started. Please check https://www.mediawiki.org/wiki/WikiDev17/Topic_ideas and the discussion page. Rachel and I have decided to make a soft-launch of the Summit by opening the registration and travel sponsorship requests only, because we cannot keep postponing this step. You should hear news from us tomorrow. We will launch properly opening the call for participation and promoting the Summit widely in Wikimedia channels and beyond hopefully next week, when we have a "good enough" list of main topics. -- Quim Gil Engineering Community Manager @ Wikimedia Foundation http://www.mediawiki.org/wiki/User:Qgil

1 0

The Revision Scoring weekly update
by Aaron Halfaker 22 Sep '16

22 Sep '16

Hey, This is the 22nd weekly update from revision scoring team that we have sent to this mailing list. UI work: - We configured the default threshold for the ORES review tool on Wikidata to be more strict (higher recall, lower precision)[1] - We fixed a display issue on Special:Contributions where the filters would not wrap[2] Increasing model fitness: - We finished demonstrating model fitness gains using hash-vector features[3]. Next, we'll be working to get the hash-vector features implemented in revscoring/ORES[4]. - We implemented a new strategy for training and testing on all data using cross-validation[5]. This will both increase the fitness of the models and make the statistics reported more robust. Maintenance and robustness - We fixed an indexing issues in ores_model that prevented the deployment of updated models[6]. - We did a minor investigation to a short period of degraded service quality on WMF Labs[7] 1. https://phabricator.wikimedia.org/T144784 -- Change default threshold for Wikidata to high 2. https://phabricator.wikimedia.org/T143518 -- Filter on user contribs has nowrap, causing issues 3. https://phabricator.wikimedia.org/T128087 -- [Spike] Investigate HashingVectorizer 4. https://phabricator.wikimedia.org/T145812 -- Implement ~100 most important hash vector features in editquality models 5. https://phabricator.wikimedia.org/T142953 -- Train on all data, Report test statistics on cross-validation 6. https://phabricator.wikimedia.org/T144432 -- oresm_model index should not be unique 7. https://phabricator.wikimedia.org/T145353 -- Investigate short period of ores-web-03 insanity Sincerely, Aaron from the Revision Scoring team

1 0

Re: [Wikitech-l] [Engineering] Long running tasks/scripts now included on [[wikitech:Deployments]]
by Guillaume Lederrey 22 Sep '16

22 Sep '16

I assumed the same, but better be explicit about those assumption :) On Thu, Sep 22, 2016 at 4:27 PM, Alex Monk <amonk(a)wikimedia.org> wrote: > I had been assuming that puppetised crons were not really relevant... > > On 22 September 2016 at 15:19, Guillaume Lederrey <glederrey(a)wikimedia.org> > wrote: >> >> Hello! >> >> Increasing visibility sounds like a great idea! How far do we want to >> go in that direction? In particular, I'm thinking of a few of the >> crons we have for Cirrus. For example, we do have daily crons on >> terbium that re-generate the suggester indices. Those can run for > >> 1h. >> >> My understanding is that those kind of crons should not be considered >> scripts, but standard working parts of the system. Adding them will >> probably generate more noise than useful information. Is this a >> reasonable understanding? >> >> Thanks! >> >> Guillaume >> >> >> >> On Wed, Sep 21, 2016 at 12:29 AM, Greg Grossmeier <greg(a)wikimedia.org> >> wrote: >> > In an effort to reduce surprises and potential mishaps it is now >> > required to include any long running tasks in the deployment >> > calendar[0]. >> > >> > "Long running tasks" include any script that is run on production 'work >> > machines' such as terbium that last for longer than ~1 hour. Think: >> > migration and maintenance scripts. >> > >> > This was discussed and proposed in T144661[1]. >> > >> > Best, >> > >> > Greg >> > >> > [0] https://wikitech.wikimedia.org/wiki/Deployments >> > Relevant diff: >> > https://wikitech.wikimedia.org/w/index.php?diff=850923&oldid=850244 >> > [1] https://phabricator.wikimedia.org/T144661 >> > >> > -- >> > | Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E | >> > | Release Team Manager A18D 1138 8E47 FAC8 1C7D | >> > >> > _______________________________________________ >> > Engineering mailing list >> > Engineering(a)lists.wikimedia.org >> > https://lists.wikimedia.org/mailman/listinfo/engineering >> > >> >> >> >> -- >> Guillaume Lederrey >> Operations Engineer, Discovery >> Wikimedia Foundation >> UTC+2 / CEST >> >> _______________________________________________ >> Wikitech-l mailing list >> Wikitech-l(a)lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/wikitech-l > > > > > -- > Alex Monk > VisualEditor/Editing team > https://wikimediafoundation.org/wiki/User:Krenair_(WMF) > > _______________________________________________ > Engineering mailing list > Engineering(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/engineering > -- Guillaume Lederrey Operations Engineer, Discovery Wikimedia Foundation UTC+2 / CEST

1 0

Re: [Wikitech-l] [Engineering] Long running tasks/scripts now included on [[wikitech:Deployments]]
by Greg Grossmeier 22 Sep '16

22 Sep '16

What Alex said. This is for one off type things. -- Sent from my phone, please excuse brevity.

1 0

Re: [Wikitech-l] [Engineering] Long running tasks/scripts now included on [[wikitech:Deployments]]
by Jaime Crespo 22 Sep '16

22 Sep '16

Let me clarify the reasoning for the idea: We realized that some schema changes (which used to be scheduled like other deployments) no longer take 1 hour (they can take 1 month, running continuously like https://phabricator.wikimedia.org/T139090 , because it affects 3 of our largest tables). Also, they no longer requires read-only mode or affect code in anyway (unless they are a prerequisite). On the other side, a schema change, combined with high read or write load from long-running maintenance jobs, like those of the updateCollation script, or any other (those where just an example), could potentially make lagging a worse problem: a single transaction has to store pending changes during its lifetime, or long-running reads can block and create pileups due to metadata locking. We want to avoid those, which certainly caused infrastructure issues in the past. So, in summary, regular deployments are exclusive from each others. Long-running maintenance work could affect each other. This is a way for me (and others) to have visibility of those potential negative interactions, and make sure we can coordinate: "You are doing work on enwiki? No problem, we will just run this task for commons". "you need to do an emergency data recovery? I will wait to do this other task that can wait". Even if only DBAs use it, it is already useful to not perform incompatible changes at the same time. But it will be even more useful if everybody uses it! On Thu, Sep 22, 2016 at 4:27 PM, Alex Monk <amonk(a)wikimedia.org> wrote: > I had been assuming that puppetised crons were not really relevant... > > On 22 September 2016 at 15:19, Guillaume Lederrey <glederrey(a)wikimedia.org > > wrote: > >> Hello! >> >> Increasing visibility sounds like a great idea! How far do we want to >> go in that direction? In particular, I'm thinking of a few of the >> crons we have for Cirrus. For example, we do have daily crons on >> terbium that re-generate the suggester indices. Those can run for > >> 1h. >> >> My understanding is that those kind of crons should not be considered >> scripts, but standard working parts of the system. Adding them will >> probably generate more noise than useful information. Is this a >> reasonable understanding? >> >> Thanks! >> >> Guillaume >> >> >> >> On Wed, Sep 21, 2016 at 12:29 AM, Greg Grossmeier <greg(a)wikimedia.org> >> wrote: >> > In an effort to reduce surprises and potential mishaps it is now >> > required to include any long running tasks in the deployment >> > calendar[0]. >> > >> > "Long running tasks" include any script that is run on production 'work >> > machines' such as terbium that last for longer than ~1 hour. Think: >> > migration and maintenance scripts. >> > >> > This was discussed and proposed in T144661[1]. >> > >> > Best, >> > >> > Greg >> > >> > [0] https://wikitech.wikimedia.org/wiki/Deployments >> > Relevant diff: >> > https://wikitech.wikimedia.org/w/index.php?diff=850923&oldid=850244 >> > [1] https://phabricator.wikimedia.org/T144661 >> > >> > -- >> > | Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E | >> > | Release Team Manager A18D 1138 8E47 FAC8 1C7D | >> > >> > _______________________________________________ >> > Engineering mailing list >> > Engineering(a)lists.wikimedia.org >> > https://lists.wikimedia.org/mailman/listinfo/engineering >> > >> >> >> >> -- >> Guillaume Lederrey >> Operations Engineer, Discovery >> Wikimedia Foundation >> UTC+2 / CEST >> >> _______________________________________________ >> Wikitech-l mailing list >> Wikitech-l(a)lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/wikitech-l >> > > > > -- > Alex Monk > VisualEditor/Editing team > https://wikimediafoundation.org/wiki/User:Krenair_(WMF) > > _______________________________________________ > Engineering mailing list > Engineering(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/engineering > > -- Jaime Crespo <http://wikimedia.org>

1 0

Long running tasks/scripts now included on [[wikitech:Deployments]]
by Greg Grossmeier 22 Sep '16

22 Sep '16

In an effort to reduce surprises and potential mishaps it is now required to include any long running tasks in the deployment calendar[0]. "Long running tasks" include any script that is run on production 'work machines' such as terbium that last for longer than ~1 hour. Think: migration and maintenance scripts. This was discussed and proposed in T144661[1]. Best, Greg [0] https://wikitech.wikimedia.org/wiki/Deployments Relevant diff: https://wikitech.wikimedia.org/w/index.php?diff=850923&oldid=850244 [1] https://phabricator.wikimedia.org/T144661 -- | Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E | | Release Team Manager A18D 1138 8E47 FAC8 1C7D |

2 2

REMINDER: No deploys next week (week of Sept 26th)
by Greg Grossmeier 22 Sep '16

22 Sep '16

Due to the Wikimedia Technical Operations Team having their team offsite that week and generally being less than normally available there will be no non-emergency deploys the week of September 26th (aka: next week). See also: https://wikitech.wikimedia.org/wiki/Deployments#Week_of_September_26th Normal schedule will resume the following week. Teaser: The week of October 17th will be a "no train deploys" weeks as the Wikimedia Release Engineering Team will be at their offsite that week. I'll send a reminder about this as well closer to that date. Best, Greg -- | Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E | | Release Team Manager A18D 1138 8E47 FAC8 1C7D |

2 1

ORES article quality data as a database table
by Amir Ladsgroup 22 Sep '16

22 Sep '16

One of ORES [1] applications is determining article quality. For example, What would be the best assessment of an article in the given revision. Users in wikiprojects use ORES data to check if articles need re-assessment. e.g. if an article is in "Start" level and now good it's enough to be a "B" article. As part of Q4 goals, we made a dataset of article quality scores of all articles in English Wikipedia [2] (Here's the link to download the dataset [3]) and we are publishing it in figshare as something you can cite [4] also we are working on publishing monthly data for researchers to track article quality data change over time. [5] As a pet project of mine, I always wanted to put these data in a database. So we can query the database and get much more useful data. For example quality of articles in category 'History_of_Essex' [6] [7]. The weighed sum is a measure of quality which is a decimal number between 0 (really stub) to 5 (a definitely featured article). We have also prediction column which is a number in this map [8] for example if prediction is 5, it means ORES thinks it should be a featured article. I leave more use cases to your imagination :) I'm looking for a more permanent place to put these data, please tell me if it's useful for you. [1] ORES is not a anti-vandalism tool, it's an infrastructure to use AI in Wikipedia. [2] https://phabricator.wikimedia.org/T135684 [3] (117 MBs) https://datasets.wikimedia.org/public-datasets/enwiki/article_quality/wp10-… [4] https://phabricator.wikimedia.org/T145332 [5] https://phabricator.wikimedia.org/T145655 [6] https://quarry.wmflabs.org/query/12647 [7] https://quarry.wmflabs.org/query/12662 [8] https://github.com/wiki-ai/wikiclass/blob/3ff2f6c44c52905c7202515c5c8b525fb… Have fun! Amir

2 1

MediaWiki-Codesniffer 0.8.0-alpha.1 released
by Legoktm 22 Sep '16

22 Sep '16

Hi, As the subject line hints, we're doing something a little different with this release. There was a lot of work done on MW-CS over the summer by our GSoC student, Lethexie. There are a significant number of changes, especially to do with documentation comments. So we are releasing this as alpha quality to ask for more feedback on the changes to the ruleset. You can upgrade in the same manner as normal - by updating the version number in the composer.json file. Feel free to provide feedback by responding to this thread or by filing a bug in the MediaWiki-Codesniffer Phabricator project. We would like to release 0.8.0 by mid-October. Thanks! -- Legoktm

1 0

Upgrade of 1.28.0-wmf.19 to group 1 is on hold
by Antoine Musso 21 Sep '16

21 Sep '16

Hello, We are holding the deployment of MediaWiki version 1.28.0-wmf.19 due to a couple of bugs that have surfaced. The first one is that renaming a user was blocked [T145596]. Reported by K6ka, triaged by MarcoAurelio. The issue is fixed now thanks to Aaron Schulz and Kunal Mehta. The second blocker is way nastier. I have pushed the upgrade to group1. matanya (a long volunteer with a lot of technical patches) immediately reported the infoboxes on the Hebrew Wiki were on the wrong side, which prompted a rollback. [T145673]. Timo Tijhof and Kunal Mehta have found the root cause. I can speak for them as to who/when we will get a solution. For now. I am holding the train. Will reassess tomorrow and ideally push group1 at 19:00 UTC then follow with group2 at 20:00UTC. Stay tuned! Up-to-date status: https://tools.wmflabs.org/versions/ MW-1.28.0-wmf.19 deployment blockers https://phabricator.wikimedia.org/T143328 [T145596] Renames getting stuck on mediawiki.org (Sept 13, 2016) https://phabricator.wikimedia.org/T145596 [T145673] 1.28.0-wmf.19 broke template RTL placement https://phabricator.wikimedia.org/T145673 -- Antoine "hashar" Musso

4 7

← Newer
1
2
3
4
5
6
7
8
...
12
Older →

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Wikitech-l September 2016