Wikitech-l August 2017

wikitech-l@lists.wikimedia.org

63 participants
72 discussions

2017-08-16 Scrum of Scrums meeting notes
by Grace Gellerman 16 Aug '17

16 Aug '17

https://www.mediawiki.org/wiki/Scrum_of_scrums/2017-08-16 = 2017-08-16 = contact: https://www.mediawiki.org/wiki/Wikimedia_Engineering == Callouts == * Releng: Trebuchet's days are numbered. If you care about these projects please get in touch with #RelEng ** These are the last remaining trebuchet projects: *** iegreview *** scholarships *** ocg *** dropwizard * Wikidata: * For weeks we see thousands of hard to track user-facing Lua errors, see https://phabricator.wikimedia.org/T170039. We believe https://phabricator.wikimedia.org/T171166 might fix it, but need help by devs willing to touch LuaSandbox build. == Wikidata == * Callout: For weeks we see thousands of hard to track user-facing Lua errors, see https://phabricator.wikimedia.org/T170039. We believe https://phabricator.wikimedia.org/T171166 might fix it, but need help by devs willing to touch LuaSandbox build. === Readers === ==== Web ==== * Completing roll out of Page previews to all projects but enwiki and dewiki (likely to be stalled until post Wikimania) * RelatedArticles now renders articles in footer rather than side bar. ( https://phabricator.wikimedia.org/T165991) ==== Reading Infrastructure ==== * Blocked by: * Blocking: * Updates: ** MCS gets geo coordinates from action API instead of Parsoid. ** Got rid of action=mobilview API calls for regular pages. === Reading Discovery ==== * Updates ** Working on search thumbnail AB test https://phabricator.wikimedia.org/T149811 === Contributors === ==== Collaboration ==== * Most of our team was at Wikimania. * However, we continued work on RC Filters and Echo * Echo: Merged a "Mark all as read" patch from community member and worked on additional patches (e.g. edit-user notification) * RCFilters: Small fixes * Flow: In-progress work on a feature to allow you to add inline Flow discussion linked to specific text. == Technology == === Analytics === *2 team members in wikimania, 2 on family leave, 1 of our team members coming back next week from parental leave. Not a lot happened. *We figured out issue with timeouts in varnishkafka, there was one host running too new of a version of kafka API *Still working on purging eventlogging data, moving to cluster some data out of MySQL (will make DBAs happy). Couple tables that are about 0.5 tb. *Ordering new hosts to replace eventlogging MYSQL hosts using some budget we will not need for the new Kafka cluster. *Edit History Reconstruction, made a snapshoot custom for Neil (editing data analysts) to calculate metrics on. *Wikimetrics had stop working due to some recent changes to labs db connection limits, fixed that. === Services === * Blockers: none * Updates: ** Wiped out Parsoid HTML cache in preparation for Cassandra 3 upgrade *** Elevated VE latencies for a while while new storage is populated === Technical Operations === * '''Blocked''': ** by noone * '''Blocking''': ** none * '''Updates''': ** Wikimania + Debconf week, not much happened. === Community Tech === *Updates: CodeMirror and LoginNotify are still coming! === Fundraising Tech === * looking into db replication lag - it'll get up to like 5000 sec behind, but restarting mysql on the replicated copy will make it catch up in less than a minute * continuing new API work with main credit card processor * updating our interrupted-donation-resolver to work generically so we can use it for PayPal express checkout * refinements to CiviCRM<->bulk mailer data flows * CentralNotice clone campaign feature almost ready for CR === Scoring platform === *Blocked **Feeling the pain of no git-fat in labs deployments (our git repos are huge!) ***https://phabricator.wikimedia.org/T171758 ***Releng, can this be prioritized? *Updates **New ORES model information available at https://ores-misc.wmflabs.org. Demo coming soon. ***Mostly an FYI for Collab Team === Release Engineering === * Last projects moving to scap3 **These have patches but deployment window scheduled: *** [https://phabricator.wikimedia.org/T129154 iegreview] *** [https://phabricator.wikimedia.org/T129134 scholarships] ** No idea about: *** [https://phabricator.wikimedia.org/T129142 ocg] *** [https://phabricator.wikimedia.org/T170839 dropwizard metrics(???)] - Speak up now or forever hold your peace (we'll delete it)

1 0

MW Function for parsing and modifying template arguments on a page
by Daren Welsh 15 Aug '17

15 Aug '17

I'm working on a server-side script that will periodically revise wiki pages that all use the same template. Is there a function I can use that will parse a page and then find and replace template arguments with values I determine within my script? For example, let's say I have Template:Beer like the following: {{Beer |Name= |Brewery= |Style= |ABV= |IBU= }} So if I had a page Rocket Fuel: {{Beer |Name=Rocket Fuel |Brewery=8th Wonder |Style=Vietnamese Coffee Porter |ABV=4.6 |IBU=18 }} Let's say my script "discovered" an error in one of the arguments and needed to update the page. For example, let's say the IBU was actually 28. Is there a function that will find and replace the value for IBU in the page? I'm currently using regex, but I'm wondering if there's a more trustworthy method. Thanks, Daren

4 5

Timeless skin now available on mw.org
by Isarra Yos 15 Aug '17

15 Aug '17

The Timeless skin has been deployed to test, test2, and mw.org. Anyone interested, please go ahead an enable it and find everything wrong and swear at me about it. Note that the current design is not the final design and I haven't quite worked out what the final design should be, either, so feel free to swear at me about that, too, if you have any ideas/whatever. * Bad documentation: https://www.mediawiki.org/wiki/Skin:Timeless * Bugs: https://phabricator.wikimedia.org/tag/timeless/ * To enable it: https://www.mediawiki.org/wiki/Special:Preferences#mw-prefsection-rendering If all goes well, we will move on to deploy it to the pilot wikis, as listed on the deployment task, sometime next week: https://phabricator.wikimedia.org/T154371 A huge thank you to all the insane people who have worked to get us to this point, when the entire notion of a new skin seemed something of an impossibility most of the way through. -I

12 16

Wikiscan statistics tool for Wikimedia projects
by Pine W 14 Aug '17

14 Aug '17

Wikiscan is an interesting tool for statistics fans. I suggest briefly reading this IEG page <https://meta.wikimedia.org/wiki/Grants:IEG/Wikiscan_multi-wiki>, then playing with the tool on https://wikiscan.org/ Pine

4 3

MediaWiki-GoatSniffer 0.11.1 released
by Legoktm 13 Aug '17

13 Aug '17

Hello! Straight from Wikimania, MediaWiki-GoatSniffer 0.11.1 is now available for use in your MediaWiki extensions and other projects. This release fixes bugs from 0.10.0 as well as some new features: * Add GoatSniffer ASCII art (Kunal Mehta) * Added OpeningKeywordBraceSniff (Umherirrender) * Add sniff to forbid PHP 7 scalar type hints (Kunal Mehta) * Enable Squiz.WhiteSpace.OperatorSpacing (Umherirrender) * Enforce "short" type definitions on @param in comments (Umherirrender) * Fix phpunit test on windows (Umherirrender) * Fix Undefined offset in FunctionCommentSniff (Umherirrender) I already updated most extensions to 0.11.0 a few days ago, it's up to each extension maintainer if they'd also like to update to 0.11.1. Thanks, -- Legoktm

2 1

Re: [Wikitech-l] Timeless skin now available on mw.org
by יגאל חיטרון 12 Aug '17

12 Aug '17

I see. Thank you. But I don't think it's a cache problem, because I never opened this page before. Igal On Aug 12, 2017 22:53, "bawolff" <bawolff+wn(a)gmail.com> wrote: This is kind of odd, but probably not an issue with the skin. I suspect you have some of the MediaWiki pages cached in browser cache from before you used the skin, and as a result are using them via a 304 response. But the user_touched timestamp should prevent that. Maybe central auth doesn't update the local one or something. need to look into that further. Maybe you got logged out and then central auth globally logged back in. I don't know, kind of grasping here, but definitely is a MW issue not a Timeless issue. -- bawolff On Sat, Aug 12, 2017 at 7:38 PM, יגאל חיטרון <khitron(a)gmail.com> wrote: > Hi. I did not expected to much, because I did not like any non vector > skins, but this is awesome! Thank you very much. One little bug - when > working in mediawiki and navigating to another page, sometimes it opens > again in vector and you should reopt it in in preferences. > Igal (User:IKhitron) > > On Aug 12, 2017 17:51, "Alangi Derick" <alangiderick(a)gmail.com> wrote: > >> Wow, this is wonderful. >> >> I like this skin and I already switched to it. A lot of stylish minded >> developers (like me) will start moving to the Timeless skin soon. Good work >> Team >> >> Nevertheless, will file in any bugs found :) >> >> *Kind regards* >> *Alangi Derick N* >> >> *[image: https://twitter.com/AlangiDerick] >> <https://twitter.com/AlangiDerick> >> <https://www.facebook.com/derick.alangi> * >> >> On Sat, Aug 12, 2017 at 3:46 PM, Daniel Kinzler < >> daniel.kinzler(a)wikimedia.de >> > wrote: >> >> > Wow, this looks great! >> > >> > Am 11.08.2017 um 18:26 schrieb Isarra Yos: >> > > The Timeless skin has been deployed to test, test2, and mw.org. Anyone >> > > interested, please go ahead an enable it and find everything wrong and >> > swear at >> > > me about it. >> > > >> > > Note that the current design is not the final design and I haven't >> quite >> > worked >> > > out what the final design should be, either, so feel free to swear at >> me >> > about >> > > that, too, if you have any ideas/whatever. >> > > >> > > * Bad documentation: https://www.mediawiki.org/wiki/Skin:Timeless >> > > * Bugs: https://phabricator.wikimedia.org/tag/timeless/ >> > > * To enable it: >> > > https://www.mediawiki.org/wiki/Special:Preferences#mw- >> > prefsection-rendering >> > > >> > > If all goes well, we will move on to deploy it to the pilot wikis, as >> > listed on >> > > the deployment task, sometime next week: https://phabricator.wikimedia >> . >> > org/T154371 >> > > >> > > A huge thank you to all the insane people who have worked to get us to >> > this >> > > point, when the entire notion of a new skin seemed something of an >> > impossibility >> > > most of the way through. >> > > >> > > -I >> > > >> > > >> > > _______________________________________________ >> > > Wikitech-l mailing list >> > > Wikitech-l(a)lists.wikimedia.org >> > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l >> > >> > >> > -- >> > Daniel Kinzler >> > Principal Platform Engineer >> > >> > Wikimedia Deutschland >> > Gesellschaft zur Förderung Freien Wissens e.V. >> > >> > _______________________________________________ >> > Wikitech-l mailing list >> > Wikitech-l(a)lists.wikimedia.org >> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l >> _______________________________________________ >> Wikitech-l mailing list >> Wikitech-l(a)lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/wikitech-l > _______________________________________________ > Wikitech-l mailing list > Wikitech-l(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikitech-l _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

1 0

Fwd: [Offline-l] Post-Wikimania Hackathon for producing offline content
by Chris Koerner 12 Aug '17

12 Aug '17

Hello, I had the fortune of meeting Martin during Wikimania where he shared a post-Wikmania event that I thought would be of interest to this community. If you're interested in offline content and are in the area, this seems to be a rather interesting event to attend. His post is below and you can find out more at: http://wiki.kiwix.org/wiki/Hackathon_Wikimania_2017 We're holding a Hackathon on August 13-18. right after Wikimania, in Potsdam, New York (about 2.5 hours drive from Montreal). The focus is on writing code to help produce offline collections of Wikipedia content along with other medical & educational materials, mostly for use in libraries, schools and clinics. The main wiki page can be found at http://OFF.NETWORK. We can probably help you with transportation from Montreal, and accommodation in Potsdam if you need it. Some of the attendees will be there from familiar offline groups such as Kiwix mainly to write code. Others focus on end-use and are from the non-profit/educational community, such as Computers for Kids, Internet-in-a-box, KA Lite. These people will want to ensure that the technical needs of the groups can be met, and that resources are shared. You can see the sort of thing Internet-in-a-Box does in this recent article on opensource.com: https://opensource.com/article/17/5/internet-in-a-box-raspberry-pi If you're interested in attending, it's probably best to contact me directly. Thanks! Martin A. Walker (walkerma on Wikipedia) Professor of Chemistry, SUNY Potsdam walkerma at potsdam.edu<mailto:walkerma at potsdam.edu> From: https://lists.wikimedia.org/pipermail/offline-l/2017-June/001405.html Yours, Chris Koerner Community Liaison Wikimedia Foundation

1 0

More Detailed Browser Stats for Desktop Sites
by Nuria Ruiz 12 Aug '17

12 Aug '17

Hello: Please take a look at the new browser report with more detailed desktop site data (all wikimedia projects agreggated): https://analytics.wikimedia.org/dashboards/browsers/#desktop-site-by-browser Some highlights: * Data is very stable over the last year * Chrome in the lead with 45% of traffic, closely followed by IE (18%) and FF (13%) * The bulk of IE traffic is IE11 and IE7 * Edge shows up with 4% slowly catching up to Safari (5%) * This data is still subject to fluctuations due to bot traffic not identified as such. We will be working on this next year. Thanks, Nuria

4 6

meeting of the mediawiki farmers user group at wikimania (6:30 saturday)
by Brian Wolff 12 Aug '17

12 Aug '17

There will be a meeting of the mediawiki farmers group at wikimania in ballroom west at 6:30 on saturday. This is a user group which aims to improve the experiance for using mediawiki in wikifarms. Anyone interested in developing new wikifarm orientied features, uses mediawiki in a farm setting or is just curious is encouraged to come. Thanks, Brian

1 0

Tidy will be replaced by RemexHTML on Wikimedia wikis latest by June 2018
by Subramanya Sastry 11 Aug '17

11 Aug '17

How to read this post? ---------------------- * For those without time to read lengthy technical emails, read the TL;DR section. * For those who don't care about all the details but want to help with this project, you can read sections 1 and 2 about Tidy, and then skip to section 7. * For those who like all their details, read the post in its entirety, and follow the links. Please ask follow up questions on wiki *on the FAQ’s talk page* [0]. If you find a bug, please report it *on Phabricator or on the page mentioned above*. TL;DR ----- The Parsing team wants to replace Tidy with a RemexHTML-based solution on the Wikimedia cluster by June 2018. This will require editors to fix pages and templates to address wikitext patterns that behave differently with RemexHTML. Please see 'What editors will need to do' section on the Tidy replacement FAQ [1]. 1. What is Tidy? ---------------- Tidy [2] is a library currently used by MediaWiki to fix some HTML errors found in wiki pages. Badly formed markup is common on wiki pages when editors use HTML tags in templates and on the page itself. (Ex: unclosed HTML tags, such as a <small> without a </small>, are common). In some cases, MediaWiki can generate erroneous HTML by itself. If we didn't fix these before sending it to browsers, some would display things in a broken way to readers. But Tidy also does other "cleanup" on its own that is not required for correctness. Ex: it removes empty elements and adds whitespace between HTML tags, which can sometimes change rendering. 2. Why replace it? ------------------ Since Tidy is based on HTML4 semantics and the Web has moved to HTML5, it also makes some incorrect changes to HTML to 'fix' things that used to not work; for example, Tidy will unexpectedly move a bullet list out of a table caption even though that's allowed. HTML4 Tidy is no longer maintained or packaged. There have also been a number of bug reports filed against Tidy [3]. Since Parsoid is based on HTML5 semantics, there are differences in rendering between Parsoid's rendering of a page and current read view that is based on Tidy. 3. Project status ----------------- Given all these considerations, the Parsing team started work to replace Tidy [4] around mid-2015. Tim Starling started this work and after a survey of existing options, decided to write a wrapper over a Java-based HTML5 parser. At the time we started the project, we thought we could probably have Tidy replaced by mid-2016. Alas! 4. What is replacing Tidy? -------------------------- Tidy will be replaced by a RemexHTML-based solution that uses the RemexHTML[5] library along with some Tidy-compatibility shims to ensure better parity with the current rendering. RemexHTML is a PHP library that Tim wrote with C.Scott’s input that implements the HTML5 parsing spec. 5. Testing and followup ----------------------- We knew that some pages will be affected and need fixing due to this change. In order to more precisely identify what that would be, we wanted to do some thorough testing. So, we built some new tools [6][7] and overhauled and upgraded other test infrastructure [8][9] to let us evaluate the impacts of replacing Tidy (among other such things in the future) which can be a subject of a post all on its own. You can find the details of our testing on the wiki [1][10], but we found that a large number of pages had rendering differences. We analyzed the results and categorized the source of differences. Based on that, to ease the process of replacement, we added a bunch of compatibility shims to mimic what Tidy does. I am skipping the details in this post. Even after that, newer testing showed that this nevertheless still leaves us with a few patterns that need fixing that we cannot / don't want to work around automatically. 6. Tools to assist editors: Linter & ParserMigration ---------------------------------------------------- In October 2016, at the parsing team offsite, Kunal ([[User:Legoktm (WMF)]]) dusted off the stalled wikitext linting project [11] and (with the help from a bunch of people on the Parsoid, db/security/code review areas) built the Linter extension that surfaces wikitext errors that Parsoid knows about to let editors fix them. Earlier this year, we decided to use Linter in service of Tidy replacement. Based on our earlier testing results, we have added a set of high-priority linter categories that identifies specific wikitext markup patterns on wiki pages that need to be fixed [12]. Separately, Tim built the ParserMigration extension to let editors evaluate their fixes to pages [13]. You can enable this in your editing preferences or replace '&action=edit' in your url bar with '&action=parsermigration-edit'. 7. What editors have to do -------------------------- The part that you have all been waiting for! Please see 'What editors will need to do' section on the Tidy replacement FAQ [1]. We have added simplified instructions, so that even community members who do not consider themselves "techies" can still learn about ways to fix pages. We'll keep that section up to date based on feedback and questions. But since it is a wiki, please also edit and tweak as required to make the text useful for yourselves! This is a first call for fixes and it is about the problems defined as "high priority". We'll issue other calls in the future for any other necessary Tidy fixups. Caveats: * As noted on that page, the linter categories don't cover all the possible sources of rendering differences. For example, there is still T157418 [14] left to address. For those who have an opinion about this, please chime in on that task. We are still evaluating the best solution for this without adding more cruft to wikitext behavior or kicking the cleanup can down the road. * As the issues in the identified linter categories are fixed, we might be better able to isolate other issues that need addressing. 8. So, when will Tidy actually be replaced? ------------------------------------------- We really would like to get Tidy removed from the cluster latest by June 2018 (or sooner if possible), and your assistance and prompt attention to these markup issues would be very helpful. We will do this in a phased manner on different wikis rather than all at once on all wikis. We really want to do this as smoothly as possible without disrupting the work of editors or affecting the rendering of the large corpus of pages on the various wikis. As you might have gathered from the text above, we have built and leveraged a wide variety of tools to assist with this. 9. Monitoring progress ---------------------- In order to monitor progress, we plan to do a weekly (or some such periodic frequency) test run that compares the rendering of pages with Tidy and with RemexHTML on a large sample of pages (in the 50K range) from a large subset of Wikimedia wikis (~50 or so). This will give us a pulse of how fixups are going, and when we might be able to flip the switch on different wikis. Subramanya (Subbu) Sastry Parsing Team. References ---------- 0. https://www.mediawiki.org/wiki/Talk:Parsing/Replacing_Tidy/FAQ 1. https://www.mediawiki.org/wiki/Parsing/Replacing_Tidy/FAQ#What_will_editors… 2. https://en.wikipedia.org/wiki/HTML_Tidy 3. https://phabricator.wikimedia.org/tag/tidy/ 4. https://phabricator.wikimedia.org/T89331 5. https://github.com/wikimedia/mediawiki-libs-RemexHtml 6. https://phabricator.wikimedia.org/T120345 7. https://github.com/wikimedia/integration-uprightdiff 8. https://github.com/wikimedia/integration-visualdiff 9. https://github.com/wikimedia/mediawiki-services-parsoid-testreduce 10. https://www.mediawiki.org/wiki/Parsing/Replacing_Tidy 11. https://phabricator.wikimedia.org/T48705 12. https://www.mediawiki.org/wiki/Help:Extension:Linter#Goal:_Replacing_Tidy 13. https://www.mediawiki.org/wiki/Help:Extension:Linter#Verifying_fixes_for_th… 14. https://phabricator.wikimedia.org/T157418

8 31

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Wikitech-l August 2017