Wikitech-l August 2016

wikitech-l@lists.wikimedia.org

78 participants
104 discussions

[ConfirmEdit][ReCaptcha] Call-to-Action: ReCaptcha module will be removed in the near future
by Florian Schmidt 13 Aug '16

13 Aug '16

Hi all together, first of all: If you don't use ConfirmEdit with the ReCaptcha (NOT the ReCaptchaNoCaptcha) module, you can STOP reading here :) TL;DR Please upgrade to the ReCaptchaNoCaptcha module to use a supported version in the future and please respond to [1], if you've an opinion about the deprecation/removal plan(s). Long version: As you may know, ConfirmEdit[2] (the MediaWiki extension that helps you to fight against spam) supports different CAPTCHA modules, such as a MathCaptcha (where the user has to solve a math problem) or QuestyCaptcha (where the user has to answer a pre-defined question). Another module is Google's ReCaptcha[3][4], in both versions, the old v1 and the newer v2 (also called NoCaptcha). In December 2014, Google announced a new version of the ReCaptcha CAPTCHA-service, called ReCaptcha NoCaptcha (or ReCaptcha v2). ConfirmEdit currently supports both versions, as Google did the same until recently. Now, as you can read on the FAQ page of ReCaptcha[5], Google stopped supporting the v1 of ReCaptcha. This means, that no new features are developed and that new keys registered for ReCaptcha will work with the new version 2 only. This let me come to the conclusion, that ReCaptcha v1 will not be supported by Google at all anymore in the near future. In order of this, we should ask our self, how long we want to support the old ReCaptcha module in ConfirmEdit (which, compared to the NoCaptcha module, _seems_ to be less effective). The mid-term plan is clear: remove the old ReCaptcha module. Now there are two ways to achieve this: * Deprecate the old reCaptcha in Version 1.27 of MediaWiki (would be a so called backport) and remove it in the upcoming 1.28 release * Deprecate the old reCaptcha in the upcoming Version 1.28 of MediaWiki and remove it: * in MediaWiki 1.29 or * when Google doesn't support the old reCaptcha anymore Because this is a huge problem for existing third-party wikis (and because we don't have any usage statistics), I'm not sure, which plan we should choose. That's why I sent this e-mail out, to get (hopefully) some responses and opinions. So, the call-to-Action: If you still use the old ReCaptcha module, please plan to upgrade to the new version 2 of ReCaptcha as soon as possible and let us know (the best way is a comment in the task[1]), if you need the old ReCaptcha version supported as long as Google supports it, or if you're fine with removing it in the upcoming 1.28 release of MediaWiki. To be absolutely clear: The ReCaptcha module will be removed, the only question is when: in MediaWiki 1.28 or in MediaWiki 1.29. If you've any questions, feel free to answer to this e-mail, add a comment to the linked task, contact me in IRC or write me an e-mail (if you need a private conversation). Thanks for your attention and for using ConfirmEdit! Best, Florian [1] https://phabricator.wikimedia.org/T142133 [2] https://www.mediawiki.org/wiki/Extension:ConfirmEdit [3] https://en.wikipedia.org/wiki/ReCAPTCHA [4] https://www.google.com/recaptcha/intro/index.html [5] https://developers.google.com/recaptcha/docs/faq

2 2

wikidatawiki.xml.bz2 failed integrity check
by Trung Dinh 12 Aug '16

12 Aug '16

Hi all, I am running into an issue with the dump file: wikidatawiki.xml.bz2 from https://dumps.wikimedia.org/wikidatawiki/latest/wikidatawiki-latest-pages-a… The dump file fails the integrity test "bzip2 -t wikidatawiki-latest-pages-artitcles.xml.bz2" I am not sure what is the best way to fix this. Re-downloading seems straightforward but there is no way to guarantee the re-downloaded one will pass the test. Thanks a lot !

2 1

Discovery Weekly Update for the week starting 2016-08-08
by Chris Koerner 12 Aug '16

12 Aug '16

Greetings, Here is this week's update from the Discovery department. * Trey did a quick analysis of the effects of putting ascii-folding (mostly removing ᶁỉäçȑîṭĭčś) before stemming (making it so searching for "jump" finds "jumps", "jumped", and "jumping") in English Wikipedia. Overall, it's a net positive! [1] * Blog post on our recent release of stripping question marks from search queries was released on Thursday, Aug 11. [2] * Stas created a nice demo of integrating WDQS and the vega graph designer (click the download button), and demo'd three other WDQS GUI improvements in this month's CREDIT showcase. [3] [4] * Our newest member of the Discovery Department started this Monday: Chelsy! :) [1] https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Re-Ordering_Stemming… [2] https://blog.wikimedia.org/2016/08/11/question-marks-search/ [3] https://phabricator.wikimedia.org/F4346786 [4] https://www.youtube.com/watch?v=BOj6nDbdPMA&t=53s ---- The full update, and archive of past updates, can be found on Mediawiki.org: https://www.mediawiki.org/wiki/Discovery/Status_updates Interested in getting involved? See tasks marked as Easy or volunteer needed in Phabricator. [1] https://phabricator.wikimedia.org/maniphest/query/qW51XhCCd8.7/#R [2] https://phabricator.wikimedia.org/maniphest/query/5KEPuEJh9TPS/#R -- Yours, Chris Koerner Community Liaison - Discovery Wikimedia Foundation

1 0

Incident report: Multiple breakages due to wfLoadExtension() transition
by Max Semenik 12 Aug '16

12 Aug '16

TLDR: migration of 2 extensions to wfLoadExtension() resulted in problems, Logstash wasn't displaying them. == Timeline == Previous days * In a massive effort by many people, lots of extensions were converted to extension.json, including ** Timeline in https://gerrit.wikimedia.org/r/#/c/303248/ ** ContactPage in https://gerrit.wikimedia.org/r/#/c/298084/ * These changes were not compatible with our current production configuration and thus had to be accompanied with mediawiki-config changes and probably be deployed separately to minimize the chance of screwup. * Furthermore, even a cursory testing of the above Timeline change would have shown that it is broken. August 9 * After 12:00 SF time Mukunda deploys train to stage 0 wikis * At 16:00 Max prepares for SWAT but sees errors in fatalmonitor and investigates: ** Creating default object from empty value in /srv/mediawiki/wmf-config/CommonSettings.php on line 686 ** Undefined variable: wgContactConfig in /srv/mediawiki/wmf-config/CommonSettings.php on line 968 * Max sees no such errors in Logstash. * After identifying the cause, Max starts reverting the affected extensions, however there were a lot of intermediate commits and Reedy was committing fixes so Max proceeds with deploying the fixes instead. * Fixes produced more problems. Max contemplates a revert of group0 back to wmf.13 but decides not to because he has never done that before and fixes kept on coming. In the hindsight, this was a mistake. * Config fixes to accommodate for wmf.14 started causing notices in wmf.13 so Max resets wmf.13 Timeline to wmf.14. * Errors indicating more breakages in Timeline prompt another batch of fixes. * At 17:42, everything is back to normal. == Casualties == * Max's liver. * Evening SWAT didn't happen. * For about 10 minutes, new timeline generation on production wikis was broken. == Conclusions == * Our code review practices are lax, including merging hairy patches without testing and self-merges. * Timeline has 0 (zero) tests while just a single parser test would have allowed to detect problems during code review. * Logstash fatalmonitor dashboard isn't displaying HHVM warnings/errors right now. * And Logstash is used by scap to verify error levels, rendering this check useless. * Logstash/Kibana is probably too complex a beast to be trusted to be the definitive source of MediaWiki health information, fatalmonitor is still more reliable. Invest time in improving it and merging with exceptionmonitor? * In ongoing outage with logs full of noise, testing stuff on canary servers is hard as non-fatal errors are easy to miss on fluorine. Deployers need access to HHVM logs on all appservers. * Beta cluster isn't serving its purpose of being the first line of defense against bugs (other than "oh, whole thing is down"). Errors in beta should be watched as closely as in prod and should be treated with the same level of seriousness, because otherwise the former will eventually turn into the latter. -- Best regards, Max Semenik ([[User:MaxSem]])

3 4

Advice on using mustache for templating
by Smriti Singh 12 Aug '16

12 Aug '16

Hello! I am currently working on implementing HTML E-Mail support. I needed advice on whether I should use mustache for implementing templates for the same. If not, I would love suggestions on what else I can use. Thank you! - Galorefitz

3 3

Wiki Conference India 2016 (Chandigarh hackathon)
by Santosh Shingare 12 Aug '16

12 Aug '16

Hello all, I am happy to announce, chandigarh hackathon WCI2016, close with 7 application, which is useful to all community. *1)* *WikiSpeak with native language :* (Hindi) (*Web Application* + *Android Mobile application*). *2)* *Edit Tamil Wiktionary **(Tamil community request)** :* - (*Android application*) *3)* *Audio file upload to WikiData (Punjabi community request) :* (*Android application*) *4) Wikipedia articles on google map :* find wikipedia articles on google map (with latitude and longitude) as per city. *5) OCR (Native Application) *: Convert scanned book copy to Indian language text with google doc (Tested for Hindi and Malayalam). *6) Communication platform[WebRTC] (Web Application)* : Community used this to talk or conference (Audio/video web conferencing application) *7) Notification (Event based)* : showing popup on event eg (If recent changes happen, It will show popup which article was updated) Thank you all participant. who participate in hackathon and deliver a big success *Note : I will share code, links and other details very shortly.* Thank you *Santosh Shingare.* *User:cherishsantosh*

1 0

Language switching improvements on mobile web
by Jon Katz 11 Aug '16

11 Aug '16

Hey Everyone, A big shout out to the reading web team (and Adam B), who rolled out a number of subtle, but important changes to the mobile web sites of all our projects earlier this week. Through a series of UX improvements, the team made it easier for people to switch between languages in an article on mobile web. To all you monolinguists out there, this is a very common and vital ability for people who speak more than one language. In addition to being a big improvement for our multi-lingual users, the development of this feature followed many of the best practices we aspire to at the foundation: - language switching on mobile initially discovered as a problem to be solved through quantitative research - tied to reading team's early strategic objective of better serving global readers - development stages included: - community consultation <https://www.mediawiki.org/wiki/Reading/Web/Projects/Improve_in-article_lang…> - iteration based on results - a/b testing and analysis in beta - qualitative research with live users <https://docs.google.com/a/wikimedia.org/presentation/d/1gbqV0fDHUDJjDZgYNDY…> - iteration based on results - collaboration with the language team (they are simultaneously rolling out compact language links <https://www.mediawiki.org/wiki/Universal_Language_Selector/Design/Interlang…> out of beta on desktop) - iteration based on collaboration Am I missing anything? More on the project here <https://www.mediawiki.org/wiki/Reading/Web/Projects/Improve_in-article_lang…> and a screenshot of the new language switching button is below Best, J

1 0

Schema migration for 'image' and 'oldimage' tables
by Krinkle 11 Aug '16

11 Aug '16

TL;DR: Participate on T589 and help decide what the upcoming schema change should entail, and how we'll migrate existing data. Hey all, Couple weeks ago we dedicated an IRC office hour to https://phabricator.wikimedia.org/T589 (RFC: image and oldimage tables). Updated draft at: https://www.mediawiki.org/wiki/Requests_for_comment/image_and_oldimage_tabl… We clarified scope and purpose of this particular RFC. Other issues are still important but considered orthogonal, and to be dealt with parallelly (or at a later time). Revised problem statement: 1. File revisions should have unique identifiers (better than "current file title + upload timestamp". (Subject to race conditions, hard to index/query, etc.) 2. Uploading new file revisions must not involve rows moving across tables, or rows being replaced. Participants agreed with the revised problem statement, it makes sense not to merely add primary keys to the existing tables ("Proposal 1" on the RFC draft), as that wouldn't adequately solve the Problem 2. The second proposal was to separate information about image revision from the image entity itself. Similar to the page/revisions tables. This was generally accepted as a good idea, but details are still to be determined. The general idea is that all revision-specific information (except for a pointer to the current revision) would no longer live in the 'image' table. Instead, information about all (for both current and past revisions) would live in the same table (instead of being moved around from one table to another when it's no longer the current one). Details at: https://www.mediawiki.org/wiki/Requests_for_comment/image_and_oldimage_tabl… Some open questions I'd like to see discussed on Phabricator (or here on wikitech): 1. Which fields do we keep in the 'image' table (img_id, img_name, img_latest, anything else?). All fields currently being queried from both tables, will probably only stay in the image revision table. But there are a few fields that we intentionally only want to query about current versions. For example 'img_sha1'. For duplicate-detection, we need to only consider the latest revisions. Doing this by keeping img_sha1 means uploading a new revision will involve updating two fields instead of one (img_latest and img_sha1). This isn't unprecedented as we do this for page as well (WikiPage::updateRevisionOn; page_latest, page_touched, page_is_redirect, page_len). Are there other fields we need to keep besides img_sha1? Or should we can solve the img_sha1 use case in a different manner? 2. img_metadata This field is a blob of serialised PHP (typically representing the Exif data of an image). Tim (correct me if I got it wrong) mentioned we could potentially make migration easier by changing img_metadata to be stored in a separate table and change the img_metadata field (in the image revision table) to instead be a pointer to a primary key. This could potentially be done separately later, but if it helps migration, we should consider doing it now. How will this interact with file deletion? Will it be difficult to garbage collect this? Do we need to? (We don't seem to do it for the 'text' table / external store; is it worth moving this an external store?) 3. Migration If we rename both tables (image/oldimage -> file/filerevision), we'd have the ability to run migration in the background without interfering with the live site, and without requiring a long read-only period and/or duplicate and additional code complexity to be developed. Is there a way we can do the migration without creating two new tables? Using the oldimage table as import destination for current rows isn't straight forward as existing scan queries would need to skip the current rows somehow while in the midst of this migration. Seems possible, but is it worth the complexity? (We'd need extra code that knows about that migration field, and how long do we keep that code? Also complicates migration for third-parties using update.php). Is creating the new tables separately viable for the scale of Wikimedia Commons? (and dropping the old ones once finished). Is this a concern from a DBA perspective with regards to storage space? (We'd temporarily need about twice the space for these tables). So far I understood that it wouldn't be a problem per se, but that there are also other options we can explore for Wikimedia. For example we could use a separate set of slaves and alter those while depooled (essentially using entirely separate set of db slaves instead of a separate table within each slave). Do we create the new table(s) separately and switch over once it's caught up? This would require doing multiple passes as we depool slaves one by one (we've done that before at Wikimedia). Switch-over could be done by migrating before the software upgrade, with a very short read-only period after the last pass is finished. It wouldn't require maintaining multiple code paths, which is attractive. Other ideas? -- Timo

2 1

CI is having issues, hang with us
by Chad Horohoe 10 Aug '16

10 Aug '16

Hi, Nodepool isn't happy, so our CI is having trouble spinning up new instances for testing. We're well aware and working on it. Since a couple of people have already asked us what they can do to help...best thing you can do is avoid pushing changes for review (if they're not urgent) for a bit. And *definitely* don't run "recheck" on existing changes for right now. Thanks for understanding! -Chad

3 3

2016-08-10 Scrum of Scrums meeting notes
by Grace Gellerman 10 Aug '16

10 Aug '16

https://www.mediawiki.org/wiki/Scrum_of_scrums/2016-08-10 = 2016-08-10 = == Product == === Reading === ==== Reading Web ==== * Current sprint: https://phabricator.wikimedia.org/project/view/2115/ * Next sprint : https://phabricator.wikimedia.org/project/view/2126/ * New language switcher button deployed this morning (about 5 minutes ago, as pages are purged naturally changes will be visible) * Lazy loaded images for mobile web Wikipedias shipping next sprint in 3 deployments: all small wikis, all medium wikis, all large wikis * Adding a contribution tab to the hamburger menu next sprint * Enhancing watchstar next sprint, primarily for non-js users ==== iOS native app ==== * 5.0.6 Passed Regression last week ** Holding while we diagnose a memory crash ** Working with beta testers to verify fixes https://phabricator.wikimedia.org/tag/ios-app-v5.0.6-hotfix/ * 5.1 is in Development ** Major features are iPad and Find in Page ** Expectd to be feature complete end of week ** Expected to go to beta next week https://phabricator.wikimedia.org/project/board/1736/query/open/ * 5.2 Scheduled to begin development next week ** iOS 10 release ** Major features are Widgets and Feed improvements ** Expected to go to beta in early september ==== Android native app ==== * Current sprint: https://phabricator.wikimedia.org/project/view/2091/ * Next sprint: https://phabricator.wikimedia.org/project/board/2142/ * Working on navigation overhaul ==== Mobile Content Service ==== * Working on fixing links with + signs ( https://phabricator.wikimedia.org/T136223) ==== Reading Infrastructure ==== * Gergo working on pywikibot issue, Brad is OOO ===Community Tech=== * Requested new hardware for cross-wiki watchlists ( https://phabricator.wikimedia.org/T142538 ) * Will be switching Macedoian and/or Swedish Wikipedias to numeric sorting next week for wider testing * Working on Programs Dashboard (https://outreachdashboard.wmflabs.org/) * Working on IABot improvements/stability == Editing == === VisualEditor === * Blocked: ** We're waiting on Design Research with T141069 so we can work with them on T141068 for the new wikitext editor work. * Blocking: ** Parsing team are waiting for our response on "native" Parsoid <gallery> implementation. Thalia will look at it. * Updates: ** Now live for logged-in users on Arabic-script Wikipedias; logged-out users planned for next week, then Indic script Wikipedias ** Lots of the team in town this week. Working on a few things. Come talk to us if you're also in town and want to say hi. ** HTML diffs for partial page saving (so faster saves); needs input from Services team on https://github.com/edg2s/restbase/tree/sections branch of RESTbase; PR forthcoming ** New wikitext editor work continues; https://phabricator.wikimedia.org/T142138 and so on === Parsing === * Blocked: ** https://phabricator.wikimedia.org/T141591 -- Security review needed ** https://gerrit.wikimedia.org/r/#/c/264026/ -- Parsoid-native implementation of <gallery>; need review and feedback from VE team * Blocking: ** Shadow Namespaces: Please review my comments https://phabricator.wikimedia.org/T91162#2408633 -- I think this is a RFC discussion matter, but happy to read through it and review it. :) * Updates: ** Parsoid cluster upgraded to node v4 & Jessie (thanks ops) - deployments resumed yesterday ** Next step: migrate deployment process to scap3 -- to be scheduled with Services ** https://gerrit.wikimedia.org/r/#/c/303912/ Implement magic word based opt-out of global user pages -- bikeshed on __NOGLOBAL__ or __NOFOREIGN__ ... === Multimedia === * Blocked: ** Is there an update on thumbor production status? (Performance?) It's still blocking ImageTweaks deployment. * Blocking: None of which we're aware. * Updates: ** Continuing our work on FileAnnotations; currently pending on === Language === * Blocked: * Blocking: * Updates: ** ContentTranslation dumps are available at, https://dumps.wikimedia.org/other/contenttranslation/ ** CX MT card and templates getting new changes this and next week. ** Apertium -> Jessie work in progress. Few packages left to upload. === Collaboration === * Blocked * Blocking ** We are working on rewriting the Flow caching for the data center migration - https://phabricator.wikimedia.org/T120009 * Updates ** Finishing the ptwikibooks LiquidThreads -> Flow conversion, which had some issues - https://phabricator.wikimedia.org/T119509 ** Working on allowing dynamic actions in the Echo secondary links (e.g. unwatching a Flow board through AJAX) - https://phabricator.wikimedia.org/T132975 == Technology == === Analytics === * refinery deployment migrated to scap3, tested and everything works well * sqooping all mediawiki databases out to hdfs, from dbstore1002, so far seems to not impact that machine much * pagecounts-raw and pagecounts-all-sites dataset updates have been stopped, old files remain, upgrade to the new pageviews or pagecounts-ez datasets as per: https://dumps.wikimedia.org/other/analytics/ * eventlogging kafka pipeline upgraded to fix broker restart bug * follow-up on pageview spike: Windows update caused a problem with the TLS handshake in Chrome 41 === Architecture === ** **https://www.mediawiki.org/wiki/Architecture_committee/Status* <https://www.mediawiki.org/wiki/Architecture_committee/Status> ** Last week: ArchCom meetings 2016W31: 2016-08-03 (Wed) *** 1pm PDT (20 UTC) Planning meeting: [[Phab:E250]] *** 2pm PDT (21 UTC) IRC #wikimedia-office [[Phab:*E251]]* ***** [[Phab:T128351]]: Notifications in core* ** This week: ArchCom meetings 2016W32: 2016-08-10 (Wed) *** 1pm PDT (20 UTC) Planning meeting: [[Phab:E258]] *** 2pm PDT (21 UTC) IRC #wikimedia-office [[Phab:*E259]]* ***** [[Wikitext]] *(and whether Wikimedia should invest in a spec) === Services === * Parsoid move to Jessie and node 4 complete * Electron PDF rendering service discussion: *https://phabricator.wikimedia.org/T142226* <https://phabricator.wikimedia.org/T142226> * Working on new kafka driver for Change-Prop - deploy as soon as kafka upgrade to 0.9 === Technical Operations === * '''Blocked''' ** None * '''Blocking''' ** None * Updates ** Parsoid clusters upgraded to jessie throughout the week ** Worked with research to get ORES workers their own dedicated cluster. Hardware estimations done ** xkey meeting held. ** Review of ContentTranslation apertium packages ongoing ** labs got openstack liberty upgrade === Discovery === * No blockers * Working on BM25 implementation * Working on multi-wiki indexes * Question mark removal deployed * Bugfixes for cross-language search deployed * Portal stats updated + fixes for small languages === Interactive Team === * Working on deploying <mapframe> and <maplink> to all non-Wikipedia sites * Working on replacing GeoHack with <maplink> + info screen on all Wikipedias * Working on deploying Tabular data on Commons. Already synced up with Wikidata. === Security === * MediaWiki 1.27.1 will be released this week * Notifications for OAuth consumer proposals are being deployed this week: ** https://phabricator.wikimedia.org/T61772 ** https://phabricator.wikimedia.org/T62528 * Planning for captcha improvements continues ** https://phabricator.wikimedia.org/T125132 ** https://phabricator.wikimedia.org/T141490 ** First step will be re-generating images using current script == Wikidata == * No blockers. * New features on query.wikidata.org (live preview of example queries, map layers). * Working on first feedback we got from publishing the Commons MediaInfo prototype. * Layout tweaks to the default Wikidata UI. == Fundraising Tech == * No blockers * Deploying and monitoring first activemq replacement changes * Minor Central Notice deploy with tonight's SWAT * Testing background contact de-duping script for CiviCRM == Performance == * No blockers * Ori still out * Still doing heavy refactoring in ResourceLoader/Outputpage for the critical rendering path * Still working on transaction bugfixes and improvements for multi-DC * WebpageTest now supports Opera Mini & UC Mini * Thumbor plugins test suite complete, production work to resume next week * Team offsite to be in NY area end of November

1 0

← Newer
1
...
4
5
6
7
8
9
10
11
Older →

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Wikitech-l August 2016