Wikitech-l July 2016

wikitech-l@lists.wikimedia.org

81 participants
83 discussions

Jenkins building Debian packages
by Antoine Musso 27 Jul '16

27 Jul '16

Hello, I have added a Jenkins job that attempt to build a Debian package when a patch is proposed. I have applied it on all of the operations/debs/ repositories as well as to a few others (such as mediawiki/debian). The newcomer is named debian-glue it is based on a set of shell script to integrated the Debian packaging toolchain in Jenkins: http://jenkins-debian-glue.org/ The first version got originally introduced by Carl "Azatoth" Fürstenberg exactly three years ago. It is not generalized but still highly experimental. Please fill feedback/issues in Phabricator against #continuous-integration-config , ideally with 'debian-glue' in the subject: https://phabricator.wikimedia.org/project/board/1208/ -- Antoine "hashar" Musso

2 1

Wikipedia seeks to speak your language
by Deborah Tankersley 27 Jul '16

27 Jul '16

Using language detection to search the right Wikipedia Wikipedia readers speak many languages, so it’s not a surprise that sometimes they search for phrases not in the language of the wiki that they’re currently reading. This, unfortunately, can lead to poor search results. A recent survey we completed on English Wikipedia identified searches done in 40 different languages <https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/TextCat_Re-optimizat…> [1]! The Wikimedia Discovery department <http://www.mediawiki.org/wiki/Wikimedia_Discovery> [2] wants to help people easily find what they are looking for. In order to do this, the Discovery Search team is rolling out new language identification software to the Wikipedia search engine. This new software will detect when a search is unsuccessful, but appears to be in a different language. When this happens, the search results page will include results from the Wikipedia of the automatically detected language. These new cross-wiki results will be displayed along with the local-wiki results, if there are any. We’ve recently enabled the language identification and search results for the English, French, German, Italian, and Spanish-language Wikipedias. The next group of Wikipedias to have language detection enabled will includeIndonesian, Japanese, Portuguese, and Russian <http://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/TextCat_Optimization_…> [3]. We are investigating ways to bring language detection to more Wikipedias and to other Wikimedia projects. The Search team has other language detection ideas and plans in the works. We’re thinking about ways to improve language detection with smarter measures of confidence <https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/TextCat_and_Confiden…> [4]. We are also exploring detection of search in one character set while using a keyboard from another character set. Early experiments with English and Russian <https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Typing_on_the_Wrong_…> [5] are promising! You can find technical details about our new language detection module (TextCat) onMediaWiki.org <https://www.mediawiki.org/wiki/TextCat> [6]. PHP <https://github.com/wikimedia/wikimedia-textcat> [7] and updated Perl <https://github.com/Trey314159/TextCat> [8] libraries are also available and the libraries include language models for dozens of languages. You can also test the language detection using our online demo <https://tools.wmflabs.org/textcatdemo/> [9]. The demo lets you try all the different language models on your own text. It also includes tutorials and lots of additional information about TextCat’s internal workings. Let’s get searching - now with language detection and better results! You can read theblog post <https://blog.wikimedia.org/2016/07/27/wikipedia-language-search/> [10] and more detailed information is here <https://commons.wikimedia.org/wiki/File:Wikipedia_Seeks_to_Speak_Your_Langu…> [11]. *Here's some nice screenshots of what it looked like before we added in the language detection...[12]* *and after we added in the language detection for a Russian query on English Wikipedia [13]:* *Thanks for reading - from the Discovery Search Team Gnomes!* [1] https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/TextCat_Re-optimizat… [2] http://www.mediawiki.org/wiki/Wikimedia_Discovery [3] http://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/TextCat_Optimization_… [4] https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/TextCat_and_Confiden… [5] https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Typing_on_the_Wrong_… [6] https://www.mediawiki.org/wiki/TextCat [7] https://github.com/wikimedia/wikimedia-textcat [8] https://github.com/Trey314159/TextCat [9] https://tools.wmflabs.org/textcatdemo/ [10] https://blog.wikimedia.org/2016/07/27/wikipedia-language-search/ [11] https://commons.wikimedia.org/wiki/File:Wikipedia_Seeks_to_Speak_Your_Langu… [12] https://commons.wikimedia.org/wiki/File%3AExisting-search_no-textcat.png [13] https://commons.wikimedia.org/wiki/File%3ANew-search_with-textcat.png -- Deb Tankersley Product Manager, Discovery IRC: debt Wikimedia Foundation

1 0

Scrum of Scrums - 2016-07-27
by Greg Grossmeier 27 Jul '16

27 Jul '16

https://www.mediawiki.org/wiki/Scrum_of_scrums/2016-07-27 = 2016-07-27 = == Product == === Reading === ==== iOS native app ==== * 5.0.5 ** passed regression, but we foud a bug in beta. Patching and pushing a new build today. ** expected release to Apple store later this week * Development of 5.1 is in progress * Planning of 5.2 is in progress ==== Android native app ==== * Stable release containing the feed planned for Thursday * Work on navigation overhaul is underway ==== Mobile Content Service (MCS) ==== * Added public feed endpoints to MCS documentation: https://www.mediawiki.org/wiki/Wikimedia_Apps/Team/RESTBase_services_for_ap… ==== Reading Web ==== ==== Reading Infrastructure ==== * Waiting on translatewiki to be ready for https://gerrit.wikimedia.org/r/#/c/280945/ === Community Tech === * Deploying numeric category sorting to testwiki today, others to follow * Working with CLs to find guinea pig wiki for PageAssessments before deploying to enwiki * Finishing work on CopyPatrol tool (https://tools.wmflabs.org/copypatrol) === Editing === ==== Collaboration ==== * Blocked - None * Blocking - Florian and I are working on refactoring how ConfirmEdit handles OutputPage, in preparation for a refactoring Timo is doing. This may be a breaking change, so follow at https://phabricator.wikimedia.org/T141300 . Otherwise, no change. * Updates ** Mention notifications: Wikimedia Germany is working on notifications for mention failures (when you apparently intended a mention, but it is not counted as one), and successful mentions. They will also allow you to mention yourself. We're coordinating with them on these tasks. ** Echo changes related to moderation. ** Warning when you leave a Flow board without saving. ** Working on updating the Echo badges. ==== Language ==== * Blocked: Apertium Jessie Packages (Tech Ops). * Updates: ** Improvements in CX and CLL (ULS) continue. ** Parallel Corpora Dumps work is restarted. == Technology == === Analytics === * edit history reconstruction algorithms done, short demo tomorrow, scaling them to handle all wikis is next * event bus schema work is approaching completion, the new schemas feed nicely into edit history and handle the requirements of the services team * work on moving refinery deployment to scap3 is ongoing * loading of the new more performant AQS cluster is progressing nicely now === Technical Operations === * '''Blocked''' ** None * '''Blocking''' ** None * Updates ** Got some problems during a network maintenance. T140770 ** Minor issues in labs with OpenLDAP memory leaks ** Working on upgrading puppetmaster infrastructure to 3.8 ** Started looking into vmod_xkey for varnish ** Planning for a parsoid host (wtp*) upgrade to jessie with Parsing. Scheduled for 1st week of August === Security === * Working on pingback privacy statement with Legal * Mediawiki 1.27.1 nearly complete; to be released August 3 * Mediawiki 1.27.2 tracking and prep has begun * Security releases wlll now be on a two month schedule === Services === * '''Blocked''' ** None * '''Blocking''' ** None * Updates ** Kafka upgrade to 0.9 ongoing. ** Session storage & authentication service planning and prototyping: https://phabricator.wikimedia.org/T140813 === Architecture / ArchCom === * https://www.mediawiki.org/wiki/Architecture_committee/Status * Recent 2016W29: 2016-07-20 (last week) ** ArchCom Planning meeting: [https://phabricator.wikimedia.org/E234 E234] *** Notes: [https://www.mediawiki.org/wiki/Architecture_committee/2016-07-20 Architecture committee/2016-07-20] ** ArchCom-RFC office hour: [https://phabricator.wikimedia.org/E235 E235] *** Agenda: [https://phabricator.wikimedia.org/T126641 T126641]: [RFC] Devise plan for a cross-wiki watchlist back-end * Upcoming 2016W30: 2016-07-27 (today) ** ArchCom Planning meeting: [https://phabricator.wikimedia.org/E236 Phab:E236] *** Notes: [https://www.mediawiki.org/wiki/Architecture_committee/2016-07-27 Architecture committee/2016-07-27] ** ArchCom-RFC office hour: [https://phabricator.wikimedia.org/E237 Phab:E237] *** Agenda: [https://phabricator.wikimedia.org/T128602 T128602 Create and deploy an extension that implements an authenticated key-value store] === RelEng === * Blocking ** None * Blocked ** None * Updates ** Chad proposed a new icinga alert to detect files owned by root in the mediawiki deployment repository: https://gerrit.wikimedia.org/r/#/c/301327/ ** Zeljko on vacation until Aug 15th ** Antione going on vacation next week until Aug 21st == Discovery == * '''Blocking''': none * '''Blocked''': none * Mostly working on refactoring & technical debt * Trey published survey of top unsuccessful searches: https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Top_Unsuccessful_Sea… * Vacations * State of the Map & DataViz hackathon (https://www.mediawiki.org/wiki/DataViz_Seattle_hackathon) == Wikidata == * working on prototype for structured commons * Daniel working on multi-content revisions == Fundraising Tech == * Not blocked. * EMcNaughton is on the cusp of running a dedupe script on CiviCRM--it'll still take several weeks to finish, going at a few hundred contacts per minute, but now we think that even the first iteration will catch a majority of the duplicates. * LZia might be doing some analysis on CentralNotice banner numbers, in a part-time volunteer project for WLM. She'll be the first analyst to use some of our exciting new data streams, like "banner history". * Queue overhaul is coming along steadily, no blockers. -- | Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E | | identi.ca: @greg A18D 1138 8E47 FAC8 1C7D |

1 0

[[Link]]s deprecated?
by Bináris 27 Jul '16

27 Jul '16

At https://phabricator.wikimedia.org/T47126 Verdy_p wrote: "So I think that writing "[[Link]]s" (or "[[Target|Link]]s") instead of "[[Target|Links]]" (or "[[Link|Links]]") is just an old (deprecated) trick of MediaWiki used by (lazy) wiki redactors, we cannot recommand using it:" I think this may be of a wider interest. Is [[Link]]s syntax really deprecated? I never read about this anywhere. >From when? Is it subject to removal? Should all guides be updated and links corrected by bot? -- Bináris

5 5

2016W30 ArchCom-RFC meeting: authenticated key-value store
by Rob Lanphier 27 Jul '16

27 Jul '16

Hi everyone In our upcoming ArchCom-RFC IRC office hour meeting, we plan to discuss T128602, which discusses how to implement an authenticated key-value store in MediaWiki. Dmitry Brant wrote the original RFC, and Brad Jorsch refined it, listing out many questions on the Phab task (e.g. Action API endpoint or RESTbase service, how should we store the data, what limits should we have, what sort of abuse prevention strategy should we employ) Discussion time/place: #wikimedia-office: 2016-07-27 (Wednesday) 21:00 UTC (2pm PDT, 23:00 CEST) More details about the meeting: <https://phabricator.wikimedia.org/E237> More details about the topic: <https://phabricator.wikimedia.org/T128602> Rob p.s. ArchCom status updates continue on <https://www.mediawiki.org/wiki/Architecture_committee/Status>

1 0

"Cases" section approved, finishing "Committee" section of Code of Conduct
by Matthew Flaschen 26 Jul '16

26 Jul '16

The community has approved the "Cases" (https://www.mediawiki.org/wiki/Code_of_Conduct/Draft#Page:_Code_of_Conduct.…) section of the draft Code of Conduct. The next section is "Committee". * Section: https://www.mediawiki.org/wiki/Code_of_Conduct/Draft#Page:_Code_of_Conduct.… * Talk: https://www.mediawiki.org/wiki/Talk:Code_of_Conduct/Draft * Alternatively, you can provide anonymous feedback to conduct-discussion at wikimedia.org . This is the best time to make any final necessary changes to the Committee section (and explain why, in edit summaries and/or talk) and discuss it on the talk page. After this last call, I will send out emails seeking approval of this section, organized by sub-section. Thanks, Matt Flaschen

1 0

Microsoft adding research tool to Office
by Pine W 26 Jul '16

26 Jul '16

I'm wondering if MS will pull citations from Wikipedia and/or make use of Wikidata. I'm also wondering if this will decrease Wikimedia site traffic in a similar way to how search engine knowledge panels may have decreased Wikimedia site traffic, particularly in this case from users who have access to MS Office. https://blogs.office.com/2016/07/26/the-evolution-of-office-apps-new-intell… Pine

1 0

The Revision Scoring weekly update
by Amir Ladsgroup 26 Jul '16

26 Jul '16

Hey, This is the 14th weekly update from revision scoring team that we have sent to this mailing list. New developments: - We have a new grafana dashboard for requests that the ORES beta feature makes [1] [2] - There is a grafana chart in ORES service showing total number of requests (including requests hitting cache). Showing we had 20 million requests last month! [3] - ORES extension can be set to score edits only in a given set of namespaces. This was useful to reduce failure rate of jobs in ORES extension from 4% to 1% [4] - Scap3 now automatically deploys to canary node for ORES first (it's scb2001.codfw.wmnet). So we can better test deployments beforehand. [5] Maintenance and robustness: - We are removing a redundant index on ores_classification. No harm would be made but we'll save disk space. [6] - In footer of ores.wikimedia.org, we had "Hosted in Wikimedia Labs". It got fixed now. [7] - Cache.php in ORES extension which stores the scores into database has a better structure for handling errors.[8] 1. https://phabricator.wikimedia.org/T141169 2. https://grafana.wikimedia.org/dashboard/db/ores-extension 3. https://grafana.wikimedia.org/dashboard/db/ores 4. https://phabricator.wikimedia.org/T139660 5. https://phabricator.wikimedia.org/T139825 6. https://phabricator.wikimedia.org/T140803 7. https://phabricator.wikimedia.org/T139028 8. https://phabricator.wikimedia.org/T137880 Sincerely, Amir from the Revision Scoring team

1 0

How to find a repository?
by Purodha Blissenbach 26 Jul '16

26 Jul '16

Hi, https://phabricator.wikimedia.org/T129442 says to belong to Tool-Labs-tools-Pageviews but how to find the repository for it, if I want to work on it? Gerrit does not find anything when I enter this sring in its repository search. Is there any tool or tutorial for this question? Purodha

6 5

Canary Deploys for MediaWiki
by Tyler Cipriani 25 Jul '16

25 Jul '16

tl;dr: Scap will deploy to canary servers and check for error-log spikes in the next version (to be released Soon™). In light of recent incidents[0] which have created outages accompanied by large, easily detectable, error-rate spikes, a patch has recently landed in Scap[1] that will: 1. Push changes to a set of canary servers[2] before syncing to proxy servers 2. Wait a configurable length of time (currently 20 seconds[3]) for any errors to have time to make themselves known 3. Query Logstash (using a script written by Gabriel Wicke[4]) to determine if the error rate has increased over a configurable threshold (currently 10-fold[5]) Big thanks to the folks that helped in this effort: Gabriel Wicke, Filippo Giunchedi and Giuseppe Lavagetto, Bryan Davis and Erik Bernhardson (for their mad Logstash skillz)! It is noteworthy, that in instances where expedience is required—we're in the middle of an outage and who cares what Logstash has to say—the `--force` flag can be added to skip canary checks all together (i.e. `scap sync-file --force wmf-config/InitialiseSettings 'Panic!!'`). The RelEng team's eventual goal is still to move MediaWiki deployments to the more robust and resillient Scap3 deployment framework. There is some high-priority work that has to happen before the Scap3 move. In the interim, we are taking steps (like this one) to respond to incidents and keep deployments safe. Hopefully, this work and the error-rate alert work from Ori last week[6] will allow everyone to be more conscientious and more keenly aware of deployments that cause large aberrations in the rate of errors. <3, Your Friendly Neighborhood Release Engineering Team [0]. https://wikitech.wikimedia.org/wiki/Incident_documentation/20160601-MediaWi… is the recent example I could find, but there have been others. [1]. https://phabricator.wikimedia.org/D248 [2]. https://gerrit.wikimedia.org/r/#/c/294742/ [3]. https://github.com/wikimedia/scap/blob/master/scap/config.py#L19 [4]. https://gerrit.wikimedia.org/r/#/c/292505/ [5]. https://github.com/wikimedia/scap/blob/master/scap/config.py#L18 [6]. https://gerrit.wikimedia.org/r/#/c/300327/

5 5

← Newer
1
2
3
4
5
6
7
8
9
Older →

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Wikitech-l July 2016