Hi folks,
Just letting you know that wm-bot logs will not be available for a day
or two, reason is required migration to different OS as current OS is
no longer accepted on wikimedia labs and all instances that are
running it have to be terminated.
If you want to track progress see https://phabricator.wikimedia.org/T157838
Bot is already up and running on new Debian instance and new logs are
being collected, but the web server and web scripts are still not
fully operational, we need to migrate about 3GB of text log files from
IRC channels and also postgre DB which has similar size (not sure how
to do that on-line, probably not even possible).
Having say that, there may be some more outages because of postgre
server switch, but they shouldn't be so much visible.
Sorry for turbulences and thank you for your patience
Hello Wikimedia technical contributors & developers:
Just sending one last reminder. The Wikimedia Foundation is asking for
your feedback
in a survey.
***The survey will close on February 15, 2017.***
We want to know how well we are supporting your contributions on and off
wiki, and how we can change or improve things in the future.[1] The
opinions you share will directly affect the current and future work of the
Wikimedia Foundation.
To say thank you for your time, we are giving away 10 Wikimedia T-shirts to
randomly selected people who take the survey.[2] The survey is available in
various languages and will take between 20 and 40 minutes.
Use this link to take the survey now:
https://wikimedia.qualtrics.com/SE/?SID=SV_6mTVlPf6O06r3mt&Aud=DEV&Src=DEV
You can find more information about this project here[3]. This survey is
hosted by a third-party service and governed by this privacy statement[4].
Please visit our frequently asked questions page to find more information
about this survey[5]. If you need additional help or have questions about
this survey, send an email to surveys(a)wikimedia.org.
Thank you!
Edward Galvez
Community Engagement
Wikimedia Foundation
[1] This survey is primarily meant to get feedback on the Wikimedia
Foundation's current work, not long-term strategy.
[2]Legal information we have to share: No purchase necessary. Must be the
age of majority to participate. Sponsored by the Wikimedia Foundation
located at 149 New Montgomery, San Francisco, CA, USA, 94105. Ends February
16, 2017. Void where prohibited. Follow this link for the contest rules:
https://meta.wikimedia.org/wiki/Community_Engagement_
Insights/2017_second_contest_rules
[3] About this survey:
https://meta.wikimedia.org/wiki/Community_Engagement_Insight
s/About_CE_Insights
[4] Privacy statement: https://wikimediafoundation.org/wiki/
Community_Engagement_Insights_2016_Survey_Privacy_Statement
[5] FAQ:
https://meta.wikimedia.org/wiki/Community_Engagement_Insight
s/Frequently_asked_questions
--
Edward Galvez
Evaluation Strategist (Survey Specialist), and
Affiliations Committee Liaison
Learning & Evaluation
Community Engagement
Wikimedia Foundation
There is a security fix to ensure that EnableFlow is always properly
attributed.
This may be an issue if you see users maliciously using
Special:EnableFlow on pages that already exist.
It should be merged shortly, but in the meantime, you can download it
from Gerrit (https://gerrit.wikimedia.org/r/#/c/333301/):
git fetch ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/Flow
refs/changes/01/333301/1 && git checkout FETCH_HEAD
Matt Flaschen
Hello,
In the second half of January 2017, the version of Node running in WMF
production has been updated from v4 to v6~[1]. Parsoid, RESTBase, AQS and
the services running on the SCB cluster (Graphoid, Mathoid, Mobile Content
Service, to mention just a few) are now running on Node v6. The only
outstanding Node service which is still running Node v4 is Maps, but we are
making good progress on moving it to Node v6 and expect it to happen
soon~[2].
In the case of services running in WMF production, we have seen a slight
increase in performance as well as a substantial decrease in memory
consumption across the board, which improves the stability of our services.
You can read more about the experience of the update and its results in the
post published on Wikimedia's blog~[3].
Big thanks to all of the people that helped to make it happen!
Happy Friday,
Marko && the Services Team
[1] https://phabricator.wikimedia.org/T149331
[2] https://phabricator.wikimedia.org/T150354
[3] https://blog.wikimedia.org/2017/02/17/node-6-wikimedia/
--
Marko Obrovac, PhD
Senior Services Engineer
Wikimedia Foundation
Hi all!
Logspam makes it difficult to glance at error logs after a deployment and
reason about a deployment's impact [0]. Release Engineering is making a
conscious effort (in Scrum of Scrums, in Phabricator, and on mailing lists) to
connect logspam tasks with folks that can make an impact on these tasks (and,
consequentially, make an impact on deployments). Sometimes, as now, our effort
takes the form of a broad appeal to help investigate high impact logspam tasks.
This week in logspam:
* wfShellExec errors end up in HHVM log [1]
These errors are the noisiest of the noisy recently. They take various forms,
but many relate to PDF handling and start with "SyntaxError"
* Warning: Cannot modify header information - headers already sent [2]
The latest iteration of this error seems to have started with the
release of 1.29.0-wmf.10
* Warning: timed out after 0.2 seconds when connecting to rdb1001.eqiad.wmnet [110]: Connection timed out [3]
saw a bit of movement last week, but there are some unanswered questions and
the message is still going strong.
* Couple of session related ones
** Session "{session}": Unverified user provided and no metadata to auth it [4]
** Session "{session}": Metadata merge failed: {exception} [5]
* Throttler: throttler data not found for {user} [6]
If you or anyone you know has information that can lead to the cessation of
these errors, please add that person or a comment on the tasks listed here.
Thus concludes another exciting week in logspam.
<3
-- RelEngers
[0] <https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Logspam>
[1] <https://phabricator.wikimedia.org/T157646>
[2] <https://phabricator.wikimedia.org/T157392>
[3] <https://phabricator.wikimedia.org/T125735>
[4] <https://phabricator.wikimedia.org/T158367>
[5] <https://phabricator.wikimedia.org/T158365>
[6] <https://phabricator.wikimedia.org/T158364>
https://www.mediawiki.org/wiki/Scrum_of_scrums/2017-02-15
= 2017-02-15 =
== Technology ==
== Analytics ==
- Finished rebuilding pageview API for resiliency (10x improvments on
throughput and latency were done last quarter). A “programatic way to get
pageviews” was one of the most requested items on community wishlist last
year.
- Still working with DBAs to get data from labs, working functionally,
now working on performance
- AB testing document design done, will publish design document.
- Working on changes to parse user agent on eventlogging data so it is
not raw in tables
- Merged many patches to delete data aon hadoop on our 60 days schedule
=== Security===
Reviews:
- PageForms
- 3D extension
- Internal File Server
== Product ==
=== Reading ===
==== iOS native app ====
* Last Week
** Continued work on 5.4
https://phabricator.wikimedia.org/project/view/2326/
*** Nearby map view is feature complete
*** Completed login updates and addition of 2FA
* This week
** Continue work on 5.4
*** Location permission changes (friendlier prompt, users to opt-in instead
of an intrusive pop-up)
*** Other bug fixes and enhancements
==== Web ====
* Current sprint: https://phabricator.wikimedia.org/project/board/2460/
* Last week:
** Create Reading Depth schema: T155639 (WIP)
** Spikes related to problems we're trying to solve from technical, design,
and product points of view: T156259, T157071, and T157071.
* Next week:
** Page Previews to be rolled out to 90% of users of italian, russian,
greek, catalan, and hebrew wikis: T157700
** Deploy new branding work to Catalan and Italian wikipedia mobile
website: T156794
====Android====
* Last week:
** 2.5.188 beta released; epicly low crash rates. Production is coming
soon! (Final beta release likely today)
** Mapbox library upgrade in progress (some issues with method limit)
** Miscellaneous release time fixes
* Next week (https://phabricator.wikimedia.org/project/view/2352/ ):
** Continue improving the offline experience
====Reading Infrastructure====
* working on TemplateStyles, ORES, random log/test errors
* TemplateStyles RfC: https://phabricator.wikimedia.org/T155813
* https://gerrit.wikimedia.org/r/#/c/336963/ (more machine-readability in
MediaWiki change list pages) could use review
* not blocking/blocked:
* Mobile Content Service: Kanban Board:
https://phabricator.wikimedia.org/project/view/2445/
** New endpoint for OnThisDay (anniversary) deployed soon (example:
/feed/onthisday/selected/mm/dd)
** Working on: minor improvements for onthisday + more tests
** Next: expanding TFA (today's featured article) to more languages using
FeaturedFeed extension
====Community Tech====
* Cookie blocks will be tested on testwiki, and hopefully out to enwiki
next week https://phabricator.wikimedia.org/T158129
* Bot that generates popular pages report, BRFA soon
https://phabricator.wikimedia.org/T156857
* Starting rewrite on XTools ArticleInfo tool
https://phabricator.wikimedia.org/T157706
* Other minor feature additions to CopyPatrol, along with porting to
Spanish Wikipedia
** https://phabricator.wikimedia.org/T157120 and
https://phabricator.wikimedia.org/T155933
* Starting investigation on increasing length of edit summaries for
non-Latin languages https://phabricator.wikimedia.org/T153333
Blockers:
* Collation version numbering patch needs review
https://gerrit.wikimedia.org/r/#/c/272419/
** Only the authors are apparently that familiar with it, blocking effort
to make UCA default for MediaWiki https://phabricator.wikimedia.org/T47611
* Range Contributions test queries and the plan for the infrastructure
needs feedback https://phabricator.wikimedia.org/T156318#3023578
=== Discovery ===
* No blockers
* A/B test for inter-wiki search results is running on Persian, Italian,
Catalan & Polish Wikis. https://phabricator.wikimedia.org/T149806
* TextCat language detection enabled on Dutch wiki and improved on German,
English, Spanish, French, Italian, Japanese, Portuguese, and Russian
Wikipedias (https://phabricator.wikimedia.org/T149324)
* Published analysis for Stempel text analyzer in Polish:
https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Stempel_Analyzer_Ana…
* Wikidata Query Service data reloaded, many old data errors should be gone
now.
* Working on ES5 upgrade & wikidata search
=== Fundraising Tech ===
* Hiring replacement for Casey, who's turning devops
* More work on new PayPal integration:
https://phabricator.wikimedia.org/T107372,
https://phabricator.wikimedia.org/T153720
* Method to purge CentralNotice banners from caches:
https://phabricator.wikimedia.org/T154954
== Wikidata ==
* Still focussing on "federation" (allow using properties and items from an
other Wikibase installation, needed for Commons) …
* … as well as support for all the freedom new entity types need (for
Wiktionary).
* Enabled search engine indexing for a few ArticlePlaceholder pages.
== German Technical Wishlist ==
* TwoColConflict betafeature deployed to dewiki
* Work on an extension to copy files to Commons from other wikis while
maintaining file & description histories
* Drafting a prototype for an improved Special:Search form
* No blockers
== RelEng ==
* Blocking
** Dashiki extension for meta wiki (Analytics)
* Blocked
** [none]
* Updates
** Logspam in current train, please help:
https://phabricator.wikimedia.org/T155527#3029895
** new version of git-fat (thanks otto!), for those that care
=== Services ===
* Blockers: none
* Updates:
** RFC Discussion *REST API language variant support*
*** https://phabricator.wikimedia.org/T154190
** RFC Discussion *Thumb API*
*** https://phabricator.wikimedia.org/T66214
** Deployed /feed/onthisday endpoint
*** request for MCS: add an x-ample in your spec so we can remove it from RB
** REST API 1.0
*** Deprecation and removal of a series of endpoints
**** will send announcement soon, deletion on May 1st
**** /page/title/ and sub-endpoints -
https://phabricator.wikimedia.org/T158100
**** /page/mobile-text - https://phabricator.wikimedia.org/T158128
** Container infrastructure requirements
****
https://docs.google.com/a/wikimedia.org/document/d/1QsCVooqxkeE6tKYTxgoRvRd…
== Editing ==
=== Parsing team ===
* No blockers
* Starting to work on audio/video support in Parsoid (VE will follow)
* Continuing work on language variant support, needed for zhwiki and others
=== Collaboration ===
* Blocking
* Blocked
* Updates
** Continued improvements to RCFilters.
*** More front-end patches have been merged, e.g. handling for conflicting
and interacting filters, AJAX support so results update immediately.
*** Back-end patch is up for review. This sends structured filter
information to the client, and changes how RecentChanges and Watchlist
filters are registered. It's still a WIP, but you might want to take a
look if your extension uses or might use such filters:
https://gerrit.wikimedia.org/r/#/c/337546/
Other changes in Collaboration-maintained extensions
** Echo
*** Fix mentions failing editing empty pages
*** Use internal diff engine
** Flow
*** Bottom-positioned toolbar
Hi all,
voting for the Developer Wishlist [1] has ended. Thanks to everyone who
participated! And extra thanks to everyone who helped. [2] You can find the
the results, with links to the full proposals, at:
https://www.mediawiki.org/wiki/User:Community_Tech_bot/WishlistSurvey/Votes
As you might have read on the wishlist page, this is a primarily-volunteer
run experiment. It can only be successful with your help! [3] If you can
afford the time, please look through the highly-voted propsals and see if
you can find something you would be interested to work on. I also invite
everyone to participate in these three follow-up tasks:
https://phabricator.wikimedia.org/T158148 - promote wishlist proposals
https://phabricator.wikimedia.org/T158149 - find owners for the top10
proposals
https://phabricator.wikimedia.org/T158150 - post-mortem
thanks
Gergő
[1] https://www.mediawiki.org/wiki/Developer_Wishlist
[2] Including, but not limited to: Srishti Sethi who handled most of the
communications; Quim Gil and James Forrester who helped a lot with the
planning; Leon Ziemba who ran the survey maintenance bot; Jeph Paul and
Jonathan Morgan who wrote the original version of the voting button gadget;
all the people working on the Community Wishlist from which I stole the
idea and most of the process/design.
[3] https://i.imgflip.com/1jnzy6.jpg
TL;DR See <https://phabricator.wikimedia.org/T136429> and try the new
jessie-migration branch.
The Wikimedia Foundation has been working on converting the majority
of its production server farm from Ubuntu to Debian for quite some
time. In August of 2016 that project advanced to the point of actually
converting the 252 servers running MediaWiki [0]. Today that work has
largely been done and the vast majority of MediaWiki is running on
Jessie.
Now it's time (past time really) for MediaWiki-Vagrant to catch up. We
have done this sort of thing once before with the switch from using
Ubuntu Precise to Ubuntu Trusty as our base image [1] in mid-2014. We
had fewer users then and only one supported virtual machine type. The
switch from Trusty to Jessie is also slightly more complicated because
the init system used is changing as well. This means that we have more
Puppet code changes to make than last time and more people who will be
impacted by the change.
For a little over a month several of us have been working on the
jessie-migration branch to get ready for the change over. I think we
are at the point that we could change over, but I want to have a
period of testing by a wider audience to see if we can iron out a few
more common issues before forcing everyone to either upgrade or
explicitly pin their MediaWiki-Vagrant virtual machines to the git tag
for the last Trusty compatible build.
== Testing the Jessie base image and Puppet profiles ==
Its recommended to test with a fresh MediaWiki-Vagrant checkout so if
things go badly you can easily switch back to your original install
and keep working.
$ git clone --recursive
https://gerrit.wikimedia.org/r/mediawiki/vagrant mwv-jessie
$ cd mwv-jessie
$ git checkout jessie-migration
$ ./setup.sh
$ vagrant up
You can run vagrant roles list -e -1 to get a nice list of the roles
you have enabled on your normal Trusty VM install to copy over to your
Jessie testing VM. This one-liner liner might even do it for you:
$ cd mwv-jessie
$ vagrant roles enable $(cd ../vagrant; vagrant roles list -e -1)
$ vagrant provision
Give things a try and report issues as children of the tracking task
for this migration [2]. Barring major issues effecting many people, I
would like to merge the jessie-migration branch with the master branch
the week of March 13th.
[0]: https://phabricator.wikimedia.org/T143536
[1]: https://phabricator.wikimedia.org/rMWVAdc73e2bee9dff1e0755d15cfe1376ee2dc6e…
[2]: https://phabricator.wikimedia.org/T136429
Bryan
--
Bryan Davis Wikimedia Foundation <bd808(a)wikimedia.org>
[[m:User:BDavis_(WMF)]] Sr Software Engineer Boise, ID USA
irc: bd808 v:415.839.6885 x6855
