Just a heads up that "Automated Security Fixes" have been disabled on the
Wikimedia GitHub org. See [1]
The reason for this is that it generates pull requests on non canonical
repositories (ie where Gerrit is the default development location) that
require developers to close them.
If this is a feature you want on your repo generally, because you
canonically develop on GitHub, you can re-enable these on your repo by
clicking the "Security" tab, and then selecting "Automated Security Fixes"
in the top right corner. See [2] for more info. If you do develop
canonically in GitHub, please let us know at [3].
Note, this doesn't affect the security alerts related to outdated packages
etc in a repo.
Thanks!
Sam
[1] https://phabricator.wikimedia.org/T237337
[2]
https://help.github.com/en/github/managing-security-vulnerabilities/configu…
[3] https://phabricator.wikimedia.org/T237470
[4]
https://help.github.com/en/github/managing-security-vulnerabilities/viewing…
📘 Read on Phabricator at
https://phabricator.wikimedia.org/phame/post/view/178/
-------
How’d we do in our strive for operational excellence last month? Read on to
find out!
## 📊 Month in numbers
* 3 documented incidents. [1]
* 33 new Wikimedia-prod-error reports. [2]
* 30 Wikimedia-prod-error reports closed. [3]
* 207 currently open Wikimedia-prod-error reports in total. [4]
There were three recorded incidents last month, which is slightly below our
median of the past two years. Explore this data: <
https://codepen.io/Krinkle/full/wbYMZK>. To read more about these
incidents, their investigations, and pending actionables; check
https://wikitech.wikimedia.org/wiki/Incident_documentation#2019
-------
## *️⃣ To Log or not To Log
MediaWiki uses the PSR-3 compliant Monolog library to send messages to
Logstash (via rsyslog and Kafka [5]). These messages are used to
automatically detect (by quantity) when the production cluster is in an
unstable state. For example, due to an increase in application errors when
deploying code, or if a backend system is failing. Two distinct issues
hampered the storing of these messages this month, and both affected us
simultaneously.
*Elasticsearch mapping limit*
The Elasticsearch storage behind Logstash optimises responses to Logstash
queries with an index. This index has an upper limit to how many distinct
fields (or columns) it can have. When reached, messages with fields not yet
in the index are discarded. Our Logstash indexes are sharded by date and
source (one for “mediawiki”, one for “syslog”, and one for everthing else).
This meant that error messages were only stored if they only contained
fields used before, by other errors stored that day. Which in turn would
only succeed if that day’s columns weren’t already fully taken. A seemingly
random subset of error messages was then rejected for a full day. Each day
it got a new chance at reserving its columns, so long as the specific kind
of error is triggered early enough.
To unblock deployment automation and monitoring of MediaWiki, an interim
solution was devised. The subset of messages from “mediawiki” that deal
with application errors now have their own index shard. These error reports
follow a consistent structure, and contain no free-form context fields. As
such, this index (hopefully) can’t reach its mapping limit or suffer
message loss.
The general index mapping limit was also raised from 1000 to 2000. For now
that means we’re not dropping any non-critical/debug messages. More
information about the incident at https://phabricator.wikimedia.org/T234564.
The general issue with accommodating debug messages in Logstash long-term,
is tracked at https://phabricator.wikimedia.org/T180051. Thanks Bartosz,
and Keith Herron.
*Crash handling*
Wikimedia’s PHP configuration has a “crash handler” that kicks in if
everything else fails. For example, when the memory limit or execution
timeout is reached, or if some crucial part of MediaWiki fails very early
on. In that case our crash handler renders a Wikimedia-branded system error
page (separate from MediaWiki and its skins). It also increments a counter
metric for monitoring purposes, and sends a detailed report to Logstash. In
migrating the crash handler from HHVM to PHP7, one part of the puzzle was
forgotten. Namely the Logstash configuration that forwards these reports
from php-fpm’s syslog channel to the one for mediawiki.
As such, our deployment automation and several Logstash dashboards were
blind to a subset of potential fatal errors for a few days. Regressions
during that week were instead found by manually digging through the raw
feed of the php-fpm channel instead. As a temporary measure, Scap was
updated to consider the php-fpm’s channel as well in its automation that
decides whether a deployment is “green”.
We’ve created new Logstash configurations that forward PHP7 crashes in a
similar way as we did for HHVM in the past. Bookmarked MW
dashboards/queries you have for Logstash now provide a complete picture
once again. Thanks Effie, and Cole White! –
https://phabricator.wikimedia.org/T234283
-------
## 📉 Outstanding reports
Take a look at the workboard and look for tasks that might need your help.
The workboard lists error reports, grouped by the month in which they were
first observed.
→ https://phabricator.wikimedia.org/tag/wikimedia-production-error/
Or help someone that’s already started with their patch:
→ https://phabricator.wikimedia.org/maniphest/query/CFzrDj3vFbE_/#R
Breakdown of recent months (past two weeks not included):
* March: 1 report fixed. (3 of 10 reports left).
* April: 8 of 14 reports left (unchanged). ⚠️
* May: (All clear!)
* June: 9 of 11 reports left (unchanged). ⚠️
* July: 13 of 18 reports left (unchanged).
* August: 2 reports were fixed! (6 of 14 reports left).
* September: 2 reports were fixed! (10 of 12 new reports left).
* October: 12 new reports survived the month of October.
-------
##### 🎉 Thanks!
Thank you, to everyone else who helped by reporting, investigating, or
resolving problems in Wikimedia production. Thanks!
Until next time,
– Timo Tijhof
🌴“Gotta love crab. In time, too. I couldn't take much more of
those coconuts. Coconut milk is a natural laxative. That's something
Gilligan never told us.”
-------
Footnotes:
[1] Incidents. –
https://wikitech.wikimedia.org/wiki/Special:PrefixIndex?prefix=Incident+doc…
[2] Tasks created. –
https://phabricator.wikimedia.org/maniphest/query/sJI9Af6LqvKL/#R
[3] Tasks closed. –
https://phabricator.wikimedia.org/maniphest/query/scW28HMEJemU/#R
[4] Open tasks. –
https://phabricator.wikimedia.org/maniphest/query/47MGY8BUDvRD/#R
[5] https://wikitech.wikimedia.org/wiki/Logstash/Interface
Today we are releasing a new dataset meant to help us understand the impact
of grants and programs on editing. This data was requested several years
ago, and we took a long time to bring in the privacy and security experts
whose help we needed to release it. With that work done, you can download
the data here: https://dumps.wikimedia.org/other/geoeditors/ and read about
it here:
https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Edits/Geoeditors/Pu…
You can send questions or comments on this thread or on the discussion page.
Hello all,
-and sorry for cross-posting-
This message is important to everyone running an instance of Wikibase
including the Query Service GUI.
We just released a new version of the Wikidata Query Service GUI. This
release is primarily to fix a security issue described in T233213
<https://phabricator.wikimedia.org/T233213> (hidden task, will be made
public soon). The fix has been successfully deployed for the Wikidata Query
Service.
In order to keep your instance safe, please make sure to update your Query
Service GUI!
Git repositories, releases and currently active version docker images also
include the latest fixed code (see links below). If you have a local test
setup using the docker-compose example then see:
https://gist.github.com/addshore/36f8d6fe2331d28ca8f70df5abda20fd
Gerrit repositories:
-
wikidata/query/gui after commit d9f964b88c01748e278ca8c4b8929a8ef0ef0267
-
wikidata/query/gui-deploy after commit
7445472ab0ec61890b42e4d524416fbc6a18aa8a
-
wikidata/query/deploy after 094d9cda98f3fb706cf9c25aefa3eb33f9f6999a
Docker images:
-
wdqs:0.3.6 (all versions of this tag)
-
wdqs:latest from digest
sha256:04237b42d0b904a2c49ecb7059c82ace8265ba0b7f690ee2d4b3004ad39517ee
-
wdqs-frontend:latest from digest
sha256:1308a7d6622b1e141783336fb52cd6993973321077f58359fbf907b77e105ca3
-
wdqs-frontend:legacy from digest
sha256:f830abd53fe5e79299011211a2aab7ad947181e56785c06eed6e9bd6b430d4ce
Downloadable releases:
-
https://archiva.wikimedia.org/repository/snapshots/org/wikidata/query/rdf/s…
If you have any questions or issues updating your code, please let us know
(you can write me an email, or ask in the Wikibase Telegram group
<https://t.me/joinchat/HGjGexZ9NE7BwpXzMsoDLA>)
Thanks for your understanding,
Cheers,
--
Léa Lacroix
Project Manager Community Communication for Wikidata
Wikimedia Deutschland e.V.
Tempelhofer Ufer 23-24
10963 Berlin
www.wikimedia.de
Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.
Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für
Körperschaften I Berlin, Steuernummer 27/029/42207.
Hi all,
Here are the minutes from this week's TechCom meeting:
* Discussed: RFC: Where to implement Desktop Improvements project
<https://phabricator.wikimedia.org/T234907>. This RFC discusses the
approach for
implementing broad UI changes to the desktop interface of Wikimedia
projects.
* Discussed: Schema change procedure documentation blanked, no alternative
given
for 8 months <https://phabricator.wikimedia.org/T237229>. This task
proposes re-adding
the procedure for schema changes to the Wikimedia development policy.
* Last call: RFC: ParallelMaintenance helper class for multi-process
maintenance scripts
<https://phabricator.wikimedia.org/T201970>. This RFC was placed on last
call on
2019-10-23 with the intent to approve on 2019-11-13. Comments in support of
or in
opposition to this RFC are welcome during the last call period.
* TechCom will not meet next week due to the 2019 Wikimedia Technical
Conference.
* No IRC meeting next week or on 2019-11-20.
You can also find our meeting minutes at
<https://www.mediawiki.org/wiki/Wikimedia_Technical_Committee/Minutes>.
See also the TechCom RFC board
<https://phabricator.wikimedia.org/tag/mediawiki-rfcs/>.
If you prefer you can subscribe to our newsletter here
<https://www.mediawiki.org/wiki/Newsletter:TechCom_Radar>.
Thanks,
- Alex
--
Alex Paskulin
Technical Writer
Wikimedia Foundation
Hi,
for HTML version see
https://www.mediawiki.org/wiki/Scrum_of_scrums/2019-11-06
Željko
--
= 2019-11-06 =
== Callouts ==
* Parsoid-PHP is about ready to be rolled out in the coming weeks for all
live traffic and will serve client requests (VE, CX, Flow, MCS for Android
app) instead of the current Parsoid/JS service. See Parsing team section of
this etherpad for more notes about testing requests. ( [[phab:T229015]] is
the tracking task )
* Product Infrastructure
** blocked by RelEng: Ensure all PI engineers have +2 rights in
operations/deployment-charts [[phab:T232794]]
** Product Infrastructure blocked by SRE: Configure Google Cloud Vision
credentials in production [[phab:T236426]]
** MachineVision extension is being enabled today, please let us know if
anyone sees any new errors on Commons
* Release Engineering
** This week train last train (1.35.0-wmf.5 - [[phab:T233853]]) is the last
train before two week break
** v3 of architecture document for new continuous integration wants review
and feedback [[User:LarsWirzenius/NewCI]]
* Language
** Blocked by Core: Preemptive refresh in getMultiWithSetCallback() and
getMultiWithUnionSetCallback() pollutes cache: [[phab:T235188]]
== Product ==
=== Editing ===
* Blocking:
** Parsing: [[phab:T229074]] -- Prepare VE for Parsoid-PHP switch ( Editing
team )
=== Growth ===
* Blocking:
** Parsing: [[phab:T229078]] -- Prepare Flow for Parsoid-PHP switch ( which
team? but, heads up Roan )
=== iOS native app ===
* Updates:
** 6.4.1 in beta/testing
***3D touch fix in article view
***Seeing crashes in beta so may wait on this until 6.5
** 6.5 in active development - [[phab:project/view/4245]]
***Wrapping up history & diffs feature - integrating live endpoints this
week
=== Android native app ===
* Updates:
** Had a minor release with an important bug fix related to 2FA login.
** Wrapped up our server side changes for Suggested Edits V3 feature and
all set to integrate with it for a major release
**Picking back momentum on mobile-html related
work:[[phab:project/view/4318]]
=== Product Infrastructure ===
* Blocked by:
** RelEng: Ensure all PI engineers have +2 rights in
operations/deployment-charts [[phab:T232794]]
** SRE: Configure Google Cloud Vision credentials in production
[[phab:T236426]]
* Blocking:
** Parsing: [[phab:T229077]] -- Prepare MCS for Parsoid-PHP switch (
Product Infrastructure team )
* Updates:
** MachineVision extension is being enabled today, please let us know if
anyone sees any new errors on Commons
=== Structured Data ===
* Blocking:
** Search Platform: Data dumps for SDC: [[phab:T221917]]
* Updates:
** machine vision released on testcommonswiki
** can now access MediaInfo items via Lua
=== Parsing ===
* Blocked by:
** [[phab:T236930]] needs resolution for using VE with Parsoid/PHP on
non-RESTBase wikis (ex: officewiki, wikitech). Need Core Platform Team to
take a look at it
** These next 4 testing / QA tasks aren't strictly a blocker yet, but will
quickly become blockers as we are aiming to roll out Parsoid/PHP for all
live traffic in the coming weeks (before Thanksgiving). Some teams are
already working on them, but just flagging them all here for completeness'
sake - thanks to you all for addressing them.
*** [[phab:T229078]] -- Prepare Flow for Parsoid-PHP switch ( which team?
but, heads up Roan )
*** [[phab:T229074]] -- Prepare VE for Parsoid-PHP switch ( Editing team )
*** [[phab:T229075]] -- Prepare CX for Parsoid-PHP switch ( Language team )
*** [[phab:T229077]] -- Prepare MCS for Parsoid-PHP switch ( Product
Infrastructure team )
* Updates:
** As of Nov 4, Parsoid/PHP cluster is receiving 100% of Parsoid/JS change
prop reparse traffic (which makes up > 90% of all traffic that Parsoid
handles). Performance, error rates are all looking good. ( [[phab:T235902]]
)
** So, we are getting ready to roll this out everywhere in the coming weeks
in a phased manner (wiki-by-wiki for *all* services). Appreciate testing
against Parsoid/PHP in the beta cluster. See description of
[[phab:T229074]] for how to test against Parsoid/PHP.
=== Language ===
* Blocked by:
** Core: [[phab:T235188]]
* Blocking:
** Parsing: [[phab:T229075]] -- Prepare CX for Parsoid-PHP switch (
Language team )
* Updates:
** Testing Parsoid-PHP switch: [[phab:T229075]]
=== Inuka ===
* Updates:
** KaiOS app Quick Facts feature: [[phab:T234619]]
== Technology ==
=== Fundraising Tech ===
* Updates:
** CiviCRM:
*** Moving more contact deduplication features from local custom hooks to a
community-installable extension
*** adding new data conflict resolution rules
*** Adding contact locking mechanism to avoid trying to deduplicate the
same contact from two different jobs
** Investigating payment processor iframe load issues
** CentralNotice: Puzzling out differences between stats from old vs new
data pipelines
** CentralNotice: reviewing sub-national targeting
** End of year summary receipt changes
=== Core Platform ===
* Blocking:
** Parsing: [[phab:T236930]] needs resolution for using VE with Parsoid/PHP
on non-RESTBase wikis (ex: officewiki, wikitech). Need Core Platform Team
to take a look at it
** Language: [[phab:T235188]]
=== Engineering Productivity ===
==== Performance ====
* Updates:
** Rolled out the new synthetic testing setup ready with a new Graphite
instance. WebPageReplay tests has moved and WebPageTest is the next thing.
All old WebPageTest dashboard will work as before but with more metrics and
data.
** Gilles presenting at Velocity Berlin today: How to make sense of real
user performance metrics
==== Release Engineering ====
* Blocking:
** Product Infrastructure: Ensure all PI engineers have +2 rights in
operations/deployment-charts [[phab:T232794]]
* Updates:
** v3 of architecture document for new continuous integration wants review
and feedback [[User:LarsWirzenius/NewCI]]
** Train Health
*** Last week: 1.35.0-wmf.4 - [[phab:T233852]]
*** This week: 1.35.0-wmf.5 - [[phab:T233853]] - last train before two week
break
*** Next week: no train because of [[Wikimedia Technical Conference/2019]]
=== Search Platform ===
* Blocked by:
** Structured Data: Data dumps for SDC: [[phab:T221917]]
=== Site Reliability Engineering ===
* Blocking:
** Product Infrastructure: Configure Google Cloud Vision credentials in
production [[phab:T236426]]
Sorry for cross-posting!
Reminder: Technical Advice IRC meeting this week **Wednesday 4-5 pm UTC**
on #wikimedia-tech.
*Note the time change due to Berlin having switched to winter time!*
Questions can be asked in English.
The Technical Advice IRC Meeting (TAIM) is a weekly support event for
volunteer developers. This week we have a special theme, related to the
upcoming Wikimedia Technical Conference focusing on Developer Productivity.
In particular we're interested in hearing about your experiences with
developer productivity in the context of your work with on-wiki tools
(gadgets, templates, modules, etc). Your input is going to be useful for
the "Developer Productivity & onwiki tooling" (
https://phabricator.wikimedia.org/T234661) session held at the Conference
next week.
Hope to see you there!
--
Leszek Manicki
Engineering Manager
Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin
Phone: +49 (0)30 219 158 26-0
http://wikimedia.de
Imagine a world in which every single human being can freely share in the
sum of all knowledge. Help us to achieve our vision!
https://spenden.wikimedia.de
Wikimedia Deutschland – Gesellschaft zur Förderung Freien Wissens e. V.
Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für
Körperschaften I Berlin, Steuernummer 27/029/42207.
(sorry for cross-posting!)
Hello all,
We’re running a session at next week’s Wikimedia Technical Conference
<https://www.mediawiki.org/wiki/Wikimedia_Technical_Conference/2019> around
the topic: Developer Productivity and onwiki tooling - userscripts,
gadgets, templates, modules <https://phabricator.wikimedia.org/T234661>.
For this we’re looking for more input from folks who won’t be at the
conference.
The goal of the conference is to identify changes to tooling and processes
to support Wikimedia developers in working more efficiently. One aspect of
that is to explore what makes it currently difficult for technical
contributors working with templates, modules, userscripts and gadgets, and
to discuss what could be improved or done differently.
It would be wonderful if you could share your experience and comment on the
following 2 questions in the phabricator ticket for the session
<https://phabricator.wikimedia.org/T234661> (or send me an email, and we’ll
add it to the phabricator ticket):
1.
What is your background, what do you do as a technical contributor?
2.
In what way your productivity as a technical contributor is affected in
the context of on-wiki tooling (what slows you down, what makes your life
complicated, what helps you …)?
To give you two examples (made-up):
1.
I am a volunteer developer and have developed several user scripts for
frwiki.
2.
When I’ve developed a userscript, I don’t know how many people are
copying the code to use my script. When I make changes to the script,
others often still have older versions. When people report bugs, I need to
first find out which version they are using which is very time-consuming.
1.
I am a developer of Wikibase extension, WMDE staff member
2.
When I develop a new feature in Wikibase, I am often informed AFTER the
feature has been released that Wikidata gadgets had been broken. Then I
need to stop my current tasks, go back to my previous work and change the
feature. This makes the process of my work on new features longer.
Many thanks for your support!
The developer productivity and onwiki tooling session crew
--
Birgit Müller (she/her)
Director of Technical Engagement
Wikimedia Foundation <https://wikimediafoundation.org/>
*Note that the time is different for this session because of Daylight
Savings time and a scheduling conflict.*
The Search Platform Team
<https://www.mediawiki.org/wiki/Wikimedia_Search_Platform> usually holds
office hours the first Wednesday of each month. Come talk to us about
anything related to Wikimedia search!
Feel free to add your items to the Etherpad Agenda for the next meeting.
Details for our next meeting:
Date: Wednesday, Nov 6th, 2019
Time: 17:00-18:00 GMT / 09:00-10:00 PST / 12:00-13:00 EST / 18:00-19:00 CST
Etherpad: https://etherpad.wikimedia.org/p/Search_Platform_Office_Hours
Google Meet link: https://meet.google.com/vyc-jvgq-dww
Hope to talk to you in a week!
—Trey
Trey Jones
Sr. Software Engineer, Search Platform
Wikimedia Foundation
UTC-4 / EDT