Hi folks,
I got this message while running a bot:
Result: 503 Backend fetch failed
I know HTTP 503 message, but I never got it during botwork. I restarted it
successfully. Is this normal?
--
Bináris
📘 Read on Phabricator at
https://phabricator.wikimedia.org/phame/post/view/172
-------
How’d we do in our strive for operational excellence in August? Read on to
find out!
## 📊 Month in numbers
* 3 documented incidents. [1]
* 42 new Wikimedia-prod-error reports. [2]
* 31 Wikimedia-prod-error reports closed. [3]
* 210 currently open Wikimedia-prod-error reports in total. [4]
The number of recorded incidents in August, at three, was below average for
the year so far. However, in previous years (2017-2018), August also has
2-3 incidents. – Explore the data at https://codepen.io/Krinkle/full/wbYMZK
To read more about these incidents, their investigations, and pending
actionables; check
https://wikitech.wikimedia.org/wiki/Incident_documentation#2019
## *️⃣ When you have eliminated the impossible...
Reports from Logstash indicated that some user requests were aborted by a
fatal PHP error from the MessageCache class. The user would be shown a
generic system error page. The affected requests didn’t seem to have
anything obvious in common, however. This made it difficult to diagnose.
MessageCache is responsible for fetching interface messages, such as the
localised word “Edit” on the edit button. It calls a “load()” function and
then tries to access the loaded information. However, sometimes the load
function would claimed to have finished its work, but yet the information
was not there.
When the load function initialises all the messages for a particular
language, it keeps track of this, so as to not do the same a second time.
>From any one angle I could look at this code, no obvious mistakes stood
out. A deeper investigation revealed that two unrelated changes (more than
a year apart), each broke 1 assumption that was safe to break. But, put
together, and this seemingly impossible problem emerges. Check out the
details of the investigation at
https://phabricator.wikimedia.org/T208897#5373846.
## 📉 Outstanding reports
Take a look at the workboard and look for tasks that might need your help.
The workboard lists error reports, grouped by the month in which they were
first observed.
→ https://phabricator.wikimedia.org/tag/wikimedia-production-error/
Or help someone that’s already started with their patch:
→ https://phabricator.wikimedia.org/maniphest/query/pzVPXPeMfRIz/#R
Breakdown of recent months (past two weeks not included):
* January: 1 report left (unchanged).
* February: 2 reports left (unchanged). ⚠️
* March: 4 reports left (unchanged). ⚠️
* April: 2 reports got fixed! (8 of 14 reports left).
* May: 4 of 10 reports left (unchanged). ⚠️
* June: 1 report got fixed! (8 of 11 reports left).
* July: 2 reports got fixed (17 of 18 reports left).
* August: 14 new reports remain unsolved.
* September: 11 new reports remain unsolved.
-------
## 🎉 Thanks!
Thank you to Aaron Schulz, Daimona, David Barratt, James Forrester, Kosta
Harlan, Piotr Miazga, Roan Kattouw, Tom Arrow, Željko Filipin, and everyone
else who helped by reporting, investigating, or resolving problems in
Wikimedia production. Thanks!
Until next time,
– Timo Tijhof
-------
Footnotes:
[1] Incidents. –
https://wikitech.wikimedia.org/wiki/Special:PrefixIndex?prefix=Incident+doc…
[2] Tasks created. –
https://phabricator.wikimedia.org/maniphest/query/8fpsoBLrmlFu/#R
[3] Tasks closed. –
https://phabricator.wikimedia.org/maniphest/query/U9.KRVNW52Yb/#R
[4] Open tasks. –
https://phabricator.wikimedia.org/maniphest/query/47MGY8BUDvRD/#R
Hi,
for HTML version go to
https://www.mediawiki.org/wiki/Scrum_of_scrums/2019-10-02
Željko
--
= 2019-10-02 =
== Callouts ==
* Rel Eng: The Growth team is planning deployments for the newcomer tasks
projects and we need to know November black out dates
* Release Engineering: REMINDER: We're at 1.34.0-wmf.24 this week. The last
branch for this release will be wmf.25 on 30 September. Teams who want to
ship things for MW 1.34 should land them now. (If you do not mark code as
deprecated in the next few days, you have to maintain it for another nine
months.)
* Looking for someone to review a new MW API module (Scoring Platform /
Jade)
* Fundraising emails to English-speaking countries started going out this
week - that traffic goes through donate.wikimedia.org (on the main cluster)
before it hits payments-wiki (on the fr cluster), so let's nobody break it.
*Structured Data team need review on a patch for [[phab:T231855]] (probably
from CPT)
== Product ==
=== Editing ===
* Updates:
** Talk pages enhancements: WIP and research ongoing for legacy signature
parsing, signature parser function, autocomplete notifications
** Fix loading icons for historical diff mode switcher
** Fix mw:MediaLink support (task [[phab:T232754]])
** mobile.editor.overlay: Use errorformat=html and display all error
messages (task [[phab:T202460]]) (task [[phab:T228897]]) (task
[[phab:T232685]])
=== Growth ===
* Updates:
** Homepage: Add startediting placeholder module for suggested edits (task
[[phab:T232419]])
** Add JS and dialog for startediting module (task [[phab:T232419]])
** Homepage: Stop querying non-existent help module storage (task
[[phab:T232548]])
** Selenium: Add regression test for NONE group survey submission (task
[[phab:T233263]])
** EditorJourney: Redact sensitive values for diff and oldid (task
[[phab:T234084]])
=== iOS native app ===
* Updates:
** 6.5 in active development - [[phab:project/view/4245]]
*** Includes history & diffs feature
**** Core Platform working on Rest APIs for this
**** wikidiff2 PR to output inline JSON merged [[gerrit:534897]]
**** followup wikidiff2 PR for section titles [[gerrit: 539906]]
=== Android native app ===
* Updates:
**Had a minor release with many bug fixes.
**Extended our User survey for suggested edits feature, by a week
**In active development of Suggested Edits V3 features
=== Product Infrastructure ===
* Updates:
** Offsite concluded last week
** Machine Vision: scripts to create image tagging candidate lists
** Machine Vision: code review
** Wikifeeds: fixed raw preq call to public endpoint that broke deployment
last week
** Wikifeeds: need to update Helm chart with new stuff and try deploying
again
** mobile-html: minor performance tweaks
** BUOD: triaged remaining issues for event platform client prototypes
** BUOD: moving event platform client prototypes into Gerrit
=== Structured Data ===
* Blocking:
** RecentChange support for SDC: [[phab:T230862]] ... working on it
* Updates:
** nothing new ... input types, MachineVision, campaigns continuing
=== UI Standardization ===
* Blocking:
** Scoring platform: [[phab:T219238]]
* Updates:
** DSG Components: this and next week finish for publish iteration
** Special:Contributions began to roll out yesterday
** Vector updates in context of [[phab:T121730]] and [[phab:T63099]] Vector
perf improvements and it uses CSS gradients now (hey, it’s 2019). Also
fixed [[phab:T4013]]
== Technology ==
=== Analytics ===
* Blocking:
** Search Platform: [[phab:T229882]]
=== Fundraising Tech ===
* Blocked by:
** Not blocked, but nice-to-have from release engineering: php7.0 docker
image for CRM tests [[phab:T230446]]
* Updates:
** CiviCRM
*** Adopted an orphan extension to fix typo-ed email domains, refactoring
it and installing it locally [[phab:T231332]]
*** Deploying first draft of automated year-end summary receipt emails for
recurring donors [[phab:T195907]]
*** Adding a UI button for on-demand send of year-end summary emails
[[phab:T221271]]
** CentralNotice
*** Found a couple last nits to pick with campaign fallback code, fixing
them [[phab:T234248]]
*** Fixing issues found in new data pipeline after first real-data tests
[[phab:T234352]]
** Payments-wiki
*** Card selection form tweaks [[phab:T231452]]
*** Revising content and amounts on form that asks one-time donors to
convert to recurring [[phab:T234059]]
=== Core Platform ===
* Blocked by:
** Release Engineering [[phab:T233092]]
* Blocking:
** Wikidata: (not sure which team is actually to call out on this, Core
Platform is our best guess): In the Data Bridge work (editing Wikidata
items from Wikipedia) we have encountered problems with Special:EntityData
serving stale (old) data. The problem is quite old: [[phab:T128486]]. We
are looking for a team/person with whom we could get some clarity on the
current state of the cache solutions on WMF cluster, and figure out how to
get [[phab:T128486]] unstuck.
** Search Platform: RecentChange support for SDC: [[phab:T230862]]
* Updates:
** RESTBagOStuff changes for 1.34
** New Echo notification storage
** Page History REST API for iOS team
** Parsoid/PHP in parsoid cluster
=== Engineering Productivity ===
==== Performance ====
* Blocked by:
** Wikidata: We believe we are actually not blocking you on
https://gerrit.wikimedia.org/r/c/mediawiki/core/+/497537 but never heard
back from you what do you think about what was said in
https://gerrit.wikimedia.org/r/c/mediawiki/core/+/497537#message-2cc44c252b….
Can we please get a clear information again if we are actually blocking you?
==== Release Engineering ====
* Blocking:
** The Growth team is planning deployments for the newcomer tasks projects
and we need to know November black out dates
** CPT [[phab:T233092]]
** Kartotherian folks: [[phab:T233316]]
** Not blocked, but nice-to-have for Fundraising tech: php7.0 docker image
for CRM tests [[phab:T230446]]
* Updates:
** REMINDER: We're at 1.34.0-wmf.25 this week. This is the last branch for
this release. Teams who want to ship things for MW 1.34 should land them
now. [[phab:T232026]] (If you do not mark code as deprecated in the next
few days, you have to maintain it for another nine months.)
** Train Health
*** Last week: 1.34.0-wmf.24 - [[phab:T220749]]
*** This week: 1.34.0-wmf.25 - [[phab:T220750]]
*** Next week: 1.35.0-wmf.1 - [[phab:T233849]]
=== Scoring Platform ===
* Blocked by:
** UI Standardization: [[phab:T219238]]
* Updates:
** Victory on WMFLabs ORES install
** Looking for someone to review MWAPI work for Jade (Core Platform? Brad
maybe?) [[phab:T199834]]
** [[phab:T232494]] (Now using Git-lfs on ORES hosts)
** [[phab:T230917]] (Blocked on improving Celery Monitoring -- Should we
use PS to count processes?)
** Kevin Bazira has started as an Associate Engineer
=== Search Platform ===
* Blocked by:
** Core Platform (or Multimedia?): RecentChange support for SDC:
[[phab:T230862]]
** Analytics: [[phab:T229882]]
* Updates:
** HighlightBuilder: Introduce a way to control the highlighting behaviors
with keywords [[phab:T190130]]
** Deferred update AutoCommitUpdate_LinksUpdate::doUpdate failed: Could not
enqueue jobs: Unable to serialize events (CirrusSearch LinksUpdate)
[[phab:T230728]]
=== Security ===
* Updates:
** Development of Privacy Review process; Anticipated completion of first
draft this month
** GRC Tool Eval
** GDPR / Data Map updates and reviews are in progress
** three new policies drafted and in review
** Parsoid-PHP, in progress - [[phab:T227209]]
== Wikimedia DE ==
=== Wikidata ===
* Blocked by:
** (not sure who, Core Platform? Performance?): In the Data Bridge work
(editing Wikidata items from Wikipedia) we have encountered problems with
Special:EntityData serving stale (old) data. The problem is quite old:
[[phab:T128486]]. We are looking for a team/person with whom we could get
some clarity on the current state of the cache solutions on WMF cluster,
and figure out how to get [[phab:T128486]] unstuck.
* Blocking:
** Performance (we actually believe we are not blocking them, read
further): We believe we are not blocking
https://gerrit.wikimedia.org/r/c/mediawiki/core/+/497537 but never heard
back from Performance team what do they think about what was said in
https://gerrit.wikimedia.org/r/c/mediawiki/core/+/497537#message-2cc44c252b….
Can we please get a clear information again if we are actually blocking you?
Hello everyone,
Wikimedia has listed seven projects in Outreachy Round 19:
https://www.outreachy.org/apply/project-selection/#wikimedia :)
One of these projects is documentation-related, one related to quality
assurance and remaining five are coding projects:
1. A system for releasing data dumps from a classifier detecting
unsourced sentences in Wikipedia, mentored by *Sam Walton, Miriam Redi
and Guilherme Gonçalves*
2. Convert Campaign pages to React, mentored by *Khyati Soneji and Sage
Ross*
3. Create command-line runner for MediaWiki maintenance tasks, mentored
by *Will Doran*
4. Create regression automated tests for Special:Homepage functionality
testing, mentored by *Elena Tonkovidova*
5. Documentation improvements to the ~20 top 100 most viewed MediaWiki
Action API pages on-wiki, mentored by *Jerop Brenda*
6. Improve MediaWiki Action API Integration Tests, mentored by *Kate
Chapman, Clara Andrew-Wani and Daniel Kinzler*
7. Improvements and User Testing of Wiki Education Dashboard Android
App, mentored by *Ujjwal Agrawal and Sage Ross*
Only applicants with an account on the Outreachy site
<https://outreachy.org> will be able to see the details of these projects.
You can learn more about the timeline, roles, and responsibilities of
participants and mentors, and ways to get in touch with the coordinators
here: https://www.mediawiki.org/wiki/Outreachy/Round_19. You can also view
these projects under "Featured projects" column here on Phabricator:
https://phabricator.wikimedia.org/project/view/4265/.
Spread the news among your friends both in the Wikimedia world and outside
and encourage them to apply :)
If you have any questions, please reach out to me!
Cheers,
Srishti
*Srishti Sethi*
Developer Advocate
Wikimedia Foundation <https://wikimediafoundation.org/>
What's the best place (wiki page) to start learning how to use the DockerHub image for MediaWiki? The image source is at https://github.com/wikimedia/mediawiki-docker and I found Docker/SIG. I understand the basic setup, but am new to Docker and don't understand how to approach Extension:Visual Editor, which seems to need a separate Docker image with NodeJS. My goal is to evaluate whether to switch from virtual machine hosting (which has been very successful) to Docker-based hosting (now that a PaaS supporting containers is now available to me). Thank you!
---
Michael Hogan
I've published a new version of Fresh. Fresh is a simple way to create
light and fast isolated contexts in your Terminal. For example, when you
need to run 'npm' commands that install and run code needed for ESLint,
Grunt or Selenium tests.
Get started at https://github.com/wikimedia/fresh
See also:
*
https://www.mediawiki.org/wiki/Manual:JavaScript_unit_testing#Getting_start…
*
https://www.mediawiki.org/wiki/Selenium/Node.js/Target_Local_MediaWiki_(Con…
Background:
Last month I wrote [1] about the risk and dangers involved with running
"npm install" and "npm test" commands as developers. In a nut shell: There
are no built-in protections. At risk are your personal data, web browser
session, and more. Interactions with 'git', 'sudo' or 'ssh' are also easy
to spy on or influence. This all in addition to the "normal" risk of
packages having undiscovered malicious (or non-malicious) security problems
in indirect dependencies that have never been audited for security by
anyone you'd know or trust. In particular, I think it is important to
understand that npm is different from Debian or PyPi in terms of social
etiquette and curation. More about that at [1].
-- Timo
[1]
https://medium.com/@timotijhof/how-to-protect-yourself-from-vulnerable-npm-…
The German Technical Wishes team is planning to implement a feature we're
calling "book referencing", which supports one level of nested references.
This makes it possible to reference the same book several times in an
article, pointing to various pages, without repeating the full citation.
A more complete description of the feature and a screenshot is available
below, please feel free to comment in Phabricator or on this thread.
https://phabricator.wikimedia.org/T234030
There has already been some community discussion, but since we're proposing
a small change to wikitext (the <ref> tag will accept a new attribute), I
thought it would be appropriate to wait for a round of technical feedback
before we begin coding.
Regards,
Adam
--
Adam Wight - Developer - Wikimedia Deutschland e.V. - https://wikimedia.de
