Hello everyone,
The second workshop on the topic of "Running Pywikibot scripts" is coming
up - it will take place on Friday, March 25th at 16:00 UTC. You can find
more details on the workshop and a link to join here: <
https://meta.wikimedia.org/wiki/Small_wiki_toolkits/Workshops#How_to_run_a_…>
[1].
This workshop will introduce participants to the bot scripts available via
the Pywikibot framework and how to use them. If you missed attending the
first one, it would be beneficial to have Pywikibot installed on your
computer before the workshop.
We look forward to your participation!
Best,
Srishti
On behalf of the SWT Workshops Organization team
[1]
https://meta.wikimedia.org/wiki/Small_wiki_toolkits/Workshops#How_to_run_a_…
*Srishti Sethi*
Senior Developer Advocate
Wikimedia Foundation <https://wikimediafoundation.org/>
I'd like to draw some attention to the current outstanding release blockers.
From https://phabricator.wikimedia.org/tag/mw-1.38-release/,
8 tasks as queried from https://phabricator.wikimedia.org/maniphest/query/2YXgSfsvECyO/#R
1. T261329 Prepare Parsoid for MW 1.38 (ideally) <https://phabricator.wikimedia.org/T261329>
2. T265518 Move Parsoid ServiceWorker.php and extension/src/Config into core <https://phabricator.wikimedia.org/T265518>
3. T302117 ZeroConf VE for MW 1.38 <https://phabricator.wikimedia.org/T302117>
> Component: Parsoid.
> Assigned to: WMF Content Transformation, formerly known as Parsing team.
If this is finished in master, it may be worth checking that relevant docs are up-to-date, and that relevant changes have made it to the branch in time (or are backported).
4. T275246 Populate rev_actor and rev_comment_id <https://phabricator.wikimedia.org/T275246>
> Component: MediaWiki-Revision-backend.
> Assigned to: WMF Platform Engineering (@tstarling).
Originally planned for MW 1.36. Last year, change 684142 <https://gerrit.wikimedia.org/r/c/mediawiki/core/+/684142> implemented the migration.
I moved it to the MW 1.39 milestone as this isn't something we can (nor should) rush through code changes alone. Per Tim's comment <https://phabricator.wikimedia.org/T275246#7512017>, we should add this to the installer/updater after we've succesfully migrated WMF's.
5. T288686 WVUI's TypeaheadSearch should work with a non-default `$wgScriptPath` <https://phabricator.wikimedia.org/T288686>
> Component: Vector 22.
> Assigned to: Readers Web.
The new Vector 22 skin isn't yet used by default, but it currently hardcodes WMF-specific configuration and is bundled the release. There is an open patch pending review.
6. T294612 Raw HTML from Language Converters' title conversion displayed <https://phabricator.wikimedia.org/T294612>
> Component: MediaWiki-Parser, MediaWiki-Language-converter.
> Assigned to: WMF Content Transformation, formerly known as Parsing team.
A regression reported October, that appears unsolved in prod (the linked example still displays raw HTML) and currently set to block the release.
7. T295187 Chinese conversion no longer work in the table of content <https://phabricator.wikimedia.org/T295187>
> Component: MediaWiki-Parser, MediaWiki-Language-converter.
> Assigned to: WMF Content Transformation (@cscott).
This appears to be resolved from my very rudimentary testing.
8. T303029 Revert ParsoidSiteConfigInit hook creation <https://phabricator.wikimedia.org/T303029>
> Component: Parsoid.
> Assigned to: WMF Content Transformation, formerly known as Parsing team.
Best,
-- Timo
Hello, all!
This is a follow-up on our earlier announcement[0] of the above.
Thanks to those who have already migrated their tool(s) from Debian Stretch
grid or are
in the process of doing this.
At the start of this process, there were 867 tools running on Stretch grid.
The current number is 821.
=== Recap ===
We are migrating away from Debian Stretch[1] to Debian Buster for all of
Toolforge servers,
and the most affected piece is the Grid Engine backend in particular.
We need to shut down all Stretch hosts before the end of support date to
ensure that
Toolforge remains a secure platform. This migration will take several
months because many people still use the Stretch hosts and our users
are working on tools in their spare time.
== What should I do? ==
You should migrate your Toolforge tool to a newer environment.
You have two options:
* migrate from Toolforge Stretch Grid Engine to Toolforge Kubernetes[2].
* migrate from Toolforge Stretch Grid Engine to Toolforge Buster Grid
Engine.[3]
== Timeline ==
* 2022-02-15: Availability of Debian Buster grid announced to community -
DONE
* 2022-03-21: Weekly reminders via email to tool maintainers for tools
still running on Stretch - IN PROGRESS
* Week of 2022-04-21:
** Daily reminders via email to tool maintainers for tools still running on
Stretch
** Switch login.toolforge.org to point to Buster bastion
* Week of 2022-05-02: Evaluate migration status and formulate plan for
final shutdown of Stretch grid
* Week of 2022-05-21: Shutdown Stretch grid
We thank all of you for your support during this migration process.
You can always reach out via any of our communication channels[4]
[0]
https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.…
[1] https://wikitech.wikimedia.org/wiki/News/Toolforge_Stretch_deprecation
[2]
https://wikitech.wikimedia.org/wiki/Help:Toolforge/Jobs_framework#Grid_Engi…
[3]
https://wikitech.wikimedia.org/wiki/News/Toolforge_Stretch_deprecation#Move…
[4]
https://wikitech.wikimedia.org/wiki/Portal:Toolforge/About_Toolforge#Commun…
Thanks.
--
Seyram Komla Sapaty
Developer Advocate
Wikimedia Cloud Services
🚂🌈Summary of 1.38.0-wmf.26 train deployment
This email is a summary of the Wikimedia production deployment of
1.38.0-wmf.26
- Conductor: Jeena Huneidi
- Backup Conductor: Ahmon Dancy
- Blocker Task: T300202 <https://phabricator.wikimedia.org/T300202>
- Current Status <https://versions.toolforge.org>
📊 Numbers
Sparklines comparing with the last 5 trains.
- 314 Patches ▅▁▄█▅
- 0 Rollbacks ▁▁▁█▁
- 1 Days of delay ▁▁▁▁█
- 6 Blockers ▁▁▂█▄
🥰 Trainbow Love 🌈 Thanks to folks who reported or resolved blockers:
- Antoine "hashar" Musso
- Amir Sarabadani
- Timo Tijhof
- Zabe
- Kosta Harlan
- Jon Robson
--
Jeena Huneidi
Software Engineer, Release Engineering
Wikimedia Foundation
The 1.39.0-wmf.1 version of MediaWiki is blocked[0].
The new version is deployed to testwikis, but can proceed no
further until these issues are resolved:
* Wikimedia\Rdbms\DBTransactionError: Transaction round stage must be
'cursory' (not 'within-rollback-callbacks') -
https://phabricator.wikimedia.org/T281451
* Wikimedia\Timestamp\TimestampException:
Wikimedia\Timestamp\ConvertibleTimestamp::getTimestamp: The timestamp
cannot be represented in the specified format -
https://phabricator.wikimedia.org/T304307
Once these issues are resolved train can resume. If these issues are
resolved on a Friday the train will resume Monday.
Thank you for your help resolving these issues!
-- Your humble train toiler
[0]. https://phabricator.wikimedia.org/T300203
[1]. <https://versions.toolforge.org/>
--
Jaime Nuche
Software Engineer III
Wikimedia Foundation <https://wikimediafoundation.org/>
<https://wikimediafoundation.org/>
The 1.38.0-wmf.26 version of MediaWiki is blocked[0].
The new version is deployed to group(s)0[1], but can proceed no
further until these issues are resolved:
* Wikimedia\Rdbms\DBUnexpectedError:
MWExceptionHandler::rollbackPrimaryChangesAndLog: Database is owned by ID
'1923155107' (got '') - https://phabricator.wikimedia.org/T303885
* Wikimedia\Rdbms\DBTransactionError: Transaction round stage must be
'cursory'
(not 'within-rollback-session') - https://phabricator.wikimedia.org/T303885
Once these issues are resolved, train can resume. If these issues are
resolved on a Friday the train will resume Monday.
Thank you for your help resolving these issues!
-- Your humble train toiler
[0]. https://phabricator.wikimedia.org/T300202
[1]. <https://versions.toolforge.org/>
--
Jeena Huneidi
Software Engineer, Release Engineering
Wikimedia Foundation
The 1.38.0-wmf.26 version of MediaWiki is blocked[0].
The new version is deployed to group(s)0[1], but can proceed no
further until these issues are resolved:
* Wikimedia\Rdbms\DBUnexpectedError:
MWExceptionHandler::rollbackPrimaryChangesAndLog: Database is owned by ID
'1923155107' (got '') - https://phabricator.wikimedia.org/T303885
* Wikimedia\Rdbms\DBTransactionError: Transaction round stage must be 'cursory'
(not 'within-rollback-session') - https://phabricator.wikimedia.org/T303885
Once these issues are resolved train can resume. If these issues are
resolved on a Friday the train will resume Monday.
Thank you for your help resolving these issues!
-- Your humble train toiler
[0]. https://phabricator.wikimedia.org/T300202
[1]. <https://versions.toolforge.org/>
--
Jeena Huneidi
Software Engineer, Release Engineering
Wikimedia Foundation
How’d we do in our strive for operational excellence last month? Read on to find out!
Incidents
3 documented incidents last month.
2022-02-01 ulsfo network <https://wikitech.wikimedia.org/wiki/Incident_documentation/2022-02-01_ulsfo…>
Impact: For 3 minutes, clients served by the ulsfo POP were not able to contribute or display un-cached pages.
2022-02-22 wdqs updater codfw <https://wikitech.wikimedia.org/wiki/Incident_documentation/2022-02-22_wdqs_…>
Impact: For 2 hours, WDQS updates failed to be processed. Most bots and tools were unable to edit Wikidata during this time.
2022-02-22 vrts <https://wikitech.wikimedia.org/wiki/Incident_documentation/2022-02-22_vrts>
Impact: For 12 hours, incoming emails to a specific recently created VRTS queue were not processed with senders receiving a bounce with an SMTP 550 Error.
See also Incident graphs <https://codepen.io/Krinkle/full/wbYMZK>.
Incident follow-up
Remember to review and schedule Incident Follow-up work <https://phabricator.wikimedia.org/project/view/4758/> in Phabricator, which are preventive measures and tech debt mitigations written down after an incident is concluded. Read about past incidents at Incident status <https://wikitech.wikimedia.org/wiki/Incident_status> on Wikitech.
Recently conducted incident follow-up:
Create a dashboard for Prometheus metrics about health of Prometheus itself. <https://phabricator.wikimedia.org/T222102>
Pitched by CDanis after an April 2019 incident, carried by Filippo (@fgiunchedi).
Improve wording around AbuseFilter messages about throttling functionality. <https://phabricator.wikimedia.org/T200036>
Originally filed in 2018. This came up last month during an incident where the wording may've led to a misunderstanding. Now resolved by @Daimona.
Exclude restart procedure from automated Elasticsearch provisioning. <https://phabricator.wikimedia.org/T290902>
There can be too much automation. Filed after an incident last September. Fixed by @RKemper.
Outstanding errors
Take a look at the workboard and look for tasks that could use your help.
→ https://phabricator.wikimedia.org/tag/wikimedia-production-error/
I skip breakdowns most months as each breakdown has its flaws. However, I hear people find them useful, so I'll try to do them from time to time with my noted caveats. The last breakdown was in the December edition <https://phabricator.wikimedia.org/phame/post/view/265/production_excellence…>, which focussed on throughput during a typical month. Important to recognise is that neither high nor low throughput is per-se good or bad. It's good when issues are detected, reported, and triaged correctly. It's also good if a team's components are stable and don't produce any errors. A report may be found to be invalid or a duplicate, which is sometimes only determined a few weeks later.
The below "after six months" breakdown takes more of that into consideration by looking at what's on the table after six months (tasks upto Sept 2021). This may be considered "fairer" in some sense, although has the drawback of suffering from hindsight bias, and possibly not highlighting current or most urgent areas.
WMF Product:
* Anti Harassment Tools (3): 1 MW Blocks, 2 SecurePoll.
* Community Tech (0).
* Design Systems (1): 1 WVUI.
* Editing Team (15): 14 VisualEditor, 1 OOUI.
* Growth Team (13): 11 Flow, 1 GrowthExperiments, 1 MW Recent changes.
* Language Team (6): 4 ContentTranslation, 1 CX-server, 1 Translate extension.
* Parsoid Team (9): 8 Parsoid, 1 ParserFunctions extension .
* Product Infrastructure: 2 JsonConfig, 1 Kartographer, 1 WikimediaEvents.
* Reading Web (0).
* Structured Data (4): 2 MW Uploading, 1 WikibaseMediaInfo, 1 3D extension.
WMF Tech:
* Data Engineering: 1 EventLogging.
* Fundraising Tech: 1 CentralNotice.
* Performance: 1 Rdbms.
* Platform MediaWiki Team (19): 4 MW-Page-data, 1 MW-REST-API, 1 MW-Action-API, 1 MW-Snapshots, 1 MW-ContentHandler, 1 MW-JobQueue, 1 MW-libs-RequestTimeout, 9 Other.
* Search Platform: 1 MW-Seach.
* SRE Service Operations: 1 Other.
WMDE:
* WMDE-Wikidata (7): 5 Wikibase, 2 Lexeme.
* WMDE-TechWish: 1 FileImporter.
Other:
* Missing steward (7): 2 Graph, 2 LiquidThreads, 2 TimedMediaHandler, MW Contributions 1 page.
* Individually maintained (2): 1 WikimediaIncubator, 1 Score extension.
Trends
In February, we reported 25 new production errors <https://phabricator.wikimedia.org/maniphest/query/1B79KZ8KkRj6/#R>. Of those, 13 have since been resolved, and 12 remain open as of today (two weeks into the following month). We also resolved 22 errors that remained open from previous months. The overall workboard has grown slightly to a total of 301 outstanding error reports.
For the month-over-month graph, refer to the spreadsheet. <https://docs.google.com/spreadsheets/d/e/2PACX-1vTrUCAI10hIroYDU-i5_8s7pony…>
Thanks!
Thank you to everyone who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!
Until next time,
– Timo Tijhof
🔗 Share or read online via https://phabricator.wikimedia.org/phame/post/view/267/