Hello,
If you missed the previous updates, there's the first
<https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/…>and
the second
<https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/…>one.
This is the last one I'm sending but in a good way.
We now finished migrating all 57 core tables to abstract schema. Now tables.sql
is empty for MySQL
<https://gerrit.wikimedia.org/g/mediawiki/core/+/c60ccf4e6d4932dddc2efd72a8a…>.
We will soon remove the tables.sql files and links to them.
We have also cleaned more than hundreds of old schema change files.
<https://phabricator.wikimedia.org/T272199> That enabled us to actually
look for unused sql files and drop tens of unused ones that have not been
used since 2002 <https://gerrit.wikimedia.org/r/c/mediawiki/core/+/668552>,
2004 <https://gerrit.wikimedia.org/r/c/mediawiki/core/+/668761>, or 2005
<https://gerrit.wikimedia.org/r/c/mediawiki/core/+/670176> (and much more)
and they got lost in the sheer number of our sql patch files.
The next update is exciting for me. With abstraction in place, we can now
have a proper tracking of drifts between schema in paper and our production
(This is a follow up from a major incident in 2018). Now we have
https://drift-tracker.toolforge.org/ that keeps track of these drifts. Our
schema has been around for more than twenty years and we have hundreds of
database hosts, making sure everything is using the right database schema
(and stays correct) is impossible manually and we have been finding and
fixing these drifts since 2018, see this comment onwards
<https://phabricator.wikimedia.org/T104459#4314828>.). Of course more work
in improving the tracker is welcome (here's the list
<https://phabricator.wikimedia.org/project/board/5350/>)
If you want to enjoy the benefits of abstract schema [1] in extension(s)
you or your team maintains, Please abstract the schema of your extension.
There's a long list of WMF deployed extensions that are not using abstract
schema <https://phabricator.wikimedia.org/T261912> and some even already
have a patch that only needs reviewing. Once that's done, we can add that
to drift tracking and have a more comprehensive list of potential issues.
If you need help with the abstraction work, just ping me.
This also helped us resolve several long-standing tickets like T104459
<https://phabricator.wikimedia.org/T104459> (5.5 years old), T62962
<https://phabricator.wikimedia.org/T62962> (7 years old), and T42626
<https://phabricator.wikimedia.org/T42626> (soon reaching its ninth
birthday) and will help us to address even more tech debt in future.
There is more to be done, improving the abstract schema *change* system,
finding a home of schema documentation, improving the drift tracker and
making it more automated, so much more. But the biggest chunk of work is
now finally done.
I really would like to thank Ammarpad for great work on abstracting the
tables and handling all sorts of edge cases, James Forrester and Tgr for
their reviews which without them this wouldn't be possible and Sam Reed who
wrote a script to speed up migration
<https://github.com/Ladsgroup/db-analyzor-tools/blob/master/db_abstractor.py>.
This was a team work to its core.
[1] In more details, by abstracting you will have automated checks for
dirfits of the schema of extension(s) you maintain and production. You will
have Postgres support for free. Also, you can have automated documentation
generation, ability to test the schema itself, and have better consistency
of your data types (like one datatype for timestamps).
Until the next adventure.
--
Amir (he/him)
Oh, hi there
tl;dr: we’re still moving MediaWiki into Kubernetes in Wikimedia
Production; there are resources on Wikitech[0] for folks interested in
learning more about the specifics of our Kubernetes pipeline.
⏳ Progress
Kubernetes is the future deployment platform for most Wikimedia code.
The developer and deployment experience under Kubernetes will
(hopefully) be easier and safer and developers will have more
autonomy.
The first Wikimedia production service (Mathoid) moved to Kubernetes
in July, 2018. Today there are 25 services using our production
Kubernetes infrastructure serving 30,000 requests per second, about
67-ish% of our overall traffic. All services are deployed by their
service owners without much oversight.
Our Docker registry[1] contains 311 Docker images with 7,690 tags. Our
production images are built via Continuous Integration (CI), using our
deployment pipeline. Blubber[2] and PipelineLib[3] were introduced in
2018 to allow developers to specify how they wanted their CI to run
and when to publish images to our shared registry.
Now we’re in the process of applying all we’ve learned from our
service migration to the MediaWiki migration to Kubernetes.
🏆 Goal
* Open Container Initiative (OCI)-compatible MediaWiki images
* Built, tested, and promoted by the image pipeline
* Deployed to Kubernetes via Helm
* Serving production traffic
🧐 Why
* Development - Standard platform for development and production
* Deployment - Safer, simpler, industry standard
* Production - Increased capacity, redundancy, reliability
👏 Now
* We’re building Docker images and Helm charts
** Release Engineering crammed all 184 branched extensions, 4 skins,
vendor, and core Mediawiki code into a single image alongside our
massive localization of 435 languages
** ServiceOps is working on php-fpm, apache, nutcracker, mcrouter and
envoy images and their corresponding helm charts
* We’re making our Kubernetes better
** ServiceOps has upgraded our base images and Kubernetes cluster
🔜 Next
* Figuring out how deployments work
** Backports, security releases, and train
* Figuring out how image upgrades work
** Is a php-fpm upgrade a deployment now? What does that look like?
* Figuring out how our Wikimedia configuration changes
** Currently a change to configuration requires a code change, does
this code change kick off a whole image build?
🎓 Resources
Kubernetes will be very impactful in our production services and we
would like to encourage those interested in this change to become
familiar with its concepts.
Please have a look at our collected Wikitech Kubernetes education
resources, tutorials, and guides[0]. Add material if you have had a
good experience with a class or tutorial.
Otherwise stay tuned and watch that page for additional resources to
be added. You will hear from us again as we have additional things to
report.
– <3
Tyler Cipriani (he/him) (On behalf of all the fine folks working on
Kubernetes for MediaWiki)
Engineering Manager, Release Engineering
Wikimedia Foundation
[0]: <https://wikitech.wikimedia.org/wiki/Kubernetes/Kubernetes_Education>
[1]: <https://docker-registry.wikimedia.org>
[2]: <https://wikitech.wikimedia.org/wiki/Blubber>
[3]: <https://wikitech.wikimedia.org/wiki/PipelineLib>
*subject line reference: https://en.wikipedia.org/wiki/The_More_You_Know
Hello,
On Tuesday June 1st at 7:00 UTC we will switch Gerrit from Java 8 to
Java 11. The system will be briefly unavailable has it restarts with the
JVM.
The window would be roughly half an hour, the service interruption
should be just a few minutes. If anything is suspicious we will roll
back to Java 8 and restart which would lead another short interruption.
The task is: https://phabricator.wikimedia.org/T268225
--
Antoine "hashar" Musso
= 2021-05-26=
== Callouts ==
* The Structured Data team will be removing MediaSearch UI code from
WikibaseMediaInfo soon. If you want to look at or use our reusable Vue
components, you can find them in the new MediaSearch extension:
https://gerrit.wikimedia.org/r/admin/repos/mediawiki/extensions/MediaSearch
* No train week of 2021-05-31
== Gerrit patches or GitHub Pull Requests for reviews or feedback ==
*
=== No updates ===
CommTech, AHT, Editing, iOS, Android, Prod Infra, Parsing, Library,
Analytics, Cloud Services, Security, WMDE Technical Wishes
== SoS Meeting Bookkeeping ==
* Updates: today is the last instance of this meeting in its current form
== Product ==
=== Growth ===
* Blocked by:
* Blocking:
* Thank yous:
* Updates:
** Continuing work on Add Link https://wikitech.wikimedia.org/wiki/Add_Link
*** planning to enable for 50% of new users on pilot wikis (arwiki, bnwiki,
cswiki, viwiki) this week
** Continuing work on on-wiki configuration
https://www.mediawiki.org/wiki/Growth/Community_configuration
*** planning to enable on pilot wikis this week
=== Web ===
* Blocked by:
* Blocking:
* Thank yous:
** Huge thanks to Trey Jones for their help with the highlighting of query
highlighting in the WVUI typeahead-search component:
https://phabricator.wikimedia.org/T281797
* Updates:
** Instrument QA for the language switcher treatment A/B test:
https://phabricator.wikimedia.org/T280770
** Continuing work on on the new user links (personal tools) treatment:
https://phabricator.wikimedia.org/T266536
=== Structured Data ===
* Blocked by:
* Blocking:
* Thank yous:
* Updates:
** New MediaSearch extension is on production, will disable WBMI version of
the UI this week
** We will soon remove our Vue code from WBMI, so if you want to look at or
use our reusable components, you can find them in the MediaSearch
extension:
https://gerrit.wikimedia.org/r/admin/repos/mediawiki/extensions/MediaSearch
=== Abstract Wikipedia ===
* Blocked by:
** None.
* Blocking:
** None known.
* Thank yous:
** Thanks to Jen Wright for help with the TDMP process alignment.
* Updates:
** Close to finishing Phase ε (epsilon):
https://meta.wikimedia.org/wiki/Abstract_Wikipedia/Phases
** Security & Performance reviews scheduled to end this Phase; our thanks
to them!
=== Language ===
* Blocked by:
* Blocking:
* Thank yous:
** Big thanks to the DBAs (Jaime and Manuel) for helping to find the cause
for https://phabricator.wikimedia.org/T282905
** To Antoine "hashar" Musso for quickly fixing:
https://phabricator.wikimedia.org/T283665 that was failing ULS patches from
being merged
* Updates:
** No major updates
=== Inuka ===
* Blocked by:
* Blocking:
* Thank yous:
* Updates:
** Icon / Banner for Wikipedia Preview Wordpress Plugin:
https://phabricator.wikimedia.org/T283623
** High memory consume issue for KaiOS v1.5.0 on Jio phone:
https://phabricator.wikimedia.org/T283725
=== Vue.js ===
* Blocked by:
* Blocking:
* Thank yous:
* Updates:
** Revewing feedback from the Technical Desicion Making forum on adding a
build step to mediawiki, working on an additional proof of concept
** Radio component is now available in WVUI:
https://doc.wikimedia.org/wvui/master/ui/?path=/story/components-radio--rad…
** Evaluating WVUI icon system: https://phabricator.wikimedia.org/T282625
** Evaluating the use of TypeScript in WVUI:
https://phabricator.wikimedia.org/T282835
** Preparing to cut a new release of WVUI
== Technology ==
=== Fundraising Tech ===
[From last week.]
* Blocked by:
* Blocking:
* Thank yous:
* Updates:
** Working on new API integration for backup card processor
** Moving lots of CiviCRM code off drupal 7
=== Engineering Productivity ===
==== Quality and Test Engineering ====
[From last week.]
* Blocked by:
* Blocking:
* Thank yous:
* Updates:
** Blog post by Željko Filipin: macOS vs Ubuntu Linux - `npm ci` & `npm run
selenium-test` with and without Fresh
https://phabricator.wikimedia.org/phame/post/view/241/macos_vs_ubuntu_linux/
==== Release Engineering ====
* Blocked by:
**
* Blocking:
**
* Thank yous:
** Thank you to Zabe for lots of quick fixes for train blockers.
** Thank you to James Forrester for code reviews.
* Updates:
** [All] Deployments/Covid-19
https://wikitech.wikimedia.org/wiki/Deployments/Covid-19
** Train Health
*** Last week: 1.37.0-wmf.6 [[phab:T281147]] <!--
https://phabricator.wikimedia.org/T281147 -->
*** This week: 1.37.0-wmf.7 [[phab:T281148]] <!--
https://phabricator.wikimedia.org/T281148 -->
*** Next week: no train
https://www.mediawiki.org/wiki/Scrum_of_scrums/2021-05-26
=== Search Platform ===
[From last week.]
* Blocked by:
* Blocking:
* Thank yous:
** Alex, Effie and Janis for helping navigate deploying Flink to k8s and
for being flexible and finding creating solutions!
* Updates:
** Service implementation for relforge100[34] -
https://phabricator.wikimedia.org/T262211
** Generate SSL certification for relforge1003.eqiad.wmnet and
relforge1004.eqiad.wmnet - https://phabricator.wikimedia.org/T275885
** Expose wdqs1009 to wdqs users and gather feedback -
https://phabricator.wikimedia.org/T266470
** Reboot elasticsearch* and relforge* to apply kernel security updates -
https://phabricator.wikimedia.org/T280563
** Unpack Spanish Elasticsearch Analyzer -
https://phabricator.wikimedia.org/T277699
** Cleanup duplicate indices in cloudelastic -
https://phabricator.wikimedia.org/T279009
** SonarQube should analyze all Search Platform projects -
https://phabricator.wikimedia.org/T264877
** Transfer one codfw wdqs-internal host over to codfw wdqs (public) -
https://phabricator.wikimedia.org/T281498
=== Site Reliability Engineering ===
[From last week.]
* Blocked by:
** None
* Blocking:
** None
* Thank yous:
* Updates:
** Shellbox deployment ongoing
** new caching PoP (point of presence - 2nd EMEA site) named drmrs buildout
to begin, puppet patches started trickling in
== Cross-cutting ==
* Blocked by:
** [long term] Search Platform: PHP 8.0 work is long-term blocked on the
migration to ElasticSearch 7.0 https://phabricator.wikimedia.org/T263142
(or at least 6.7).
* Blocking:
** None known.
* Thank yous:
** Special thanks to bd808 and legoktm for their work helping with the
freenode crisis.
* Updates:
** Lots of energy spent on the freenode -> Libera migration; disruption not
ideal. :(
** REL1_36 release process:
*** 1.36.0 should be released soon. Thanks to all who have helped.
** CI tools:
*** mediawiki-codesniffer v36 upgrade completed. Thanks to everyone who has
made this happen, especially Umherirrender.
*** CI tools' upgrade status:
https://libraryupgrader2.wmcloud.org/status?branch=master
** PHP 8.0:
*** No changes this week.
*** Our target remains REL1_35 (and thus also REL1_36), as well as master.
*** Upstream libraries: Elastica-related PHP code is theoretically the last
one.
*** Core: Some unit and integration tests still fail; thank you to everyone
working on fixing them.
[[Category:Scrum of scrums{{#translation:}}|*]]
Hi All
There will be no train 2021-05-31 (2021-05-31–2021-06-04) — having our
Engineering Productivity offsite.
There is a long-term calendar of upcoming known deployment disruptions
available on Wikitech:
https://wikitech.wikimedia.org/wiki/Deployments/Yearly_calendar
Thanks!
Tyler Cipriani (he/him)
Engineering Manager, Release Engineering
Wikimedia Foundation
[X-posting from ops-l]
Hi all,
Starting today, we are building our base container images using
debuerreotype instead than bootstrap-vz, which is unmaintained [1]. This is
the same tool that is used for the dockerhub debian images, and our images
are now completely equivalent to the debian base images, plus our own apt
configuration[2].
With this change, we're also introducing a simpler nomenclature for our
base images:
we will tag our images with "$codename" instead than with
"wikimedia-$codename". Thus:
- the base stretch image is now docker-registry.wikimedia.org/stretch
- the base buster image is now docker-registry.wikimedia.org/buster
We have also added a new image based on the (yet unreleased, caveat emptor)
debian bullseye.
We will keep tagging the latest version of those images as
"wikimedia-stretch" and "wikimedia-buster" for the time being, in order to
allow for backwards compatibility, but we encourage everyone to migrate
eventually to the new naming.
Cheers,
Giuseppe
[1] https://phabricator.wikimedia.org/T281984
[2] Our very simple build script is here:
https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/hea…
--
Giuseppe Lavagetto
Principal Site Reliability Engineer, Wikimedia Foundation
tl;dr:
* Wikimedia IRC channels are moving from Freenode to Libera.Chat.
* Register an account on Libera.Chat and join us there!
There has been a lot of activity over the past week related to
staffing changes on the Freenode IRC network [0]. The Wikimedia IRC
Group Contacts (GCs) [1] evaluated the situation and decided that
moving the Wikimedia IRC channels from Freenode to the brand new
Libera.Chat IRC network [2] would be the best course of action [3].
So, we are moving!
There is a new subpage on Meta-Wiki [4] for information on how to create
a new account for yourself on Libera.Chat and other related information.
Earlier we had planned on supporting the Freenode channels for longer
with bridges across the networks, but supposedly "draft" changes to
Freenode policies and recent actions by their new staff have pushed us
to close our channels and fully leave their network. We need help with
updating all references to point to the Libera Chat channels instead -
this work is being tracked as [5].
[0]: https://www.kline.sh/
[1]: https://meta.wikimedia.org/wiki/IRC/Group_Contacts
[2]: https://libera.chat/
[3]: https://meta.wikimedia.org/w/index.php?diff=21476411
[4]: https://meta.wikimedia.org/wiki/IRC/Migrating_to_Libera_Chat
[5]: https://phabricator.wikimedia.org/T283247
-- Legoktm
Hello all,
The Wikimedia Hackathon
<https://www.mediawiki.org/wiki/Wikimedia_Hackathon_2021> is taking place
right now, and you can still join us spontaneously!
First of all, we are happy to present our online venue, a spatialized chat
where you can walk around, spontaneously interact with other participants,
and discover the various surprises or the map that we designed specially
for the hackathon. You can find more information here
<https://www.mediawiki.org/wiki/Wikimedia_Hackathon_2021/Discussions#Online_…>,
and join us at any time to work on your projects or just socialize with
other Wikimedians.
During the hackathon, we have a schedule of sessions
<https://www.mediawiki.org/wiki/Wikimedia_Hackathon_2021/Schedule> running:
the second day is now over, and you can watch the main track in replay here
<https://www.youtube.com/watch?v=VujfdGnUnp0&ab_channel=MediaWiki>. On
Sunday, more sessions will take place in the hacking rooms, you can still
join or even schedule your own discussions.
If you would like to work on a project, have a look at the Phabricator board
<https://phabricator.wikimedia.org/project/board/5303/>, where you can
create new tasks or join existing projects.
Finally, a showcase will take place Sunday at 14:00 UTC
<https://www.mediawiki.org/wiki/Wikimedia_Hackathon_2021/Showcase>. If you
are working on a project, feel free to add it in the related Etherpad. You
are also welcome to watch the presentations.
As usual, for questions, feel free to use the talk page
<https://www.mediawiki.org/wiki/Talk:Wikimedia_Hackathon_2021> or our social
channels
<https://www.mediawiki.org/wiki/Wikimedia_Hackathon_2021/Discussions>.
Have a great time at the hackathon,
For the coordination team,
--
Léa Lacroix
Community Engagement Coordinator
Wikimedia Deutschland e.V.
Tempelhofer Ufer 23-24
10963 Berlin
www.wikimedia.de
Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.
Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für
Körperschaften I Berlin, Steuernummer 27/029/42207.
This is a summary of this week's deployment of the 1.37.0-wmf.6 branch
of MediaWiki and its extensions (also known as "the train"). The
primary person in charge this week is Antoine Musso, with Ahmon Dancy
as backup, both from the Release Engineering team.
The summary task for this week is
https://phabricator.wikimedia.org/T281147
The new version is running all sites. There were a total of four
blockers reported and resolved:
- T282825 - PHP Deprecated: Use of RevisionListBase::$title was deprecated
in MediaWiki 1.37. [Called from RevDelRevisionItem::getHTML]
https://phabricator.wikimedia.org/T282825
- T283167 - InvalidArgumentException: Unable to normalize the provided
actor name x.y.z.v/16
https://phabricator.wikimedia.org/T283167
- T283170 - Special:RecentChanges in it.wikiversity dies with an internal
error
https://phabricator.wikimedia.org/T283170
- T283196 - TypeError: Argument 2 passed to UploadStash::__construct() must
implement interface MediaWiki\User\UserIdentity or be null, boolean given,
called in
/srv/mediawiki/php-1.37.0-wmf.6/includes/upload/UploadFromStash.php on line
66
https://phabricator.wikimedia.org/T283196
As usual, a whole bunch of people helped to find, triage, analyze,
fix, or work around problems this. Release Engineering thanks
everyone, without help we wouldn't be able to deploy MediaWiki.
- Zabe
- Tim Starling
- James Forrester
- DannyS712
- Petr Pchelko
- Valerio Bozzolan
- Giacomo Alessandroni
- Sam Reed
- Daniel Kinzler
- Amir Sarabadani
- Bartosz Dziewoński
- Timo Tijhif
There may have been other people, and if so, I apologize for not
including them on the list above.
Have a good weekend. Be well. Be safe.
For more information, please see:
- https://phabricator.wikimedia.org/T281147
- https://wikitech.wikimedia.org/wiki/Heterogeneous_deployment/Train_deploys
- https://versions.toolforge.org/
--
Jeena Huneidi
Software Engineer, Release Engineering
Wikimedia Foundation