Hey all,
This is a quick note to highlight that in six weeks' time, the REL1_39
branch will be created for MediaWiki core and each of the extensions and
skins in Wikimedia git, with some (the 'tarball') included as sub-modules
of MediaWiki itself[0]. This is the first step in the release process for
MediaWiki 1.39, which should be out in November 2022, approximately
six months after MediaWiki 1.38.
The branches will reflect the code as of the last 'alpha' branch for the
release, 1.39.0-wmf.28, which will be deployed to Wikimedia wikis in the
week beginning 12 September 2022 for MediaWiki itself and those extensions
and skins available there.
After that point, patches that land in the main development branch of
MediaWiki and its bundled extensions and skins will be slated for the
MediaWiki 1.40 release unless specifically backported[1].
If you are working on a new feature that you wish to land for the release,
you now have a few days to finish your work and land it in the development
branch; feature changes should not be backported except in an urgent case.
If your work might not be complete in time, and yet should block release
for everyone else, please file a task against the `mw-1.39-release` project
on Phabricator.[2]
If you have tickets that are already tagged for `mw-1.39-release`, please
finish them, untag them, or reach out to get them resolved in the next few
weeks.
We hope to issue the first release candidate, 1.39.0-rc.0, two weeks after
the branch point, and if all goes well, to release MediaWiki 1.39.0 a few
weeks after that.
Tyler Cipriani (he/him)
Engineering Manager, Release Engineering
Wikimedia Foundation
[0]: <https://www.mediawiki.org/wiki/Bundled_extensions_and_skins>
[1]: <https://www.mediawiki.org/wiki/Backporting_fixes>
[2]: <https://phabricator.wikimedia.org/project/board/5694/>
Hello,
We have a proposal to make small adjustments to the MediaWiki database
policy which would make abstract schema an explicit requirement.
Abstract schema was approved by TechCom in
https://phabricator.wikimedia.org/T191231 and now core and all WMF-deployed
extensions have migrated to abstract schema.
You can find the proposal in here. Please take a look and comment:
https://www.mediawiki.org/wiki/Talk:MediaWiki_database_policy
If there are no major objections by the end of August 2022, I will codify
this into the policy.
Best
--
*Amir Sarabadani (he/him)*
Staff Database Architect
Wikimedia Foundation <https://wikimediafoundation.org/>
Hi,
Tomorrow at the HOPE 2022 conference, I'm giving a talk titled, "How to
Run a Top-10 Website, Publicly and Transparently", discussing the impact
of transparency in Wikimedia's technical spaces. A number of people have
expressed interest in watching, including non-technical users, so I'm
advertising it a bit more broadly.
I apologize for the short notice, I didn't realize the stream would be
free to watch until yesterday (thanks Ori!).
Time: 2022-07-23 17:00 UTC (1pm ET) -
https://zonestamp.toolforge.org/1658595637
Stream: https://hope.net/416dac.html
If you can't watch it live, a recording will be uploaded later on.
I've documented all of this on-wiki, including the full abstract:
<https://meta.wikimedia.org/wiki/User:Legoktm/HOPE_2022>.
I am of course happy to answer any questions people might have after the
talk!
Thanks,
-- Kunal / Legoktm
Hello everyone,
The sixth workshop on the topic of "How to maintain bots" is coming up - it
will take place on Friday, July 29th at 16:00 UTC. You can find more
details on the workshop and a link to join here: <
https://meta.wikimedia.org/wiki/Small_wiki_toolkits/Workshops#How_to_mainta…>
[1].
This session will focus on best practices for maintaining bots and tools in
the Wikimedia ecosystem. It will cover a few practices that can help
developers run a bot or a tool with help from others, such as picking a
license, adding co-maintainers to the project, publishing source code,
writing docs, and much more.
To participate in this workshop, you would need basic familiarity with bots
or tools development. You can add your discussion ideas in the etherpad doc
linked from the workshops page.
We look forward to your participation!
Best,
Srishti
On behalf of the SWT Workshops Organization team
[1]
https://meta.wikimedia.org/wiki/Small_wiki_toolkits/Workshops#How_to_mainta…
*Srishti Sethi*
Senior Developer Advocate
Wikimedia Foundation <https://wikimediafoundation.org/>
Hello all,
The Wikimania Hackathon is August 12-14, just over 2 weeks away! Details
are listed below.
Registration
Registration for the Wikimania Hackathon has now opened [1]! To
register, submit
your information to the Wikimania organizers, which will give you access to
the Wikimania platform [2]. You can also optionally add your name to
the participants
page on the Wikimania wiki [3].
Platform
The Hackathon will take place virtually on Pheedloop, the Wikimania
platform [4]. This platform complies with WCAG 2.1 AA, and will support
screen readers, font adjustments, and many other accessibility features.
Video sessions will be held in Jitsi through this platform.
Format of the event
The Hackathon consists of events spread over three days [5]:
-
On the first day, there will be a pre-Hacking showcase to share project
ideas and find collaborators. Anyone can present a project, and anyone can
come as an observer.
-
Throughout the next two days, there will be open hacking, social events,
and technical sessions. Anyone can offer a session; just claim a slot on
the schedule!
-
Finally, there will be a final showcase to share the projects worked on
during the Hackathon.
Preparing for the Hackathon
There are many ways to take part in the event. Think about what project you
might want to work on (see examples from past Hackathons [6]), add your
idea to Phabricator [7], and consider presenting at the pre-Hacking
showcase. Host a session by adding information to the schedule [5]. Check
out information for newcomers [8]. And don’t forget to register [2]!
Best wishes,
Haley and the Developer Advocacy Team
[1] https://wikimania.wikimedia.org/wiki/Hackathon
[2] https://pheedloop.com/register/wikimania2022/attendee/
[3] https://wikimania.wikimedia.org/wiki/Hackathon/Participants
[4]
https://diff.wikimedia.org/2022/07/20/the-platform-powering-wikimania-2022/
[5] https://wikimania.wikimedia.org/wiki/Hackathon/Schedule
[6] https://www.mediawiki.org/wiki/Wikimedia_Hackathon_2022/Showcase
[7] https://phabricator.wikimedia.org/project/board/6030/
[8] https://wikimania.wikimedia.org/wiki/Hackathon/Newcomers
(If you don’t work with links tables such as templatelinks, pagelinks and
so on, feel free to ignore this message)
TLDR: The schema of links tables (starting with templatelinks) will change
to have numeric id pointing to linktarget table instead of repeating
namespace and title.
Hello,
The current schema and storage of most links tables are: page id (the
source), namespace id of the target link and title of the target. For
example, if a page with id of 1 uses Template:Foo, the row in the database
would be 1, 6, and Foo (Template namespace has id of 6)
Repeating the target’s title is not sustainable, for example more than half
of Wikimedia Commons database is just three links tables. The sheer size of
these tables makes a considerable portion of all queries slower, backups
and dumps taking longer and taking much more space than needed due to
unnecessary duplication. In Wikimedia Commons, on average a title is
duplicated around 100 times for templatelinks and around 20 times for
pagelinks. The numbers for other wikis depend on the usage patterns.
Moving forward, these tables will be normalized, meaning a typical row will
hold mapping of page id to linktarget id instead. Linktarget is a new table
deployed in production and contains immutable records of namespace id and
string. The major differences between page and linktarget tables are: 1-
linktarget values won’t change (unlike page records that change with page
move) 2- linktarget values can point to non-existent pages (=red links).
The first table being done is templatelinks, then pagelinks, imagelinks and
categorylinks will follow. During the migration phase both values will be
accessible but we will turn off writing to the old columns once the values
are backfilled and switched to be read from the new schema. We will
announce any major changes beforehand but this is to let you know these
changes are coming.
While the normalization of all links tables will take several years to
finish, templatelinks will finish in the next few months and is the most
pressing one.
So if you:
-
… rely on the schema of these tables in cloud replicas, you will need to
change your tools.
-
… rely on dumps of these tables, you will need to change your scripts.
Currently, templatelinks writes to both data schemes for new rows in most
wikis. This week we will start backfilling the data with the new schema but
it will take months to finish in large wikis.
You can keep track of the general long-term work in
https://phabricator.wikimedia.org/T300222 and the specific work for
templatelinks in https://phabricator.wikimedia.org/T299417. You can also
read more on the reasoning in https://phabricator.wikimedia.org/T222224.
Thanks
--
*Amir Sarabadani (he/him)*
Staff Database Architect
Wikimedia Foundation <https://wikimediafoundation.org/>
How are we doing in our strive for operational excellence? Read on to find out!
Incidents
There were 6 incidents in June this year. That's double the median of three per month, over the past two years (Incident graphs <https://codepen.io/Krinkle/full/wbYMZK>).
2022-06-01 cloudelastic <https://wikitech.wikimedia.org/wiki/Incidents/2022-06-01_Lost_index_in_clou…>
Impact: For 41 days, Cloudelastic was missing search results about files from commons.wikimedia.org.
2022-06-10 overload varnish haproxy <https://wikitech.wikimedia.org/wiki/Incidents/2022-06-10_overload_varnish_h…>
Impact: For 3 minutes, wiki traffic was disrupted in multiple regions for cached and logged-in responses.
2022-06-12 appserver latency <https://wikitech.wikimedia.org/wiki/Incidents/2022-06-12_appserver_latency>
Impact: For 30 minutes, wiki backends were intermittently slow or unresponsive, affecting a portion of logged-in requests and uncached page views.
2022-06-16 MariaDB password <https://wikitech.wikimedia.org/wiki/Incidents/2022-06-16_MariaDB_password_l…>
Impact: For 2 hours, a current production database password was publicly known. Other measures ensured that no data could be compromised (e.g. firewalls and selective IP grants).
2022-06-21 asw-a2-codfw power <https://wikitech.wikimedia.org/wiki/Incidents/2022-06-21_asw-a2-codfw_accid…>
Impact: For 11 minutes, one of the Codfw server racks lost network connectivity. Among the affected servers was an LVS host. Another LVS host in Codfw automatically took over its load balancing responsibility for wiki traffic. During the transition, there was a brief increase in latency for regions served by Codfw (Mexico, and parts of US/Canada).
2022-06-30 asw-a4-codfw power <https://wikitech.wikimedia.org/wiki/Incidents/2022-06-30_asw-a4-codfw_accid…>
Impact: For 18 minutes, servers in the A4-codfw rack lost network connectivity. Little to no external impact.
Incident follow-up
Recently completed incident follow-up:
Audit database usage of GlobalBlocking extension <https://phabricator.wikimedia.org/T307648>
Filed by Amir (Ladsgroup) in May following an outage due to db load from GlobalBlocking. Amir reduced the extensions' DB load by 10%, through avoiding checks for edit traffic from WMCS and Toolforge. And he implemented stats for monitoring GlobalBlocking DB queries going forward.
Reduce Lilypond shellouts from VisualEditor <https://phabricator.wikimedia.org/T312319>
Filed by Reuven (RLazarus) and Kunal (Legoktm) after a shellbox incident. Ed (Esanders) and Sammy (TheresNoTime) improved the Score extension's VisualEditor plugin to increase its debounce duration.
Remember to review and schedule Incident Follow-up work <https://phabricator.wikimedia.org/project/view/4758/> in Phabricator! These are preventive measures and tech debt mitigations written down after an incident is concluded. Read more about past incidents at Incident status <https://wikitech.wikimedia.org/wiki/Incident_status> on Wikitech.
Trends
In June and July (which is almost over), we reported 27 new production errors <https://phabricator.wikimedia.org/maniphest/query/WDqlrITVmIoX/#R> and 25 production errors <https://phabricator.wikimedia.org/maniphest/query/pzOAOpbnF3PX/#R> respectively. Of these 52 new issues, 27 were closed in weeks since then, and 25 remain unresolved and will carry over to August.
We also addressed 25 stagnant problems that we carried over from previous months, thus the workboard overall remains at exactly 299 unresolved production errors.
Take a look at the Wikimedia-production-error <https://phabricator.wikimedia.org/tag/wikimedia-production-error/> workboard and look for tasks that could use your help.
💡 *Did you know?* To zoom in and find your team's error reports, use the appropriate "Filter" link in the sidebar of the workboard .
For the month-over-month numbers, refer to the spreadsheet data <https://docs.google.com/spreadsheets/d/e/2PACX-1vTrUCAI10hIroYDU-i5_8s7pony…>.
Thanks!
Thank you to everyone who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!
Until next time,
– Timo Tijhof
🔗 Share or read later via https://phabricator.wikimedia.org/phame/post/view/292/
Hi everyone,
We’re happy to share the July 2022 edition of the Technical Community
Newsletter:
https://www.mediawiki.org/wiki/Technical_Community_Newsletter/2022/July
The newsletter is compiled by the Wikimedia Developer Advocacy Team. It
aims to share highlights, news, and information of interest from and about
the Wikimedia technical community.
The Wikimedia Technical Community is large and diverse, and we know we
can't capture everything perfectly. We would love to hear your ideas for
future newsletters. Got something you would like to see or something you
want to highlight in the next quarterly newsletter? Add your suggestion to
the talk page:
https://www.mediawiki.org/wiki/Talk:Technical_Community_Newsletter
If you'd like to keep up with updates and information, subscribe to the
Technical Community Newsletter:
https://www.mediawiki.org/wiki/Newsletter:Technical_Community_Newsletter
Thanks,
Melinda
--
Melinda Seckington
Developer Advocacy Manager
Wikimedia Foundation <https://wikimediafoundation.org/>
The Search Platform Team
<https://www.mediawiki.org/wiki/Wikimedia_Search_Platform> usually holds an
open meeting on the first Wednesday of each month. Come talk to us about
anything related to Wikimedia search, Wikidata Query Service (WDQS),
Wikimedia Commons Query Service (WCQS), etc.!
Feel free to add your items to the Etherpad Agenda for the next meeting.
Details for our next meeting:
Date: Wednesday, August 3rd, 2022
Time: 15:00-16:00 UTC / 08:00-09:00 PDT / 11:00-12:00 EDT / 16:00-17:00 WAT
/ 17:00-18:00 CEST
Etherpad: https://etherpad.wikimedia.org/p/Search_Platform_Office_Hours
Google Meet link: https://meet.google.com/vgj-bbeb-uyi
Join by phone: https://tel.meet/vgj-bbeb-uyi?pin=8118110806927
Hope to talk to you next week!
—Trey
Trey Jones
Staff Computational Linguist, Search Platform
Wikimedia Foundation
UTC–4 / EDT