Hello all,
The next language community meeting is scheduled in a few weeks - May 31st
at 16:00 UTC. If you're interested, you can sign up on this wiki page: <
https://www.mediawiki.org/w/index.php?title=Wikimedia_Language_engineering/…
>.
This is a participant-driven meeting, where we share language-specific
updates related to various projects, collectively discuss technical issues
related to language wikis, and work together to find possible solutions.
For example, in the last meeting, the topics included the machine
translation service (MinT) and the languages and models it currently
supports, localization efforts from the Kiwix team, and technical
challenges with numerical sorting in files used on Bengali Wikisource.
Do you have any ideas for topics to share technical updates related to your
project? Any problems that you would like to bring for discussion during
the meeting? Do you need interpretation support from English to another
language? Please reach out to me at ssethi(a)wikimedia.org and add
agenda items to the document here: <
https://etherpad.wikimedia.org/p/language-community-meeting-may-2024>.
We look forward to your participation!
Cheers,
Jon, Mary, Oscar, Amir and Srishti
*Srishti Sethi*
Senior Developer Advocate
Wikimedia Foundation <https://wikimediafoundation.org/>
Hello all!
The Search Platform Team usually holds an open meeting on the first
Wednesday of each month. Come talk to us about anything related to
Wikimedia search, Wikidata Query Service (WDQS), Wikimedia Commons Query
Service (WCQS), etc.!
Feel free to add your items to the Etherpad Agenda for the next meeting.
Details for our next meeting:
Date: Wednesday, May 8, 2024
Time: 15:00-16:00 UTC / 08:00 PDT / 11:00 EDT / 17:00 CEST
Etherpad: https://etherpad.wikimedia.org/p/Search_Platform_Office_Hours
Google Meet link: https://meet.google.com/vgj-bbeb-uyi
Join by phone: https://tel.meet/vgj-bbeb-uyi?pin=8118110806927
Have fun and see you soon!
Guillaume
--
*Guillaume Lederrey* (he/him)
Engineering Manager
Wikimedia Foundation <https://wikimediafoundation.org/>
Hi All, welcome to the monthly MediaWiki Insights email!
First of all, congratulations to Ederporto, SD hehua and Labster who got
their first patch in MW core or Wikimedia deployed extensions and services
merged during the month of April! Many thanks also to the reviewers for
your support and thoughtfulness! It's the collaborative effort that makes
MediaWiki scale for what it's built for.
10 months ago
<https://www.mediawiki.org/wiki/MediaWiki_Product_Insights/Reports/August_20…>
we started the journey of ramping up MediaWiki as a product and thinking
about ways to systemically tackle the challenges around the MediaWiki
software platform. The initial announcement
<https://phabricator.wikimedia.org/T336990> and first conversations already
happened before we officially kicked things off: At the Wikimedia Hackathon
2023 in Athens. Many interviews, conversations, explorations, investments
and decisions later we’re coming back to the Wikimedia Hackathon
<https://www.mediawiki.org/wiki/Wikimedia_Hackathon_2024> this year to
connect the dots.
MediaWiki, the software platform and interfaces that allow Wikipedia and
other projects to function, needs ongoing support for the next decade in
order to provide creation, moderation, storage, discovery, and consumption
of open, multilingual content at scale.
What decisions and platform improvements can we make to ensure that
MediaWiki is sustainable? - This has been a key guiding question over the
past months. Platform improvements have been the subject of many
initiatives mentioned in the monthly MediaWiki Insights emails
<https://www.mediawiki.org/wiki/MediaWiki_Product_Insights/Reports>, and
the focus on sustainability is also reflected in the draft for the
Wikimedia Foundation’s annual plan
<https://meta.wikimedia.org/wiki/Wikimedia_Foundation_Annual_Plan/2024-2025/…>
for the upcoming fiscal year (July 2024 – June 2025).
We plan to present an overview on the state of things at the Hackathon, and
hope to use the “hallway track” to discuss questions, plans and ideas with
some of you! A related presentation has already been given at the recent
MediaWiki users and developers conference.
We will follow up with links to presentations, a summary and reflections in
the next MediaWiki insights email. Stay tuned!
MediaWiki track at the Wikimedia Hackathon in Tallinn
The Wikimedia Hackathon starts this week! Beyond wanting to talk about
MediaWiki, we’ve also put a MediaWiki track
<https://www.mediawiki.org/wiki/Wikimedia_Hackathon_2024/MediaWiki_Track>
together to help people get started - or continue - with working on
MediaWiki (core), discuss and collaborate with each other (workboard
<https://phabricator.wikimedia.org/project/board/7117/>). Whether you want
to get started, join as a mentor, or talk about MediaWiki - if you’re
attending the Hackathon, please come find us at the MediaWiki track table!
Project snapshots: ObjectCacheFactory introduced to MediaWiki core, SUL3
and more work on REST API
Caching: We introduced ObjectCacheFactory to MediaWiki core (T358346
<https://phabricator.wikimedia.org/T358346>). ObjectCache is responsible
for creating and fetching various cache instances for MediaWiki making use
of BagOStuff, which is the mechanism for caching in the software's
ecosystem. The introduction of the Cache Factory aims to reduce
inconsistencies in our codebase and improve stability. Many thanks to
Derick for leading on this work and to Timo, Piotr and Gergö for their
support!
Help with replacing the deprecated static factory methods is much
appreciated! Please see this tracking ticket for more information on which
repositories need updates: T363770
<https://phabricator.wikimedia.org/T363770>.
MediaWiki REST API: We completed T359652
<https://phabricator.wikimedia.org/T359652>, which disallows optional path
parameters <https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1016820> in
the MediaWiki REST API <https://www.mediawiki.org/wiki/API:REST_API> (aka
rest.php) at the PathMatcher
<https://gerrit.wikimedia.org/g/mediawiki/core/+/eab7628c59be589c8bba2b29dab…>
level. This is important for compatibility with API modules that we are
moving from RESTBase <https://www.mediawiki.org/wiki/RESTBase> into MW Core
<https://www.mediawiki.org/wiki/Core> as part of RESTBase sunset
<https://phabricator.wikimedia.org/T262315> (such as Reading Lists
<https://phabricator.wikimedia.org/T336693>), and avoids issues with our
generated OpenAPI/swagger specs
<https://meta.wikimedia.beta.wmflabs.org/w/rest.php/> (swagger does not
allow optional path parameters
<https://swagger.io/docs/specification/describing-parameters/#path-parameters>).
A big thanks to Bill for his work on this, and to all the people who
provided support!
Database: SelectQueryBuilder and Expression Builder: We shared about the
work that the Data Persistence team has been doing on the MediaWiki Rdbms
library’s interface to improve consistency, security and ease for getting
database connections and performing common queries in an earlier email
<https://www.mediawiki.org/wiki/MediaWiki_Product_Insights/Reports/October_2…>.
A big thanks to Amir for all his work on this to date! Since then, a lot of
progress has been made on updating MediaWiki repositories in Wikimedia
production to use the new SelectQuery and Expression Builders: Many thanks
to all the people who helped with this! A special thanks to Umherirrender
for migrating an impressive number of extensions to use SelectQuery
<https://phabricator.wikimedia.org/T311866> and Expression
<https://phabricator.wikimedia.org/T350075> builders over the past month,
and to DannyS712 for many reviews!
Another highlight is the work done by Taavi over the past year: Allowing
multiple different 2FA devices (T242031
<https://phabricator.wikimedia.org/T242031>) is about to wrap up, which
should be a nice improvement to make 2FA easier to use! Many thanks to
Taavi and everyone involved for this work!
SUL3: Browsers increasingly roll out anti-tracking measures and limitations
on third-party cookie use. A side effect of this is that it also impacts
CentralAuth autologin. We aim to transition to a single sign-on domain
to minimize
the number of times users need to enter their credentials when changing
wikis as well as for other benefits; and are about to move from the
research to the coding phase. The implementation plan (with some question
marks) is in T348388 <https://phabricator.wikimedia.org/T348388> and its
subtasks - feedback is very welcome!
The Language team just released MediaWiki Language Extension Bundle 2024.04
(announcement
<https://lists.wikimedia.org/hyperkitty/list/mediawiki-l@lists.wikimedia.org…>).
They are also looking into changing how and when we release MLEB. Please
see T356847 <https://phabricator.wikimedia.org/T356847> for more
information and feedback.
MediaWiki Release: The 1.42.0-rc.0 announcement will be out soon. MW
1.42-alpha has been branched since April 9th and added to the on-wiki
documentation as the development snapshot. If you have changes that need to
go to 1.42, they should be backported. New tasks with commits since April
16th have been targeted to the 1.43 unstable branch
<https://phabricator.wikimedia.org/project/profile/7083/>.
See some of you in Tallinn!
Thanks all for reading,
Birgit
--
Birgit Müller (she/her)
Director of Product, MediaWiki and Developer Experiences
Wikimedia Foundation <https://wikimediafoundation.org/>
Hi Community Metrics team,
This is your automatic monthly Phabricator statistics mail.
Accounts created in (2024-04): 255
Active Maniphest users (any activity) in (2024-04): 1114
Task authors in (2024-04): 596
Users who have closed tasks in (2024-04): 324
Projects which had at least one task moved from one column to another on
their workboard in (2024-04): 313
Tasks created in (2024-04): 2424
Tasks closed in (2024-04): 2442
Open and stalled tasks in total: 53891
* Only open tasks in total: 52917
* Only stalled tasks in total: 974
Median age in days of open tasks by priority:
Unbreak now: 20
Needs Triage: 997
High: 1236
Normal: 2049
Low: 2586
Lowest: 3066
(How long tasks have been open, not how long they have had that priority)
To see the names of the most active task authors:
* Go to https://wikimedia.biterg.io/
* Choose "Phabricator > Overview" from the top bar
* Adjust the time frame in the upper right corner to your needs
* See the author names in the "Submitters" panel
TODO: Numbers which refer to closed tasks might not be correct, as
described in https://phabricator.wikimedia.org/T1003 .
Yours sincerely,
Fab Rick Aytor
(via community_metrics.sh on phab1004 at Wed 01 May 2024 12:00:40 AM UTC)
Esteemed technical community,
The WMF’s SRE Observability team
<https://wikitech.wikimedia.org/wiki/SRE/Observability> invites you to join
our ongoing effort to migrate MediaWiki metrics to Prometheus
<https://wikitech.wikimedia.org/wiki/Prometheus>, utilizing StatsLib
<https://www.mediawiki.org/wiki/Manual:Stats>, an internally developed,
Prometheus-capable metrics interface. This initiative is fundamental in
unifying our metrics, improving MediaWiki observability, and reducing tool
fragmentation. Your participation is crucial to drive this effort forward.
The Ask
We invite you to contribute to this project
<https://phabricator.wikimedia.org/T350592>. Your expertise can drive the
success of this migration, helping us streamline and improve our monitoring
capabilities.
We appreciate your support in migrating your component’s metrics to
StatsLib (*T350592*) <https://phabricator.wikimedia.org/T350592>; this
involves:
-
Look up your component, extension, or module in the task above, claim
(or create) the corresponding sub-task for your metrics, and follow the
examples/docs available to migrate your metrics to the new metrics
interface.
-
Help deprecate and clean up/remove outdated metrics not in use (or
graphed in dashboards).
-
Collaborate with testing and provide feedback for a seamless transition.
Why Prometheus <https://prometheus.io/docs/introduction/overview/>?
We have been using Prometheus in production for several years as it offers
several benefits over Graphite
<https://prometheus.io/docs/introduction/comparison/>. Migrating MediaWiki
off Graphite ensures we stay ahead with a supported, scalable metrics
platform for more effective long-term, multidimensional metrics analysis
and storage. The new stack provides more robust data labeling, storage, and
query capabilities. This project facilitates the improvement of our
production metrics infrastructure and deprecates older systems. The general
thought process is outlined in T249164 RFC: Better interface for
generating metrics in MediaWiki <https://phabricator.wikimedia.org/T249164>.
Support and Resources:
We can assist via various channels such as email, phabricator and IRC
(#wikimedia-observability on Libera.Chat), including periodic technical
office hours (to be scheduled soon). Please contact me or visit the SRE
Observability Team Interface Page
<https://office.wikimedia.org/wiki/Team_interfaces/SRE_-_Observability> for
ways to get involved.
Timeline
Please prioritize and schedule this work within the next few quarters to
ensure a seamless transition and the sustainability of the MW production
ecosystem. Together, we can achieve a more efficient and reliable
Observability platform. Thank you in advance for your understanding and
support.
Hackathon 2024!
We will be at this year’s hackathon
<https://www.mediawiki.org/wiki/Wikimedia_Hackathon_2024> for those
attending who are interested in participating or have any questions.
We appreciate your support and help; many thanks for your attention and to
all those who have/are participating already!
Respectfully,
Leo
*Leo Mata* (he/him)
Engineering Manager - Observability
Wikimedia Foundation <https://wikimediafoundation.org/>
*[[Note: I've received feedback that the previous announcement email may
have been buried in older threads, so I'm resending this for visibility.
Apologies if you are seeing this more than once.]]*
Hello all,
The TDMP Retro Committee has finalized the gathering and analysis of
the Technical
Decision-Making Process Retrospective and Consultation phase
<https://www.mediawiki.org/wiki/Technical_decision_making/Technical_Decision…>
.
This document presents the findings of the survey conducted among the
Wikimedia technical community regarding their perspectives on a technical
decision-making process in July 2023.
The purpose of this document is to serve as a comprehensive summary,
offering insights from the retrospective collection of opinions and voices
within the technical community. It is not intended to provide a new
decision-making methodology or devise a fresh process.
It serves to inform the next steps of the process of re-evaluating and
adjusting the current technical decision making process.
The report is now available on MediaWiki at Retrospective and Consultation:
Results and Analysis
<https://www.mediawiki.org/wiki/Technical_decision_making/Technical_Decision…>
Mark Bergsma (VP SRE), Tajh Taylor (VP Data Sci and Eng), and Suman
Cherukuwada (Senior Dir Feature Eng) are taking on the responsibility to
determine what changes to make to the TDMP. They will guide a process to
solicit input and feedback on a proposal that will be open to staff and
volunteer technical contributors.
Thank you,
Moriel, on behalf of the TDMP Retro Core Group.
Core group:
- Moriel Schottlender (chair)
- Daniel Kinzler
- Chris Danis
- Kosta Harlan
- Temilola Adeleye
--
Moriel Schottlender (she/her <https://pronoun.is/she>)
Principal Software Engineer
Wikimedia Foundation https://wikimediafoundation.org/
We operate Wikipedia <https://wikipedia.org> and its sister sites
Hi all!
- Results:
https://www.mediawiki.org/wiki/Developer_Satisfaction_Survey/2024
- The developer satisfaction survey launched in December 2023 and closed in
January 2024[0].
- 171 members of the Wikimedia developer community responded to questions
about developer tools and processes.
____
The developer satisfaction survey results show what's important to our
developer community—the parts of the developer experience that work well,
and the parts that need more resources to improve.
➡️ Read it. Talk about it. And ask questions! (please ;))!
____
🎉 Thanks to everyone who took the time to fill out our survey.
And huge thanks to the Wikimedia Research Team for helping Cloud
Services, Release
Engineering, Technical Documentation and other Wikimedia Research team
members build the survey and process the data. In particular, Yu-Ming Liou
for survey-building expertise. And Caroline Myrick who was an absolute hero
processing, exploring, graphing, and patiently explaining all the data.
<3
Tyler Cipriani (he/him)
Engineering Manager, Release Engineering
Wikimedia Foundation
[0]: <
https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/…
>
i've never contributed to any mailing lists, or partaken much, so i don't know how to start this off...
hi? i've realized i can compressed a dataset of lossless .png files down to between a third- or a fourth of the initial size on disk.
that said: in compressing my own backups i find that lossless .png files converted to .xpm (not .ppm, not .bmp) before being compressed result in a compression that of 3- to 4 times smaller in size on disk than the initial .png with it's own compression --- irregardless of any 'PNG optimization' done beforehand: the resulting .xpm files remain identical in size and compress to precisely the same size.
the following commands¹ can surely be tinkered with to greater effect:
---start of shell commands---
mkdir -p xpm
magick convert image.png xpm/image.xpm
mkdwarfs -i xpm/ -o compressed.dfs -l9
---end of shell commands---
now you have a compressed image, three to four times smaller in size on disk, to inspect.
here i openly wonder how a comparison - to assert whether the resulting .xpm file and the lossless .png are indeed the same picture still - would be carried out.
likewise i see this quality of compression extends to the .xgm .xbm formats.
in sharing i wish to bring up the above observation to my best ability. insights welcome as to why this happens and if indeed the image resulting from a lossless .png to .xpm conversion is the same - if this compressed bitmap outperforms the PNG compression significantly without compromising image integrity. (do reply with saying if this is irrelavant information and or presented inadequately in any way: i don't wish to bring red herrings to this mailing list.)
ultimately this is about if a large portion of WikiMedia imagedata indeed can be compressed further by this process - in a 'Pareto improvement' kind of way.
-Ivy, 25
for interest: sources to programs referenced and my brief notes on these.
[1]
magick: https://imagemagick.org/index.php
mkdwarfs, part of the inappropriately named dwarfs toolset:
https://github.com/mhx/dwarfs/blob/main/doc/mkdwarfs.md
note that the -l flag is given the option 9 in the command - this means LZMA is used in this program.
also note that while i use 'mkdwarfs' with LZMA here, i realize the same result on any other program using LZMA or XZ occurs - like the following 'dar', more adequate for single file extraction from an archive and analyzing individual file compression values en masse.
dar: http://dar.linux.free.fr/doc/man/dar.html
unfortunately the projects' website doesn't use HTTPS, so a wayback machine link with HTTPS:
https://web.archive.org/web/20240423233825/http://dar.linux.free.fr/doc/man…
---start of referenced dar command---
dar -c output -zxz9 -R input/
---end of referenced dar command---
lastly two clarifications:
'a lossless .png file' here means an imagefile which never has been converted in a lossy way. a .jpg file converted to a .png used in this procedure produces an output taking more space on disk.
'the procedure' here refers to conversion of a lossless .png to .xpm imagefile format then compressed with either LZMA or XZ in any program.
thank you for reading.