Dear Wikitechians,
On *Wednesday March 1st*, the SRE team will run a planned data center
switchover, moving all wikis from our primary data center in Virginia to
the secondary data center in Texas. This is an important periodic test of
our tools and procedures, to ensure the wikis will continue to be available
even in the event of major technical issues in our primary home. It also
gives all our SRE and ops teams a chance to do maintenance and upgrades on
systems in Virginia that normally run 24 hours a day.
The switchover process requires a *brief read-only period for all
Foundation-hosted wikis*, which will start at *14:00 UTC on Wednesday March
1st*, and will last for a few minutes while we execute the migration as
efficiently as possible. All our public and private wikis will be
continuously available for reading as usual, but no one will be able to
save edits during the process. Users will see a notification of the
upcoming maintenance, and anyone still editing will be asked to try again
in a few minutes.
CommRel has already begun notifying communities of the read-only window. A
similar event will follow a few weeks later, when we move back to Virginia.
This is currently scheduled for *Wednesday, April 26th*.
If you like, you can follow along on the day in the public
#wikimedia-operations channel on IRC (instructions for joining here
<https://meta.wikimedia.org/wiki/IRC/Instructions>). To report any issues,
you can reach us in #wikimedia-sre on IRC, or file a Phabricator ticket
with the *datacenter-switchover* tag (pre-filled form here
<https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=Data…>);
we'll be monitoring closely for reports of trouble during and after the
switchover. (If you're new to Phab, there's more information at
Phabricator/Help.) The switchover and its preparation are tracked tracked
in Phabricator Task T327920 <https://phabricator.wikimedia.org/T327920>
On behalf of the SRE team, please excuse the disruption, and our thanks to
everyone in a number of departments who've been involved in planning this
work for the past weeks. Feel free to reply directly to me with any
questions.
Thank you,
--
Clément Goubert (they/them)
Senior SRE
Wikimedia Foundation
*Fixing my list addressing errors...*
TLDR: The Foundation will be conducting a retrospective on the Technical
Decision Making Process.
To the entire Wiki technical community,
For quite some time now, we have experienced issues with the Technical
Decision Making Process (TDMP). Volunteer contributors and staff have asked
if we are still operating the Technical Decision Forum (TDF, the member
body that participates in the TDMP). Communication about it from the
Foundation has been inconsistent, and interest from the volunteer community
in joining has been low. Some of our most senior engineers on Foundation
staff have expressed that the process is flawed, doesn’t create room for
discussion about the technical issues surrounding a decision, and doesn’t
ensure participation by all stakeholders who may be affected by the
decision. Suffice it to say, the current state of affairs leaves many
participants wanting more.
We must also remind ourselves of the purpose of a decision making process.
The decisions are not meant to be random or isolated. They should be
aligned to our technical strategy, and we should be able to look at the
decisions we have made and understand how they advance our progress against
that strategy. If the process is working as it should, the decisions that
are produced should represent settled wisdom, and not need to be revisited
too quickly. The goals for a well-run process include:
-
A straightforward, widely understood decision making process, that
-
Facilitates impactful technical decisions to be made in a timely manner,
-
Incorporates input from staff and volunteers in our technical community,
with
-
Decisions that align with accountability for decision outcomes, and
-
Clear communication and transparent operations throughout the process.
On examination of the contributing factors that have led us to this point,
the factor that stands out to me is the need for clear accountability:
accountability for the TDMP itself and accountability for each of the
decisions we make. Technical decision making, beyond a certain magnitude,
is a core organizational process for any engineering organization. It is
therefore important for us to examine and improve this process from time to
time to ensure organizational effectiveness. Not unrelated, regular
retrospectives are a routine agile software engineering practice to enact
continuous improvement. To keep our decision making process effective and
efficient, we need to conduct regular retros. Overall accountability for
maintaining an effective decision making process should rest with a person
who is sufficiently able to marshal resources and address problems at a
large scale – here at the Foundation, that resides in the executive level.
The Foundation will be conducting a retro on the TDMP over the next couple
of months. Because we don’t yet have a habit of doing retros on this
process, and because there is a wide range of stakeholders we seek to hear
from, the process will be a bit more structured than an ordinary retro, and
will take more time. As we do more of these, we should get better at them.
The feedback gathered through the retro will be used to make changes to
improve the TDMP.
Foundation staff will follow up with more information about the kickoff of
the retro and what steps will follow. I am looking forward to wide
participation in this retro.
Here are the links to the relevant wiki page and Phab ticket:
- Wiki page
<https://www.mediawiki.org/wiki/Technical_decision_making/Technical_Decision…>
- Phabricator ticket <https://phabricator.wikimedia.org/T333235>
Thank you! And apologies for all the crossposting.
Tajh Taylor (he/him/his)
VP, Data Science & Engineering
Wikimedia Foundation <https://wikimediafoundation.org/>
*Sorry for cross-posting*
Hello, this is Wikimedia Deutschland’s Technical Wishes team with news on
Kartographer:
As was requested by many community members, it is now possible to see
nearby articles on a Kartographer map. This offers an additional way of
exploring content on the wikis. The feature was deployed to most Wikimedia
wikis <https://phabricator.wikimedia.org/T334079> on April 11. To try it
out, open any Kartographer map into full screen mode and click on “Show
nearby articles” in the footer.
Some links if you would like to learn more:
-
The help page on mediawiki.org with a short overview of this feature:
https://www.mediawiki.org/wiki/Help:Extension:Kartographer#Show_nearby_arti…
-
Our project page with more detailed information:
https://meta.wikimedia.org/wiki/WMDE_Technical_Wishes/Geoinformation/Nearby…
German, Catalan, Russian and Hebrew Wikipedia, and English and German
Wikivoyage have already been using this feature since October, and have
given plenty of feedback – thanks a lot! If you have any feedback, please
let us know on this talk page:
https://meta.wikimedia.org/wiki/Talk:WMDE_Technical_Wishes/Geoinformation/N…
With this deployment, our focus area “Better support for geo information
<https://meta.wikimedia.org/wiki/WMDE_Technical_Wishes/Geoinformation>” is
coming to an end. On behalf of the whole Technical Wishes team, many many
thanks to everyone who tested, gave feedback, was interviewed, reviewed
tickets, or contributed in any other way. It is deeply appreciated!
Johanna
for the Technical Wishes team at Wikimedia Deutschland
--
Johanna Strodt Project Manager Community Communications for the Technical
Wishes project
Hello Everyone,
Today it was identified that the Graph extension [1], which uses the older
Vega 1 & Vega 2 libraries, had a number of security vulnerabilities.
In the interest of the security of our users, the Graph extension was
disabled on Wikimedia wiki's. We know that this is disruptive for editors
and readers, and WMF teams are working quickly on a plan to respond to
these vulnerabilities.
We recommend that any other third party users of the Graph extension should
disable the use of that extension on their wikis.
A configuration change will suppress the exposed raw tags and graph json
definition to avoid excess disruption to the end user experience when the
extension is disabled. [2] This also provides a tracking category
"Category:Pages with disabled graphs" showing the pages that used to
contain graphs. Local administrators can localise the name of the category
and its description by editing [[MediaWiki:Graph-disabled-category]],
[[MediaWiki:Graph-disabled-category-desc]] interface messages on your local
wiki.
On Wikimedia projects, graphs created via the extension will remain
unavailable. This means that pages that were formerly displaying graphs
will now display a small blank area. To help readers understand this
situation, communities can now define a brief message that can be displayed
to readers in place of each graph until this is resolved. That message can
be defined on each wiki at [[MediaWiki:Graph-disabled]] by local
administrators.
Wikimedia Foundation staff are looking at options available and expected
timelines. For updates on this topic, follow the public Phabricator task
for this issue: https://phabricator.wikimedia.org/T334940
We are seeking translation to be able to distribute a massmessage about
this to various language wikis tomorrow. Please help us translate here:
https://meta.wikimedia.org/wiki/User:Seddon_(WMF)/Graph_massmessage
Thank you,
Seddon
[1] https://www.mediawiki.org/wiki/Extension:Graph
[2]
https://gerrit.wikimedia.org/r/plugins/gitiles/operations/mediawiki-config/…
Seddon (he/him they/them)
*Engineering Manager*
*Wikimedia Foundation*
Greetings. The Wikimedia Foundation is pleased to share a summary of our
draft annual plan with you. It's available on Meta-wiki, and we encourage
you to take a look.[0] We'd also love to hear your thoughts, so feel free
to leave a comment on the talk page.[1] The comment period is open from now
until May 19th. Thank you for your interest in our work.
[0]
https://meta.wikimedia.org/wiki/Wikimedia_Foundation_Annual_Plan/2023-2024
[1]
https://meta.wikimedia.org/wiki/Talk:Wikimedia_Foundation_Annual_Plan/2023-…
Yours,
Chris Koerner (he/him)
Senior Movement Communications Specialist
Wikimedia Foundation
[also posted to wikimedia-l]
Hi everyone,
I joined the Wikimedia Foundation on August 1 of last year in a newly
created role as the Chief Product and Technology Officer (CPTO). (For the
first few weeks, some of the staff called me C3PO as they got used to the
new title :) The role was created to bring both the Product and Technology
departments back under a single accountable leader for the first time since
about 2015. Like Maryana
<https://meta.wikimedia.org/wiki/Wikimedia_Foundation_Chief_Executive_Office…>,
I decided to spend the first few months of my time at Wikimedia listening
and learning. Although I come from the open source technology field, and
have worked with volunteers and communities in prior jobs, it felt
important to start here with curiosity and openness about what’s working
well and what needs to change.
Since then, I have met one on one and in small groups with more than 360
people, who spoke with me from 38 different countries. I also attended 22
large and small convenings and events which included about 3,150 people.
This includes members of the Foundation’s product and technology teams,
other Foundation staff, editors, functionaries, affiliates, movement
organizers and open internet partners. I tried to approach every
conversation with curiosity, openness, and eagerness, letting go of any
preconceptions I may have had (intentionally embracing beginner’s mind
<https://en.wikipedia.org/wiki/Shoshin>) about the Foundation, the
Wikimedia projects, and communities worldwide that contribute to creating
and sharing free knowledge. I can confirm that I quickly found myself awash
in details, experiencing a firehose of information from all sides! My
husband and two young children have also learned a lot more about this
movement in the last six months than you might expect.
To provide myself with some structure, I asked everyone the same kind of
questions about: (1) the impact our product and technology organizations
have had on the movement and/or the world in the last five years, and what
people were most proud of; (2) the current vision and strategy and if they
will take us where we need to go; and (3) the most promising opportunities
that people see in our work, and what is needed to realize that potential.
I want to thank everyone who took the time to share with me, and I’ve
included some direct, anonymized quotes in this letter from the
conversations I had. And I want to confirm that the listening continues — I
will create more spaces in the year ahead for dedicated conversations about
some of the important topics I have highlighted below. I will also be
posting this letter to Meta.
Pulling in the same direction: More visible and shared metrics
On a page of the first notebook I had for my onboarding, I quoted a person
who said they just wanted "meaningful common goals." This was a theme
repeated over and over — a clear desire from everyone to do work together
that was linked by common purpose, and with all the volunteers that have
created all Wikimedia projects. I got to hear so many different voices, and
I heard the details from every side — what’s working, what hasn’t been
working for a long time — some of the problems we face are over ten years
old. People shared what’s missing, what’s extra, who’s fighting to be heard
and who’s feeling lost at sea.
"I think there are lots of promising opportunities to incentivise people to
pay off technical debt and make our existing stack more sustainable. Right
now there are no incentives for engineers in this regard."
"Are we really having impact?"
How can we unite behind meaningful common goals? And which metrics matter
the most? We have so much data, but we really need lodestar
<https://en.wiktionary.org/wiki/lodestar> (or some refer to this as north
star) metrics across the whole Foundation, a system for reviewing and
reflecting on what we learn from them, and then a way to connect those
metrics with the day to day work everyone is doing.
To get at that, we’re doing two main things — one is deepening our
understanding of volunteer activities and the health of the volunteer
communities. This will be through working closely with volunteers using
existing processes and sharing what we’re learning, as well as qualitative
and quantitative research workstreams, including reviewing existing
research of volunteer activities and typical work profiles. The other is
working to establish a set of Foundation-wide lodestar metrics. Shared
metrics help everyone understand how we’re measuring success across the
Foundation, and we’re sharing these publicly as part of our Annual Plan.
Over time, we plan to bring our measures of success for important
initiatives to communities for conversations and debate to help everyone
align what success might look like. Shared metrics and data will empower us
to make more effective and better decisions, along with collaboration with
those who are working on changes and those who may be directly affected by
them.
What does our open source strategy look like for today’s world?
"I strongly believe that Wikipedia will be obsolete by 2030 if we don’t fix
MediaWiki now."
What is our open source strategy?
We have to make some harder choices about what it means to be an
open-source organization, and shift the conversation to resolve historic
debates that prevent us from making important, strategic choices.
Two big areas to resolve are:
-
What is our strategy for MediaWiki support? Today there is a tug of war
about whether we should support MediaWiki for third-party users, even
though their use cases have diverged significantly from those of Wikimedia
projects. I’m planning a MediaWiki convening in late 2023 to begin tackling
this issue.
-
What is our strategy for third-party re-use of Wikimedia content? There
are a lot of nuances around rate limiting and updating the existing API
policies in line with our values around open access. How can we coordinate
more across the Foundation and technical volunteers to build greater
understanding and alignment? Wikimedia project content also has become a
cornerstone of artificial intelligence (AI) products. Wikipedias have long
used machine learning (ML) to improve content and detect vandalism. How can
we help support the use of ML and AI that is a public good? We have
started some
conversations
<https://meta.wikimedia.org/wiki/Wikimedia_Foundation_Annual_Plan/2023-2024/…>
about this but need to go further.
What will it take to have impact at scale?
"Before we can think about strategy, we need to answer ‘do we want to
change this culture to work with a unified strategy, or do we want to
change the strategy methodology to work with a decentralized culture? Or
some combination thereof?’"
What is our strategy for scaling that will allow us to have the most impact
with limited resources?
Today we support over 750+ distinct Wikimedia projects, with over 300 of
those including language versions of Wikipedia, Wikisource, multiple
language versions of Wiktionary, and many other free knowledge projects.
What is an efficient and responsible way to steward the limited resources
we have towards Wikipedia and/or the sister projects? And similar to the
earlier conversation about Foundation metrics – we must do this in a way
that can have an impact on our mission of bringing free knowledge to the
whole world.
Some of the big questions that came up included consolidation of projects
and the technology underpinning them where it makes sense, and from a
prompt given to me by the Commons community – how can we think even bigger,
and question elephants in the room, which in part would be to examine the
long-standing and seemingly unquestioned assumption that MediaWiki is the
best software to solve all problems we face. And if we do solve big
problems in different ways, what does that look like? What can we learn
from projects like WikiLearn, which uses free software not made by the
Foundation, as well as people and organizations outside our movement? This
is definitely a multi-year, rich problem space to explore.
Everyone’s relationship with English Wikipedia, including the Wikimedia
Foundation’s
"For various reasons, the Foundation and some parts of the communities are
stuck in an uneasy relationship where the Foundation admires but fears the
communities’ power, like a beautiful but dangerous animal – the tiger might
attack you – and the communities, not least English Wikipedia, distrust the
Foundation."
My experience so far has been that we have a very contentious relationship
with English Wikipedia. The Foundation raises most of the revenue to
support a global movement from English Wikipedia, and it’s often where
volunteers raise most of the concerns and objections to the Foundation’s
work.
It's painfully affecting volunteers and staff that are trying to maintain
content and code, and make important improvements to all the websites, as
with the launch of Vector 2022 this year. It has made product and
engineering teams very conservative in their approach to rolling out
features, making each change take 12 or 18 months, or years!, to get
valuable features to users. And it impacts our ability to collaborate with
communities on and off English Wikipedia on big goals like knowledge equity
and the movement strategy recommendations. As Yoda noted
<https://en.wikiquote.org/wiki/Fear#L>, fear is the path to the dark side.
This is a bummer, and I’d like it to change.
So how do we break this cycle? What I’m doing now is directly engaging.
Today, for example, I participated in an office hours session to talk about
Vector 2022. Some of the product senior leadership in the recent past have
specifically avoided talking directly with people on English Wikipedia, and
this approach will no longer be applied. Engaging human to human is the
best way I know to help resolve some of the mystery, fear and anger that
are present. However, that will absolutely not fix what’s wrong here. We
need systemic solutions. Today, there’s no way to make lasting and mutually
binding agreements with volunteers, and that isn’t a sustainable way to
create and maintain infrastructure software. My hope is that, with a more
open and direct approach to engage and also through the work of the
Movement Charter Drafting Committee, we will chart out a path for more
lasting, productive collaboration.
Being more intentional, and also clear, in our technical support for
volunteers
"We lack clear governance and communication for most of our tech
components, squandering a lot of the opportunities we have for more and
better participation from long-time and new volunteers."
How can the Foundation be more intentional about our relationship with all
volunteers?
Today we have few and incomplete policies about what volunteers can do in
technical spaces. We need to chart clearer boundaries, and move more toward
rational and practical policy instead of precedent guiding our work.
Similarly, the technical spaces where the Foundation "stays out" have felt
ad hoc, which led to volunteers stepping in to do important work. The
Foundation needs to exhibit better accountability in maintaining essential
services (e.g. 2-factor authentication), and to be explicit about the
technical tasks that it is definitely leaving for volunteers to own.
Finally, we really need to embrace a product development model that’s more
collaborative and efficient. This calls into question feedback tools like
RfCs, and takes into consideration movement "technology council" proposals.
What will really make us better together? I’m really interested in finding
an answer to this question.
Three Priorities for the Coming Year
What I have identified above are complex issues that cannot be solved in a
single year. We all need to take a multi-year view, especially in order to
define the precise issues that need to be solved more carefully.
For now, you have seen the draft annual plan priorities for the
Foundation’s Product/Tech teams and they include:
- *Volunteers*: We need closer connections, with a focus on making all
time spent volunteering fulfilling and productive. I will continue to talk
directly to volunteers, on-wiki and in person. I am making a shift in our
Annual Plan to support the work and improve the experience of "editors with
extended rights" (inclusive of admins, stewards, patrollers, and moderators
of all kinds, which are also known as functionaries). The work done by this
group on mis- and dis-information and on enforcing our Universal Code of
Conduct is crucial to the functioning of all Wikimedia projects. Success
requires that we are able to have metrics to guide our progress, identify
ways of measuring the health of communities, and that we do this work hand
in hand with volunteers.
- *Maintenance*: Staff and volunteers have both identified that we have
far too many unfinished technical migrations. This means that we continue
to support both old and new tools and ways of doing our technical work.
This increases the workload of everyone, without necessarily adding
features or improving our technical systems overall. Challenges include
issues with Foundation staff and volunteer community decision making,
accountability for that decision making and the best projects to pursue,
and, on the Foundation's side, a desire to not cause upset among
volunteers. As a result, we have many abandoned or poorly maintained tools.
We must be able to choose maintenance and technical migration areas for
prioritization, and then be ok with not doing work on others in order to
complete some of these big projects. For example, we have big work to do on
our data infrastructure, which is aging and made up of more than 40
distinct and fragile systems supported by a tiny team. We also have big
work to do on MediaWiki to ensure it can support our projects for the next
20 years.
- *Decision making*: From the very start of my time with the Foundation,
a common theme that kept coming up was the confusion that internal teams
had around decision making structures and accountability. I heard stories
about teams being indefinitely stuck, unclear decisions from the past, and
an inability to make and keep a decision. I view decision making like
lifting weights: you get good at it by doing it, incrementally and
consistently, over a long period of time. To start, I am making decisions
around the structure and organization of the Product and Technology teams
within the Foundation in order to make decision owners more clear, direct
and transparent. We’re collaborating better together internally, and
raising long-standing unresolved issues between teams in order to resolve
them, one by one. As I look ahead, clarity of decision making and how we
align our work towards our three objectives will be a core part of how I
organize teams.
In addition, I believe that decision making and achieving lasting positive
results needs to be rooted in data. We will identify essential metrics to
evaluate progress and assess impact on the three objectives of our work.
This allows us to stay focused on our most important goals, make
adjustments as needed, and track our progress over time.
I am committed to promoting transparent and accountable decision making at
all levels of management and individual contributor leadership. As I wrote
earlier in this letter, I also welcome ideas on how to build well-defined
processes for engaging with communities and making decisions that endure.
These changes to how we make decisions will allow us to move more quickly,
be more responsive, and create a larger impact for our goals over time.
What’s Next
During my listening tour, some staff asked me an "elephant in the room"
question: why should they trust me? Given the number of different
executives who have come to the Foundation and left within a year or two,
skepticism about yet another new leader is high. My answer was: I believe
the problems we face, as a Foundation and volunteers striving to bring free
knowledge to the world, are complex puzzles that cannot be solved by one
person, and I’m committed to a multi-year approach to collaboratively solve
them together.
Success requires more than a product roadmap. We need deep and effective
collaboration between the Foundation and all volunteers and communities,
shared ways to learn and be successful together, and constant adaptation to
changes in the internet and world, so we can solve the big puzzles we face
together.
Trust is built over time and through consistency, so I don’t ask for trust
as I begin my work. I ask that people be open to working closely together,
learning as much as we can about the important problems we face, and that
we regularly review our work in a data-informed way.
I would like to be direct about how difficult I know some of these topics
are, even for a discussion. But it is our job to tackle the most difficult
questions, especially where inaction due to fear has led to stagnation and
demotivation amongst both our staff and communities. This is not going to
be a quick turnaround. None of these issues will be a quick or easy fix.
Building and improving systems will be a lot of work, and will take a lot
of patience. But the payoff for solving each of these puzzles will be that
we’re able to engage more fully, and maybe even more joyfully, in our work.
My listening tour was an invaluable opportunity to get candid information
about what exactly is working, what isn’t, and what ideas everyone has for
creating something great together. We have a lot of work ahead of us, but
I’m encouraged by the energy and enthusiasm and I know we’ll be able to
tackle this together.
Next time you’ll hear from me is August to share the outcomes of community
discussions related to annual planning
<https://meta.wikimedia.org/wiki/Wikimedia_Foundation_Annual_Plan/2023-2024>,
and where I think we’re going to have impact in the coming year. In the
meantime, I want to share a few questions that I’ll be returning to
regularly: Are there examples of big issues that we've tackled well as a
movement? Where would you suggest I draw inspiration? What's worked well?
These are the complex issues that will guide my priorities over the coming
years. What elephants am I missing?
As I shared when I joined
<https://lists.wikimedia.org/hyperkitty/list/wikimedia-l@lists.wikimedia.org…>,
I came to work for the Wikimedia Foundation because free access to
knowledge is the most important thing I can be doing right now. Our work
empowers the people who have knowledge to share. By involving youth, women,
and underrepresented identities to contribute their unique knowledge, we
will continue our journey to share the sum of all human knowledge. And this
kind of mission cannot be accomplished by any one person alone; we are
called to – and I feel strongly committed to – collaborate and truly be in
this mission together.
-selena
Hello
*Phabricator's aphlict service[0] will be down for 30 minutes tomorrow
(Tue, 18 Apr 2023) between 13:00–15:00 UTC[1]* due to network switch
upgrades in its data center row[2].
During this window, real-time notifications on Phabricator will not work.
This affects:
- Pop-up notifications on Phabricator
- Real-time workboard updates—you'll need to refresh the page to see other
people's workboard changes.
Apologies for any inconvenience.
Tyler Cipriani (he/him)
Engineering Manager, Release Engineering
Wikimedia Foundation
[0]: <https://wikitech.wikimedia.org/wiki/Aphlict>
[1]: <https://zonestamp.toolforge.org/1681822803> (thanks Sebastian Berlin
for reminding me of this tool :D)
[2]: <https://phabricator.wikimedia.org/T333377>