Hi all
A few questions to provoke discussion/share knowledge better: * Why does the train run Tue,Wed, Thur rather than Mon,Tue,Wed * Why do we only have 2 group 1 Wikipedia's (Catalan and Hebrew) * Should there be a backport window Friday mornings for certain changes?
Longer spiel:
A few weeks ago a change I made led to a small but noticeable UI regression. The site was perfectly usable, but looked noticeably off. It was in a more obscure part of the UI so we missed it during QA/code review.
Late Wednesday a ticket was reported against Wikimedia commons, but I only became aware of it late Thursday when the regression rolled out to English Wikipedia. A village pump discussion was started and several duplicate tickets were created. While the site could still be used it didn't look great and upset the experience of many editors.
Once aware of the problem, the issue was easy to fix. A patch was written on Friday.
I understand Friday backports are possible, but my team tend to use them as a last resort in fear of creating more work for my fellow maintainers over weekend periods. As a result, given the site was still usable, the fix wasn't backported until the first available backport window on Monday. This is unfortunately a regular pattern, particularly for small UI regressions.
We addressed the issue on Monday, but I got feedback from several users that this particular issue took too long to get backported. I mentioned the no Friday deploy policy. One user asked me why we don't run the train Monday-Wednesday and to be honest I wasn't sure. I couldn't find anything on https://wikitech.wikimedia.org/wiki/Deployments/Train.
My team tries to avoid big changes on Mondays as Monday merged patches are more likely to have issues since they don't always get the time to go through QA during the week by our dedicated QA engineer.
So... Why don't we run the train Monday-Wednesday? Having a Thursday buffer during which we can more comfortably backport any issues not caught in testing, particularly UI bugs would be extremely helpful to my team and I don't think we'd lose much by losing the Monday to rush last-minute changes.
Assuming there are good reasons for Tuesday-Thursday train, I think there is another problem with our deploy process which is the size of group 1. Given the complexity of our interfaces (several skins, gadgets, multiple special pages, user preferences, gadgets, multiple extensions, and different user rights), generally, many obscure UI bugs get missed in QA by people who don't use the software every day and have a clear mental model of how it looks and behaves. My team mostly works on visible user interface changes and we rely heavily on Catalan and Hebrew Wikipedia users - our only group 1 wikis to notice errors with UI before they go out to a wider audience. Given the size of those audiences, that often doesn't work, and it's often group 2 wikis that make us aware of issues. If we are going to keep the existing train of Tue-Thur, I think it's essential we have at least one larger Wikipedia in our group 1 deploy to give us better protection against UI regressions living over the weekend. My understanding is for some reason this is not a decision release engineering can make, but one that requires an on-wiki RFC by the editors themselves. Is that correct? While I can understand the reluctance of editors to experience bugs, I'd argue that it's better to have a bug for a day than to have it for an entire weekend, and definitely something we need to think more deeply about.
בתאריך יום ג׳, 22 ביוני 2021 ב-23:04 מאת Jon Robson < jrobson@wikimedia.org>:
Hi all
A few questions to provoke discussion/share knowledge better:
- Why does the train run Tue,Wed, Thur rather than Mon,Tue,Wed
- Why do we only have 2 group 1 Wikipedia's (Catalan and Hebrew)
It was my idea, kind of.
Earlier, all Wikipedias were on Thursday. Wikipedias are a bit different because they have some extensions other sites don't, such as Content Translation, which interests me, and some others. It happened a few times that a bad change was rolled out to Wikipedias on Thursday and it was difficult to roll it back until the next week. Moving all Wikipedias or just English a day earlier looked risky, so I proposed moving just two Wikipedias that are fairly active and have a community that is known to be friendly to early adoption and bug reporting. Everyone in these Wikipedias and in Ops thought it was a good idea and it was done.
I have a vague recollection that it helped catch a few bugs early on Wednesday and get them fixed before the Thursday train, although I don't remember the details.
This is not written in stone, and I don't think anyone in these two Wikipedias will strongly object to changing some things, although if anyone seriously wants to change it, it should be brought up on Village pumps there.
I'm not a train conductor, and I don't know anything about the decision-making process that went into the current train schedule. But I'll note that Mondays tend to be observed as holidays, which could cause complications for train scheduling. By my quick count from officewiki, there are 9 US holidays on Mondays in 2021 (not counting the end-of-year holiday).
I've been on teams that switched sprint start dates from Monday to Tuesday because we found the number of Monday holidays disruptive to our cadence.
Bill Pirkle Software Engineer www.wikimediafoundation.org
On Tue, Jun 22, 2021 at 3:04 PM Jon Robson jrobson@wikimedia.org wrote:
Hi all
A few questions to provoke discussion/share knowledge better:
- Why does the train run Tue,Wed, Thur rather than Mon,Tue,Wed
- Why do we only have 2 group 1 Wikipedia's (Catalan and Hebrew)
- Should there be a backport window Friday mornings for certain changes?
Longer spiel:
A few weeks ago a change I made led to a small but noticeable UI regression. The site was perfectly usable, but looked noticeably off. It was in a more obscure part of the UI so we missed it during QA/code review.
Late Wednesday a ticket was reported against Wikimedia commons, but I only became aware of it late Thursday when the regression rolled out to English Wikipedia. A village pump discussion was started and several duplicate tickets were created. While the site could still be used it didn't look great and upset the experience of many editors.
Once aware of the problem, the issue was easy to fix. A patch was written on Friday.
I understand Friday backports are possible, but my team tend to use them as a last resort in fear of creating more work for my fellow maintainers over weekend periods. As a result, given the site was still usable, the fix wasn't backported until the first available backport window on Monday. This is unfortunately a regular pattern, particularly for small UI regressions.
We addressed the issue on Monday, but I got feedback from several users that this particular issue took too long to get backported. I mentioned the no Friday deploy policy. One user asked me why we don't run the train Monday-Wednesday and to be honest I wasn't sure. I couldn't find anything on https://wikitech.wikimedia.org/wiki/Deployments/Train.
My team tries to avoid big changes on Mondays as Monday merged patches are more likely to have issues since they don't always get the time to go through QA during the week by our dedicated QA engineer.
So... Why don't we run the train Monday-Wednesday? Having a Thursday buffer during which we can more comfortably backport any issues not caught in testing, particularly UI bugs would be extremely helpful to my team and I don't think we'd lose much by losing the Monday to rush last-minute changes.
Assuming there are good reasons for Tuesday-Thursday train, I think there is another problem with our deploy process which is the size of group 1. Given the complexity of our interfaces (several skins, gadgets, multiple special pages, user preferences, gadgets, multiple extensions, and different user rights), generally, many obscure UI bugs get missed in QA by people who don't use the software every day and have a clear mental model of how it looks and behaves. My team mostly works on visible user interface changes and we rely heavily on Catalan and Hebrew Wikipedia users - our only group 1 wikis to notice errors with UI before they go out to a wider audience. Given the size of those audiences, that often doesn't work, and it's often group 2 wikis that make us aware of issues. If we are going to keep the existing train of Tue-Thur, I think it's essential we have at least one larger Wikipedia in our group 1 deploy to give us better protection against UI regressions living over the weekend. My understanding is for some reason this is not a decision release engineering can make, but one that requires an on-wiki RFC by the editors themselves. Is that correct? While I can understand the reluctance of editors to experience bugs, I'd argue that it's better to have a bug for a day than to have it for an entire weekend, and definitely something we need to think more deeply about.
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
On Tue, Jun 22, 2021 at 3:03 PM Jon Robson jrobson@wikimedia.org wrote:
A few questions to provoke discussion/share knowledge better:
- Why does the train run Tue,Wed, Thur rather than Mon,Tue,Wed
I'd note here that the standard security deployment window is Monday between 21:00 and 23:00 UTC. That date and time is not a hard requirement by any means, but having such a window exist early in the week, prior to the start of the train, has worked out well for a few reasons. It's both convenient and less risky to only deploy security patches to a single wmf production branch, which is the case most Mondays. It's also less risky having the space to monitor patches and roll them back or re-patch during the week, as opposed to say, on a Friday, with substantially reduced coverage going into most weekends. Of course there are times when critical security issues need to be dealt with on a Friday or even over the weekend, but in general, the Security Team likes to avoid this. Moving the train to a Mon, Tue, Wed cadence would imply the security window be moved to the previous Friday or possibly Thursday, which is doable, but not desired for the aforementioned reasons.
Jon, I think you're misunderstanding the point of the "No Deployment on Friday" policy.
Let's look from far, Why is a work day a no-deploy day? Why are we limiting ourselves to a four-day week while we can use our full potential? The reason is that if we deploy something to production on Friday and if it breaks production, then there is no one around to fix the issue over the weekend. So far it's rather obvious.
In other words, the requirement for having a time to deploy changes is to have some buffer until the weekend *to fix issues caused by changes deployed*. That buffer is Friday. It's made a no deployment day so you could push urgent fixes (including train blockers). Of course, the fix should not be too big to cause issues later.
Fridays are not "no deployment day" in the same of sense Sunday is a "no deployment day". It's the buffer to fix UBN issues, not a long weekend. If you're fixing an UBN issue, then please go ahead and deploy on Friday. The ultimate goal of RelEng policies is to have major issues live in production for the shortest possible time. Refusing to deploy a major fix on a Friday does the exact opposite of that.
Regarding Catalan and Hebrew Wikipedia, the other Amir said it well, I don't think I have much to add beside the fact that I have personally seen them finding major issues before they hit all Wikipedia languages many times, more than I can count.
HTH
On Tue, Jun 22, 2021 at 11:11 PM Scott Bassett sbassett@wikimedia.org wrote:
On Tue, Jun 22, 2021 at 3:03 PM Jon Robson jrobson@wikimedia.org wrote:
A few questions to provoke discussion/share knowledge better:
- Why does the train run Tue,Wed, Thur rather than Mon,Tue,Wed
I'd note here that the standard security deployment window is Monday between 21:00 and 23:00 UTC. That date and time is not a hard requirement by any means, but having such a window exist early in the week, prior to the start of the train, has worked out well for a few reasons. It's both convenient and less risky to only deploy security patches to a single wmf production branch, which is the case most Mondays. It's also less risky having the space to monitor patches and roll them back or re-patch during the week, as opposed to say, on a Friday, with substantially reduced coverage going into most weekends. Of course there are times when critical security issues need to be dealt with on a Friday or even over the weekend, but in general, the Security Team likes to avoid this. Moving the train to a Mon, Tue, Wed cadence would imply the security window be moved to the previous Friday or possibly Thursday, which is doable, but not desired for the aforementioned reasons.
-- Scott Bassett sbassett@wikimedia.org _______________________________________________ Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
Thanks for all the input so far.
On Tue, Jun 22, 2021 at 2:41 PM Amir Sarabadani ladsgroup@gmail.com wrote:
Jon, I think you're misunderstanding the point of the "No Deployment on Friday" policy.
I don't think I'm misunderstanding the policy? I'm talking explicitly about high priority issues UI regressions, not unbreak now (ie. site is still functional but styled incorrectly e.g. imagine a link is the wrong color). I've used Friday deployments historically, but only for really really bad issues.
To give an example, if an icon is visually overlapping text I don't consider that an UBN, I consider that unfortunate. If the icon overlap is on the edit icon and that's not clickable, definitely UBN and I'm happy to go the extra lengths to get a Friday patch out.
I understand the Friday is a buffer, but it's not a great buffer, particularly now at the Wikimedia foundation most people observe "Silent Fridays", and many people in teams that are involved in decision making work in different timezones. Side note what constitutes a UBN UI regression is being discussed on the talk page [1].
Regarding Catalan and Hebrew Wikipedia, the other Amir said it well, I
don't think I have much to add beside the fact that I have personally seen them finding major issues before they hit all Wikipedia languages many times, more than I can count.
Apologies if I didn't make myself clear, but it seems I didn't given both Amir's comments. I am very happy that we have these, and my question was *not *why do we have them, but rather* why do we only have 2*. I want more of them and every time I've asked why we don't have more, I've been told it's a community decision and that seems odd to me.
[1] https://wikitech.wikimedia.org/wiki/Talk:Deployments/Holding_the_train
Regarding Catalan and Hebrew Wikipedia, the other Amir said it well, I
don't think I have much to add beside the fact that I have personally seen them finding major issues before they hit all Wikipedia languages many times, more than I can count.
Apologies if I didn't make myself clear, but it seems I didn't given both Amir's comments. I am very happy that we have these, and my question was *not *why do we have them, but rather* why do we only have 2*. I want more of them and every time I've asked why we don't have more, I've been told it's a community decision and that seems odd to me.
Anyone is welcome to correct me if I'm wrong, but I'm quite sure that there is no community decision to have *only* two. I suggested these two back then, because I felt that it is *enough*, I can read these two languages and I thought these two communities would not be opposed to it, and I even happened to be right. (Hebrew also represents RTL languages well.)
So there is probably no blocker to have more than two. You should probably ask the relevant wikis similarly to how I did back then, get an agreement from them and from Rel. Eng. and SRE, and that's it.
How did I ask them? I just raised it myself with my personal account in the Village pumps, there were a few positive responses, and that was it, it worked and no one complained. But don't take my word for it :) You can do the same or maybe you can go through a more formal process. Up to you. Which languages to pick? - I'm also not sure. Maybe some wikis that represent special features, like the Growth extensions, or language converter, etc. Also up to you.
On Wed, Jun 23, 2021 at 12:10 AM Jon Robson jrobson@wikimedia.org wrote:
I understand the Friday is a buffer, but it's not a great buffer, particularly now
I don't have any suggestions, as I am not super-familiar with the current situation, but I can understand (and suffered) from some hidden bugs (not immediately obvious) taking some time to surface and/or reach the right people, but I have some questions:
* How often are issues surfaced in the group0 -> group1 vs group1 -> group2, are there any stats to back the need for a change there? * Without changing the actual deploying days or the frequency, would there be any benefit of shifting the deploy over multiple weeks? (random example Tu: group1->group2, (new branch) We: group0, Th: group0-> group1) or would that make things worse? * You mention commons. I am guessing Commons, and Wikidata, to some extent- are both large sites with a lot of visibility but also very different from the core features that are similar to most other wikis, but the test version of those on group0 may not be enough to catch all issues. Is there something that could be improved specifically for those sites? * Can we do something to improve the speed from "a user notices an issue with the site" to "the right team/owner is aware of it and acts on it"?
(Late reply: I was out the week this was sent, then another week of vacation happened)
On Wed, Jun 23, 2021 at 2:59 AM Jaime Crespo jcrespo@wikimedia.org wrote:
- How often are issues surfaced in the group0 -> group1 vs group1 ->
group2, are there any stats to back the need for a change there?
The closest number I have to issues/group is the count of new "blocker" tasks filed in phabricator per group.
Progressive rollout to each group gives us more confidence in the code being deployed, so for each group we should see progressively fewer blockers.
🚂📈*Group vs blocker discovery over the past 153 trains**:* [image: README_10_0.png]
- *Before group0*: 233 - *Group0*: 180 - *Group1*: 230 - *Group2*: 91
"Before group0" means that before we've rolled out the train to any wiki in wikiprod, there's a blocker on the train task (just like today 1.37.0-wmf.14 is not deployed anywhere, but there's a blocker on the train task: https://phabricator.wikimedia.org/T281155 ).
If we want each group to have progressively fewer blockers for each group then the data shows that group0 is too small and/or group1 is too big. There are other considerations. Deployers have a lot of work to do on group0 day vs group1 day: so making group0 bigger/more useful for developers makes the lives of deployers harder.
* Without changing the actual deploying days or the frequency, would there
be any benefit of shifting the deploy over multiple weeks? (random example Tu: group1->group2, (new branch) We: group0, Th: group0-> group1) or would that make things worse?
I wonder what impact this change would have on blocker reports. For instance, is it a function of the time left in the week that group2 surfaces relatively few blockers?
- You mention commons. I am guessing Commons, and Wikidata, to some
extent- are both large sites with a lot of visibility but also very different from the core features that are similar to most other wikis, but the test version of those on group0 may not be enough to catch all issues. Is there something that could be improved specifically for those sites?
This is a subset of a question I've been asking folks: why does the train give us confidence? What does a train give us that a testing environment like beta or a local environment can't give us? I think some of the magic of train is the amount of traffic, but if that were the case then artificial traffic should suffice. I think the other aspect of the train is Hyrum's law[0]—all observable properties of a system are hammered with traffic: even observable properties that were not built intentionally.
- Can we do something to improve the speed from "a user notices an issue
with the site" to "the right team/owner is aware of it and acts on it"?
Or can we do something to improve how many issues users notice? :)
Thanks for all of these great questions. – Tyler
On Mon, 12 Jul 2021 at 19:26, Tyler Cipriani tcipriani@wikimedia.org wrote:
(Late reply: I was out the week this was sent, then another week of vacation happened)
- Can we do something to improve the speed from "a user notices an issue
with the site" to "the right team/owner is aware of it and acts on it"?
Or can we do something to improve how many issues users notice? :)
As someone who's been around for a long time as an editor, I can say honestly that having most of the issues addressed before they hit the really big projects has resulted in a huge improvement. The train really works, and the only challenge I really see is what Jon mentions in his original post. Some of those issues aren't really that significant in the great scheme of things, but there's a big leap when something takes two business days to fix from the Tuesday deployment and two business days to fix from the Thursday deployment.
It's not always possible for even the best developer and the best testing systems to catch an issue that will be spotted by a hands-on user, several of whom are much more familiar with the purpose, expected outcomes and change impact on extensions than the people who have written them or QA'd them. That's why there will always be plenty of issues that are identified by users, and it is in no way a problem that a small number of them (compared to what we saw 10-15 years ago) get through to the end of the train before being identified as needing to be addressed (for different values of "addressed").
Other people on this list are in a better position to understand the implications of backporting on Fridays, but I think it would be worthwhile to give serious consideration to expanding the list of what could be included in a Friday backport. Limiting it to essentially "breaks the site" or "major impact to accessibility of the content" doesn't really include most of the noticeable user experience issues (for either editors or readers), and leaving those sitting for a minimum of half a week is suboptimal.
Risker/Anne
On Mon, Jul 12, 2021 at 10:47 PM Risker risker.wp@gmail.com wrote:
On Mon, 12 Jul 2021 at 19:26, Tyler Cipriani tcipriani@wikimedia.org wrote:
- Can we do something to improve the speed from "a user notices an issue
with the site" to "the right team/owner is aware of it and acts on it"?
Or can we do something to improve how many issues users notice? :)
As someone who's been around for a long time as an editor, I can say honestly that having most of the issues addressed before they hit the really big projects has resulted in a huge improvement. The train really works, and the only challenge I really see is what Jon mentions in his original post. Some of those issues aren't really that significant in the great scheme of things, but there's a big leap when something takes two business days to fix from the Tuesday deployment and two business days to fix from the Thursday deployment.
It's not always possible for even the best developer and the best testing systems to catch an issue that will be spotted by a hands-on user, several of whom are much more familiar with the purpose, expected outcomes and change impact on extensions than the people who have written them or QA'd them. That's why there will always be plenty of issues that are identified by users, and it is in no way a problem that a small number of them (compared to what we saw 10-15 years ago) get through to the end of the train before being identified as needing to be addressed (for different values of "addressed").
Thank you for this response! The train existed before I started thinking about MediaWiki-software deployment. The impression that it has had a positive impact on the number of problems seen by users is important information. Your response is a fantastic answer to a different question I wonder about a lot: why does the train process give us confidence in the code being released?
The next part of that question is: are there ways we can gain this confidence with less disruption? I'd be interested in trying to catalog the types of problems that are only spotted by hands-on users in the interest of seeing if patterns emerge.
Thank you again! – Tyler
On Tue, 22 Jun 2021 at 23:10, Jon Robson jrobson@wikimedia.org wrote:
Apologies if I didn't make myself clear, but it seems I didn't given both Amir's comments. I am very happy that we have these, and my question was *not *why do we have them, but rather* why do we only have 2*. I want more of them and every time I've asked why we don't have more, I've been told it's a community decision and that seems odd to me.
It's a community decision in the sense that it wasn't something the WMF decided for them, it was those communities decided it for themselves. They're open to it probably as a result of being generally less conservative when it comes to changes, and more willing to try things out early. But, it's not a community decision in the sense that you, as staff, can't get involved and try to recruit more wikis!
So, if other wikis are asked if they want to be in group 1, understand the advantages and disadvantages of that, and they said yes, I doubt anyone would raise any objections. Having more wikis in that group to test things on early would definitely be a benefit.
Amir's suggestion to contact wikis that have been open to Growth features, like Czech and Korean (I think?), sounds like a great one, especially if there are community ambassadors that could help explain what it means to be in group 1.
Dan
Thanks all for the interesting discussion. I think the most immediate actionable here is expanding group 1 wikis, so I'm looking into that here https://phabricator.wikimedia.org/T286664. Ideally, it's my belief that 2 top ten wikis that are not English would give us the visibility of problems with UI changes that we need.
Some thoughts that come to mind: * What if group 0 and 1 were merged? How would that impact the train? From the data for the last 153 trains it seems those usually capture the majority of our issues. It wouldn't help capturing issues in group 2, but it would give more of a buffer to fix them. * What if group 0 happened after the security window on a Monday? Would that be too stressful/not feasible for those involved?
For me, regarding backporting on Fridays, ideally, as an engineer I appreciate a buffer going into the weekend to reduce any anxiety around my work having incapacitated our editors in some way. Deploying code on Friday's only increases that anxiety in some cases.
On Tue, Jun 22, 2021 at 3:09 PM Jon Robson jrobson@wikimedia.org wrote:
Thanks for all the input so far.
On Tue, Jun 22, 2021 at 2:41 PM Amir Sarabadani ladsgroup@gmail.com wrote:
Jon, I think you're misunderstanding the point of the "No Deployment on Friday" policy.
I don't think I'm misunderstanding the policy? I'm talking explicitly about high priority issues UI regressions, not unbreak now (ie. site is still functional but styled incorrectly e.g. imagine a link is the wrong color). I've used Friday deployments historically, but only for really really bad issues.
To give an example, if an icon is visually overlapping text I don't consider that an UBN, I consider that unfortunate. If the icon overlap is on the edit icon and that's not clickable, definitely UBN and I'm happy to go the extra lengths to get a Friday patch out.
I understand the Friday is a buffer, but it's not a great buffer, particularly now at the Wikimedia foundation most people observe "Silent Fridays", and many people in teams that are involved in decision making work in different timezones. Side note what constitutes a UBN UI regression is being discussed on the talk page [1].
Regarding Catalan and Hebrew Wikipedia, the other Amir said it well, I
don't think I have much to add beside the fact that I have personally seen them finding major issues before they hit all Wikipedia languages many times, more than I can count.
Apologies if I didn't make myself clear, but it seems I didn't given both Amir's comments. I am very happy that we have these, and my question was *not *why do we have them, but rather* why do we only have 2*. I want more of them and every time I've asked why we don't have more, I've been told it's a community decision and that seems odd to me.
[1] https://wikitech.wikimedia.org/wiki/Talk:Deployments/Holding_the_train
This makes me wonder if there could ever be a way to allow some requests to be on one group and other requests for the same wiki could be another group.
e.g. users could opt-in to being a canary or some users could be randomly selected.
When would this not work? If a newer version required schema change then have to stop using old version after applying new schema? but that isn't very common? any other complications?
-Jeremyb
You can already opt in to testing undployed code by using the WikimediaDebug[1] browser extension (available for firefox[2] and chrome[3])
Maybe we should add an option to force a request to be served from the latest branch instead of the one assigned in wikiversions.json.
[1] https://wikitech.wikimedia.org/wiki/WikimediaDebug [2] https://addons.mozilla.org/en-US/firefox/addon/wikimedia-debug-header/ [3] https://chrome.google.com/webstore/detail/wikimediadebug/binmakecefompkjggik...
On Tue, Jul 13, 2021 at 12:21 AM Jeremy Baron jeremy@tuxmachine.com wrote:
This makes me wonder if there could ever be a way to allow some requests to be on one group and other requests for the same wiki could be another group.
e.g. users could opt-in to being a canary or some users could be randomly selected.
When would this not work? If a newer version required schema change then have to stop using old version after applying new schema? but that isn't very common? any other complications?
-Jeremyb _______________________________________________ Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
wikitech-l@lists.wikimedia.org