Hi all A few questions to provoke discussion/share knowledge better: * Why does the train run Tue,Wed, Thur rather than Mon,Tue,Wed * Why do we only have 2 group 1 Wikipedia's (Catalan and Hebrew) * Should there be a backport window Friday mornings for certain changes? Longer spiel: A few weeks ago a change I made led to a small but noticeable UI regression. The site was perfectly usable, but looked noticeably off. It was in a more obscure part of the UI so we missed it during QA/code review. Late Wednesday a ticket was reported against Wikimedia commons, but I only became aware of it late Thursday when the regression rolled out to English Wikipedia. A village pump discussion was started and several duplicate tickets were created. While the site could still be used it didn't look great and upset the experience of many editors. Once aware of the problem, the issue was easy to fix. A patch was written on Friday. I understand Friday backports are possible, but my team tend to use them as a last resort in fear of creating more work for my fellow maintainers over weekend periods. As a result, given the site was still usable, the fix wasn't backported until the first available backport window on Monday. This is unfortunately a regular pattern, particularly for small UI regressions. We addressed the issue on Monday, but I got feedback from several users that this particular issue took too long to get backported. I mentioned the no Friday deploy policy. One user asked me why we don't run the train Monday-Wednesday and to be honest I wasn't sure. I couldn't find anything on https://wikitech.wikimedia.org/wiki/Deployments/Train. My team tries to avoid big changes on Mondays as Monday merged patches are more likely to have issues since they don't always get the time to go through QA during the week by our dedicated QA engineer. So... Why don't we run the train Monday-Wednesday? Having a Thursday buffer during which we can more comfortably backport any issues not caught in testing, particularly UI bugs would be extremely helpful to my team and I don't think we'd lose much by losing the Monday to rush last-minute changes. Assuming there are good reasons for Tuesday-Thursday train, I think there is another problem with our deploy process which is the size of group 1. Given the complexity of our interfaces (several skins, gadgets, multiple special pages, user preferences, gadgets, multiple extensions, and different user rights), generally, many obscure UI bugs get missed in QA by people who don't use the software every day and have a clear mental model of how it looks and behaves. My team mostly works on visible user interface changes and we rely heavily on Catalan and Hebrew Wikipedia users - our only group 1 wikis to notice errors with UI before they go out to a wider audience. Given the size of those audiences, that often doesn't work, and it's often group 2 wikis that make us aware of issues. If we are going to keep the existing train of Tue-Thur, I think it's essential we have at least one larger Wikipedia in our group 1 deploy to give us better protection against UI regressions living over the weekend. My understanding is for some reason this is not a decision release engineering can make, but one that requires an on-wiki RFC by the editors themselves. Is that correct? While I can understand the reluctance of editors to experience bugs, I'd argue that it's better to have a bug for a day than to have it for an entire weekend, and definitely something we need to think more deeply about. _______________________________________________ Wikitech-l mailing list -- wikitech-l(a)lists.wikimedia.org To unsubscribe send an email to wikitech-l-leave(a)lists.wikimedia.org https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

On Tue, Jun 22, 2021 at 3:03 PM Jon Robson <jrobson(a)wikimedia.org> wrote: I'd note here that the standard security deployment window is Monday between 21:00 and 23:00 UTC. That date and time is not a hard requirement by any means, but having such a window exist early in the week, prior to the start of the train, has worked out well for a few reasons. It's both convenient and less risky to only deploy security patches to a single wmf production branch, which is the case most Mondays. It's also less risky having the space to monitor patches and roll them back or re-patch during the week, as opposed to say, on a Friday, with substantially reduced coverage going into most weekends. Of course there are times when critical security issues need to be dealt with on a Friday or even over the weekend, but in general, the Security Team likes to avoid this. Moving the train to a Mon, Tue, Wed cadence would imply the security window be moved to the previous Friday or possibly Thursday, which is doable, but not desired for the aforementioned reasons. -- Scott Bassett sbassett(a)wikimedia.org _______________________________________________ Wikitech-l mailing list -- wikitech-l(a)lists.wikimedia.org To unsubscribe send an email to wikitech-l-leave(a)lists.wikimedia.org https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

Regarding Catalan and Hebrew Wikipedia, the other Amir said it well, I

Apologies if I didn't make myself clear, but it seems I didn't given both Amir's comments. I am very happy that we have these, and my question was *not *why do we have them, but rather* why do we only have 2*. I want more of them and every time I've asked why we don't have more, I've been told it's a community decision and that seems odd to me.

Apologies if I didn't make myself clear, but it seems I didn't given both Amir's comments. I am very happy that we have these, and my question was *not *why do we have them, but rather* why do we only have 2*. I want more of them and every time I've asked why we don't have more, I've been told it's a community decision and that seems odd to me.

(Late reply: I was out the week this was sent, then another week of vacation happened) Or can we do something to improve how many issues users notice? :)

This makes me wonder if there could ever be a way to allow some requests to be on one group and other requests for the same wiki could be another group. e.g. users could opt-in to being a canary or some users could be randomly selected. When would this not work? If a newer version required schema change then have to stop using old version after applying new schema? but that isn't very common? any other complications? -Jeremyb _______________________________________________ Wikitech-l mailing list -- wikitech-l(a)lists.wikimedia.org To unsubscribe send an email to wikitech-l-leave(a)lists.wikimedia.org https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

On Mon, 12 Jul 2021 at 19:26, Tyler Cipriani <tcipriani(a)wikimedia.org> wrote: As someone who's been around for a long time as an editor, I can say honestly that having most of the issues addressed before they hit the really big projects has resulted in a huge improvement. The train really works, and the only challenge I really see is what Jon mentions in his original post. Some of those issues aren't really that significant in the great scheme of things, but there's a big leap when something takes two business days to fix from the Tuesday deployment and two business days to fix from the Thursday deployment. It's not always possible for even the best developer and the best testing systems to catch an issue that will be spotted by a hands-on user, several of whom are much more familiar with the purpose, expected outcomes and change impact on extensions than the people who have written them or QA'd them. That's why there will always be plenty of issues that are identified by users, and it is in no way a problem that a small number of them (compared to what we saw 10-15 years ago) get through to the end of the train before being identified as needing to be addressed (for different values of "addressed").