<quote name="John" date="2015-05-27" time="16:24:17 -0400"> No, the time to Wikipedias from branch cut is 2 days. I trust our code review, integration, and testing workflows. If it turns out this is too aggressive, we'll switch back to the previous cadence. Greg -- | Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E | | identi.ca: @greg A18D 1138 8E47 FAC8 1C7D |

<quote name="S Page" date="2015-05-27" time="14:58:16 -0700"> right right, I just mentally lump mw.org with "test wikis" ;) -- | Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E | | identi.ca: @greg A18D 1138 8E47 FAC8 1C7D |

This is strictly a question from an uninvolved observer. Does this schedule provide for sufficient time and real-time/hands-on testing before changes hit the big projects? An IRC discussion I was following last evening suggested to me that the first deploy (to test wikis and mw.org) probably did not get sufficient hands-on testing/utilization to surface many issues that would be significant on production wikis, which means only 24 hours on smaller non-wikipedia wikis, hoping that any problems will pop up before it's applied to dewiki, frwiki and enwiki. I recognize the challenges in balancing continuous improvement and uptime - but if problems aren't surfaced before they hit wikipedias simply because the changes aren't activated by user actions or the problems aren't reported quickly enough, then it's probably going to make more work at the other end of the chain, with more likelihood that changes will need to be rolled back or patches having to be written on the fly. I have a lot of admiration for all of you who address these unplanned situations (it really is impressive to watch!), but I'd hate to see a lot of people constantly being pulled away from other tasks to problem-solve downtimes on big projects. Risker/Anne On 28 May 2015 at 07:51, Dan Garry &lt;dgarry(a)wikimedia.org&gt; wrote:

On May 28, 2015 4:21 PM, "Jon Robson" &lt;jdlrobson(a)gmail.com&gt; wrote: tuning Significantly better than three years ago, sure. However I would not use the phrase great place. There are still significant gaps in our coverage. For browser tests in particular Im given to understand that the asynchronous nature and somewhat high false positive rate make them not be taken as seriously as they should. Fwiw, i have on several occasions recieved reports from users that I broke something (obviously i try to avoid that, but im far from perfect). I have never once had a browser test succesfully tell me I broke something in advanced (albeit it could be because i do backend things, but backend things do affect the front end when they explode). Occasionally the unit tests do, but i would still say there is a lot they dont. Tl;dr: imo tests are great, but no where near replacing actual testing, at least for now. That said, i dont think that the new deployment schedule will cause any problems, and is at the very least worth trying. --bawolff P.s. anyone remember writing code in the time between 1.16 and 1.17? You are all spoiled :p

<quote name="Dan Garry" date="2015-05-28" time="13:51:47 +0200"> :) 1) the above (happier teams) 2) It's going to be hard to measure "success" but it'll be much easier to identify failure. I'll be talking with many PMs from WMF and looking through the SWAT deploys and incidents over the next two or more weeks (and on going, of course) to determine if this has caused any unmitigated pain. Greg -- | Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E | | identi.ca: @greg A18D 1138 8E47 FAC8 1C7D |

<quote name="Legoktm" date="2015-05-28" time="10:17:19 -0700"> Emergency bug fixes are already OK on Fridays (just not "I want my new feature out"). -- | Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E | | identi.ca: @greg A18D 1138 8E47 FAC8 1C7D |

<quote name="John Mark Vandenberg" date="2015-05-29" time="01:39:52 +0700"> Which shows the tooling works, but not the social aspects. The backport process (eg SWAT and related things) will improve soon as well which should address much of this. Not-a-great-response-but: can you specifically ping me in phabricator (I'm @greg) for issues like that above? -- | Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E | | identi.ca: @greg A18D 1138 8E47 FAC8 1C7D |

<quote name="John Mark Vandenberg" date="2015-05-29" time="04:11:05 +0700"> That's not "my" tooling, that's pywikibot's ;). But, the point is, there was a problem identified in your testing that was reported and fix submitted in a reasonable amount of time. The failure to get it merged, however, was the failure. I would support having pywikibot use WMF hosted integration testing. Please file a task with your current setup in the #continuous-integration-config project: https://phabricator.wikimedia.org/project/profile/1208/ Concur. -- | Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E | | identi.ca: @greg A18D 1138 8E47 FAC8 1C7D |

Le 28/05/2015 20:39, John Mark Vandenberg a écrit : <snip> I cant stress how important pywikibot is! It covers so many functionalities and use cases that it is an excellent test stress for the API. A low hanging fruit would be to run its test suite against beta (which runs tip of master) on an hourly basis. -- Antoine "hashar" Musso

I'll echo those nervous about the faster pace for deploys -- although not so nervous as to dig my feet in and yell stop. Mostly my concerns boil down to the fact that the beta environment isn't really a good test for anything other than absolute crashers. Hardly anyone uses the Collection extension (for example) on beta, for example. I'd *really* like to see some effort put into improving beta. In particular, running beta with an up-to-date (but sanitized, perhaps) mirror of the main WP databases would ensure that we have a decent amount of test cases on beta. We recently spent a couple of hours trying to test Parsoid on beta, only to find out that the behavior we were chasing down was caused by the fact that beta's copy of the IPA formatting templates on enwiki (!) were out-of-date and incomplete. We really need to do better than [[English language]] as far as articles to test on enbeta. --scott

On Fri, May 29, 2015 at 9:36 PM, Brad Jorsch (Anomie) &lt;bjorsch(a)wikimedia.org&gt; wrote: In the proposed deploy sequence, 12 hours is a serious amount of time. So if a shorter deploy process is implemented, we need to find ways to get bug reports to you sooner, and ensure you are not the only one who can notice and fix bugs related to API breakages, etc. On a positive note, the API breakage this week was rectified much quicker and pywikibot test builds are green again. https://phabricator.wikimedia.org/T100775 https://travis-ci.org/wikimedia/pywikibot-core/builds/64631025 Hmm. You still don't appreciate that you actually really truly fair-dinkum broke pywikibot....? warnings are part of the API, and adding/changing them can break clients. warnings are one of the more brittle part of the API. The impact on pywikibot core users wasnt so apparent, as the pywikibot core devs fixed the problem when it hit the test servers and it was merged before it hit production servers. Not all users had 'git pull' the latest pywikibot core code, and they informed us their bots were broken, but as far as I am aware we didnt get any pywikibot core bug reports submitted because (often after mentioning problems on IRC) their problems disappeared after they ran 'git pull'. However pywikipedia / compat isnt actively maintained, and it broke badly in production, with some scripts being broken for over a month: https://phabricator.wikimedia.org/T74667 https://phabricator.wikimedia.org/T74749 pywikibot core is gradually improving its understanding of the API warning system, but it isnt well supported yet. As a result, generally pywikibot reports warnings to the user. IIRC your API warnings system can send multiple distinct warnings as a single string, with each warning separated by only a new line, which is especially nasty for user agents to 'understand'. (but this may only be in older versions of the API - I'm not sure) So adding a new warning to the API can result in the same warning appearing many many times on the user console / logs, and thousands of warnings on the screen sends the users into panic mode. I strongly recommend fixing the warning system before using it again aggressively like was done for rawcontinue. e.g. It would be nice if the API emitted codes for each warning scenario (dont reuse codes for similar scenarios), so we don't need to do string matching to detect & discard expected warnings, and you can i18n those messages without breaking clients. (I think there is already a phab task for this.) I also strongly recommend that Wikimedia gets heavily involved in decommissioning pywikibot compat bots on Wikimedia servers, and any other very old unmaintained clients, so that the API can be aggressively updated without breaking the many bots still using compat. pywikibot devs did some initial work with WMF staff at Lyon on this front, and we need to keep that moving ahead. Yes, some of the code was raising an exception when it detected an API warning. However another part of the breakage was that the JSON structure of the new rawcontinue warnings was not what was expected. Some pywikipedia / compat code assumed that the presence of warnings implied that there would be a warning related to the module used, e.g. result['warnings']['allpages'] , but that didnt exist because this warning was in result['warnings']['query']. https://gerrit.wikimedia.org/r/#/c/176910/4/wikipedia.py,cm https://gerrit.wikimedia.org/r/#/c/170075/2/wikipedia.py,cm I presume you mean we could have just added rawcontinue='' . Our very limited testing at the time (we didnt have much time to fix the bugs before it would hit production), and subsequent testing, indicates that the rawcontinue parameter can be used even for very old versions of MediaWiki. It appears to be silently ignored by earlier versions of MediaWiki. If this is true, it would have been good to very explicitly mention in the mediawiki-api-announce message that adding rawcontinue='' is backwards compatible , and to which version it will work. To be safe, we implemented version detection so that rawcontinue is only used on the MediaWiki versions that require it. That was not trivial, as detecting the version means requesting action=query&meta=siteinfo , and that would cause this warning if rawcontinue wasnt used. Do you notice the loop there? In order to avoid using a feature on the wrong version, we need to use the feature. In case you are interested, the pywikibot core merged change was https://gerrit.wikimedia.org/r/#/c/168529/ (there were two other attempts at solving it which were abandoned.) Thankfully, you eventually reduced the occurrences of those warnings (https://gerrit.wikimedia.org/r/#/c/201193/ ), so that a single simple meta=siteinfo query doesn't cause an unnecessary warning to appear in the result. I am not sure whether we can rely on that though, as you strenuously assert that meta=siteinfo could require continuation - if so, our site info abstraction layer is probably broken whenever meta=siteinfo actually does ever require continuation. As a result of this, I ask that more care is given to changes to the output of siteinfo, as that is how clients detect your version, which is a critical step in negotiating the 'protocol version' used. pywikibot tries to use feature detection when sensible instead of version detection, but that is harder to implement and involves shooting off more requests to the server to detect what fails, slowing down the bootstrapping sequence for each site. unit tests, development processes, test process, and production usage. Everything broke. Jenkins was -1'ing all patches uploaded into Gerrit pywikibot because a very simple doctest failed. As a result, a solution needed to be merged before normal development activities could resume. ;-) https://lists.wikimedia.org/pipermail/pywikipedia-l/2014-October/009140.html Yea, we're also to blame for not picking up on this. We are paying a bit more attention to that mailing list now as a result;-) It is hard to know what would have worked better, but the 'its just a warning' mentality feels like the underlying problem. A single notice and a month delay is probably not enough, and "About a month from now" might have got a better response than "Sometime during the MediaWiki 1.25 development cycle". IMO Wikimedia should have be doing impact analysis before flicking the switch. i.e. tracking how many Wikimedia wiki clients were switching to the new format to avoid the warnings during that month, and which user agents were not responding. I understand that is happening now before the final switch to 'continue' as default, but it would have been useful to start that process before releasing the warning. This change didnt affect pywikibot, so I dont know of any actual impact, and there may have been none, but afaik there was not appropriate notice of this intentional breakage, removing functionality. An RFC is great, and in that RFC there was a concern raise, albeit not backed with an actual usage which would be broken. A real notice of the actual breakage is still needed, even if there was no concerns raised during the RFC. My point is not that these changes were catastrophic - tools were updated, the wikis lived on, and a few pywikibot compat users probably switched to pywikibot core as a result of a month long breakage. My point is that shortening the timeframes of deploying reduces the ability for volunteers to detect and raise bugs about breakages, and/or update their client code. -- John Vandenberg