I am sorry to hear that. It looks like something that we will have to take into account for the next switchovers. That being said, we had deliberations across the involved teams months ago to come up with those exact dates and have been communicating them via at least SoS since 2018-08-01.
I am curious about something though. How does the deployment train (cause that's what we are talking about) impact the software release exactly ? On Wed, Oct 3, 2018 at 6:19 PM C. Scott Ananian cananian@wikimedia.org wrote:
Oct 8 seems to be a particularly bad time to freeze the train given that we are forking for the MW 1.32 release on Oct 15, and a lot of folks have last-minute things they want to get into the release (eg deprecations, etc). --scott
On Thu, Aug 30, 2018 at 10:57 AM Pine W wiki.pine@gmail.com wrote:
+1 to DJ's question about timing. Also, one might wish to be mindful of the number of recent trains that were supposed to be boring but involved interesting surprises; this makes me wonder whether trains that one thinks will be boring are actually OK in this circumstance even if they turn out to be "interesting".
Pine ( https://meta.wikimedia.org/wiki/User:Pine )
-------- Original message --------From: Derk-Jan Hartman < d.j.hartman+wmf_ml@gmail.com> Date: 8/30/18 2:54 AM (GMT-08:00) To: Wikimedia developers wikitech-l@lists.wikimedia.org Subject: Re: [Wikitech-l] Datacenter switchover and switchback While I think these regular switches are a very good idea, from an outside perspective I do have to question a process that puts a significant plug in the velocity of various teams working on major projects (esp. in a time of year that could probably be seen as one of the most productive). What are plans to reduce the disruption of this exercise in the future ?
DJ
On Thu, Aug 30, 2018 at 8:38 AM Jaime Crespo jcrespo@wikimedia.org wrote:
Let me explain the rationale of the bellow request for clarification:
On Wed, Aug 29, 2018 at 11:30 PM MA strigiwm@gmail.com wrote:
Hello:
For the duration of the switchover (1 month), deployers are kindly requested to refrain from large db schema changes and avoid deploying any kind of new feature that requires creation of tables. There will be a train freeze in the week of Sept 10th and Oct 8th.
During the failover, some schema changes will be finalized on the current active datacenter (plus some major server and network maintenance may be done)- our request is mostly to refrain from quickly enabling those large new unlocked features (e.g. the ongoing comment refactoring, actor/user refactoring, Multi Content Revision, JADE, major wikidata or structured comons structure changes, new extensions not ever deployed to the
cluster,
etc.) at the same time than the ongoing maintenance to reduce variables
of
things that can go bad- enabling those features may be unblocked during
the
switchover time, but we ask you to hold until being back on the current active datacenter. Basically, ask yourself if you are enabling a large
new
core feature or want to start a heavy-write maintenance script and there
is
a chance you will need DBA/system support. Sadly, we had some instances
of
this happening last year and we want to explicitly discourage this during these 2 weeks.
In own my opinion, enabling existing features on smaller projects (size here is in amount of server resources, not that they are less important)
is
equivalent to a swat change, and I am not against it happening. I would
ask
contributors to use their best judgement on every case, and ask people on the #DBA tag on phabricator or add me as reviewers on gerrit if in doubt. My plea is to not enable major structural changes during that time may affect thousands of edits per minute. Swat-like changes and "boring" :-) trains are ok.
For new wiki creations I would prefer if those were delayed but CC #DBA s on the phabricator task to check with us. _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
-- (http://cscott.net) _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l