I do appreciate that the ops team is working to improve reliability and performance of the database access. Unfortunately it seems to me that there is a disconnect between ops and tool devs. I wonder if the ops actually looked at how many user databases have been created and how frequently they got accessed (all that info should be readily available to them). The logs would also have told the ops which users relied in user DBs on the project DB servers. A direct email ahead of time would have gone a long way. 
The phabricator post contains the same language I've heard many times before: The tools devs shouldn't have used the feature anyways. To that I say, well, we still did and it worked great. Volunteer developers have a limited time budged with which they create tools that large amounts of users (editors and readers alike) rely on. That is just the reality of things, and it is not the ideal op fantasy, I know. The ops seem to be in an asymmetric position of power here. It sure sounds a lot like a take it or leave it situation to me.

On Sat, Dec 23, 2017 at 12:13 PM John <phoenixoverride@gmail.com> wrote:
why did this happen?

Martin

See https://phabricator.wikimedia.org/phame/post/view/70/new_wiki_replica_servers_ready_for_use/
_______________________________________________
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud