On Tue, Nov 20, 2018 at 4:41 PM Mukunda Modell <mmodell(a)wikimedia.org> wrote: The beta cluster wikis are working again. It turns out that there was some confusion when moving/removing servers because of implementation drift between our production clusters and the beta cluster. Before the move to the new region we had both "memc*" and "redis*" servers in the beta cluster project. The "memc*" servers are the equivalent of our production "mc*" servers. In production the "mc*" servers run both memcached and redis services. In the beta cluster our "memc*" servers were only providing memcached and the configuration relied on the "redis*" servers for session storage. The "redis*" servers were removed while migrating virtual machines to the eqiad1-r region under the assumption that they were legacy servers from the time when we used redis as storage for the job queue. The fix was to setup the "memc*" servers with both memcached for arbitrary data caching and redis for session storage. If you are interested in the gory details see notes left on https://phabricator.wikimedia.org/T210030. Bryan -- Bryan Davis Wikimedia Foundation <bd808(a)wikimedia.org> [[m:User:BDavis_(WMF)]] Manager, Technical Engagement Boise, ID USA irc: bd808 v:415.839.6885 x6855

Also: * Puppet is disabled on deployment-mediawiki-07 for unknown reasons https://phabricator.wikimedia.org/T208101#4770736 * Confusion about the status of deployment-redis*, they seem completely broken but have important references? https://phabricator.wikimedia.org/T210301 On Fri, 23 Nov 2018 at 17:15, Alex Monk <krenair(a)gmail.com> wrote: