Forwarding to wider reach.
Cheers,
CT
---------- Forwarded message ----------
From: Ct Woo <ctwoo(a)wikimedia.org>
Date: Thu, Oct 4, 2012 at 11:24 AM
Subject: Update on Ashburn data center migration/switchover date
To: ops(a)wikimedia.org, WMF Engineering Management <emgt(a)wikimedia.org>
All,
We set a Q2 goal to migrate our primary data center from Tampa to
Ashburn, and earlier, communicated the planned switchover to be on
15th Oct., 2012.
While we have been making good progress and completing the work on
several key components such as databases, image-scalers, apache
servers, and /home, we have also been encountering issues with
Swift software (i.e., cross data center replication), Swift hardware
failures (meltdown actually), Varnish video streaming and unexpected
network cabling issues with the memcached servers. Resources have been
diverted to work on resolving those issues and caused some delay in
the schedule as a result.
This week, Mark and Asher performed an assessment on the migration
readiness state and concluded they are increasingly uncomfortable
meeting the migration date, and even if we were to complete the rest
of the outstanding tasks, the migration risks are high especially when
we will be deploying new technologies (i.e., Redis and NetApp as
multimedia store) that are inadequately tested.
Since the Fundraising team is ramping up A/B testings now, and will be
going full scale Fundraising mode this November till December (or
January, ) we have decided to postpone the switchover till after all
the Fundraising activities are over.
The migration team will continue to work on completing the
outstanding tasks, and to ready our Ashburn infrastructure for the
big switchover day. Meantime, we will set up Ashburn data center to
be a 'warm' backup site, in a standby mode, ready to take over full
production traffic within a short period of time, should the need
arises.
Thanks.
CT