Forwarding to wider reach.

Cheers, CT ---------- Forwarded message ---------- From: Ct Woo ctwoo@wikimedia.org Date: Thu, Oct 4, 2012 at 11:24 AM Subject: Update on Ashburn data center migration/switchover date To: ops@wikimedia.org, WMF Engineering Management emgt@wikimedia.org

All,

We set a Q2 goal to migrate our primary data center from Tampa to Ashburn, and earlier, communicated the planned switchover to be on 15th Oct., 2012.

While we have been making good progress and completing the work on several key components such as databases, image-scalers, apache servers, and /home, we have also been encountering issues with Swift software (i.e., cross data center replication), Swift hardware failures (meltdown actually), Varnish video streaming and unexpected network cabling issues with the memcached servers. Resources have been diverted to work on resolving those issues and caused some delay in the schedule as a result.

This week, Mark and Asher performed an assessment on the migration readiness state and concluded they are increasingly uncomfortable meeting the migration date, and even if we were to complete the rest of the outstanding tasks, the migration risks are high especially when we will be deploying new technologies (i.e., Redis and NetApp as multimedia store) that are inadequately tested.

Since the Fundraising team is ramping up A/B testings now, and will be going full scale Fundraising mode this November till December (or January, ) we have decided to postpone the switchover till after all the Fundraising activities are over.

The migration team will continue to work on completing the outstanding tasks, and to ready our Ashburn infrastructure for the big switchover day. Meantime, we will set up Ashburn data center to be a 'warm' backup site, in a standby mode, ready to take over full production traffic within a short period of time, should the need arises.

Thanks.

CT