Just in case this is news to you:  WMF is in the process of shutting down our Tampa datacenter.  The stat1 server that you know and love is in Tampa, and will be shutdown along with the rest of most of Tampa in a couple of weeks.  stat1003 is a new replacement server for stat1 in our Ashburn datacenter.

stat1003.wikimedia.org is up and running now!  Over the last week we did an audit of user accounts on stat1.  We wanted to trim down the list of users that had access to ones that actually used that access.  (The complete list of migrated accoutns is in this etherpad: http://etherpad.wikimedia.org/p/stat1_accounts, under the 'Keep' heading.)

For the most part, everything will be the same on stat1003 as it was on stat1.  Home directories have been rsynced over (as of April 3), and /a has been fully rsynced over as well (as of April 2nd).  I will rsync /a again once last time before stat1 is to be decommissioned.  Crontabs have also been migrated, so any cronjobs you had on stat1 are now also running on stat1003.


There are a very few differences:

- stat1003.wikimedia.org is the new hostname.
If there is a desire for a stat1 redirect/cname to stat1003, let me know.  I don't plan on setting one up otherwise.

- stat1003 does not allow direct ssh.
You must use bastion hosts (bast1001.wikimedia.org) to ssh in.  Add the following to your .ssh/config file to do this:

  Host stat1003.wikimedia.org
  ProxyCommand ssh -e none bast1001.wikimedia.org exec nc -w 3600 %h %p

This will fail if you don't have an account on bast1001.  You should have one!  If this doesn't work for you, let me know and we will fix that asap.

- /a has been renamed to /srv
We are trying to use /srv rather than /a on all new servers, in order to keep more in line with Linux FHS: http://www.pathname.com/fhs/.  I have set up a symlink from /a -> /srv on stat1003, so if you have scripts that rely on the the /a absolute path, they should continue to work on stat1003 without modification.

- Firewall!
stat1003 still has a public IP, but it also has pretty restrictive firewall rules in place.  If you need access to a service on stat1003, please submit an RT ticket to open a hole in this firewall.  This will allow us to be more careful about what is running on stat1003 accessible to the outside world.


Tampa will be shut down soon, and I need time to let you all migrate, and also time enough to decommission stat1 before everything is turned off.  Please make sure stat1003 works for you and everything is as it should be before Friday April 11th.  After that date I plan to shutdown stat1.

Thanks!  Don't hesitate to let me know if you need any help.

-Andrew Otto



---------- Forwarded message ----------
From: Andrew Otto <otto@wikimedia.org>
Date: Tue, Mar 25, 2014 at 12:19 PM
Subject: stat1 account audit
To: Analytics List <analytics@lists.wikimedia.org>, Development and Operations Engineers <engineering@lists.wikimedia.org>, matanya <matanya@foss.co.il>, Operations Engineers <ops@lists.wikimedia.org>


Hi all!

We will soon be migrating everything on stat1 over to a new server in eqiad: stat1003.  For the most part, data, accounts and cronjobs will be copied over exactly as they are.  However, stat1 has been around for a while, and there are quite a few accounts on there, may of which are probably not used.  We’re doing a little audit to see which accounts we don’t need to migrate to the new server.  

I’ve pasted a list of names below that we are not sure about.  None of these users have logged in in the last few weeks at least.

If you see a name there and you know that it SHOULD DEFINITELY have an account on the new stat1003 server, please let me know via a reply by Tuesday April 1.

See also: https://rt.wikimedia.org/Ticket/Display.html?id=6789

Thanks!
-Andrew Otto