Just in case this is news to you: WMF is in the process of shutting down our Tampa datacenter. The stat1 server that you know and love is in Tampa, and will be shutdown along with the rest of most of Tampa in a couple of weeks. stat1003 is a new replacement server for stat1 in our Ashburn datacenter. stat1003.wikimedia.org is up and running now! Over the last week we did an audit of user accounts on stat1. We wanted to trim down the list of users that had access to ones that actually used that access. (The complete list of migrated accoutns is in this etherpad: http://etherpad.wikimedia.org/p/stat1_accounts, under the 'Keep' heading.) For the most part, everything will be the same on stat1003 as it was on stat1. Home directories have been rsynced over (as of April 3), and /a has been fully rsynced over as well (as of April 2nd). I will rsync /a again once last time before stat1 is to be decommissioned. Crontabs have also been migrated, so any cronjobs you had on stat1 are now also running on stat1003. There are a very few differences: - stat1003.wikimedia.org is the new hostname. If there is a desire for a stat1 redirect/cname to stat1003, let me know. I don't plan on setting one up otherwise. - stat1003 does not allow direct ssh. You must use bastion hosts (bast1001.wikimedia.org) to ssh in. Add the following to your .ssh/config file to do this: Host stat1003.wikimedia.org ProxyCommand ssh -e none bast1001.wikimedia.org exec nc -w 3600 %h %p This will fail if you don't have an account on bast1001. You should have one! If this doesn't work for you, let me know and we will fix that asap. - /a has been renamed to /srv We are trying to use /srv rather than /a on all new servers, in order to keep more in line with Linux FHS: http://www.pathname.com/fhs/. I have set up a symlink from /a -> /srv on stat1003, so if you have scripts that rely on the the /a absolute path, they should continue to work on stat1003 without modification. - Firewall! stat1003 still has a public IP, but it also has pretty restrictive firewall rules in place. If you need access to a service on stat1003, please submit an RT ticket to open a hole in this firewall. This will allow us to be more careful about what is running on stat1003 accessible to the outside world. Tampa will be shut down soon, and I need time to let you all migrate, and also time enough to decommission stat1 before everything is turned off. Please make sure stat1003 works for you and everything is as it should be before Friday April 11th. After that date I plan to shutdown stat1. Thanks! Don't hesitate to let me know if you need any help. -Andrew Otto ---------- Forwarded message ---------- From: Andrew Otto <otto(a)wikimedia.org> Date: Tue, Mar 25, 2014 at 12:19 PM Subject: stat1 account audit To: Analytics List <analytics(a)lists.wikimedia.org>rg>, Development and Operations Engineers <engineering(a)lists.wikimedia.org>rg>, matanya <matanya(a)foss.co.il>il>, Operations Engineers <ops(a)lists.wikimedia.org> Hi all! We will soon be migrating everything on stat1 over to a new server in eqiad: stat1003. For the most part, data, accounts and cronjobs will be copied over exactly as they are. However, stat1 has been around for a while, and there are quite a few accounts on there, may of which are probably not used. We're doing a little audit to see which accounts we don't need to migrate to the new server. I've pasted a list of names below that we are not sure about. None of these users have logged in in the last few weeks at least. If you see a name there and you know that it SHOULD DEFINITELY have an account on the new stat1003 server, please let me know via a reply by Tuesday April 1. See also: https://rt.wikimedia.org/Ticket/Display.html?id=6789 Thanks! -Andrew Otto _______________________________________________ Engineering mailing list Engineering(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering

Just in case this is news to you: WMF is in the process of shutting down our Tampa datacenter. The stat1 server that you know and love is in Tampa, and will be shutdown along with the rest of most of Tampa in a couple of weeks. stat1003 is a new replacement server for stat1 in our Ashburn datacenter. stat1003.wikimedia.org is up and running now! Over the last week we did an audit of user accounts on stat1. We wanted to trim down the list of users that had access to ones that actually used that access. (The complete list of migrated accoutns is in this etherpad: http://etherpad.wikimedia.org/p/stat1_accounts, under the 'Keep' heading.) For the most part, everything will be the same on stat1003 as it was on stat1. Home directories have been rsynced over (as of April 3), and /a has been fully rsynced over as well (as of April 2nd). I will rsync /a again once last time before stat1 is to be decommissioned. Crontabs have also been migrated, so any cronjobs you had on stat1 are now also running on stat1003. There are a very few differences: - stat1003.wikimedia.org is the new hostname. If there is a desire for a stat1 redirect/cname to stat1003, let me know. I don't plan on setting one up otherwise. - stat1003 does not allow direct ssh. You must use bastion hosts (bast1001.wikimedia.org) to ssh in. Add the following to your .ssh/config file to do this: Host stat1003.wikimedia.org ProxyCommand ssh -e none bast1001.wikimedia.org exec nc -w 3600 %h %p This will fail if you don't have an account on bast1001. You should have one! If this doesn't work for you, let me know and we will fix that asap. - /a has been renamed to /srv We are trying to use /srv rather than /a on all new servers, in order to keep more in line with Linux FHS: http://www.pathname.com/fhs/. I have set up a symlink from /a -> /srv on stat1003, so if you have scripts that rely on the the /a absolute path, they should continue to work on stat1003 without modification. - Firewall! stat1003 still has a public IP, but it also has pretty restrictive firewall rules in place. If you need access to a service on stat1003, please submit an RT ticket to open a hole in this firewall. This will allow us to be more careful about what is running on stat1003 accessible to the outside world. Tampa will be shut down soon, and I need time to let you all migrate, and also time enough to decommission stat1 before everything is turned off. Please make sure stat1003 works for you and everything is as it should be before Friday April 11th. After that date I plan to shutdown stat1. Thanks! Don't hesitate to let me know if you need any help. -Andrew Otto ---------- Forwarded message ---------- From: Andrew Otto <otto(a)wikimedia.org> Date: Tue, Mar 25, 2014 at 12:19 PM Subject: stat1 account audit To: Analytics List <analytics(a)lists.wikimedia.org>rg>, Development and Operations Engineers <engineering(a)lists.wikimedia.org>rg>, matanya <matanya(a)foss.co.il>il>, Operations Engineers <ops(a)lists.wikimedia.org> Hi all! We will soon be migrating everything on stat1 over to a new server in eqiad: stat1003. For the most part, data, accounts and cronjobs will be copied over exactly as they are. However, stat1 has been around for a while, and there are quite a few accounts on there, may of which are probably not used. We're doing a little audit to see which accounts we don't need to migrate to the new server. I've pasted a list of names below that we are not sure about. None of these users have logged in in the last few weeks at least. If you see a name there and you know that it SHOULD DEFINITELY have an account on the new stat1003 server, please let me know via a reply by Tuesday April 1. See also: https://rt.wikimedia.org/Ticket/Display.html?id=6789 Thanks! -Andrew Otto _______________________________________________ Engineering mailing list Engineering(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering

I get a key error when I try to ssh into bast1001. Where can I upload my rsa key? - J On Fri, Apr 4, 2014 at 10:54 AM, Maryana Pinchuk <mpinchuk(a)wikimedia.org> wrote: Thanks, Andrew! A bunch of us non-engineer interlopers who have stat1 accounts (aka, most of the Product team) use a GUI called Sequel Pro to ssh in. I gave it the old college try (...that is, about 5 minutes of poking around in settings), but I couldn't figure out how to update the host/proxy per your instructions. I'm also fairly sure none of us have accounts on bastion... Anybody in the office who knows what's up care to help those of us who are tragically unhip to the command line? :) On Fri, Apr 4, 2014 at 8:32 AM, Andrew Otto <otto(a)wikimedia.org> wrote: -- Maryana Pinchuk Product Manager, Wikimedia Foundation wikimediafoundation.org _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics -- Jonathan T. Morgan Learning Strategist Wikimedia Foundation jmorgan(a)wikimedia.org +1 (206) 914 - 8358

Ok, there are some ops discussions about this right now, and we're going to have to work out some policy details over the next week. I'll spare everyone the full context here, and continue that discussion on the ops@mailing list. For now, the firewall on stat1003 has been disabled. This means that you can ssh directly into stat1003, just like you used to on stat1. Use of SQL GUIs will work the same. If you already have access to bast1001, then you should continue to use that. The firewall will be reenabled sometime within a week or two, and you will have to use bastions then. There are 7 users on stat1003 that do not have bastion access. For you 7, I have been asked to ask you to read this page carefully https://wikitech.wikimedia.org/wiki/Server_access_responsibilities , and confirm to me that you have read and understand the details. Once you have done that, I can grant you bastion access. Again, you'll need to do this ASAP. In order to give ASAP a (slightly arbitrary) deadline, I'm asking that you do this before Friday of next week, April 11th. The 7 people I need confirmations from are: howief jdlrobson jforrester jmorgan maryana msyed swalling Thanks all! Sorry for any confusion and back and forth around this! We'll get this settled soon. -Ao On Apr 4, 2014, at 2:47 PM, Andrew Otto <aotto(a)wikimedia.org> wrote: Turns out most of you don't have accounts on bast1001. Working on it, trying to find someone in ops to review that change now. Stay tuned... On Apr 4, 2014, at 2:44 PM, Jonathan Morgan <jmorgan(a)wikimedia.org> wrote: I get a key error when I try to ssh into bast1001. Where can I upload my rsa key? - J On Fri, Apr 4, 2014 at 10:54 AM, Maryana Pinchuk <mpinchuk(a)wikimedia.org>wrote;wrote: -- Jonathan T. Morgan Learning Strategist Wikimedia Foundation jmorgan(a)wikimedia.org +1 (206) 914 - 8358

Turns out most of you don’t have accounts on bast1001. Working on it, trying to find someone in ops to review that change now. Stay tuned… On Apr 4, 2014, at 2:44 PM, Jonathan Morgan <jmorgan(a)wikimedia.org> wrote: