Just in case this is news to you: WMF is in the process of shutting down our Tampa datacenter. The stat1 server that you know and love is in Tampa, and will be shutdown along with the rest of most of Tampa in a couple of weeks. stat1003 is a new replacement server for stat1 in our Ashburn datacenter.
stat1003.wikimedia.org is up and running now! Over the last week we did an audit of user accounts on stat1. We wanted to trim down the list of users that had access to ones that actually used that access. (The complete list of migrated accoutns is in this etherpad: http://etherpad.wikimedia.org/p/stat1_accounts, under the 'Keep' heading.)
For the most part, everything will be the same on stat1003 as it was on stat1. Home directories have been rsynced over (as of April 3), and /a has been fully rsynced over as well (as of April 2nd). I will rsync /a again once last time before stat1 is to be decommissioned. Crontabs have also been migrated, so any cronjobs you had on stat1 are now also running on stat1003.
There are a very few differences:
- stat1003.wikimedia.org is the new hostname. If there is a desire for a stat1 redirect/cname to stat1003, let me know. I don't plan on setting one up otherwise.
- stat1003 does not allow direct ssh. You must use bastion hosts (bast1001.wikimedia.org) to ssh in. Add the following to your .ssh/config file to do this:
Host stat1003.wikimedia.org ProxyCommand ssh -e none bast1001.wikimedia.org exec nc -w 3600 %h %p
This will fail if you don't have an account on bast1001. You should have one! If this doesn't work for you, let me know and we will fix that asap.
- /a has been renamed to /srv We are trying to use /srv rather than /a on all new servers, in order to keep more in line with Linux FHS: http://www.pathname.com/fhs/. I have set up a symlink from /a -> /srv on stat1003, so if you have scripts that rely on the the /a absolute path, they should continue to work on stat1003 without modification.
- Firewall! stat1003 still has a public IP, but it also has pretty restrictive firewall rules in place. If you need access to a service on stat1003, please submit an RT ticket to open a hole in this firewall. This will allow us to be more careful about what is running on stat1003 accessible to the outside world.
Tampa will be shut down soon, and I need time to let you all migrate, and also time enough to decommission stat1 before everything is turned off. Please make sure stat1003 works for you and everything is as it should be before Friday April 11th. After that date I plan to shutdown stat1.
Thanks! Don't hesitate to let me know if you need any help.
-Andrew Otto
---------- Forwarded message ---------- From: Andrew Otto otto@wikimedia.org Date: Tue, Mar 25, 2014 at 12:19 PM Subject: stat1 account audit To: Analytics List analytics@lists.wikimedia.org, Development and Operations Engineers engineering@lists.wikimedia.org, matanya < matanya@foss.co.il>, Operations Engineers ops@lists.wikimedia.org
Hi all!
We will soon be migrating everything on stat1 over to a new server in eqiad: stat1003. For the most part, data, accounts and cronjobs will be copied over exactly as they are. However, stat1 has been around for a while, and there are quite a few accounts on there, may of which are probably not used. We're doing a little audit to see which accounts we don't need to migrate to the new server.
I've pasted a list of names below that we are not sure about. None of these users have logged in in the last few weeks at least.
If you see a name there and you know that it SHOULD DEFINITELY have an account on the new stat1003 server, please let me know via a reply by Tuesday April 1.
See also: https://rt.wikimedia.org/Ticket/Display.html?id=6789
Thanks! -Andrew Otto
Thanks, Andrew!
A bunch of us non-engineer interlopers who have stat1 accounts (aka, most of the Product team) use a GUI called Sequel Pro to ssh in. I gave it the old college try (...that is, about 5 minutes of poking around in settings), but I couldn't figure out how to update the host/proxy per your instructions. I'm also fairly sure none of us have accounts on bastion... Anybody in the office who knows what's up care to help those of us who are tragically unhip to the command line? :)
On Fri, Apr 4, 2014 at 8:32 AM, Andrew Otto otto@wikimedia.org wrote:
Just in case this is news to you: WMF is in the process of shutting down our Tampa datacenter. The stat1 server that you know and love is in Tampa, and will be shutdown along with the rest of most of Tampa in a couple of weeks. stat1003 is a new replacement server for stat1 in our Ashburn datacenter.
stat1003.wikimedia.org is up and running now! Over the last week we did an audit of user accounts on stat1. We wanted to trim down the list of users that had access to ones that actually used that access. (The complete list of migrated accoutns is in this etherpad: http://etherpad.wikimedia.org/p/stat1_accounts, under the 'Keep' heading.)
For the most part, everything will be the same on stat1003 as it was on stat1. Home directories have been rsynced over (as of April 3), and /a has been fully rsynced over as well (as of April 2nd). I will rsync /a again once last time before stat1 is to be decommissioned. Crontabs have also been migrated, so any cronjobs you had on stat1 are now also running on stat1003.
There are a very few differences:
- stat1003.wikimedia.org is the new hostname.
If there is a desire for a stat1 redirect/cname to stat1003, let me know. I don't plan on setting one up otherwise.
- stat1003 does not allow direct ssh.
You must use bastion hosts (bast1001.wikimedia.org) to ssh in. Add the following to your .ssh/config file to do this:
Host stat1003.wikimedia.org ProxyCommand ssh -e none bast1001.wikimedia.org exec nc -w 3600 %h %p
This will fail if you don't have an account on bast1001. You should have one! If this doesn't work for you, let me know and we will fix that asap.
- /a has been renamed to /srv
We are trying to use /srv rather than /a on all new servers, in order to keep more in line with Linux FHS: http://www.pathname.com/fhs/. I have set up a symlink from /a -> /srv on stat1003, so if you have scripts that rely on the the /a absolute path, they should continue to work on stat1003 without modification.
- Firewall!
stat1003 still has a public IP, but it also has pretty restrictive firewall rules in place. If you need access to a service on stat1003, please submit an RT ticket to open a hole in this firewall. This will allow us to be more careful about what is running on stat1003 accessible to the outside world.
Tampa will be shut down soon, and I need time to let you all migrate, and also time enough to decommission stat1 before everything is turned off. Please make sure stat1003 works for you and everything is as it should be before Friday April 11th. After that date I plan to shutdown stat1.
Thanks! Don't hesitate to let me know if you need any help.
-Andrew Otto
---------- Forwarded message ---------- From: Andrew Otto otto@wikimedia.org Date: Tue, Mar 25, 2014 at 12:19 PM Subject: stat1 account audit To: Analytics List analytics@lists.wikimedia.org, Development and Operations Engineers engineering@lists.wikimedia.org, matanya matanya@foss.co.il, Operations Engineers ops@lists.wikimedia.org
Hi all!
We will soon be migrating everything on stat1 over to a new server in eqiad: stat1003. For the most part, data, accounts and cronjobs will be copied over exactly as they are. However, stat1 has been around for a while, and there are quite a few accounts on there, may of which are probably not used. We're doing a little audit to see which accounts we don't need to migrate to the new server.
I've pasted a list of names below that we are not sure about. None of these users have logged in in the last few weeks at least.
If you see a name there and you know that it SHOULD DEFINITELY have an account on the new stat1003 server, please let me know via a reply by Tuesday April 1.
See also: https://rt.wikimedia.org/Ticket/Display.html?id=6789
Thanks! -Andrew Otto
Engineering mailing list Engineering@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering
If you are using a GUI for SQL stuff, you should probably just be using bast1001.wikimedia.org instead of stat1/stat1003 anyway. Try changing your host in your connection settings to ‘bastion1001.wikimedia.org’. I betcha everything else will be the same.
Oh! That is, if you have an account on bast1001! I’m on that now, gimme just a few…
On Apr 4, 2014, at 1:54 PM, Maryana Pinchuk mpinchuk@wikimedia.org wrote:
Thanks, Andrew!
A bunch of us non-engineer interlopers who have stat1 accounts (aka, most of the Product team) use a GUI called Sequel Pro to ssh in. I gave it the old college try (...that is, about 5 minutes of poking around in settings), but I couldn't figure out how to update the host/proxy per your instructions. I'm also fairly sure none of us have accounts on bastion... Anybody in the office who knows what's up care to help those of us who are tragically unhip to the command line? :)
On Fri, Apr 4, 2014 at 8:32 AM, Andrew Otto otto@wikimedia.org wrote:
Just in case this is news to you: WMF is in the process of shutting down our Tampa datacenter. The stat1 server that you know and love is in Tampa, and will be shutdown along with the rest of most of Tampa in a couple of weeks. stat1003 is a new replacement server for stat1 in our Ashburn datacenter.
stat1003.wikimedia.org is up and running now! Over the last week we did an audit of user accounts on stat1. We wanted to trim down the list of users that had access to ones that actually used that access. (The complete list of migrated accoutns is in this etherpad: http://etherpad.wikimedia.org/p/stat1_accounts, under the 'Keep' heading.)
For the most part, everything will be the same on stat1003 as it was on stat1. Home directories have been rsynced over (as of April 3), and /a has been fully rsynced over as well (as of April 2nd). I will rsync /a again once last time before stat1 is to be decommissioned. Crontabs have also been migrated, so any cronjobs you had on stat1 are now also running on stat1003.
There are a very few differences:
- stat1003.wikimedia.org is the new hostname.
If there is a desire for a stat1 redirect/cname to stat1003, let me know. I don't plan on setting one up otherwise.
- stat1003 does not allow direct ssh.
You must use bastion hosts (bast1001.wikimedia.org) to ssh in. Add the following to your .ssh/config file to do this:
Host stat1003.wikimedia.org ProxyCommand ssh -e none bast1001.wikimedia.org exec nc -w 3600 %h %p
This will fail if you don't have an account on bast1001. You should have one! If this doesn't work for you, let me know and we will fix that asap.
- /a has been renamed to /srv
We are trying to use /srv rather than /a on all new servers, in order to keep more in line with Linux FHS: http://www.pathname.com/fhs/. I have set up a symlink from /a -> /srv on stat1003, so if you have scripts that rely on the the /a absolute path, they should continue to work on stat1003 without modification.
- Firewall!
stat1003 still has a public IP, but it also has pretty restrictive firewall rules in place. If you need access to a service on stat1003, please submit an RT ticket to open a hole in this firewall. This will allow us to be more careful about what is running on stat1003 accessible to the outside world.
Tampa will be shut down soon, and I need time to let you all migrate, and also time enough to decommission stat1 before everything is turned off. Please make sure stat1003 works for you and everything is as it should be before Friday April 11th. After that date I plan to shutdown stat1.
Thanks! Don't hesitate to let me know if you need any help.
-Andrew Otto
---------- Forwarded message ---------- From: Andrew Otto otto@wikimedia.org Date: Tue, Mar 25, 2014 at 12:19 PM Subject: stat1 account audit To: Analytics List analytics@lists.wikimedia.org, Development and Operations Engineers engineering@lists.wikimedia.org, matanya matanya@foss.co.il, Operations Engineers ops@lists.wikimedia.org
Hi all!
We will soon be migrating everything on stat1 over to a new server in eqiad: stat1003. For the most part, data, accounts and cronjobs will be copied over exactly as they are. However, stat1 has been around for a while, and there are quite a few accounts on there, may of which are probably not used. We're doing a little audit to see which accounts we don't need to migrate to the new server.
I've pasted a list of names below that we are not sure about. None of these users have logged in in the last few weeks at least.
If you see a name there and you know that it SHOULD DEFINITELY have an account on the new stat1003 server, please let me know via a reply by Tuesday April 1.
See also: https://rt.wikimedia.org/Ticket/Display.html?id=6789
Thanks! -Andrew Otto
Engineering mailing list Engineering@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering
-- Maryana Pinchuk Product Manager, Wikimedia Foundation wikimediafoundation.org
I get a key error when I try to ssh into bast1001. Where can I upload my rsa key?
- J
On Fri, Apr 4, 2014 at 10:54 AM, Maryana Pinchuk mpinchuk@wikimedia.orgwrote:
Thanks, Andrew!
A bunch of us non-engineer interlopers who have stat1 accounts (aka, most of the Product team) use a GUI called Sequel Pro to ssh in. I gave it the old college try (...that is, about 5 minutes of poking around in settings), but I couldn't figure out how to update the host/proxy per your instructions. I'm also fairly sure none of us have accounts on bastion... Anybody in the office who knows what's up care to help those of us who are tragically unhip to the command line? :)
On Fri, Apr 4, 2014 at 8:32 AM, Andrew Otto otto@wikimedia.org wrote:
Just in case this is news to you: WMF is in the process of shutting down our Tampa datacenter. The stat1 server that you know and love is in
Tampa,
and will be shutdown along with the rest of most of Tampa in a couple of weeks. stat1003 is a new replacement server for stat1 in our Ashburn datacenter.
stat1003.wikimedia.org is up and running now! Over the last week we
did an
audit of user accounts on stat1. We wanted to trim down the list of
users
that had access to ones that actually used that access. (The complete
list
of migrated accoutns is in this etherpad: http://etherpad.wikimedia.org/p/stat1_accounts, under the 'Keep'
heading.)
For the most part, everything will be the same on stat1003 as it was on stat1. Home directories have been rsynced over (as of April 3), and /a
has
been fully rsynced over as well (as of April 2nd). I will rsync /a again once last time before stat1 is to be decommissioned. Crontabs have also been migrated, so any cronjobs you had on stat1 are now also running on stat1003.
There are a very few differences:
- stat1003.wikimedia.org is the new hostname.
If there is a desire for a stat1 redirect/cname to stat1003, let me
know. I
don't plan on setting one up otherwise.
- stat1003 does not allow direct ssh.
You must use bastion hosts (bast1001.wikimedia.org) to ssh in. Add the following to your .ssh/config file to do this:
Host stat1003.wikimedia.org ProxyCommand ssh -e none bast1001.wikimedia.org exec nc -w 3600 %h %p
This will fail if you don't have an account on bast1001. You should have one! If this doesn't work for you, let me know and we will fix that
asap.
- /a has been renamed to /srv
We are trying to use /srv rather than /a on all new servers, in order to keep more in line with Linux FHS: http://www.pathname.com/fhs/. I have
set
up a symlink from /a -> /srv on stat1003, so if you have scripts that
rely
on the the /a absolute path, they should continue to work on stat1003 without modification.
- Firewall!
stat1003 still has a public IP, but it also has pretty restrictive
firewall
rules in place. If you need access to a service on stat1003, please
submit
an RT ticket to open a hole in this firewall. This will allow us to be
more
careful about what is running on stat1003 accessible to the outside
world.
Tampa will be shut down soon, and I need time to let you all migrate, and also time enough to decommission stat1 before everything is turned off. Please make sure stat1003 works for you and everything is as it should be before Friday April 11th. After that date I plan to shutdown stat1.
Thanks! Don't hesitate to let me know if you need any help.
-Andrew Otto
---------- Forwarded message ---------- From: Andrew Otto otto@wikimedia.org Date: Tue, Mar 25, 2014 at 12:19 PM Subject: stat1 account audit To: Analytics List analytics@lists.wikimedia.org, Development and Operations Engineers engineering@lists.wikimedia.org, matanya matanya@foss.co.il, Operations Engineers ops@lists.wikimedia.org
Hi all!
We will soon be migrating everything on stat1 over to a new server in
eqiad:
stat1003. For the most part, data, accounts and cronjobs will be copied over exactly as they are. However, stat1 has been around for a while,
and
there are quite a few accounts on there, may of which are probably not
used.
We're doing a little audit to see which accounts we don't need to
migrate to
the new server.
I've pasted a list of names below that we are not sure about. None of
these
users have logged in in the last few weeks at least.
If you see a name there and you know that it SHOULD DEFINITELY have an account on the new stat1003 server, please let me know via a reply by Tuesday April 1.
See also: https://rt.wikimedia.org/Ticket/Display.html?id=6789
Thanks! -Andrew Otto
Engineering mailing list Engineering@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering
-- Maryana Pinchuk Product Manager, Wikimedia Foundation wikimediafoundation.org
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Turns out most of you don’t have accounts on bast1001. Working on it, trying to find someone in ops to review that change now. Stay tuned…
On Apr 4, 2014, at 2:44 PM, Jonathan Morgan jmorgan@wikimedia.org wrote:
I get a key error when I try to ssh into bast1001. Where can I upload my rsa key?
- J
On Fri, Apr 4, 2014 at 10:54 AM, Maryana Pinchuk mpinchuk@wikimedia.org wrote: Thanks, Andrew!
A bunch of us non-engineer interlopers who have stat1 accounts (aka, most of the Product team) use a GUI called Sequel Pro to ssh in. I gave it the old college try (...that is, about 5 minutes of poking around in settings), but I couldn't figure out how to update the host/proxy per your instructions. I'm also fairly sure none of us have accounts on bastion... Anybody in the office who knows what's up care to help those of us who are tragically unhip to the command line? :)
On Fri, Apr 4, 2014 at 8:32 AM, Andrew Otto otto@wikimedia.org wrote:
Just in case this is news to you: WMF is in the process of shutting down our Tampa datacenter. The stat1 server that you know and love is in Tampa, and will be shutdown along with the rest of most of Tampa in a couple of weeks. stat1003 is a new replacement server for stat1 in our Ashburn datacenter.
stat1003.wikimedia.org is up and running now! Over the last week we did an audit of user accounts on stat1. We wanted to trim down the list of users that had access to ones that actually used that access. (The complete list of migrated accoutns is in this etherpad: http://etherpad.wikimedia.org/p/stat1_accounts, under the 'Keep' heading.)
For the most part, everything will be the same on stat1003 as it was on stat1. Home directories have been rsynced over (as of April 3), and /a has been fully rsynced over as well (as of April 2nd). I will rsync /a again once last time before stat1 is to be decommissioned. Crontabs have also been migrated, so any cronjobs you had on stat1 are now also running on stat1003.
There are a very few differences:
- stat1003.wikimedia.org is the new hostname.
If there is a desire for a stat1 redirect/cname to stat1003, let me know. I don't plan on setting one up otherwise.
- stat1003 does not allow direct ssh.
You must use bastion hosts (bast1001.wikimedia.org) to ssh in. Add the following to your .ssh/config file to do this:
Host stat1003.wikimedia.org ProxyCommand ssh -e none bast1001.wikimedia.org exec nc -w 3600 %h %p
This will fail if you don't have an account on bast1001. You should have one! If this doesn't work for you, let me know and we will fix that asap.
- /a has been renamed to /srv
We are trying to use /srv rather than /a on all new servers, in order to keep more in line with Linux FHS: http://www.pathname.com/fhs/. I have set up a symlink from /a -> /srv on stat1003, so if you have scripts that rely on the the /a absolute path, they should continue to work on stat1003 without modification.
- Firewall!
stat1003 still has a public IP, but it also has pretty restrictive firewall rules in place. If you need access to a service on stat1003, please submit an RT ticket to open a hole in this firewall. This will allow us to be more careful about what is running on stat1003 accessible to the outside world.
Tampa will be shut down soon, and I need time to let you all migrate, and also time enough to decommission stat1 before everything is turned off. Please make sure stat1003 works for you and everything is as it should be before Friday April 11th. After that date I plan to shutdown stat1.
Thanks! Don't hesitate to let me know if you need any help.
-Andrew Otto
---------- Forwarded message ---------- From: Andrew Otto otto@wikimedia.org Date: Tue, Mar 25, 2014 at 12:19 PM Subject: stat1 account audit To: Analytics List analytics@lists.wikimedia.org, Development and Operations Engineers engineering@lists.wikimedia.org, matanya matanya@foss.co.il, Operations Engineers ops@lists.wikimedia.org
Hi all!
We will soon be migrating everything on stat1 over to a new server in eqiad: stat1003. For the most part, data, accounts and cronjobs will be copied over exactly as they are. However, stat1 has been around for a while, and there are quite a few accounts on there, may of which are probably not used. We're doing a little audit to see which accounts we don't need to migrate to the new server.
I've pasted a list of names below that we are not sure about. None of these users have logged in in the last few weeks at least.
If you see a name there and you know that it SHOULD DEFINITELY have an account on the new stat1003 server, please let me know via a reply by Tuesday April 1.
See also: https://rt.wikimedia.org/Ticket/Display.html?id=6789
Thanks! -Andrew Otto
Engineering mailing list Engineering@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering
-- Maryana Pinchuk Product Manager, Wikimedia Foundation wikimediafoundation.org
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
-- Jonathan T. Morgan Learning Strategist Wikimedia Foundation jmorgan@wikimedia.org +1 (206) 914 - 8358
Cool. Thanks for the update Andrew!
- J
On Fri, Apr 4, 2014 at 11:47 AM, Andrew Otto aotto@wikimedia.org wrote:
Turns out most of you don't have accounts on bast1001. Working on it, trying to find someone in ops to review that change now. Stay tuned...
On Apr 4, 2014, at 2:44 PM, Jonathan Morgan jmorgan@wikimedia.org wrote:
I get a key error when I try to ssh into bast1001. Where can I upload my rsa key?
- J
On Fri, Apr 4, 2014 at 10:54 AM, Maryana Pinchuk mpinchuk@wikimedia.orgwrote:
Thanks, Andrew!
A bunch of us non-engineer interlopers who have stat1 accounts (aka, most of the Product team) use a GUI called Sequel Pro to ssh in. I gave it the old college try (...that is, about 5 minutes of poking around in settings), but I couldn't figure out how to update the host/proxy per your instructions. I'm also fairly sure none of us have accounts on bastion... Anybody in the office who knows what's up care to help those of us who are tragically unhip to the command line? :)
On Fri, Apr 4, 2014 at 8:32 AM, Andrew Otto otto@wikimedia.org wrote:
Just in case this is news to you: WMF is in the process of shutting
down
our Tampa datacenter. The stat1 server that you know and love is in
Tampa,
and will be shutdown along with the rest of most of Tampa in a couple of weeks. stat1003 is a new replacement server for stat1 in our Ashburn datacenter.
stat1003.wikimedia.org is up and running now! Over the last week we
did an
audit of user accounts on stat1. We wanted to trim down the list of
users
that had access to ones that actually used that access. (The complete
list
of migrated accoutns is in this etherpad: http://etherpad.wikimedia.org/p/stat1_accounts, under the 'Keep'
heading.)
For the most part, everything will be the same on stat1003 as it was on stat1. Home directories have been rsynced over (as of April 3), and /a
has
been fully rsynced over as well (as of April 2nd). I will rsync /a
again
once last time before stat1 is to be decommissioned. Crontabs have also been migrated, so any cronjobs you had on stat1 are now also running on stat1003.
There are a very few differences:
- stat1003.wikimedia.org is the new hostname.
If there is a desire for a stat1 redirect/cname to stat1003, let me
know. I
don't plan on setting one up otherwise.
- stat1003 does not allow direct ssh.
You must use bastion hosts (bast1001.wikimedia.org) to ssh in. Add the following to your .ssh/config file to do this:
Host stat1003.wikimedia.org ProxyCommand ssh -e none bast1001.wikimedia.org exec nc -w 3600 %h %p
This will fail if you don't have an account on bast1001. You should
have
one! If this doesn't work for you, let me know and we will fix that
asap.
- /a has been renamed to /srv
We are trying to use /srv rather than /a on all new servers, in order to keep more in line with Linux FHS: http://www.pathname.com/fhs/. I
have set
up a symlink from /a -> /srv on stat1003, so if you have scripts that
rely
on the the /a absolute path, they should continue to work on stat1003 without modification.
- Firewall!
stat1003 still has a public IP, but it also has pretty restrictive
firewall
rules in place. If you need access to a service on stat1003, please
submit
an RT ticket to open a hole in this firewall. This will allow us to be
more
careful about what is running on stat1003 accessible to the outside
world.
Tampa will be shut down soon, and I need time to let you all migrate,
and
also time enough to decommission stat1 before everything is turned off. Please make sure stat1003 works for you and everything is as it should
be
before Friday April 11th. After that date I plan to shutdown stat1.
Thanks! Don't hesitate to let me know if you need any help.
-Andrew Otto
---------- Forwarded message ---------- From: Andrew Otto otto@wikimedia.org Date: Tue, Mar 25, 2014 at 12:19 PM Subject: stat1 account audit To: Analytics List analytics@lists.wikimedia.org, Development and Operations Engineers engineering@lists.wikimedia.org, matanya matanya@foss.co.il, Operations Engineers ops@lists.wikimedia.org
Hi all!
We will soon be migrating everything on stat1 over to a new server in
eqiad:
stat1003. For the most part, data, accounts and cronjobs will be copied over exactly as they are. However, stat1 has been around for a while,
and
there are quite a few accounts on there, may of which are probably not
used.
We're doing a little audit to see which accounts we don't need to
migrate to
the new server.
I've pasted a list of names below that we are not sure about. None of
these
users have logged in in the last few weeks at least.
If you see a name there and you know that it SHOULD DEFINITELY have an account on the new stat1003 server, please let me know via a reply by Tuesday April 1.
See also: https://rt.wikimedia.org/Ticket/Display.html?id=6789
Thanks! -Andrew Otto
Engineering mailing list Engineering@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering
-- Maryana Pinchuk Product Manager, Wikimedia Foundation wikimediafoundation.org
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
-- Jonathan T. Morgan Learning Strategist Wikimedia Foundation jmorgan@wikimedia.org +1 (206) 914 - 8358
Ok, there are some ops discussions about this right now, and we’re going to have to work out some policy details over the next week. I’ll spare everyone the full context here, and continue that discussion on the ops@ mailing list.
For now, the firewall on stat1003 has been disabled. This means that you can ssh directly into stat1003, just like you used to on stat1. Use of SQL GUIs will work the same. If you already have access to bast1001, then you should continue to use that. The firewall will be reenabled sometime within a week or two, and you will have to use bastions then.
There are 7 users on stat1003 that do not have bastion access. For you 7, I have been asked to ask you to read this page carefully https://wikitech.wikimedia.org/wiki/Server_access_responsibilities , and confirm to me that you have read and understand the details. Once you have done that, I can grant you bastion access. Again, you’ll need to do this ASAP. In order to give ASAP a (slightly arbitrary) deadline, I’m asking that you do this before Friday of next week, April 11th.
The 7 people I need confirmations from are:
howief jdlrobson jforrester jmorgan maryana msyed swalling
Thanks all! Sorry for any confusion and back and forth around this! We’ll get this settled soon.
-Ao
On Apr 4, 2014, at 2:47 PM, Andrew Otto aotto@wikimedia.org wrote:
Turns out most of you don’t have accounts on bast1001. Working on it, trying to find someone in ops to review that change now. Stay tuned…
On Apr 4, 2014, at 2:44 PM, Jonathan Morgan jmorgan@wikimedia.org wrote:
I get a key error when I try to ssh into bast1001. Where can I upload my rsa key?
- J
On Fri, Apr 4, 2014 at 10:54 AM, Maryana Pinchuk mpinchuk@wikimedia.org wrote: Thanks, Andrew!
A bunch of us non-engineer interlopers who have stat1 accounts (aka, most of the Product team) use a GUI called Sequel Pro to ssh in. I gave it the old college try (...that is, about 5 minutes of poking around in settings), but I couldn't figure out how to update the host/proxy per your instructions. I'm also fairly sure none of us have accounts on bastion... Anybody in the office who knows what's up care to help those of us who are tragically unhip to the command line? :)
On Fri, Apr 4, 2014 at 8:32 AM, Andrew Otto otto@wikimedia.org wrote:
Just in case this is news to you: WMF is in the process of shutting down our Tampa datacenter. The stat1 server that you know and love is in Tampa, and will be shutdown along with the rest of most of Tampa in a couple of weeks. stat1003 is a new replacement server for stat1 in our Ashburn datacenter.
stat1003.wikimedia.org is up and running now! Over the last week we did an audit of user accounts on stat1. We wanted to trim down the list of users that had access to ones that actually used that access. (The complete list of migrated accoutns is in this etherpad: http://etherpad.wikimedia.org/p/stat1_accounts, under the 'Keep' heading.)
For the most part, everything will be the same on stat1003 as it was on stat1. Home directories have been rsynced over (as of April 3), and /a has been fully rsynced over as well (as of April 2nd). I will rsync /a again once last time before stat1 is to be decommissioned. Crontabs have also been migrated, so any cronjobs you had on stat1 are now also running on stat1003.
There are a very few differences:
- stat1003.wikimedia.org is the new hostname.
If there is a desire for a stat1 redirect/cname to stat1003, let me know. I don't plan on setting one up otherwise.
- stat1003 does not allow direct ssh.
You must use bastion hosts (bast1001.wikimedia.org) to ssh in. Add the following to your .ssh/config file to do this:
Host stat1003.wikimedia.org ProxyCommand ssh -e none bast1001.wikimedia.org exec nc -w 3600 %h %p
This will fail if you don't have an account on bast1001. You should have one! If this doesn't work for you, let me know and we will fix that asap.
- /a has been renamed to /srv
We are trying to use /srv rather than /a on all new servers, in order to keep more in line with Linux FHS: http://www.pathname.com/fhs/. I have set up a symlink from /a -> /srv on stat1003, so if you have scripts that rely on the the /a absolute path, they should continue to work on stat1003 without modification.
- Firewall!
stat1003 still has a public IP, but it also has pretty restrictive firewall rules in place. If you need access to a service on stat1003, please submit an RT ticket to open a hole in this firewall. This will allow us to be more careful about what is running on stat1003 accessible to the outside world.
Tampa will be shut down soon, and I need time to let you all migrate, and also time enough to decommission stat1 before everything is turned off. Please make sure stat1003 works for you and everything is as it should be before Friday April 11th. After that date I plan to shutdown stat1.
Thanks! Don't hesitate to let me know if you need any help.
-Andrew Otto
---------- Forwarded message ---------- From: Andrew Otto otto@wikimedia.org Date: Tue, Mar 25, 2014 at 12:19 PM Subject: stat1 account audit To: Analytics List analytics@lists.wikimedia.org, Development and Operations Engineers engineering@lists.wikimedia.org, matanya matanya@foss.co.il, Operations Engineers ops@lists.wikimedia.org
Hi all!
We will soon be migrating everything on stat1 over to a new server in eqiad: stat1003. For the most part, data, accounts and cronjobs will be copied over exactly as they are. However, stat1 has been around for a while, and there are quite a few accounts on there, may of which are probably not used. We're doing a little audit to see which accounts we don't need to migrate to the new server.
I've pasted a list of names below that we are not sure about. None of these users have logged in in the last few weeks at least.
If you see a name there and you know that it SHOULD DEFINITELY have an account on the new stat1003 server, please let me know via a reply by Tuesday April 1.
See also: https://rt.wikimedia.org/Ticket/Display.html?id=6789
Thanks! -Andrew Otto
Engineering mailing list Engineering@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering
-- Maryana Pinchuk Product Manager, Wikimedia Foundation wikimediafoundation.org
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
-- Jonathan T. Morgan Learning Strategist Wikimedia Foundation jmorgan@wikimedia.org +1 (206) 914 - 8358
On Fri, Apr 4, 2014 at 1:00 PM, Andrew Otto aotto@wikimedia.org wrote:
There are 7 users on stat1003 that do not have bastion access. For you 7, I have been asked to ask you to read this page carefully https://wikitech.wikimedia.org/wiki/Server_access_responsibilities , and confirm to me that you have read and understand the details. Once you have done that, I can grant you bastion access. Again, you’ll need to do this ASAP. In order to give ASAP a (slightly arbitrary) deadline, I’m asking that you do this before Friday of next week, April 11th.
Yes, I've read this several times in the past. It's all fine. Most of the PMs, designers, and such are not doing anything that would remotely need to be puppetized/outside of our home directories/requiring sudo.
K yeah, I was just asked to double check, thanks for confirming!
On Apr 4, 2014, at 5:03 PM, Steven Walling swalling@wikimedia.org wrote:
On Fri, Apr 4, 2014 at 1:00 PM, Andrew Otto aotto@wikimedia.org wrote: There are 7 users on stat1003 that do not have bastion access. For you 7, I have been asked to ask you to read this page carefully https://wikitech.wikimedia.org/wiki/Server_access_responsibilities , and confirm to me that you have read and understand the details. Once you have done that, I can grant you bastion access. Again, you’ll need to do this ASAP. In order to give ASAP a (slightly arbitrary) deadline, I’m asking that you do this before Friday of next week, April 11th.
Yes, I've read this several times in the past. It's all fine. Most of the PMs, designers, and such are not doing anything that would remotely need to be puppetized/outside of our home directories/requiring sudo.
-- Steven Walling, Product Manager https://wikimediafoundation.org/ _______________________________________________ Engineering mailing list Engineering@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering
I have yet to hear from:
howief jdlrobson jmorgan msyed
If you are one of those 4 people, that means you have stat1003 access but no bast1001 access. You need to confirm to me that you have read https://wikitech.wikimedia.org/wiki/Server_access_responsibilities and understand the details before I can give you bast1001 access. (Yes, you may have already read it, but I have been asked to double check before I grant more bast1001 access).
I'm going to turn the stat1003 firewall back on on this Friday, April 11th. If you haven't confirmed by then you won't be able to reach stat1003. That's ok! You can always confirm later and we can get you access then.
-Ao
On Fri, Apr 4, 2014 at 4:00 PM, Andrew Otto aotto@wikimedia.org wrote:
Ok, there are some ops discussions about this right now, and we're going to have to work out some policy details over the next week. I'll spare everyone the full context here, and continue that discussion on the ops@mailing list.
For now, the firewall on stat1003 has been disabled. This means that you can ssh directly into stat1003, just like you used to on stat1. Use of SQL GUIs will work the same. If you already have access to bast1001, then you should continue to use that. The firewall will be reenabled sometime within a week or two, and you will have to use bastions then.
There are 7 users on stat1003 that do not have bastion access. For you 7, I have been asked to ask you to read this page carefully https://wikitech.wikimedia.org/wiki/Server_access_responsibilities , and confirm to me that you have read and understand the details. Once you have done that, I can grant you bastion access. Again, you'll need to do this ASAP. In order to give ASAP a (slightly arbitrary) deadline, I'm asking that you do this before Friday of next week, April 11th.
The 7 people I need confirmations from are:
howief jdlrobson jforrester jmorgan maryana msyed swalling
Thanks all! Sorry for any confusion and back and forth around this! We'll get this settled soon.
-Ao
On Apr 4, 2014, at 2:47 PM, Andrew Otto aotto@wikimedia.org wrote:
Turns out most of you don't have accounts on bast1001. Working on it, trying to find someone in ops to review that change now. Stay tuned...
On Apr 4, 2014, at 2:44 PM, Jonathan Morgan jmorgan@wikimedia.org wrote:
I get a key error when I try to ssh into bast1001. Where can I upload my rsa key?
- J
On Fri, Apr 4, 2014 at 10:54 AM, Maryana Pinchuk mpinchuk@wikimedia.orgwrote:
Thanks, Andrew!
A bunch of us non-engineer interlopers who have stat1 accounts (aka, most of the Product team) use a GUI called Sequel Pro to ssh in. I gave it the old college try (...that is, about 5 minutes of poking around in settings), but I couldn't figure out how to update the host/proxy per your instructions. I'm also fairly sure none of us have accounts on bastion... Anybody in the office who knows what's up care to help those of us who are tragically unhip to the command line? :)
On Fri, Apr 4, 2014 at 8:32 AM, Andrew Otto otto@wikimedia.org wrote:
Just in case this is news to you: WMF is in the process of shutting
down
our Tampa datacenter. The stat1 server that you know and love is in
Tampa,
and will be shutdown along with the rest of most of Tampa in a couple of weeks. stat1003 is a new replacement server for stat1 in our Ashburn datacenter.
stat1003.wikimedia.org is up and running now! Over the last week we
did an
audit of user accounts on stat1. We wanted to trim down the list of
users
that had access to ones that actually used that access. (The complete
list
of migrated accoutns is in this etherpad: http://etherpad.wikimedia.org/p/stat1_accounts, under the 'Keep'
heading.)
For the most part, everything will be the same on stat1003 as it was on stat1. Home directories have been rsynced over (as of April 3), and /a
has
been fully rsynced over as well (as of April 2nd). I will rsync /a
again
once last time before stat1 is to be decommissioned. Crontabs have also been migrated, so any cronjobs you had on stat1 are now also running on stat1003.
There are a very few differences:
- stat1003.wikimedia.org is the new hostname.
If there is a desire for a stat1 redirect/cname to stat1003, let me
know. I
don't plan on setting one up otherwise.
- stat1003 does not allow direct ssh.
You must use bastion hosts (bast1001.wikimedia.org) to ssh in. Add the following to your .ssh/config file to do this:
Host stat1003.wikimedia.org ProxyCommand ssh -e none bast1001.wikimedia.org exec nc -w 3600 %h %p
This will fail if you don't have an account on bast1001. You should
have
one! If this doesn't work for you, let me know and we will fix that
asap.
- /a has been renamed to /srv
We are trying to use /srv rather than /a on all new servers, in order to keep more in line with Linux FHS: http://www.pathname.com/fhs/. I
have set
up a symlink from /a -> /srv on stat1003, so if you have scripts that
rely
on the the /a absolute path, they should continue to work on stat1003 without modification.
- Firewall!
stat1003 still has a public IP, but it also has pretty restrictive
firewall
rules in place. If you need access to a service on stat1003, please
submit
an RT ticket to open a hole in this firewall. This will allow us to be
more
careful about what is running on stat1003 accessible to the outside
world.
Tampa will be shut down soon, and I need time to let you all migrate,
and
also time enough to decommission stat1 before everything is turned off. Please make sure stat1003 works for you and everything is as it should
be
before Friday April 11th. After that date I plan to shutdown stat1.
Thanks! Don't hesitate to let me know if you need any help.
-Andrew Otto
---------- Forwarded message ---------- From: Andrew Otto otto@wikimedia.org Date: Tue, Mar 25, 2014 at 12:19 PM Subject: stat1 account audit To: Analytics List analytics@lists.wikimedia.org, Development and Operations Engineers engineering@lists.wikimedia.org, matanya matanya@foss.co.il, Operations Engineers ops@lists.wikimedia.org
Hi all!
We will soon be migrating everything on stat1 over to a new server in
eqiad:
stat1003. For the most part, data, accounts and cronjobs will be copied over exactly as they are. However, stat1 has been around for a while,
and
there are quite a few accounts on there, may of which are probably not
used.
We're doing a little audit to see which accounts we don't need to
migrate to
the new server.
I've pasted a list of names below that we are not sure about. None of
these
users have logged in in the last few weeks at least.
If you see a name there and you know that it SHOULD DEFINITELY have an account on the new stat1003 server, please let me know via a reply by Tuesday April 1.
See also: https://rt.wikimedia.org/Ticket/Display.html?id=6789
Thanks! -Andrew Otto
Engineering mailing list Engineering@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering
-- Maryana Pinchuk Product Manager, Wikimedia Foundation wikimediafoundation.org
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
-- Jonathan T. Morgan Learning Strategist Wikimedia Foundation jmorgan@wikimedia.org +1 (206) 914 - 8358
Hi Andrew,
Sorry for the delay. I have read and understand the details of the server access responsibilities document.
Thanks, Jonathan
On Wed, Apr 9, 2014 at 9:23 AM, Andrew Otto otto@wikimedia.org wrote:
I have yet to hear from:
howief jdlrobson jmorgan msyed
If you are one of those 4 people, that means you have stat1003 access but no bast1001 access. You need to confirm to me that you have read https://wikitech.wikimedia.org/wiki/Server_access_responsibilities and understand the details before I can give you bast1001 access. (Yes, you may have already read it, but I have been asked to double check before I grant more bast1001 access).
I'm going to turn the stat1003 firewall back on on this Friday, April 11th. If you haven't confirmed by then you won't be able to reach stat1003. That's ok! You can always confirm later and we can get you access then.
-Ao
On Fri, Apr 4, 2014 at 4:00 PM, Andrew Otto aotto@wikimedia.org wrote:
Ok, there are some ops discussions about this right now, and we're going to have to work out some policy details over the next week. I'll spare everyone the full context here, and continue that discussion on the ops@mailing list.
For now, the firewall on stat1003 has been disabled. This means that you can ssh directly into stat1003, just like you used to on stat1. Use of SQL GUIs will work the same. If you already have access to bast1001, then you should continue to use that. The firewall will be reenabled sometime within a week or two, and you will have to use bastions then.
There are 7 users on stat1003 that do not have bastion access. For you 7, I have been asked to ask you to read this page carefully https://wikitech.wikimedia.org/wiki/Server_access_responsibilities , and confirm to me that you have read and understand the details. Once you have done that, I can grant you bastion access. Again, you'll need to do this ASAP. In order to give ASAP a (slightly arbitrary) deadline, I'm asking that you do this before Friday of next week, April 11th.
The 7 people I need confirmations from are:
howief jdlrobson jforrester jmorgan maryana msyed swalling
Thanks all! Sorry for any confusion and back and forth around this! We'll get this settled soon.
-Ao
On Apr 4, 2014, at 2:47 PM, Andrew Otto aotto@wikimedia.org wrote:
Turns out most of you don't have accounts on bast1001. Working on it, trying to find someone in ops to review that change now. Stay tuned...
On Apr 4, 2014, at 2:44 PM, Jonathan Morgan jmorgan@wikimedia.org wrote:
I get a key error when I try to ssh into bast1001. Where can I upload my rsa key?
- J
On Fri, Apr 4, 2014 at 10:54 AM, Maryana Pinchuk mpinchuk@wikimedia.orgwrote:
Thanks, Andrew!
A bunch of us non-engineer interlopers who have stat1 accounts (aka, most of the Product team) use a GUI called Sequel Pro to ssh in. I gave it the old college try (...that is, about 5 minutes of poking around in settings), but I couldn't figure out how to update the host/proxy per your instructions. I'm also fairly sure none of us have accounts on bastion... Anybody in the office who knows what's up care to help those of us who are tragically unhip to the command line? :)
On Fri, Apr 4, 2014 at 8:32 AM, Andrew Otto otto@wikimedia.org wrote:
Just in case this is news to you: WMF is in the process of shutting
down
our Tampa datacenter. The stat1 server that you know and love is in
Tampa,
and will be shutdown along with the rest of most of Tampa in a couple
of
weeks. stat1003 is a new replacement server for stat1 in our Ashburn datacenter.
stat1003.wikimedia.org is up and running now! Over the last week we
did an
audit of user accounts on stat1. We wanted to trim down the list of
users
that had access to ones that actually used that access. (The complete
list
of migrated accoutns is in this etherpad: http://etherpad.wikimedia.org/p/stat1_accounts, under the 'Keep'
heading.)
For the most part, everything will be the same on stat1003 as it was on stat1. Home directories have been rsynced over (as of April 3), and
/a has
been fully rsynced over as well (as of April 2nd). I will rsync /a
again
once last time before stat1 is to be decommissioned. Crontabs have
also
been migrated, so any cronjobs you had on stat1 are now also running on stat1003.
There are a very few differences:
- stat1003.wikimedia.org is the new hostname.
If there is a desire for a stat1 redirect/cname to stat1003, let me
know. I
don't plan on setting one up otherwise.
- stat1003 does not allow direct ssh.
You must use bastion hosts (bast1001.wikimedia.org) to ssh in. Add
the
following to your .ssh/config file to do this:
Host stat1003.wikimedia.org ProxyCommand ssh -e none bast1001.wikimedia.org exec nc -w 3600 %h
%p
This will fail if you don't have an account on bast1001. You should
have
one! If this doesn't work for you, let me know and we will fix that
asap.
- /a has been renamed to /srv
We are trying to use /srv rather than /a on all new servers, in order
to
keep more in line with Linux FHS: http://www.pathname.com/fhs/. I
have set
up a symlink from /a -> /srv on stat1003, so if you have scripts that
rely
on the the /a absolute path, they should continue to work on stat1003 without modification.
- Firewall!
stat1003 still has a public IP, but it also has pretty restrictive
firewall
rules in place. If you need access to a service on stat1003, please
submit
an RT ticket to open a hole in this firewall. This will allow us to
be more
careful about what is running on stat1003 accessible to the outside
world.
Tampa will be shut down soon, and I need time to let you all migrate,
and
also time enough to decommission stat1 before everything is turned off. Please make sure stat1003 works for you and everything is as it should
be
before Friday April 11th. After that date I plan to shutdown stat1.
Thanks! Don't hesitate to let me know if you need any help.
-Andrew Otto
---------- Forwarded message ---------- From: Andrew Otto otto@wikimedia.org Date: Tue, Mar 25, 2014 at 12:19 PM Subject: stat1 account audit To: Analytics List analytics@lists.wikimedia.org, Development and Operations Engineers engineering@lists.wikimedia.org, matanya matanya@foss.co.il, Operations Engineers ops@lists.wikimedia.org
Hi all!
We will soon be migrating everything on stat1 over to a new server in
eqiad:
stat1003. For the most part, data, accounts and cronjobs will be
copied
over exactly as they are. However, stat1 has been around for a while,
and
there are quite a few accounts on there, may of which are probably not
used.
We're doing a little audit to see which accounts we don't need to
migrate to
the new server.
I've pasted a list of names below that we are not sure about. None of
these
users have logged in in the last few weeks at least.
If you see a name there and you know that it SHOULD DEFINITELY have an account on the new stat1003 server, please let me know via a reply by Tuesday April 1.
See also: https://rt.wikimedia.org/Ticket/Display.html?id=6789
Thanks! -Andrew Otto
Engineering mailing list Engineering@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering
-- Maryana Pinchuk Product Manager, Wikimedia Foundation wikimediafoundation.org
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
-- Jonathan T. Morgan Learning Strategist Wikimedia Foundation jmorgan@wikimedia.org +1 (206) 914 - 8358
Hi Andrew,
Just read the server access responsibilities doc and I understand the details.
Thanks!
On Wed, Apr 9, 2014 at 9:23 AM, Andrew Otto otto@wikimedia.org wrote:
I have yet to hear from:
howief jdlrobson jmorgan msyed
If you are one of those 4 people, that means you have stat1003 access but no bast1001 access. You need to confirm to me that you have read https://wikitech.wikimedia.org/wiki/Server_access_responsibilities and understand the details before I can give you bast1001 access. (Yes, you may have already read it, but I have been asked to double check before I grant more bast1001 access).
I'm going to turn the stat1003 firewall back on on this Friday, April 11th. If you haven't confirmed by then you won't be able to reach stat1003. That's ok! You can always confirm later and we can get you access then.
-Ao
On Fri, Apr 4, 2014 at 4:00 PM, Andrew Otto aotto@wikimedia.org wrote:
Ok, there are some ops discussions about this right now, and we're going to have to work out some policy details over the next week. I'll spare everyone the full context here, and continue that discussion on the ops@mailing list.
For now, the firewall on stat1003 has been disabled. This means that you can ssh directly into stat1003, just like you used to on stat1. Use of SQL GUIs will work the same. If you already have access to bast1001, then you should continue to use that. The firewall will be reenabled sometime within a week or two, and you will have to use bastions then.
There are 7 users on stat1003 that do not have bastion access. For you 7, I have been asked to ask you to read this page carefully https://wikitech.wikimedia.org/wiki/Server_access_responsibilities , and confirm to me that you have read and understand the details. Once you have done that, I can grant you bastion access. Again, you'll need to do this ASAP. In order to give ASAP a (slightly arbitrary) deadline, I'm asking that you do this before Friday of next week, April 11th.
The 7 people I need confirmations from are:
howief jdlrobson jforrester jmorgan maryana msyed swalling
Thanks all! Sorry for any confusion and back and forth around this! We'll get this settled soon.
-Ao
On Apr 4, 2014, at 2:47 PM, Andrew Otto aotto@wikimedia.org wrote:
Turns out most of you don't have accounts on bast1001. Working on it, trying to find someone in ops to review that change now. Stay tuned...
On Apr 4, 2014, at 2:44 PM, Jonathan Morgan jmorgan@wikimedia.org wrote:
I get a key error when I try to ssh into bast1001. Where can I upload my rsa key?
- J
On Fri, Apr 4, 2014 at 10:54 AM, Maryana Pinchuk mpinchuk@wikimedia.orgwrote:
Thanks, Andrew!
A bunch of us non-engineer interlopers who have stat1 accounts (aka, most of the Product team) use a GUI called Sequel Pro to ssh in. I gave it the old college try (...that is, about 5 minutes of poking around in settings), but I couldn't figure out how to update the host/proxy per your instructions. I'm also fairly sure none of us have accounts on bastion... Anybody in the office who knows what's up care to help those of us who are tragically unhip to the command line? :)
On Fri, Apr 4, 2014 at 8:32 AM, Andrew Otto otto@wikimedia.org wrote:
Just in case this is news to you: WMF is in the process of shutting
down
our Tampa datacenter. The stat1 server that you know and love is in
Tampa,
and will be shutdown along with the rest of most of Tampa in a couple
of
weeks. stat1003 is a new replacement server for stat1 in our Ashburn datacenter.
stat1003.wikimedia.org is up and running now! Over the last week we
did an
audit of user accounts on stat1. We wanted to trim down the list of
users
that had access to ones that actually used that access. (The complete
list
of migrated accoutns is in this etherpad: http://etherpad.wikimedia.org/p/stat1_accounts, under the 'Keep'
heading.)
For the most part, everything will be the same on stat1003 as it was on stat1. Home directories have been rsynced over (as of April 3), and
/a has
been fully rsynced over as well (as of April 2nd). I will rsync /a
again
once last time before stat1 is to be decommissioned. Crontabs have
also
been migrated, so any cronjobs you had on stat1 are now also running on stat1003.
There are a very few differences:
- stat1003.wikimedia.org is the new hostname.
If there is a desire for a stat1 redirect/cname to stat1003, let me
know. I
don't plan on setting one up otherwise.
- stat1003 does not allow direct ssh.
You must use bastion hosts (bast1001.wikimedia.org) to ssh in. Add
the
following to your .ssh/config file to do this:
Host stat1003.wikimedia.org ProxyCommand ssh -e none bast1001.wikimedia.org exec nc -w 3600 %h
%p
This will fail if you don't have an account on bast1001. You should
have
one! If this doesn't work for you, let me know and we will fix that
asap.
- /a has been renamed to /srv
We are trying to use /srv rather than /a on all new servers, in order
to
keep more in line with Linux FHS: http://www.pathname.com/fhs/. I
have set
up a symlink from /a -> /srv on stat1003, so if you have scripts that
rely
on the the /a absolute path, they should continue to work on stat1003 without modification.
- Firewall!
stat1003 still has a public IP, but it also has pretty restrictive
firewall
rules in place. If you need access to a service on stat1003, please
submit
an RT ticket to open a hole in this firewall. This will allow us to
be more
careful about what is running on stat1003 accessible to the outside
world.
Tampa will be shut down soon, and I need time to let you all migrate,
and
also time enough to decommission stat1 before everything is turned off. Please make sure stat1003 works for you and everything is as it should
be
before Friday April 11th. After that date I plan to shutdown stat1.
Thanks! Don't hesitate to let me know if you need any help.
-Andrew Otto
---------- Forwarded message ---------- From: Andrew Otto otto@wikimedia.org Date: Tue, Mar 25, 2014 at 12:19 PM Subject: stat1 account audit To: Analytics List analytics@lists.wikimedia.org, Development and Operations Engineers engineering@lists.wikimedia.org, matanya matanya@foss.co.il, Operations Engineers ops@lists.wikimedia.org
Hi all!
We will soon be migrating everything on stat1 over to a new server in
eqiad:
stat1003. For the most part, data, accounts and cronjobs will be
copied
over exactly as they are. However, stat1 has been around for a while,
and
there are quite a few accounts on there, may of which are probably not
used.
We're doing a little audit to see which accounts we don't need to
migrate to
the new server.
I've pasted a list of names below that we are not sure about. None of
these
users have logged in in the last few weeks at least.
If you see a name there and you know that it SHOULD DEFINITELY have an account on the new stat1003 server, please let me know via a reply by Tuesday April 1.
See also: https://rt.wikimedia.org/Ticket/Display.html?id=6789
Thanks! -Andrew Otto
Engineering mailing list Engineering@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering
-- Maryana Pinchuk Product Manager, Wikimedia Foundation wikimediafoundation.org
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
-- Jonathan T. Morgan Learning Strategist Wikimedia Foundation jmorgan@wikimedia.org +1 (206) 914 - 8358
Engineering mailing list Engineering@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering
Ok! Today is Friday, woo! I have re-enabled base::firewall on stat1003. You will have to use bast1001 to access stat1003 from now on.
I’d like to decom stat1 next week. I will do a final rsync of stat1:/a -> stat1003:/srv before I do, just in case there as been any work there that hasn’t made it over since the original rsync.
What this means: don’t use stat1 anymore! :)
Again, let me know if you have any trouble at all.
Thanks! -Ao
P.S. jdlrobson! Let me know if you have read and understand https://wikitech.wikimedia.org/wiki/Server_access_responsibilities and want an account on bast1001.wikimedia.org. You are the only person who has not gotten back to me! :)
On Apr 9, 2014, at 12:23 PM, Andrew Otto otto@wikimedia.org wrote:
I have yet to hear from:
howief jdlrobson jmorgan msyed
If you are one of those 4 people, that means you have stat1003 access but no bast1001 access. You need to confirm to me that you have read https://wikitech.wikimedia.org/wiki/Server_access_responsibilities and understand the details before I can give you bast1001 access. (Yes, you may have already read it, but I have been asked to double check before I grant more bast1001 access).
I'm going to turn the stat1003 firewall back on on this Friday, April 11th. If you haven't confirmed by then you won't be able to reach stat1003. That's ok! You can always confirm later and we can get you access then.
-Ao
On Fri, Apr 4, 2014 at 4:00 PM, Andrew Otto aotto@wikimedia.org wrote: Ok, there are some ops discussions about this right now, and we’re going to have to work out some policy details over the next week. I’ll spare everyone the full context here, and continue that discussion on the ops@ mailing list.
For now, the firewall on stat1003 has been disabled. This means that you can ssh directly into stat1003, just like you used to on stat1. Use of SQL GUIs will work the same. If you already have access to bast1001, then you should continue to use that. The firewall will be reenabled sometime within a week or two, and you will have to use bastions then.
There are 7 users on stat1003 that do not have bastion access. For you 7, I have been asked to ask you to read this page carefully https://wikitech.wikimedia.org/wiki/Server_access_responsibilities , and confirm to me that you have read and understand the details. Once you have done that, I can grant you bastion access. Again, you’ll need to do this ASAP. In order to give ASAP a (slightly arbitrary) deadline, I’m asking that you do this before Friday of next week, April 11th.
The 7 people I need confirmations from are:
howief jdlrobson jforrester jmorgan maryana msyed swalling
Thanks all! Sorry for any confusion and back and forth around this! We’ll get this settled soon.
-Ao
On Apr 4, 2014, at 2:47 PM, Andrew Otto aotto@wikimedia.org wrote:
Turns out most of you don’t have accounts on bast1001. Working on it, trying to find someone in ops to review that change now. Stay tuned…
On Apr 4, 2014, at 2:44 PM, Jonathan Morgan jmorgan@wikimedia.org wrote:
I get a key error when I try to ssh into bast1001. Where can I upload my rsa key?
- J
On Fri, Apr 4, 2014 at 10:54 AM, Maryana Pinchuk mpinchuk@wikimedia.org wrote: Thanks, Andrew!
A bunch of us non-engineer interlopers who have stat1 accounts (aka, most of the Product team) use a GUI called Sequel Pro to ssh in. I gave it the old college try (...that is, about 5 minutes of poking around in settings), but I couldn't figure out how to update the host/proxy per your instructions. I'm also fairly sure none of us have accounts on bastion... Anybody in the office who knows what's up care to help those of us who are tragically unhip to the command line? :)
On Fri, Apr 4, 2014 at 8:32 AM, Andrew Otto otto@wikimedia.org wrote:
Just in case this is news to you: WMF is in the process of shutting down our Tampa datacenter. The stat1 server that you know and love is in Tampa, and will be shutdown along with the rest of most of Tampa in a couple of weeks. stat1003 is a new replacement server for stat1 in our Ashburn datacenter.
stat1003.wikimedia.org is up and running now! Over the last week we did an audit of user accounts on stat1. We wanted to trim down the list of users that had access to ones that actually used that access. (The complete list of migrated accoutns is in this etherpad: http://etherpad.wikimedia.org/p/stat1_accounts, under the 'Keep' heading.)
For the most part, everything will be the same on stat1003 as it was on stat1. Home directories have been rsynced over (as of April 3), and /a has been fully rsynced over as well (as of April 2nd). I will rsync /a again once last time before stat1 is to be decommissioned. Crontabs have also been migrated, so any cronjobs you had on stat1 are now also running on stat1003.
There are a very few differences:
- stat1003.wikimedia.org is the new hostname.
If there is a desire for a stat1 redirect/cname to stat1003, let me know. I don't plan on setting one up otherwise.
- stat1003 does not allow direct ssh.
You must use bastion hosts (bast1001.wikimedia.org) to ssh in. Add the following to your .ssh/config file to do this:
Host stat1003.wikimedia.org ProxyCommand ssh -e none bast1001.wikimedia.org exec nc -w 3600 %h %p
This will fail if you don't have an account on bast1001. You should have one! If this doesn't work for you, let me know and we will fix that asap.
- /a has been renamed to /srv
We are trying to use /srv rather than /a on all new servers, in order to keep more in line with Linux FHS: http://www.pathname.com/fhs/. I have set up a symlink from /a -> /srv on stat1003, so if you have scripts that rely on the the /a absolute path, they should continue to work on stat1003 without modification.
- Firewall!
stat1003 still has a public IP, but it also has pretty restrictive firewall rules in place. If you need access to a service on stat1003, please submit an RT ticket to open a hole in this firewall. This will allow us to be more careful about what is running on stat1003 accessible to the outside world.
Tampa will be shut down soon, and I need time to let you all migrate, and also time enough to decommission stat1 before everything is turned off. Please make sure stat1003 works for you and everything is as it should be before Friday April 11th. After that date I plan to shutdown stat1.
Thanks! Don't hesitate to let me know if you need any help.
-Andrew Otto
---------- Forwarded message ---------- From: Andrew Otto otto@wikimedia.org Date: Tue, Mar 25, 2014 at 12:19 PM Subject: stat1 account audit To: Analytics List analytics@lists.wikimedia.org, Development and Operations Engineers engineering@lists.wikimedia.org, matanya matanya@foss.co.il, Operations Engineers ops@lists.wikimedia.org
Hi all!
We will soon be migrating everything on stat1 over to a new server in eqiad: stat1003. For the most part, data, accounts and cronjobs will be copied over exactly as they are. However, stat1 has been around for a while, and there are quite a few accounts on there, may of which are probably not used. We're doing a little audit to see which accounts we don't need to migrate to the new server.
I've pasted a list of names below that we are not sure about. None of these users have logged in in the last few weeks at least.
If you see a name there and you know that it SHOULD DEFINITELY have an account on the new stat1003 server, please let me know via a reply by Tuesday April 1.
See also: https://rt.wikimedia.org/Ticket/Display.html?id=6789
Thanks! -Andrew Otto
Engineering mailing list Engineering@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering
-- Maryana Pinchuk Product Manager, Wikimedia Foundation wikimediafoundation.org
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
-- Jonathan T. Morgan Learning Strategist Wikimedia Foundation jmorgan@wikimedia.org +1 (206) 914 - 8358
I have just done a final rsync of stat1:/a -> stat1003:/srv. I plan to decommission stat1 today or tomorrow. Please do not use stat1 anymore.
Thanks all!
-Andrew Otto
On Fri, Apr 11, 2014 at 1:32 PM, Andrew Otto aotto@wikimedia.org wrote:
Ok! Today is Friday, woo! I have re-enabled base::firewall on stat1003. You will have to use bast1001 to access stat1003 from now on.
I'd like to decom stat1 next week. I will do a final rsync of stat1:/a -> stat1003:/srv before I do, just in case there as been any work there that hasn't made it over since the original rsync.
What this means: don't use stat1 anymore! :)
Again, let me know if you have any trouble at all.
Thanks! -Ao
P.S. jdlrobson! Let me know if you have read and understand https://wikitech.wikimedia.org/wiki/Server_access_responsibilities and want an account on bast1001.wikimedia.org. You are the only person who has not gotten back to me! :)
On Apr 9, 2014, at 12:23 PM, Andrew Otto otto@wikimedia.org wrote:
I have yet to hear from:
howief jdlrobson jmorgan msyed
If you are one of those 4 people, that means you have stat1003 access but no bast1001 access. You need to confirm to me that you have read https://wikitech.wikimedia.org/wiki/Server_access_responsibilities and understand the details before I can give you bast1001 access. (Yes, you may have already read it, but I have been asked to double check before I grant more bast1001 access).
I'm going to turn the stat1003 firewall back on on this Friday, April 11th. If you haven't confirmed by then you won't be able to reach stat1003. That's ok! You can always confirm later and we can get you access then.
-Ao
On Fri, Apr 4, 2014 at 4:00 PM, Andrew Otto aotto@wikimedia.org wrote:
Ok, there are some ops discussions about this right now, and we're going to have to work out some policy details over the next week. I'll spare everyone the full context here, and continue that discussion on the ops@mailing list.
For now, the firewall on stat1003 has been disabled. This means that you can ssh directly into stat1003, just like you used to on stat1. Use of SQL GUIs will work the same. If you already have access to bast1001, then you should continue to use that. The firewall will be reenabled sometime within a week or two, and you will have to use bastions then.
There are 7 users on stat1003 that do not have bastion access. For you 7, I have been asked to ask you to read this page carefully https://wikitech.wikimedia.org/wiki/Server_access_responsibilities , and confirm to me that you have read and understand the details. Once you have done that, I can grant you bastion access. Again, you'll need to do this ASAP. In order to give ASAP a (slightly arbitrary) deadline, I'm asking that you do this before Friday of next week, April 11th.
The 7 people I need confirmations from are:
howief jdlrobson jforrester jmorgan maryana msyed swalling
Thanks all! Sorry for any confusion and back and forth around this! We'll get this settled soon.
-Ao
On Apr 4, 2014, at 2:47 PM, Andrew Otto aotto@wikimedia.org wrote:
Turns out most of you don't have accounts on bast1001. Working on it, trying to find someone in ops to review that change now. Stay tuned...
On Apr 4, 2014, at 2:44 PM, Jonathan Morgan jmorgan@wikimedia.org wrote:
I get a key error when I try to ssh into bast1001. Where can I upload my rsa key?
- J
On Fri, Apr 4, 2014 at 10:54 AM, Maryana Pinchuk mpinchuk@wikimedia.orgwrote:
Thanks, Andrew!
A bunch of us non-engineer interlopers who have stat1 accounts (aka, most of the Product team) use a GUI called Sequel Pro to ssh in. I gave it the old college try (...that is, about 5 minutes of poking around in settings), but I couldn't figure out how to update the host/proxy per your instructions. I'm also fairly sure none of us have accounts on bastion... Anybody in the office who knows what's up care to help those of us who are tragically unhip to the command line? :)
On Fri, Apr 4, 2014 at 8:32 AM, Andrew Otto otto@wikimedia.org wrote:
Just in case this is news to you: WMF is in the process of shutting
down
our Tampa datacenter. The stat1 server that you know and love is in
Tampa,
and will be shutdown along with the rest of most of Tampa in a couple
of
weeks. stat1003 is a new replacement server for stat1 in our Ashburn datacenter.
stat1003.wikimedia.org is up and running now! Over the last week we
did an
audit of user accounts on stat1. We wanted to trim down the list of
users
that had access to ones that actually used that access. (The complete
list
of migrated accoutns is in this etherpad: http://etherpad.wikimedia.org/p/stat1_accounts, under the 'Keep'
heading.)
For the most part, everything will be the same on stat1003 as it was on stat1. Home directories have been rsynced over (as of April 3), and
/a has
been fully rsynced over as well (as of April 2nd). I will rsync /a
again
once last time before stat1 is to be decommissioned. Crontabs have
also
been migrated, so any cronjobs you had on stat1 are now also running on stat1003.
There are a very few differences:
- stat1003.wikimedia.org is the new hostname.
If there is a desire for a stat1 redirect/cname to stat1003, let me
know. I
don't plan on setting one up otherwise.
- stat1003 does not allow direct ssh.
You must use bastion hosts (bast1001.wikimedia.org) to ssh in. Add
the
following to your .ssh/config file to do this:
Host stat1003.wikimedia.org ProxyCommand ssh -e none bast1001.wikimedia.org exec nc -w 3600 %h
%p
This will fail if you don't have an account on bast1001. You should
have
one! If this doesn't work for you, let me know and we will fix that
asap.
- /a has been renamed to /srv
We are trying to use /srv rather than /a on all new servers, in order
to
keep more in line with Linux FHS: http://www.pathname.com/fhs/. I
have set
up a symlink from /a -> /srv on stat1003, so if you have scripts that
rely
on the the /a absolute path, they should continue to work on stat1003 without modification.
- Firewall!
stat1003 still has a public IP, but it also has pretty restrictive
firewall
rules in place. If you need access to a service on stat1003, please
submit
an RT ticket to open a hole in this firewall. This will allow us to
be more
careful about what is running on stat1003 accessible to the outside
world.
Tampa will be shut down soon, and I need time to let you all migrate,
and
also time enough to decommission stat1 before everything is turned off. Please make sure stat1003 works for you and everything is as it should
be
before Friday April 11th. After that date I plan to shutdown stat1.
Thanks! Don't hesitate to let me know if you need any help.
-Andrew Otto
---------- Forwarded message ---------- From: Andrew Otto otto@wikimedia.org Date: Tue, Mar 25, 2014 at 12:19 PM Subject: stat1 account audit To: Analytics List analytics@lists.wikimedia.org, Development and Operations Engineers engineering@lists.wikimedia.org, matanya matanya@foss.co.il, Operations Engineers ops@lists.wikimedia.org
Hi all!
We will soon be migrating everything on stat1 over to a new server in
eqiad:
stat1003. For the most part, data, accounts and cronjobs will be
copied
over exactly as they are. However, stat1 has been around for a while,
and
there are quite a few accounts on there, may of which are probably not
used.
We're doing a little audit to see which accounts we don't need to
migrate to
the new server.
I've pasted a list of names below that we are not sure about. None of
these
users have logged in in the last few weeks at least.
If you see a name there and you know that it SHOULD DEFINITELY have an account on the new stat1003 server, please let me know via a reply by Tuesday April 1.
See also: https://rt.wikimedia.org/Ticket/Display.html?id=6789
Thanks! -Andrew Otto
Engineering mailing list Engineering@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering
-- Maryana Pinchuk Product Manager, Wikimedia Foundation wikimediafoundation.org
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
-- Jonathan T. Morgan Learning Strategist Wikimedia Foundation jmorgan@wikimedia.org +1 (206) 914 - 8358
On Fri, Apr 4, 2014 at 8:32 AM, Andrew Otto otto@wikimedia.org wrote:
Just in case this is news to you: WMF is in the process of shutting down our Tampa datacenter. The stat1 server that you know and love is in Tampa, and will be shutdown along with the rest of most of Tampa in a couple of weeks. stat1003 is a new replacement server for stat1 in our Ashburn datacenter.
stat1003.wikimedia.org is up and running now! Over the last week we did an audit of user accounts on stat1. We wanted to trim down the list of users that had access to ones that actually used that access. (The complete list of migrated accoutns is in this etherpad: http://etherpad.wikimedia.org/p/stat1_accounts, under the 'Keep' heading.)
I did not see my name (swalling) on any of the lists in that etherpad. I use stat1 access all the time, so I added myself to the keep group. Thanks!
Thanks Steven, dunno why your name wasn’t on that list, but your stuff has been migrated along with everyone else’s. Thanks!
On Apr 4, 2014, at 2:22 PM, Steven Walling swalling@wikimedia.org wrote:
On Fri, Apr 4, 2014 at 8:32 AM, Andrew Otto otto@wikimedia.org wrote: Just in case this is news to you: WMF is in the process of shutting down our Tampa datacenter. The stat1 server that you know and love is in Tampa, and will be shutdown along with the rest of most of Tampa in a couple of weeks. stat1003 is a new replacement server for stat1 in our Ashburn datacenter.
stat1003.wikimedia.org is up and running now! Over the last week we did an audit of user accounts on stat1. We wanted to trim down the list of users that had access to ones that actually used that access. (The complete list of migrated accoutns is in this etherpad: http://etherpad.wikimedia.org/p/stat1_accounts, under the 'Keep' heading.)
I did not see my name (swalling) on any of the lists in that etherpad. I use stat1 access all the time, so I added myself to the keep group. Thanks!
-- Steven Walling, Product Manager https://wikimediafoundation.org/ _______________________________________________ Ops mailing list Ops@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/ops