Hi all!
Ops would like us to move the stat* boxes inside of the analytics VLAN. I need to just pick a date for this to happen.
I’m not entirely sure how long this will all take, so I’d like to schedule an entire day for these to potentially be offline. How about Thursday December 18th? If there are objections, I can find another day.
Thanks! -Andrew Otto
Why?
On Fri, Dec 5, 2014 at 8:10 AM, Andrew Otto aotto@wikimedia.org wrote:
Hi all!
Ops would like us to move the stat* boxes inside of the analytics VLAN. I need to just pick a date for this to happen.
I’m not entirely sure how long this will all take, so I’d like to schedule an entire day for these to potentially be offline. How about Thursday December 18th? If there are objections, I can find another day.
Thanks! -Andrew Otto
wmfresearch mailing list wmfresearch@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wmfresearch
A whole day of downtime will be rough. I don't really do much without the stats machines. If it's necessary, it's necessary.
On Fri, Dec 5, 2014 at 10:11 AM, Toby Negrin tnegrin@wikimedia.org wrote:
Why?
On Fri, Dec 5, 2014 at 8:10 AM, Andrew Otto aotto@wikimedia.org wrote:
Hi all!
Ops would like us to move the stat* boxes inside of the analytics VLAN. I need to just pick a date for this to happen.
I’m not entirely sure how long this will all take, so I’d like to schedule an entire day for these to potentially be offline. How about Thursday December 18th? If there are objections, I can find another day.
Thanks! -Andrew Otto
wmfresearch mailing list wmfresearch@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wmfresearch
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Likely they will not be down for a whole day. But, I want to do them all at once (to get it over with), and I’m really not sure how long it will take. We will likely do one at a time, and can try to do stat1002 first or last, to make sure we keep it offline as little as possible.
On Dec 5, 2014, at 11:40, Aaron Halfaker ahalfaker@wikimedia.org wrote:
A whole day of downtime will be rough. I don't really do much without the stats machines. If it's necessary, it's necessary.
On Fri, Dec 5, 2014 at 10:11 AM, Toby Negrin <tnegrin@wikimedia.org mailto:tnegrin@wikimedia.org> wrote: Why?
On Fri, Dec 5, 2014 at 8:10 AM, Andrew Otto <aotto@wikimedia.org mailto:aotto@wikimedia.org> wrote: Hi all!
Ops would like us to move the stat* boxes inside of the analytics VLAN. I need to just pick a date for this to happen.
I’m not entirely sure how long this will all take, so I’d like to schedule an entire day for these to potentially be offline. How about Thursday December 18th? If there are objections, I can find another day.
Thanks! -Andrew Otto
wmfresearch mailing list wmfresearch@lists.wikimedia.org mailto:wmfresearch@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wmfresearch https://lists.wikimedia.org/mailman/listinfo/wmfresearch
Analytics mailing list Analytics@lists.wikimedia.org mailto:Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics https://lists.wikimedia.org/mailman/listinfo/analytics
Thanks Aaron -- Andrew and I are running through some scenarios but it looks like the most expedient method is to just take the downtime. I think we need to pick a day and go forward with it. It's worth it.
-Toby
On Fri, Dec 5, 2014 at 8:40 AM, Aaron Halfaker ahalfaker@wikimedia.org wrote:
A whole day of downtime will be rough. I don't really do much without the stats machines. If it's necessary, it's necessary.
On Fri, Dec 5, 2014 at 10:11 AM, Toby Negrin tnegrin@wikimedia.org wrote:
Why?
On Fri, Dec 5, 2014 at 8:10 AM, Andrew Otto aotto@wikimedia.org wrote:
Hi all!
Ops would like us to move the stat* boxes inside of the analytics VLAN. I need to just pick a date for this to happen.
I’m not entirely sure how long this will all take, so I’d like to schedule an entire day for these to potentially be offline. How about Thursday December 18th? If there are objections, I can find another day.
Thanks! -Andrew Otto
wmfresearch mailing list wmfresearch@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wmfresearch
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
wmfresearch mailing list wmfresearch@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wmfresearch
Ok, I haven’t heard any strong objections, and unless I get any before the ops meeting in 3 hours, I will schedule this for Thursday, December 18th.
Also, there was some discussion in another thread about the usefulness of stat1003’s public IP. Ops wants this go to away, and I tend to agree with them. It still possible to reach out to the internet on machines with private IP using the webproxy:
https://wikitech.wikimedia.org/wiki/Http_proxy
The only trouble I have had is when trying to reach HTTPS resources. I will ask ops about how to get around this.
That said, are there any strong objections to removing stat1003’s public IP?
-Ao
On Dec 5, 2014, at 11:10, Andrew Otto aotto@wikimedia.org wrote:
Hi all!
Ops would like us to move the stat* boxes inside of the analytics VLAN. I need to just pick a date for this to happen.
I’m not entirely sure how long this will all take, so I’d like to schedule an entire day for these to potentially be offline. How about Thursday December 18th? If there are objections, I can find another day.
Thanks! -Andrew Otto
Hi everybody! Reminder that this is happening today!
I will try to make sure downtime is as small as possible. I will email as each machine is back up and ready for use.
On Mon, Dec 8, 2014 at 11:20 AM, Andrew Otto aotto@wikimedia.org wrote:
Ok, I haven’t heard any strong objections, and unless I get any before the ops meeting in 3 hours, I will schedule this for Thursday, December 18th.
Also, there was some discussion in another thread about the usefulness of stat1003’s public IP. Ops wants this go to away, and I tend to agree with them. It still possible to reach out to the internet on machines with private IP using the webproxy:
https://wikitech.wikimedia.org/wiki/Http_proxy
The only trouble I have had is when trying to reach HTTPS resources. I will ask ops about how to get around this.
That said, are there any strong objections to removing stat1003’s public IP?
-Ao
On Dec 5, 2014, at 11:10, Andrew Otto aotto@wikimedia.org wrote:
Hi all!
Ops would like us to move the stat* boxes inside of the analytics VLAN.
I need to just pick a date for this to happen.
I’m not entirely sure how long this will all take, so I’d like to
schedule an entire day for these to potentially be offline. How about Thursday December 18th? If there are objections, I can find another day.
Thanks! -Andrew Otto
Woo! Ok I think we are done. Sorry I didn't email about each machine...it didn't quite work like that. Puppet is happy, and all 3 stat nodes now have internal IPs on the Analytics VLAN.
stat1001.eqiad.wmnet 10.64.21.101 stat1002.eqiad.wmnet 10.64.5.102 stat1003.eqiad.wmnet 10.64.36.103
Note that you will now need to addess stat1003 (and stat1001) via their backend network names. the *.wikmedia.org addresses no longer exist. You will need to ssh into them via a bastion now.
There will likely be problems that crop up in the next week or two. If you notice any jobs that aren't able to connect to something, or don't work anymore like they used to, this change is probably the cause. Let me know asap and I'll see what I can do.
Note that by being inside the Analytics VLAN, these nodes are now firewalled off from the rest of the WMF production networks. If you need access to something you don't already have access to, we'll have to explicitly whitelist it.
Many thanks to Brandon Black for all his help today!
-Ao
On Thu, Dec 18, 2014 at 9:58 AM, Andrew Otto otto@wikimedia.org wrote:
Hi everybody! Reminder that this is happening today!
I will try to make sure downtime is as small as possible. I will email as each machine is back up and ready for use.
On Mon, Dec 8, 2014 at 11:20 AM, Andrew Otto aotto@wikimedia.org wrote:
Ok, I haven’t heard any strong objections, and unless I get any before the ops meeting in 3 hours, I will schedule this for Thursday, December 18th.
Also, there was some discussion in another thread about the usefulness of stat1003’s public IP. Ops wants this go to away, and I tend to agree with them. It still possible to reach out to the internet on machines with private IP using the webproxy:
https://wikitech.wikimedia.org/wiki/Http_proxy
The only trouble I have had is when trying to reach HTTPS resources. I will ask ops about how to get around this.
That said, are there any strong objections to removing stat1003’s public IP?
-Ao
On Dec 5, 2014, at 11:10, Andrew Otto aotto@wikimedia.org wrote:
Hi all!
Ops would like us to move the stat* boxes inside of the analytics
VLAN. I need to just pick a date for this to happen.
I’m not entirely sure how long this will all take, so I’d like to
schedule an entire day for these to potentially be offline. How about Thursday December 18th? If there are objections, I can find another day.
Thanks! -Andrew Otto
Thanks Andrew. IMO this went great. I knew exactly what to expect and I appreciated the communication throughout the whole process.
On Thu, Dec 18, 2014 at 2:49 PM, Dan Andreescu dandreescu@wikimedia.org wrote:
Note that you will now need to addess stat1003 (and stat1001) via their
backend network names. the *.wikmedia.org addresses no longer exist. You will need to ssh into them via a bastion now.
Their "backend network names" are:
stat1003.eqiad.wmnet stat1001.eqiad.wmnet
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
+1 for this having gone well
Thanks, Andrew!
On Thu, Dec 18, 2014 at 2:42 PM, Aaron Halfaker ahalfaker@wikimedia.org wrote:
Thanks Andrew. IMO this went great. I knew exactly what to expect and I appreciated the communication throughout the whole process.
On Thu, Dec 18, 2014 at 2:49 PM, Dan Andreescu dandreescu@wikimedia.org wrote:
Note that you will now need to addess stat1003 (and stat1001) via their
backend network names. the *.wikmedia.org addresses no longer exist. You will need to ssh into them via a bastion now.
Their "backend network names" are:
stat1003.eqiad.wmnet stat1001.eqiad.wmnet
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics