Hi,
So the time is upon us to finally upgrade Gerrit. I thank everyone in advance for all of your patience and good work testing things out for us. The plan is outlined in detail on Phabricator[0], but I'll give everyone the short version here.
The downtime will be on Monday, July 25th from 01:00 to 04:00 UTC; that's Sunday night for those of us who are US-based. This time was picked based because it's one of our lowest traffic times on Gerrit. There's never a *good* time to bring it down, it's always being used, but this looks like it'll impact the fewest number of users. I do not anticipate the process actually taking the full 3 hours, but I'm giving us a lot of extra time just in case.
Jaime will be on hand to assist with a final DB snapshot of the old version and possible rollback, and Daniel Z. is going to assist me with the puppet work. Most of this is already prepared so the amount of "change" to do the swap has been kept to a minimum.
Gerrit is a critical service to all developers, so the plan includes a generous provision to roll back if things are not operating properly--I'd rather us be on the old version that works on a Monday morning than be stuck broken going into the work week.
I'll be sure to send a last minute reminder Sunday night prior to taking services offline.
Thanks again!
-Chad
Friendly reminder that this is happening in about 2 hours!
-Chad
On Thu, Jul 21, 2016 at 8:08 AM, Chad Horohoe chorohoe@wikimedia.org wrote:
Hi,
So the time is upon us to finally upgrade Gerrit. I thank everyone in advance for all of your patience and good work testing things out for us. The plan is outlined in detail on Phabricator[0], but I'll give everyone the short version here.
The downtime will be on Monday, July 25th from 01:00 to 04:00 UTC; that's Sunday night for those of us who are US-based. This time was picked based because it's one of our lowest traffic times on Gerrit. There's never a *good* time to bring it down, it's always being used, but this looks like it'll impact the fewest number of users. I do not anticipate the process actually taking the full 3 hours, but I'm giving us a lot of extra time just in case.
Jaime will be on hand to assist with a final DB snapshot of the old version and possible rollback, and Daniel Z. is going to assist me with the puppet work. Most of this is already prepared so the amount of "change" to do the swap has been kept to a minimum.
Gerrit is a critical service to all developers, so the plan includes a generous provision to roll back if things are not operating properly--I'd rather us be on the old version that works on a Monday morning than be stuck broken going into the work week.
I'll be sure to send a last minute reminder Sunday night prior to taking services offline.
Thanks again!
-Chad
Hi!
So things are back up. Took a tad longer than expected.
Big caveat: something is up with the Phabricator bridge and it's not logging what's up. I'll look at it first thing after I've gotten a few hours of sleep :)
Other than that, everything seems to be going great. Zuul is getting along just fine with Gerrit so changes are getting their tests run. IRC bot is back too.
Your SSH connections to Gerrit *might* complain about the host verification. While the host key has remained the same, the IP address shifted. You can either delete or update the offending line in your ~/.ssh/known_hosts if your client does have trouble with it.
Thanks again to Daniel, Antoine and Jaime for their help today.
-Chad
On Sun, Jul 24, 2016 at 4:02 PM, Chad Horohoe chorohoe@wikimedia.org wrote:
Friendly reminder that this is happening in about 2 hours!
-Chad
On Thu, Jul 21, 2016 at 8:08 AM, Chad Horohoe chorohoe@wikimedia.org wrote:
Hi,
So the time is upon us to finally upgrade Gerrit. I thank everyone in advance for all of your patience and good work testing things out for us. The plan is outlined in detail on Phabricator[0], but I'll give everyone the short version here.
The downtime will be on Monday, July 25th from 01:00 to 04:00 UTC; that's Sunday night for those of us who are US-based. This time was picked based because it's one of our lowest traffic times on Gerrit. There's never a *good* time to bring it down, it's always being used, but this looks like it'll impact the fewest number of users. I do not anticipate the process actually taking the full 3 hours, but I'm giving us a lot of extra time just in case.
Jaime will be on hand to assist with a final DB snapshot of the old version and possible rollback, and Daniel Z. is going to assist me with the puppet work. Most of this is already prepared so the amount of "change" to do the swap has been kept to a minimum.
Gerrit is a critical service to all developers, so the plan includes a generous provision to roll back if things are not operating properly--I'd rather us be on the old version that works on a Monday morning than be stuck broken going into the work week.
I'll be sure to send a last minute reminder Sunday night prior to taking services offline.
Thanks again!
-Chad
Thanks for working on a Sunday!
Pine
On Sun, Jul 24, 2016 at 10:45 PM, Chad Horohoe chorohoe@wikimedia.org wrote:
Hi!
So things are back up. Took a tad longer than expected.
Big caveat: something is up with the Phabricator bridge and it's not logging what's up. I'll look at it first thing after I've gotten a few hours of sleep :)
Other than that, everything seems to be going great. Zuul is getting along just fine with Gerrit so changes are getting their tests run. IRC bot is back too.
Your SSH connections to Gerrit *might* complain about the host verification. While the host key has remained the same, the IP address shifted. You can either delete or update the offending line in your ~/.ssh/known_hosts if your client does have trouble with it.
Thanks again to Daniel, Antoine and Jaime for their help today.
-Chad
On Sun, Jul 24, 2016 at 4:02 PM, Chad Horohoe chorohoe@wikimedia.org wrote:
Friendly reminder that this is happening in about 2 hours!
-Chad
On Thu, Jul 21, 2016 at 8:08 AM, Chad Horohoe chorohoe@wikimedia.org wrote:
Hi,
So the time is upon us to finally upgrade Gerrit. I thank everyone in advance for all of your patience and good work testing things out for us. The plan is outlined in detail on Phabricator[0], but I'll give everyone the short version here.
The downtime will be on Monday, July 25th from 01:00 to 04:00 UTC;
that's
Sunday night for those of us who are US-based. This time was picked based because it's one of our lowest traffic times on Gerrit. There's never a *good* time to bring it down, it's always being used, but this looks like it'll impact the fewest number of users. I do not anticipate the process actually taking the full 3 hours, but I'm giving us a lot of extra time just in case.
Jaime will be on hand to assist with a final DB snapshot of the old version and possible rollback, and Daniel Z. is going to assist me with the puppet work. Most of this
is
already prepared so the amount of "change" to do the swap has been kept to a minimum.
Gerrit is a critical service to all developers, so the plan includes a generous provision to roll back if things are not operating properly--I'd rather us be on the old
version
that works on a Monday morning than be stuck broken going into the work week.
I'll be sure to send a last minute reminder Sunday night prior to taking services offline.
Thanks again!
-Chad
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
wikitech-l@lists.wikimedia.org