Next Wednesday, December 4th, at 00:00 UTC (4pm San Francisco) there will be a Phabricator upgrade window that might last up to 2 hours.
While we expect downtime to be much shorter than that Phabricator might be unreachable in that time frame.
We are switching the production Phabricator server to a buster (Debian stable) machine.
https://phabricator.wikimedia.org/T238956
This has happened. The Phabricator production server is now phab1001 (again) and not phab1003 anymore.
Phabricator now runs on Debian stable (buster) and PHP 7.3.
https://phabricator.wikimedia.org/T238956 shows all the related patches and can be used to report issues.
It turned out BIOS settings on phab1001 were such that disks were running under "legacy IDE" mode and write cache was turned off.
This was really bad for performance. Changing that in BIOS unfortunately also means complete data loss.
So we had to go back to phab1003 yesterday, fix phab1001, reimage it, rsync all the data again and finally (today) switch to phab1001 for real.
The good part is that the process to switch is now simplified and there was basically no downtime for users besides the rarely used git-ssh service, which is also fixed now.
This is done now.
On Wed, Dec 4, 2019 at 12:14 AM Daniel Zahn dzahn@wikimedia.org wrote:
This has happened. The Phabricator production server is now phab1001 (again) and not phab1003 anymore.
Phabricator now runs on Debian stable (buster) and PHP 7.3.
https://phabricator.wikimedia.org/T238956 shows all the related patches and can be used to report issues.
wikitech-l@lists.wikimedia.org