Hi all,
The maintenance took around two hours of downtime. GitLab was switched over to the other datacenter.
Unfortunately due to a misconfiguration *we lost some data*. All actions (like pushes, merges, builds and comments) *between Feb 28th 0:30 UTC and Feb 28th 2:00 UTC* (so one and a half hours) are lost. If you used GitLab during that time make sure to re-apply your changes.
Apologies for the trouble. We took immediate action to prevent this from happening again.
Some more context: The gitlab replicas (passive instances) run a restore from the most recent backup every day at 2:00 UTC to test the restore and keep them in sync with the production host. This was configured manually in Puppet/hiera and was not updated during the switchover in https://phabricator.wikimedia.org/T329931. So the restore run on the production host from the most recent backup (finished at around 0:30 UTC). In the future we want to automatically enable and disable the restore, depending on the status of the machine (replica or production host). This should prevent any unwanted restores on the production host in the future.
Again, sorry for the inconveniences. If you need help feel free to reach out in T329931 https://phabricator.wikimedia.org/T329931[1] or on IRC in the libera.chat #wikimedia-gitlab channel[2].
Greetings
[1]: https://phabricator.wikimedia.org/T329931 [2]: ircs://irc.libera.chat:6697/wikimedia-gitlab
On Sun, Feb 26, 2023 at 2:30 AM Tyler Cipriani tcipriani@wikimedia.org wrote:
Hello
*GitLab will be down this coming Monday, 27 February, from 10:00 AM–noon UTC* for the datacenter switchover[0][1].
Much of this switchover window will be downtime, during which you will be unable to access GitLab through ssh or the web interface.
Apologies for any inconvenience :(
The process will involve a full backup of the GitLab server in the eqiad datacenter, which serviceops will then restore on the GitLab server in the codfw datacenter.
You can follow along on IRC in the libera.chat #wikimedia-gitlab channel[2].
Tyler Cipriani (he/him) Engineering Manager, Release Engineering Wikimedia Foundation
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/