Hi everyone,
Just letting everyone know that we're having some issues with the Gerrit server at the moment. Sometime since last night, a little over a dozen repositories (listed below) have gone a wonky and are missing their branches & tags (objects are intact).
We're working on recovering everything right now, and I'll be sure to let everyone know the status when we're done.
In the meantime, if you use any of the following repositories, please refrain from trying to use them until I give the all-clear (I don't want to exacerbate any problems):
In mediawiki/extensions/*: Comments, FacebookOpenGraph, GoogleDocs4MW, Nonlinear, OnlineStatusBar, Phalanx, RandomImageByCategory, SemanticImageInput, ShoutWikiAds, SphinxSearch, TranslationNotifcations
In operations/*: debs/mysqlatfacebook, debs/wikimedia-lvs-realserver debs/wikimedia-search-qa, debs/wikistats, software
Also note: operations/mediawiki-config was busted, but we went ahead and just rebuilt the repo from the live copy so as to not interfere with site operations.
I thank you for your patience, and please find me on IRC if you have any questions or find other repositories that are misbehaving.
-Chad
On Thu, Sep 6, 2012 at 5:11 PM, Chad innocentkiller@gmail.com wrote:
Hi everyone,
Just letting everyone know that we're having some issues with the Gerrit server at the moment. Sometime since last night, a little over a dozen repositories (listed below) have gone a wonky and are missing their branches & tags (objects are intact).
We're working on recovering everything right now, and I'll be sure to let everyone know the status when we're done.
In the meantime, if you use any of the following repositories, please refrain from trying to use them until I give the all-clear (I don't want to exacerbate any problems):
In mediawiki/extensions/*: Comments, FacebookOpenGraph, GoogleDocs4MW, Nonlinear, OnlineStatusBar, Phalanx, RandomImageByCategory, SemanticImageInput, ShoutWikiAds, SphinxSearch, TranslationNotifcations
In operations/*: debs/mysqlatfacebook, debs/wikimedia-lvs-realserver debs/wikimedia-search-qa, debs/wikistats, software
Also note: operations/mediawiki-config was busted, but we went ahead and just rebuilt the repo from the live copy so as to not interfere with site operations.
I thank you for your patience, and please find me on IRC if you have any questions or find other repositories that are misbehaving.
Just to let everyone know, I'm still working on restoring these projects, but they're not ready yet.
operations/software.git is back up, though, master was intact after all.
-Chad
On Thu, Sep 6, 2012 at 9:43 PM, Chad innocentkiller@gmail.com wrote:
On Thu, Sep 6, 2012 at 5:11 PM, Chad innocentkiller@gmail.com wrote:
Hi everyone,
Just letting everyone know that we're having some issues with the Gerrit server at the moment. Sometime since last night, a little over a dozen repositories (listed below) have gone a wonky and are missing their branches & tags (objects are intact).
We're working on recovering everything right now, and I'll be sure to let everyone know the status when we're done.
In the meantime, if you use any of the following repositories, please refrain from trying to use them until I give the all-clear (I don't want to exacerbate any problems):
In mediawiki/extensions/*: Comments, FacebookOpenGraph, GoogleDocs4MW, Nonlinear, OnlineStatusBar, Phalanx, RandomImageByCategory, SemanticImageInput, ShoutWikiAds, SphinxSearch, TranslationNotifcations
In operations/*: debs/mysqlatfacebook, debs/wikimedia-lvs-realserver debs/wikimedia-search-qa, debs/wikistats, software
Also note: operations/mediawiki-config was busted, but we went ahead and just rebuilt the repo from the live copy so as to not interfere with site operations.
I thank you for your patience, and please find me on IRC if you have any questions or find other repositories that are misbehaving.
Just to let everyone know, I'm still working on restoring these projects, but they're not ready yet.
operations/software.git is back up, though, master was intact after all.
And mediawiki/extensions/TranslationNotifications is back up too, master was intact.
The remaining ones in extensions/* and debs/* are under repair.
-Chad
2012/9/7 Chad innocentkiller@gmail.com:
And mediawiki/extensions/TranslationNotifications is back up too, master was intact.
The remaining ones in extensions/* and debs/* are under repair.
It seems TranslationNotifications extension has some problem. Git review is always failing with this error:
error: unpack failed: error Missing unknown 4ee9dc02a655cb376a8f20e6d5c1ee95a81b1a37
And remote rejected with unpack error.
ps: Some information about this corruption available at http://asheepapart.blogspot.com/2011/10/gerrit-code-review-unpack-error-miss...
Thanks Santhosh Thottingal
On Fri, Sep 7, 2012 at 11:30 AM, Santhosh Thottingal santhosh.thottingal@gmail.com wrote:
2012/9/7 Chad innocentkiller@gmail.com:
And mediawiki/extensions/TranslationNotifications is back up too, master was intact.
The remaining ones in extensions/* and debs/* are under repair.
It seems TranslationNotifications extension has some problem. Git review is always failing with this error:
error: unpack failed: error Missing unknown 4ee9dc02a655cb376a8f20e6d5c1ee95a81b1a37
And remote rejected with unpack error.
ps: Some information about this corruption available at http://asheepapart.blogspot.com/2011/10/gerrit-code-review-unpack-error-miss...
Ah indeed, that repo does need some work. This is the only thing I'm working on today, and I figured out how to resolve it late last night--things will be back to normal by the end of today.
-Chad
On Fri, 07 Sep 2012 08:30:47 -0700, Santhosh Thottingal santhosh.thottingal@gmail.com wrote:
2012/9/7 Chad innocentkiller@gmail.com:
And mediawiki/extensions/TranslationNotifications is back up too, master was intact.
The remaining ones in extensions/* and debs/* are under repair.
It seems TranslationNotifications extension has some problem. Git review is always failing with this error:
error: unpack failed: error Missing unknown 4ee9dc02a655cb376a8f20e6d5c1ee95a81b1a37
And remote rejected with unpack error.
ps: Some information about this corruption available at http://asheepapart.blogspot.com/2011/10/gerrit-code-review-unpack-error-miss...
Thanks Santhosh Thottingal
I'd like to know what kind of blobs it is that Gerrit depends on so much but git is happy to prune.
On Thu, Sep 6, 2012 at 5:11 PM, Chad innocentkiller@gmail.com wrote:
In mediawiki/extensions/*: Comments, FacebookOpenGraph, GoogleDocs4MW, Nonlinear, OnlineStatusBar, Phalanx, RandomImageByCategory, SemanticImageInput, ShoutWikiAds, SphinxSearch, TranslationNotifcations
All of these have been fixed other than Nonlinear (more heavily broken). TranslationNotifications' master is intact, but some of the changes are still in a bad state and I need to finish cleaning it up.
In operations/*: debs/mysqlatfacebook, debs/wikimedia-lvs-realserver debs/wikimedia-search-qa, debs/wikistats, software
software, wikimedia-lvs-realserver, wikimedia-search-qa and wikistats are all back and fine. mysqlatfacebook is very broken like Nonlinear.
-Chad
On Fri, Sep 7, 2012 at 2:12 PM, Chad innocentkiller@gmail.com wrote:
All of these have been fixed other than Nonlinear (more heavily broken). TranslationNotifications' master is intact, but some of the changes are still in a bad state and I need to finish cleaning it up.
TranslationNotifications has now been fixed.
software, wikimedia-lvs-realserver, wikimedia-search-qa and wikistats are all back and fine. mysqlatfacebook is very broken like Nonlinear.
I fixed mysqlatfacebook, which had been completely garbage-collected on manganese (the primary server). I restored the commits from formey (the replication slave) where they'd been orphaned but not deleted. Nonlinear is also fixed.
Roan
Am 11.09.2012 06:52, schrieb Roan Kattouw:
On Fri, Sep 7, 2012 at 2:12 PM, Chad innocentkiller@gmail.com wrote:
All of these have been fixed other than Nonlinear (more heavily broken). TranslationNotifications' master is intact, but some of the changes are still in a bad state and I need to finish cleaning it up.
TranslationNotifications has now been fixed.
Could you please check the permissions? L10n-bot is not longer allowed to merge its commits:
https://gerrit.wikimedia.org/r/#/c/23455/ https://gerrit.wikimedia.org/r/#/c/23565/
It fails with
Merging 23565... error: Verified=1 not permitted one or more approvals failed; review output above
Thanks. Raimond.
On Wed, Sep 12, 2012 at 4:53 PM, Raimond Spekking raimond.spekking@gmail.com wrote:
Am 11.09.2012 06:52, schrieb Roan Kattouw:
On Fri, Sep 7, 2012 at 2:12 PM, Chad innocentkiller@gmail.com wrote:
All of these have been fixed other than Nonlinear (more heavily broken). TranslationNotifications' master is intact, but some of the changes are still in a bad state and I need to finish cleaning it up.
TranslationNotifications has now been fixed.
Could you please check the permissions? L10n-bot is not longer allowed to merge its commits:
https://gerrit.wikimedia.org/r/#/c/23455/ https://gerrit.wikimedia.org/r/#/c/23565/
It fails with
Merging 23565... error: Verified=1 not permitted one or more approvals failed; review output above
The permissions were not inheriting from mediawiki/extensions, they were instead inheriting from All-Projects. This is now fixed.
-Chad
http://blog.wikimedia.org/2012/09/07/recovery-of-broken-gerrit-repositories/ writes:
While Git does make it very difficult to get rid of data forever, it is possible. For example, if we had let the cron run again, git gc would’ve merrily deleted all of the unreferenced objects since they were unused.
So, now it just became a matter of sorting out which refs these commits could go to. Playing around with git show makes it pretty easy to figure out where the commits belong, and so this just turned into a process of editing the appropriate refs/* files to add the commit hash (and what seems so obvious now took several hours of hand-wringing, I assure you). So for the next several hours, I took the time to sort out these dangling commits, and the repos ended up working again.
Doesn't formey have core.logAllRefUpdates set to true? Wouldn't that have prevented git gc from removing commits referenced in the reflog? (at least until two weeks without the references passed, it should probably have been run with --no-prune)
On Wed, Sep 12, 2012 at 3:51 PM, Platonides Platonides@gmail.com wrote:
Doesn't formey have core.logAllRefUpdates set to true? Wouldn't that have prevented git gc from removing commits referenced in the reflog? (at least until two weeks without the references passed, it should probably have been run with --no-prune)
I did indeed manage to recover some commits that were lost (apparently gc'ed) on manganese but still present on formey, so that sounds plausible.
Roan
wikitech-l@lists.wikimedia.org