Gerrit server issues

List overview All Threads
Download

newer

older

HTML wikipedia dumps: Could you...

New experimental feature on IRC

Chad

6 Sep 2012 6 Sep '12

9:11 p.m.

Hi everyone, Just letting everyone know that we're having some issues with the Gerrit server at the moment. Sometime since last night, a little over a dozen repositories (listed below) have gone a wonky and are missing their branches & tags (objects are intact). We're working on recovering everything right now, and I'll be sure to let everyone know the status when we're done. In the meantime, if you use any of the following repositories, please refrain from trying to use them until I give the all-clear (I don't want to exacerbate any problems): In mediawiki/extensions/*: Comments, FacebookOpenGraph, GoogleDocs4MW, Nonlinear, OnlineStatusBar, Phalanx, RandomImageByCategory, SemanticImageInput, ShoutWikiAds, SphinxSearch, TranslationNotifcations In operations/*: debs/mysqlatfacebook, debs/wikimedia-lvs-realserver debs/wikimedia-search-qa, debs/wikistats, software Also note: operations/mediawiki-config was busted, but we went ahead and just rebuilt the repo from the live copy so as to not interfere with site operations. I thank you for your patience, and please find me on IRC if you have any questions or find other repositories that are misbehaving. -Chad

Show replies by date

Chad

7 Sep 7 Sep

1:43 a.m.

On Thu, Sep 6, 2012 at 5:11 PM, Chad <innocentkiller(a)gmail.com> wrote:

...

Just to let everyone know, I'm still working on restoring these projects, but they're not ready yet. operations/software.git is back up, though, master was intact after all. -Chad

Chad

1:49 a.m.

On Thu, Sep 6, 2012 at 9:43 PM, Chad <innocentkiller(a)gmail.com> wrote:

...

On Thu, Sep 6, 2012 at 5:11 PM, Chad <innocentkiller(a)gmail.com> wrote:

Just to let everyone know, I'm still working on restoring these projects, but they're not ready yet. operations/software.git is back up, though, master was intact after all.

And mediawiki/extensions/TranslationNotifications is back up too, master was intact. The remaining ones in extensions/* and debs/* are under repair. -Chad

Santhosh Thottingal

3:30 p.m.

2012/9/7 Chad <innocentkiller(a)gmail.com>om>:

...

And mediawiki/extensions/TranslationNotifications is back up too, master was intact. The remaining ones in extensions/* and debs/* are under repair.

It seems TranslationNotifications extension has some problem. Git review is always failing with this error: error: unpack failed: error Missing unknown 4ee9dc02a655cb376a8f20e6d5c1ee95a81b1a37 And remote rejected with unpack error. ps: Some information about this corruption available at http://asheepapart.blogspot.com/2011/10/gerrit-code-review-unpack-error-mis… Thanks Santhosh Thottingal

Chad

4:45 p.m.

On Fri, Sep 7, 2012 at 11:30 AM, Santhosh Thottingal <santhosh.thottingal(a)gmail.com> wrote:

...

2012/9/7 Chad <innocentkiller(a)gmail.com>om>:

And mediawiki/extensions/TranslationNotifications is back up too, master was intact. The remaining ones in extensions/* and debs/* are under repair.

Ah indeed, that repo does need some work. This is the only thing I'm working on today, and I figured out how to resolve it late last night--things will be back to normal by the end of today. -Chad

Daniel Friesen

10:48 p.m.

On Fri, 07 Sep 2012 08:30:47 -0700, Santhosh Thottingal <santhosh.thottingal(a)gmail.com> wrote:

...

2012/9/7 Chad <innocentkiller(a)gmail.com>om>:

And mediawiki/extensions/TranslationNotifications is back up too, master was intact. The remaining ones in extensions/* and debs/* are under repair.

I'd like to know what kind of blobs it is that Gerrit depends on so much but git is happy to prune. -- ~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://daniel.friesen.name]

Chad

9:12 p.m.

On Thu, Sep 6, 2012 at 5:11 PM, Chad <innocentkiller(a)gmail.com> wrote:

...

In mediawiki/extensions/*: Comments, FacebookOpenGraph, GoogleDocs4MW, Nonlinear, OnlineStatusBar, Phalanx, RandomImageByCategory, SemanticImageInput, ShoutWikiAds, SphinxSearch, TranslationNotifcations

All of these have been fixed other than Nonlinear (more heavily broken). TranslationNotifications' master is intact, but some of the changes are still in a bad state and I need to finish cleaning it up.

...

In operations/*: debs/mysqlatfacebook, debs/wikimedia-lvs-realserver debs/wikimedia-search-qa, debs/wikistats, software

software, wikimedia-lvs-realserver, wikimedia-search-qa and wikistats are all back and fine. mysqlatfacebook is very broken like Nonlinear. -Chad

Roan Kattouw

11 Sep 11 Sep

4:52 a.m.

On Fri, Sep 7, 2012 at 2:12 PM, Chad <innocentkiller(a)gmail.com> wrote:

...

TranslationNotifications has now been fixed.

...

software, wikimedia-lvs-realserver, wikimedia-search-qa and wikistats are all back and fine. mysqlatfacebook is very broken like Nonlinear.

I fixed mysqlatfacebook, which had been completely garbage-collected on manganese (the primary server). I restored the commits from formey (the replication slave) where they'd been orphaned but not deleted. Nonlinear is also fixed. Roan

Raimond Spekking

12 Sep 12 Sep

8:53 p.m.

Am 11.09.2012 06:52, schrieb Roan Kattouw:

...

On Fri, Sep 7, 2012 at 2:12 PM, Chad <innocentkiller(a)gmail.com> wrote:

TranslationNotifications has now been fixed.

Could you please check the permissions? L10n-bot is not longer allowed to merge its commits: https://gerrit.wikimedia.org/r/#/c/23455/ https://gerrit.wikimedia.org/r/#/c/23565/ It fails with Merging 23565... error: Verified=1 not permitted one or more approvals failed; review output above Thanks. Raimond.

Chad

9:16 p.m.

On Wed, Sep 12, 2012 at 4:53 PM, Raimond Spekking <raimond.spekking(a)gmail.com> wrote:

...

Am 11.09.2012 06:52, schrieb Roan Kattouw:

On Fri, Sep 7, 2012 at 2:12 PM, Chad <innocentkiller(a)gmail.com> wrote:

TranslationNotifications has now been fixed.

The permissions were not inheriting from mediawiki/extensions, they were instead inheriting from All-Projects. This is now fixed. -Chad

Platonides

10:51 p.m.

http://blog.wikimedia.org/2012/09/07/recovery-of-broken-gerrit-repositories/ writes:

...

While Git does make it very difficult to get rid of data forever, it is possible. For example, if we had let the cron run again, git gc would’ve merrily deleted all of the unreferenced objects since they were unused.

...

So, now it just became a matter of sorting out which refs these commits could go to. Playing around with git show makes it pretty easy to figure out where the commits belong, and so this just turned into a process of editing the appropriate refs/* files to add the commit hash (and what seems so obvious now took several hours of hand-wringing, I assure you). So for the next several hours, I took the time to sort out these dangling commits, and the repos ended up working again.

Doesn't formey have core.logAllRefUpdates set to true? Wouldn't that have prevented git gc from removing commits referenced in the reflog? (at least until two weeks without the references passed, it should probably have been run with --no-prune)

Roan Kattouw

11:07 p.m.

On Wed, Sep 12, 2012 at 3:51 PM, Platonides <Platonides(a)gmail.com> wrote:

...

I did indeed manage to recover some commits that were lost (apparently gc'ed) on manganese but still present on formey, so that sounds plausible. Roan

4245

days inactive

4251

days old

wikitech-l@lists.wikimedia.org

Manage subscription

11 comments

6 participants

tags (0)

participants (6)

Chad
Daniel Friesen
Platonides
Raimond Spekking
Roan Kattouw
Santhosh Thottingal