On Wed, Jul 18, 2012 at 11:28 AM, Platonides Platonides@gmail.com wrote:

The evaluation is intended to determine if we want to continue with gerrit or change. If we decide to change, we should then see what we want to change to. Maybe we would finally decide that all the alternatives are worse and end up not changing. Or keeping an eye to anything new which gets published. Or perhaps contributing a LDAP module to some otherwise-fine tool.

I don't think you can decide to change away from gerrit without having an idea of what we want to use instead. As I understand it, we're on gerrit because it's the least-bad option. To show that it's no longer the least bad option, you need to find a product that is less bad.

What about changing gerrit to our needs? It's open source, I suppose.

On Wed, Jul 18, 2012 at 6:44 PM, Derric Atzrott datzrott@alizeepathology.com wrote:

...

I don't think you can decide to change away from gerrit without having an idea of what we want to use instead. As I understand it, we're on gerrit because it's the least- bad option. To show that it's no longer the least bad option, you need to find a product that is less bad.

Agreed. Otherwise we end up in a position where we really want to switch and everyone is excited to switch, but there is nothing to switch to. People get disappointed, eventually everyone moves on and goes back to using gerrit like nothing happened. All in all, discussing switching without any suitable options to switch to leads to disappointment and wasted time. Neither of which are helpful for us.

Thank you, Derric Atzrott

---

Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

On Wed, 18 Jul 2012 11:41:06 -0700, Ryan Lane rlane32@gmail.com wrote:

On Wed, Jul 18, 2012 at 1:18 PM, Petr Bena benapetr@gmail.com wrote:

...

What about changing gerrit to our needs? It's open source, I suppose.

That's of course the route we're going right now. The biggest hurdle to doing so is that it's written in Java, and Prolog. Neither of those languages are amazingly difficult, though, so I don't really see that as a complete blocker.

• Ryan

The blocker for me was not the language, but the codebase. I wanted to make a small tweak to Gerrit so I started looking through the code. And I had absolutely no clue where to find what I was looking for. I couldn't make sense of what most of what I found was even supposed to do. And people have pointed out a number of issues with Gerrit like the way it handles output and css which feel much more like fundamental (ie: unfixable without practically rewriting) issues with Gerrit.

...

Daniel Friesen lists@nadir-seen-fire.com wrote:

On Wed, 18 Jul 2012 11:41:06 -0700, Ryan Lane rlane32@gmail.com wrote:

...

On Wed, Jul 18, 2012 at 1:18 PM, Petr Bena benapetr@gmail.com wrote:

...

What about changing gerrit to our needs? It's open source, I suppose.

That's of course the route we're going right now. The biggest hurdle to doing so is that it's written in Java, and Prolog. Neither of those languages are amazingly difficult, though, so I don't really see that as a complete blocker.

• Ryan

The blocker for me was not the language, but the codebase. I wanted to make a small tweak to Gerrit so I started looking through the code. And I had absolutely no clue where to find what I was looking for. I couldn't make sense of what most of what I found was even supposed to do. And people have pointed out a number of issues with Gerrit like the way it handles output and css which feel much more like fundamental (ie: unfixable without practically rewriting) issues with Gerrit.

I got used to it. It's completely different Java if one is used to old-skool Java programming. Components are decoupled with the use of Guice (for "dependency injection" - http://c2.com/cgi/wiki?DependencyInjection) framework plus there is Google Web Toolkit programming, again a very special beast.

Another component is the ORM mapper, gwtorm.

Other than that it's pretty fine, with the usual Java problem that I need to cut through gazillion of classes and interfaces before I get to the core of things.

For example, to fix https://bugzilla.wikimedia.org/show_bug.cgi?id=38114 few lines need to be added before line 296 of

https://gerrit.googlesource.com/gerrit/+/05d942c324d7a17285873a468f3605cbc19...

(not sure it's a good idea but here it is)

I have attached Jython interpreter to Gerrit to play a bit with the code:

https://gerrit-review.googlesource.com/#/c/34670/

You can play live with the ORM mapper for example and retrieve Java objects from the database (not just rows).

//Saper

On Wed, Jul 18, 2012 at 9:33 AM, Andrew Garrett agarrett@wikimedia.org wrote:

I don't think you can decide to change away from gerrit without having an idea of what we want to use instead. As I understand it, we're on gerrit because it's the least-bad option. To show that it's no longer the least bad option, you need to find a product that is less bad.

Precisely. Since my stated bias is to stay on Gerrit, I'm not the best person to propose the alternative, but suffice it to say, we're going to have to have a much more buttoned-down case for the alternative if yet another migration is in the cards.

If we stay on Gerrit, we plan to make investments in those areas that we've identified as deficient. However, we shouldn't make the decision to stay on Gerrit based on vaporware - we should evaluate Gerrit as it is today, and assume that any changes we propose are harder than they appear on the surface. That also means evaluating the *other* tools as they are today, as well, and assuming that any changes we need to *those* tools are harder than they appear on the surface.

Rob

On 07/18/2012 11:35 PM, Roan Kattouw wrote:

On Wed, Jul 18, 2012 at 9:30 PM, Subramanya Sastry ssastry@wikimedia.org wrote:

...

(b) Commit amends hide evolution of an idea and the tradeoffs and considerations that went into development of something -- the reviews are all separate from git commit history. All that is seen is the final amended commit -- the discussion and the arguments even that inform the commit are lost. Instead, if as in (a), a topic branch was explicitly (or automatically created when a reviewer rejects a commit), and fixes are additional commits, then, review documentation could be part of the commit history of git and shows the considerations that went into developing something and the evolution of an idea.

There was an email recently on wikitext-l where Mark Bergsma was asked to squash his commits (http://osdir.com/ml/general/2012-07/msg20847.html) -- I personally think it is a bad idea that is a result of the gerrit review model. In a different review model, a suggestion like this would not be made.

Although squashing and amending has downsides, there is also an advantage: now that Jenkins is set up properly for mediawiki/core.git , we will never put commits in master that don't pass the tests. With the fixup commit model, intermediate commits often won't pass the tests or even lint, which leads to false positives in git bisect and makes things like rolling back deployments harder.

Agree somewhat. But, two things to consider:

(a) Not all commits that are reviewed break tests. I am not arguing that amends and squashing via rebase dont have a place. I am arguing that amends are overused right now. Where commits break an existing test suite, part of the review requirement could be that the previous commit be amended so it passes tests. But, applying this logic in all cases doesn't seem right to me.

(b) What happens if I submit a new test that breaks HEAD? Are we arguing that the test cannot be committed till HEAD passes all tests? Will my test be held in abeyance till someone gets around to fixing HEAD?

I think there is an argument to be made for a non-straitjacketed approach to review: always pre-commit review, commit is always a single unit of review, always amend commits, always squash multiple commits -- this is the gerrit model which I am chafing against.

I am new here and I also have limited exposure to review tools and processes, so I dont want to overstate my case. I also understand that this is not a simple problem, and maybe this is the best we have -- but I at least want to express my annoyance with what we have, and that we should reduce this to being entirely an UI issue (which is important in and of itself).

Also, my gratitude to all of you who have previously spent the time instituting processes and tools -- it is better than nothing, and it can be a thankless job with people like me griping :) but my intention here is to only help this forward where I can. Can I work with what we have? Yes, I can. Will I be happier with a different process / tool? Yes, I will be :).

Subbu.

On Wed, 18 Jul 2012 22:37:56 -0700, Daniel Friesen lists@nadir-seen-fire.com wrote:

On Wed, 18 Jul 2012 21:35:00 -0700, Roan Kattouw roan.kattouw@gmail.com wrote:

...

On Wed, Jul 18, 2012 at 9:30 PM, Subramanya Sastry ssastry@wikimedia.org wrote:

...

(b) Commit amends hide evolution of an idea and the tradeoffs and considerations that went into development of something -- the reviews are all separate from git commit history. All that is seen is the final amended commit -- the discussion and the arguments even that inform the commit are lost. Instead, if as in (a), a topic branch was explicitly (or automatically created when a reviewer rejects a commit), and fixes are additional commits, then, review documentation could be part of the commit history of git and shows the considerations that went into developing something and the evolution of an idea.

There was an email recently on wikitext-l where Mark Bergsma was asked to squash his commits (http://osdir.com/ml/general/2012-07/msg20847.html) -- I personally think it is a bad idea that is a result of the gerrit review model. In a different review model, a suggestion like this would not be made.

Although squashing and amending has downsides, there is also an advantage: now that Jenkins is set up properly for mediawiki/core.git , we will never put commits in master that don't pass the tests. With the fixup commit model, intermediate commits often won't pass the tests or even lint, which leads to false positives in git bisect and makes things like rolling back deployments harder.

Roan

I don't see this as a case against not-squashing commits. The fact that previous commits are broken is not a bug. It's a fact of development and history that MUST be accepted rather than covered up. Making sure that every single commit inside the repo passes test does NOT matter. The only thing that matters is that final commits that we merge into the master branch pass tests. In other words only the actual commits (ie: merge commits) directly committed to the master branch should be required to pass tests, not the potentially broken commits they are based on. If you are bisecting and trying to track down an issue you should NOT be randomly going down every topic branch and testing it. If you test the commit from which the topic branch was merged into master and it comes back OK then you should ignore the topic branch since those were never directly part of master and are not the source of your bug. You should continue down master until you hit the topic/merge commit that introduced the bug directly into master.

Frankly while it may look odd my recommendation is to make use of --no-ff to ensure that the commit that actually goes into master is not the HEAD of the topic branch but a merge commit. Even if there is no rebasing or merging needed. This helps keep a clean line where you can trust that everything inside master was explicitly committed there. And instead of having random cases where either a commit is directly put into master or you get some commit saying something like "merged from" just because of some conflicts. You instead end up with your merge commits becoming a proofread and edited version that's gone through the final review, testing, and been approved by everyone.

One of the ideas I had for Gareth was to let a commit message be web-editable on the review page. The commit message would be based on the original commit's commit message. Afterwards anyone could contribute to the commit message in the web view and make tweaks to it. Instead of forcing someone to amend a commit to change the commit message you would just make a tweak to the commit message displayed on the review page. You could easily re-word it a bit, add some items not included in the original (eg: you added a new feature in a follow up commit), and fix any typos the original committer made. When the review/topic branch is okayed and accepted into master Gareth would use --no-ff to force Git to create a brand new commit for the merge and it would use the commit message that everyone made edits to as the commit message for that merge commit instead of some "merged from" message. Then master would become a clean vetted linear history. And when you wanted to learn the history of an individual addition to the codebase you would follow the parents of that commit into the review branch that was merged into master.

Here's an example of the flow I mean: https://www.mediawiki.org/wiki/User:Dantman/Git/FastForward-vs-no-ff

On the left is the normal project flow. You make changes to the topic branch and at the end merge it into the master branch. When you do so because there are no conflicts all the commits in the topic branch become part of master. Not necessarily what you want.

On the right is the flow I consider ideal for large projects like MediaWiki. The pattern follows the exact same pattern as the left. You make changes into the topic branch and then at the end merge it into master. But this time when you do the merge (or rather the review system does) you use --no-ff and add a customized message with an overview of all the changes in the topic branch that got merged into master (This essentially becomes your proofread and edited final commit that everyone sees). As a result of --no-ff: - None of the individual changes which weren't guaranteed to have passed tests made their way into master. - You have a very clean overview of the branch as the actual commit in master leading to a tidy history that everyone can understand. - The entire history of the project is contained, including the small tweaks that everyone has made. And no-one ends up overriding a commiter's name with their own because they fixed a typo. - Anyone trying to bisect can simply ignore the individual irrelevant commits inside of a topic branch and just test master itself. (In fact at this level you can probably find some way to use a test-suite to auto-bisect a codebase using a simple test case and running it over the history of the repo)

On Jul 18, 2012, at 9:35 PM, Roan Kattouw wrote:

On Wed, Jul 18, 2012 at 9:30 PM, Subramanya Sastry ssastry@wikimedia.org wrote:

...

(b) Commit amends hide evolution of an idea and the tradeoffs and considerations that went into development of something -- the reviews are all separate from git commit history. All that is seen is the final amended commit -- the discussion and the arguments even that inform the commit are lost. Instead, if as in (a), a topic branch was explicitly (or automatically created when a reviewer rejects a commit), and fixes are additional commits, then, review documentation could be part of the commit history of git and shows the considerations that went into developing something and the evolution of an idea.

There was an email recently on wikitext-l where Mark Bergsma was asked to squash his commits (http://osdir.com/ml/general/2012-07/msg20847.html) -- I personally think it is a bad idea that is a result of the gerrit review model. In a different review model, a suggestion like this would not be made.

Although squashing and amending has downsides, there is also an advantage: now that Jenkins is set up properly for mediawiki/core.git , we will never put commits in master that don't pass the tests. With the fixup commit model, intermediate commits often won't pass the tests or even lint, which leads to false positives in git bisect and makes things like rolling back deployments harder.

Roan

I disagree. Contrary to what many think, every single patch set and amendment goes into the mediawiki/core.git repository, whether reviewed and passing, or a fresh mistake. This is easily verified by the fact that every patch set revision has its own gitweb link, and the fact that git-review downloads the merge request from a remote branch, inside the core.git repo (or whatever the repo may be).

git-bisect is not an issue now and won't be an issue in the branch-review model[1], because git-bisect only affects the HEAD's tree (naturally). The way to merge a branch would be to squash the branch into one commit when merging (e.g. the "merge" commit). This is also how jQuery lands branches most of the time, and iirc GitHub (as in, the internal repo github.com/github/github) also works this way with long-lived branches (based on a presentation they gave; "How GitHub uses GitHub to build GitHub"[2]).

And, of course, we will stick to the model of only allowing merges when tests pass. So the HEAD's history will remain free of commits that don't pass tests.

One could argue that then the branches' history will not be included in the master's history, but that's not the case now either. Because only the last amendment will be in the master's history (which is just as much a squash, except that that squash is the result if re-ammending over and over again, instead of subsequent commits being squashed afterwards).

-- Krinkle

[1] "branch-review model", as in, a model where a review is about a topic-branch, whether it contains 1 commit, or many. Or even a branch from a fork. In other words the pull-request model from GitHub. And yes, on GitHub one can also create a pull-request from one branch to another, within the same repository (e.g. mediawiki-core/feature-x -> mediawiki-core/master or mediawiki-core/feature-x-y-z -> mediawiki-core/feature-x) .

[2] http://zachholman.com/talk/how-github-uses-github-to-build-github

+1 to Subramanya.

Roan Kattouw wrote:

Although squashing and amending has downsides, there is also an advantage: now that Jenkins is set up properly for mediawiki/core.git , we will never put commits in master that don't pass the tests. With the fixup commit model, intermediate commits often won't pass the tests or even lint, which leads to false positives in git bisect and makes things like rolling back deployments harder.

Roan

It should be possible to only bisect on the left-branch of merges. If git doesn't have such feature, it should be added. In fact, it's very likely that it's what it uses when start and end are in the same line, but as the merged commits can also have that ancestor, it might also go there.

We would need to store multiple commits as non-fastforwards, even if descending from head, but that's trivial. What I have trouble with is in imagining the UI for reviewing a topic branch, with perhaps several commits per "step".

Always registering as a merge, has succh as storing the merger name as the commiter, but I'd like keeping them as fest-forwards having for one-commit changes. I think the history would easily get hard to browse. Also, I don't like how when browsing a merge (eg. gitk) you are not shown the actual changes to the repository (which for us is the left-parent). Anyone knows of a way to do so?

Regards

Roan Kattouw:

Fixup commits tend to be a blind spot when people think about pre-commit review.

How are the blind spot? You can diff each patchset with its previous one (though a rebase kind of screw it up).

Whenever I send a new patchset, I try to add a cover message introducing what was done in the new patchset so people receive a mail notification explaining what has been done.

Platonides wrote:

What I have trouble with is in imagining the UI for reviewing a topic branch, with perhaps several commits per "step".

To me the concept of topic is to lamely tag a commit as being part of specific area of code such as API, Parser, tests. We could probably tweak Gerrit to automatically assign a set of reviewer just on the basis of the topic name.

If someone is working on a big changes having several steps, I would prefer we use a branch (for now the sandbox system). Then each final commit will be one of the step and all the intermediate/fixing commits would be patchsets.

One could also submit all their patchsets and have them submitted in reversed date order. The submission of the first patch that has an already merged parent will then trigger a merge commit of all the changes. An example with some doc rewrite I did which was merged by Sam in mediawiki/core.git:

$ git log --oneline --graph d11d328^..444ab87 * 444ab87 Merge changes Icfd267a9,If956c0a1,I0e232273,Ia84494c8,Ib70e0046,If2c01a11,I40196371 |\ | * 4a69fce prettify documentation | * 12e608e converts '@fixme' to '@todo FIXME' | * d5737f8 update @param @return doc in several files | * bcf0010 prettify filerepo documentation | * 9aa825f prettify SkinTemplate:makeLink documentation | * 1dd38a7 move parameter doc under @param | * aab43dd escape tags and entity in doxygen comments * | 1a9246e ChannelFeed now marked as an abstract class * | f3f985c remove unneeded @abstract |/ * d11d328 Merge "Fixed Language::isValidBuiltInCode exception." * 75f5bda Fixed Language::isValidBuiltInCode exception. $

I made my changed in sequence and based on d11d328. Submitted all of them then Sam submitted all my change submitting aab43dd as the last. Gerrit then happily merged them.

Always registering as a merge, has such as storing the merger name as the commiter, but I'd like keeping them as fest-forwards having for one-commit changes.

A strategy could be setup where we always merge as to know from git history who actually applied the change but do a fast forward when the submitter is the patch author. Still messy though, I can live without the merge commits.

I think the history would easily get hard to browse.

One can filter out merge commit by passing `--no-merges` to `git log`

Also, I don't like how when browsing a merge (eg. gitk) you are not shown the actual changes to the repository (which for us is the left-parent). Anyone knows of a way to do so?

A merge commit does not really have any change by itself. I just do a diff against the merge parent such as:

git diff 444ab87^ 444ab87

The related changelist would be: git log --oneline 444ab87^..444ab87

On Thu, 19 Jul 2012 11:58:54 -0700, Erik Moeller erik@wikimedia.org wrote:

On Tue, Jul 17, 2012 at 5:41 PM, Rob Lanphier robla@wikimedia.org wrote:

...

It would appear from reading this page that the only alternative to Gerrit that has a serious following is GitHub. Is that the case? Personally, it seems like Phabricator or Barkeep has the best chance of dislodging Gerrit, but those won't probably get serious consideration without a champion pushing them.

What are people's experiences with Gitorious? Does it seem sufficiently hackable to perhaps meet our needs, or does it have too many architectural flaws to be worthwhile?

Gitorious does pubkey management on it's own. Something that we already do and don't want the tool to do (ie: we're trying to get rid of the fact that you have to add your key to Gerrit too). It also needs to have LDAP integration written.

The ops guys hate ruby.

Here are a few things I've been able to find out from poking at it:

• Unlike GitHub, it's fully open source (AGPL).
• It's a Ruby on Rails codebase with lots of gem dependencies and a

reputation for being hard to install (haven't tried it).

• Like GitHub/Gerrit, it's not just a code review tool, but also

manages repositories.

Gerrit doesn't really manage repositories much. Admins can create them, but it's really nothing like GitHub/Gitorious for managing them. Gerrit is a code review tool. GitHub/Gitorious are not. Though GitHub's does have a nice pull requests have come very far and are close to code review. Last I checked Gitorious doesn't have anything that close.

• Like GitHub, it makes it easy to clone/fork entire repos and send a

merge request for a series of commits, encouraging true decentralized development.

• It supports both individual clones and team clones.
• Like most tools, it supports inline comments in code review:

http://blog.gitorious.org/2009/11/06/awesome-code-review/

• Unlike Gerrit, it conveniently shows all diffs for a commit on a

single page, e.g.: https://gitorious.org/gitorious/mainline/merge_requests/208

• Unlike GitHub/Gerrit, it has no easy way to integrate merge requests

-- developers have to do so manually using git commands (see the "how to apply this merge request" box in the page above). The process seems particularly cumbersome for small patchsets, of which we get a lot.

• It appears to have limited private repo support:

http://blog.gitorious.org/2012/02/29/private-repositories-2/

• It has a semi-active upstream: https://gitorious.org/gitorious/mainline
• Its UI appears less clunky and vomit-inducing than Gerrit's, but it

doesn't exactly feel like the most responsive & efficient design either. There are some niceties, like breadcrumb navigation, avatars, etc.

From what I can tell, we have essentially three choices:

• Continue to work with the heavily centralized and clunky Gerrit

workflow, and try to beat it into shape to not cause us too much pain, while seeing people increasingly move into GitHub for doing experimental projects. Hope for Gerrit's large developer base to ultimately join a rewrite effort, or to incrementally make it better. Build Gerrit/GitHub bridges if possible.

• Drink the kool-aid and join the wonderful semi-open world of GitHub,

with all the risks and benefits it entails.

• Help build a realistic alternative to GitHub, probably by building

on the open source toolset that's the closest right now in terms of its overall architectural approach and existing functionality. (My impression: Gitorious is closer in functionality but at a lower overall quality, Phabricator/Barkeep are much more limited in functionality right now and therefore would take a long time to be usable with our current workflow assumptions.)

Erik

On Thu, Jul 19, 2012 at 11:55 PM, Antoine Musso hashar+wmf@free.fr wrote:

Daniel Friesen wrote:

...

The ops guys hate ruby.

I am pretty sure they love it. Puppet itself is a DSL based on top of ruby. The ops argument is we don't want to handle security updates and nasty performance bug for yet another language.

No. No. We hate it. I, personally, also dislike puppet, but that's another discussion totally.

- Ryan

I think the "we" is a bit unwarranted. I don't hate Ruby and I certainly don't hate Ruby more than Java :-) I also don't feel the same about puppet; I do see some problems with it, but none of them have nothing to do with the fact that it's written in Ruby.

Well, 90% of the ops team.

I think this discussion is pointless though. If we find a good tool for the job and it's clear how to install and maintain it, I don't see why we should care about its implementation language, at least from an ops perspective. I've seen horrible & difficult to operate software in Python and perfect ones in Ruby. I don't see how our personal preferences towards languages have any value here.

It wouldn't be a blocker, but it's a major negative, IMO. At minimum, it's as much of a negative as Gerrit being Java. Few of us know ruby very well, and managing the language and libraries is a pain in the ass, thanks to the ruby community.

That said, it's no point worrying about this until we actually find something that's a reasonable alternative. I'm on the pro-gerrit side, as it works well for me and I see no other reasonable alternatives.

- Ryan

On Thu, Jul 19, 2012 at 6:29 PM, Roan Kattouw roan.kattouw@gmail.com wrote:

gitlab is an open source clone of GitHub (which means you can self-host it)

Forgot to say: thanks to Timo for bringing gitlab to my attention.

Roan

From what I can tell, we have essentially three choices:

• Continue to work with the heavily centralized and clunky Gerrit

workflow, and try to beat it into shape to not cause us too much pain, while seeing people increasingly move into GitHub for doing experimental projects. Hope for Gerrit's large developer base to ultimately join a rewrite effort, or to incrementally make it better. Build Gerrit/GitHub bridges if possible.

• Drink the kool-aid and join the wonderful semi-open world of GitHub,

with all the risks and benefits it entails.

• Help build a realistic alternative to GitHub, probably by building

on the open source toolset that's the closest right now in terms of its overall architectural approach and existing functionality. (My impression: Gitorious is closer in functionality but at a lower overall quality, Phabricator/Barkeep are much more limited in functionality right now and therefore would take a long time to be usable with our current workflow assumptions.)

I agree with Rob that changing Gerrit or Gitorious or Gitlab is not something that we should get undertake lightly. In my opinion, it is more realistic to evaluate the tools as they exist today, with maybe mostly minor changes and small features that MW developers could contribute towards. Larger changes might require more dedicated developer resources and would also take time.

Subbu.

OT: I also noticed some Ruby concerns on this thread. Except Gerrit, 4 of the other 5 candidates (Github, Gitorious, Gitlab, and Barkeep) are all written in Ruby (on Rails possibly). Ruby performance has improved over the years for all 3 implementations (MRI, Rubinius, and JRuby). JRuby (Ruby implementation on the JVM) performance especially has improved quite a lot over the last couple years. Over the last 3 years, I have been an active developer on JRuby and I have also written and maintained Rails apps. So, I am familiar with Ruby and am happy to help with Ruby related issues.

On Thu, Jul 19, 2012 at 11:58:54AM -0700, Erik Moeller wrote:

From what I can tell, we have essentially three choices:

• Continue to work with the heavily centralized and clunky Gerrit

workflow, and try to beat it into shape to not cause us too much pain, while seeing people increasingly move into GitHub for doing experimental projects. Hope for Gerrit's large developer base to ultimately join a rewrite effort, or to incrementally make it better. Build Gerrit/GitHub bridges if possible.

• Drink the kool-aid and join the wonderful semi-open world of GitHub,

with all the risks and benefits it entails.

• Help build a realistic alternative to GitHub, probably by building

on the open source toolset that's the closest right now in terms of its overall architectural approach and existing functionality. (My impression: Gitorious is closer in functionality but at a lower overall quality, Phabricator/Barkeep are much more limited in functionality right now and therefore would take a long time to be usable with our current workflow assumptions.)

That's a nice summary. I (too?) like option 3. My distaste for Gerrit and closed platforms are probably on par right now.

Regards, Faidon

On Tuesday, July 24, 2012, Sumana Harihareswara wrote:

Basically, I'm thinking, let's not put so many of our eggs in the GitHub basket. GitHub is fine for FLOSS projects with fewer than a hundred repositories, ones that don't already have several communications channels, ones where privacy is less of a concern, or ones that don't run the sixth biggest website in the world practically right off trunk. But we have and will have so many strange, unforeseen needs that we should keep certain key operations on servers that we run and can hack at will.

We do need a GitHub strategy -- to make our projects more discoverable, make use of more contributions, and participate in the GitHub reputational economy. So we must figure out the right ways to mirror and sync. But I doubt our own long-term needs would work well with using GitHub as our main platform.

[0] https://bugzilla.wikimedia.org/show_bug.cgi?id=38196 [1] https://bugzilla.wikimedia.org/show_bug.cgi?id=35497 [2] https://github.com/site/terms [3] http://developer.github.com/

(Thanks to Chad and RobLa for talking through much of this with me.)

-- Sumana Harihareswara Engineering Community Manager Wikimedia Foundation

But do we have a plan for improving Gerrit in a substantial way?

I can get behind the decision to use a currently substandard tool in order to preserve Wikimedia's long term freedom. But to stick with Gerrit, we must have a plan for fixing it that does not simply declare that the ability to make changes means that the magic FOSS fairy will make it so. I don't know what it would take -- maybe a weekend in SF where we invite every Java and Prolog expert we can find? Paying a contractor or two to make the necessary fixes?

This isn't just about attracting scores of new volunteers or having a "reputational economy". It's a push for change driven by the fact that Gerrit seriously undercuts developer productivity and happiness. When we've got so many difficult, ambitious projects under way, I think those are two things we should be prioritizing. By that measuring stick, Gerrit fails miserably and GitHub is a winner.

Steven

---

Wikitech-l mailing list Wikitech-l@lists.wikimedia.org javascript:; https://lists.wikimedia.org/mailman/listinfo/wikitech-l

On Wed, Jul 25, 2012 at 12:39 AM, Ryan Lane rlane32@gmail.com wrote:

1. It would be worrisome from a security point of view to host the

operations repos on Github. I think it's very likely that those repos would stay in Gerrit if the devs decided to move to Github. This would mean that we'd have split workflows, forcing developers to use both.

What's more, we also wouldn't want to host MW deployment code (the wmf/* branches) on Github for the same reason. Those aren't even separate repos, they're branches within the mediawiki/core repo. Having a split workflow for contributing code to the project on the one hand and merging it into deployment on the other hand would be even worse.

Roan

Currently the platform guys are too busy to even report bugs upstream.

That is simply not true.

...

I'm not sure I agree with the claim that this seriously undercuts productivity. Is there any data to back this up?

There is, and I wonder how you cannot be aware of that. Let me repeat what I've said before.

I spent countless hours preparing translatewiki.net for git. Updating and comitting to hundreds of repositories takes minutes. I'm glad we were able to automate that, except that it took two months to get it working as it was supposed to work right after the switch.

3rd party users (me included) all still left out in the cold. There are docs or scripts to manage installation with subset of extensions from svn and git. Barely a day goes by without someone having some kind of problem with git or gerrit. And then we have these discussions about it. Where is the solution for my review workflow? Ever since the switch I spend more time on code review to review less code.

On individual level the wasted productivity might not be much, (but it is: [1][2]), but adding it all together - that is a lot. In your mind, do a rough estimate of all the time spent on preparing and executing the switch, all the time spend on learning and solving the issues. Then convert that to dollars and you get rough price tag for the switch.

[1] Each new person needs to spend tens of hours to learn to use our GiGeGat effectively - if they don't give up [2] Some people have waited for commit access and creation of new repositories for multiple days

None of these issues are Gerrit specific. You are complaining about the switch from svn to git. Yes, we know there was productivity lost in the switchover. We're discussing alternatives to git, not the switchover, though.

- Ryan

We're discussing alternatives to *gerrit*, not to git.

Heh. Sorry, yes. that's what I meant.

- Ryan

On Jul 25, 2012, at 12:39 AM, Ryan Lane wrote:

Many claims that "Gerrit sucks" tend to be due with misunderstandings of how Gerrit works. Many other claims are due to our workflow or our restrictions with access currently. Of course, many claims are legitimate and we are reporting the issues, tracking them upstream, and in some cases pushing in fixes.

In their defense, I think a lot of it has to do with terrible UI/UX in Gerrit. The basics is can be modified by CSS and templates (I believe we've done some), but it only goes so far. How do I modify Javascript in Gerrit? I think it starts (and ends) somewhere in the hell that is GWT… and when GWT begins with, something about how awesome it is to be able to write Ajax stuff using Java, I stop reading http://www.flickr.com/photos/tychay/1388234558/

I've not added this complaint because David already put 90% of this in the #1 reason on his list already. ;-)

Related is the fact that we seem to have a lot of PHP web dev expertise (for some reason) and Gerrit went from Python (serviceable) to Java (totally opaque). Apologies to those of you at the WMF who lurv themselves some Java… all two of you… and one of you is probably the guy who wrote the "case against"

…

But it is true a lot of the griping was related to people from a SVN/CVS model not understanding the Git model at all (in my more cynical moments, I feel that neither do Gerrit's developers :-D ).

Take care,

terry

Well there's more than pushing changes upstream. Modifying/customizing Gerri's UI is supposed to be able to be done without pushing back upstream. As a relative novice of GWT, modifying/customizing UI in gerrit seems rather opaque. But before I go whole hog on Phabricator, I'd have to look more into how it handles templating and customization… I think there is a large class of changes we might want to make that aren't so programmy as to involve using Conduit or Arcanist to get it done.

In any case, I think at the moment Gerrit does what we want and its finally usably fast. I'm sure off the top of my head I can think of a number of things that keep something written in PHP (Phabricator) from matching Gerrit's feature set (LDAP? ACLs?), and pushing upstream patches isn't high on my list pro/con any of these systems.

But things are changing here at a rapid pace so I'm not going to say Gerrit now, Gerrit forever. ;-)

On Jul 25, 2012, at 5:01 AM, Chad wrote:

On Wed, Jul 25, 2012 at 7:29 AM, Terry Chay tchay@wikimedia.org wrote:

...

   Related is the fact that we seem to have a lot of PHP web dev expertise (for some reason) and Gerrit went from Python (serviceable) to Java (totally opaque). Apologies to those of you at the  WMF who lurv themselves some Java… all two of you… and one of you is probably the guy who wrote the "case against"

The more I've thought about it, the less that I feel "language it's written in" really matters at all. The number of people contributing upstream is always going to be relatively small, and as long as those who /want/ to contribute upstream are comfortable with it, it could be written in Cobol for all I care.

It kinda struck me the other day when the subject of bug-tracking tools came up again. Had we been using $SOME_OTHER_PRODUCT and people were advocating switching to Bugzilla, I'm sure people would complain "omg, it's Perl--we can't contribute upstream." But in reality, how many people *have* contributed upstream to Bugzilla? Most people file bugs in our tracker and they get re-filed upstream, which is perfectly fine as long as there's an upstream who responds, which in this case there is.

I think the choice of platform matters when we're talking about "ease of installation/upgrading" to some degree so we don't make the ops angry, but that's a total non-issue with Gerrit because installation/uprgrades are very very easy :)

-Chad

---

Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

On Wed, Jul 25, 2012 at 3:31 PM, Chad innocentkiller@gmail.com wrote:

I haven't actually tried doing custom Javascript yet, but it should be completely doable via the GerritSite.html header that you can customize (in fact, I've got some other non-JS customizations I want to roll out there soon).

I've done this, stealing some code from the OpenStack-CI people. It's in an abandoned change in the puppet repo: https://gerrit.wikimedia.org/r/#/c/3285/2/files/gerrit/skin/GerritSiteHeader...

Roan

«Our most prolific reviewers seem to prefer Gerrit»{{citation needed}}

Do you mean the reviewers who are most prolific right now, those who were most prolific before the Git conversion or a mix of the two? And where are the stats? I know the analytics team is working on gerrit-stats but not what the project specifications include (there's only been an email here in April), see https://www.mediawiki.org/wiki/Talk:Analytics/Reportcard.

Nemo

On Jul 24, 2012, at 9:09 PM, Sumana Harihareswara wrote:

On 07/17/2012 08:41 PM, Rob Lanphier wrote:

...

It would appear from reading this page that the only alternative to Gerrit that has a serious following is GitHub. Is that the case?

There's some irony and yet so apropos in that now that Gerrit is finally stabilizing we're discussing alternatives. :-)

Oh well… Here's my 2c on GitHub…

In an "ignore reality" world, I suppose my personal choices would be 1) GitHub; 2) Phabricator; 3) everything else. But let's cross GitHub off that list (for WMF).

Maybe in some future when our development process more closely models a seat-of-the-pants startup universe of code first, break often, recover fast we could consider GitHub for hosting some of our public repositories, but since I don't see that happening anytime soon (ever?).

…

The nonstarter is that while we could host the public repositories, we do have a lot of non-public stuff in Gerrit right now. That stuff can't go into the cloud.

Well on to specifics…

But I have a lot of reservations about using GitHub as our primary source control and code review platform. There's the free-as-in-freedom issue, of course,

Personally I think this ship sailed the day we used Google Apps for e-mail. :-)

but I'm also concerned about flexibility, account management, fragmentation of community and duplication of tools, and their terms of service.

== Flexibility == I see GitHub as kind of like a Mac.

This trope is too facile. But I do agree with what you are alluding to which is while it's fine for some, that doesn't mean it's fine for us. Especially us in our current development process.

It has a nice UI for the use case that its creators envision. It's fine for personal use.

A great many very large open source projects are currently using or hosted at GitHub (including node, jQuery, and our Android/PhoneGap app ;-))

And if we try it, everything'll be great.... until we smack into an invisible brick wall. We'll want to work around one little thing, the way that we sneak around various issues in Gerrit, with hacks and searches and upgrades, if it's not in GitHub's web UI or API [3], we'll be stuck.

The API is simplistic but serviceable. However, the satellite tools that are built around it are either part of GitHub (their internal issue tracker, their own Ruby-based wiki Gollum, etc) or are mostly for commercial use/cloud-based. For instance, in tools that assist in deployment from a GitHub repositiory (even if that was feasible for us which it isn't), most seem to have a hidden assumption that these are Web 2.0 companies deploying on AWS… not to mention that usage of those tools clearly violates our policy and values.

Right now we have our primary Git repo on our own machines, which is the ultimate backdoor. The way we have been modifying our tools, automating certain kinds of commits (like with l10n-bot), troubleshooting by looking at our logfiles, and generally customizing things to suit our weird needs -- GitHub is closed source and won't let us do that. We are not the typical use case for GitHub. Since we have hundreds of extensions, each with their own repository, we would have way more repositories and members than almost any other organization on there. So, one example: arbitrary sortability of lists of repositories. We could mod Gerrit to do it, but not GitHub. How would we centralize and list the repositories so they're easy to browse, search, edit, follow, and watch them together? It looks like GitHub's less suitable for that, but I'd welcome examples of orgs that create their own sub-GitHub hubs.

Well GitHubs modality doesn't prevent operating on the git repository through the API. But I agree since where is the support on our end for doing/writing these when we already have something servicable in Gerrit?

== Accounts == By using GitHub, we would no longer be managing the user accounts. This would make single sign-on with other Wikimedia services (especially Labs) completely impossible.

Technically this integration could be done by them authorizing us to their accounts via OAuth2. It's not the same thing as what you're saying though… it's kind of the opposite of what you're saying. What you want is what GitHub Enterprise is for.

I mentioned above that GitHub seems more meant for single FLOSS projects than for confederations of related repositories. GitHub does not have the concept of "groups," so granting access to collections of repos would be a time-consuming process. GitHub does not support branch-level permissions, either (it encourages "forking" and then merging back to master), and that does not seem as suitable for long-term collaborative branches.

This isn't quite true. GitHub does have the concept of groups (you can create as many as you want and control access levels (read-only, read/write, admin) on a project-by-project and between projects). However you cannot do it as robustly as Gerrit does.

More troublesome I think is not that GitHubs's forking-merge model handles permission, but that GitHub's model is fundamentally a different modality than our gated trunk code review model. GitHub effectively allows self-review because there is no concept of review.

On the other hand, since this is handled through a Pull Request instead of a Gerrit ChangeId, it does mean the history of the code commits, etc. doesn't get lost or munged down like it does in Gerrit. Too bad because I'd like this and it's fairly transparent (not requiring Git voodoo to handle these things). It's not our workflow though.

== Duplication of tools, fragmentation of community == We don't want to fragment our communication EVEN MORE. GitHub wikis and bug management aren't such a big deal since we can probably disable those. But messaging and notification .... "oh, did you say that on GitHub? We didn't see that there." That's already a big enough headache, with Bugzilla and all the mailing lists and IRC channels and talk pages and and and. :-)

GitHubs only collabs are the repository itself, Gollum (their wiki), their Issue tracker, and the commit comments.

Assuming that Gollum and Issue tracker are turned off (pity, I don't care for Gollum, but their Issue tracker is nicely integrated), the commit comments and repository are no different than what currently is a feature in Gerrit. Dare I say it, but GitHub's commit comments are awesome. They leave Gerrit and every other review tool in the dust as far as I've seen.

I should mention that Gerrit's actual review of a change is nicer than even Phabricator's. You can step through and it will mark them as reviewed as you go. Obviously GitHub has no such thing since it has no concept of a pre-commit review or gated review.

== The Terms of Service == GitHub's ToS/Security/Privacy policies[2] pose a few problems for our needs.

One is that people under 13 can't sign up. I do not want to limit our community that way.

Makes sense. I didn't consider this.

Another is: "You may not duplicate, copy, or reuse any portion of the HTML/CSS, Javascript, or visual design elements or concepts without express written permission from GitHub." Do we really want to get into a possible situation where we have noticed a design concept or cool use of JS on GitHub but don't feel okay reusing it in our personal or professional projects?

I think this is boilerplate. In any case, that part should apply even if we were mirroring on GitHub. Besides, since we have no inclination of building a GitHub competitor, who cares.

And, considering our level of activity, check out this clause: "If your bandwidth usage significantly exceeds the average bandwidth usage (as determined solely by GitHub) of other GitHub customers, we reserve the right to immediately disable your account or throttle your file hosting until you can reduce your bandwidth consumption." We simply cannot afford to have GitHub disable our access with no notice.

We still have the code and I'm sure there will be busier repositories out there. This is to prevent abuse on their side so they don't have to guarantee service.

== A couple open questions ==

• What's the FLOSS project on GitHub that's most like us, in terms of

size, number of unique repositories, privacy concerns, robustness needs, and so on? How are they dealing with these issues?

I don't know this. I believe when it migrates the jQuery plugin repository and jQuery itself will probably be the larger in terms of number of users and size. But they don't manage in a cascading web of trust.

• What does GitHub Enterprise buy us? Which of these issues would that fix?

It's a self-hosted GitHub. It would allow us to have private repositories (good for deploys, ops, etc.) and manage our own user database (we could integrate with our own auth system) and probably waives the 13 and under rule above.

The price is too steep since its a per-seat license. A nonstarter if the WMF is going to have to pay for every potential developer who wants to attach.

We do need a GitHub strategy -- to make our projects more discoverable, make use of more contributions, and participate in the GitHub reputational economy. So we must figure out the right ways to mirror and sync. But I doubt our own long-term needs would work well with using GitHub as our main platform.

I'm 1000% with you on this.

We should definitely at some point mirror our code in GitHub like the PHP project does http://www.php.net/git.php. Being able to publish and handle pull requests coming from GitHub would be a nice feature in Gerrit or any replacement. It'd be nice if others can have their own MW extensions or versions of extensions and core on GitHub and pull from us (and us from them) esp. for extensions that may need some love or have changes that don't satisfy the WMF code quality bar.

…

As for actually dealing with the pro/con of Gerrit vs. others-than-GitHub, I suppose I'll sit down and add matrix them on the wikipage if I have some time. I haven't yet thought things through enough to bore you with an even longer e-mail. :-)

As mentioned before, we can't use github enterprise at all, since it

doesn't allow for hosting public repos. Let's ignore that it even exists.

I feel like as Wikipedia is one of the top 10 most visited sites on the Internet we might be able to work out something special with them though right? I'm not saying we have to go down that route, nor have I even examined all the advantages and disadvantages of the idea. I feel though that the possibility exists though and should be looked into.

If I was GitHub and the WMF approached me about potentially using GitHub Enterprise for MediaWiki and MediaWiki extensions and NOT for creating a competition service to GitHub, then I would likely entertain the idea of crafting a special set of terms for them. Furthermore, I personally might even charge them differently given that that charging per developer would be crippling to an open-source project. Of course, I am not GitHub, nor can I anticipate what they might do, nor their internal policies, but I can speak for myself and how I would run a FOSS focused company.

Thank you, Derric Atzrott

On Jul 25, 2012, at 11:36 AM, Faidon Liambotis wrote:

think the BitKeeper story is relevant here

Yes, good point. Honestly before we talk GitHub or GitLab, we should consider if we are willing to rethink our model of handling code submissions to be more Pull-requesty. These two systems don't really have pre commit code review in the traditional sense (correct me if I'm wrong) and I don't think there is a way to bolt this on.

Le 25/07/12 13:10, Terry Chay a écrit :

...

...

• What does GitHub Enterprise buy us? Which of these issues

would that fix?

It's a self-hosted GitHub. It would allow us to have private repositories (good for deploys, ops, etc.) and manage our own user database (we could integrate with our own auth system) and probably waives the 13 and under rule above.

The price is too steep since its a per-seat license. A nonstarter if the WMF is going to have to pay for every potential developer who wants to attach.

The per-seat license scheme can usually be negotiated with the company, I am pretty sure any company will be glad to offer a huge discount or just provide an unlimited license instead of seeing a potential customer to move to a competitor. Unless said company just do not care about your small revenues / sector ..

I am pretty sure GitHub will be glad to host Wikimedia git repositories as an in-kind donation. Helping our project is definitely a nice asset to any company willing to build a nice reputation, that will provide them with a nice reference when prospecting new customer.

Le 26/07/12 00:45, aude a écrit :

I thought we used to have a mirror of MediaWiki on github but maybe that was when we were using SVN.

Indeed, we used to have one build by Ævar Arnfjörð Bjarmason from the svn repo. We got it deleted once we have build our own git repo.

Ævar kindly gave us access to the GitHub project page at: https://github.com/mediawiki (empty for now).

Cool, Priestley is awesome. If he comes to visit we should prevent him from leaving :)

+1. We should definitely think about adopting Phabricator as a project if we're going to invest in its core developer.

Look forward to having a less painful (and as Steven aptly puts it - "a happier") solution for our developers and contributors.

-Alolita

On Tue, Jul 24, 2012 at 10:30 PM, Ori Livneh ori@wikimedia.org wrote:

On Tuesday, July 24, 2012 at 10:26 PM, Erik Moeller wrote:

...

As one quick update, we're also in touch with Evan Priestley, who's no longer at Facebook and now running Phabricator as a dedicated open source project and potential business. If all goes well, Evan's going to come visit WMF sometime soon, which will be an opportunity to seriously explore whether Phabricator could be a viable long term alternative (it's probably not a near term one). Will post more details if this meeting materializes.

Cool, Priestley is awesome. If he comes to visit we should prevent him from leaving :)

-- Ori Livneh ori@wikimedia.org

---

Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

I think Ori's comment that touched this off was tongue-in-cheek. :-)

On Jul 25, 2012, at 12:40 AM, Roan Kattouw wrote:

On Tue, Jul 24, 2012 at 10:46 PM, Alolita Sharma alolita.sharma@gmail.com wrote:

...

Cool, Priestley is awesome. If he comes to visit we should prevent him from leaving :)

+1. We should definitely think about adopting Phabricator as a project if we're going to invest in its core developer.

I think this should be reversed. If we're gonna use Phabricator, we should attract its core developer. We shouldn't switch code review systems just because we hired a famous person.

I personally think that something like gitlabs would be superior to Phabricator, but that's another discussion.

Roan

---

Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

On Tue, Jul 24, 2012 at 10:26 PM, Erik Moeller erik@wikimedia.org wrote:

As one quick update, we're also in touch with Evan Priestley, who's no longer at Facebook and now running Phabricator as a dedicated open source project and potential business. If all goes well, Evan's going to come visit WMF sometime soon, which will be an opportunity to seriously explore whether Phabricator could be a viable long term alternative (it's probably not a near term one). Will post more details if this meeting materializes.

Evan pointed me to the following pages which give some hints at Phabricator's longer term direction (requires logging in):

Project roadmap: https://secure.phabricator.com/w/roadmap/ Plans for repo/project level permission management: https://secure.phabricator.com/T603

In prep for this, I started a section about Phabricator as an alternative.

https://www.mediawiki.org/w/index.php?title=Git%2FGerrit_evaluation&diff...

Steven