On Tue, Mar 22, 2011 at 08:27, Yuvi Panda <yuvipanda(a)gmail.com> wrote: Lack of time and people is indeed a factor. The import we have now isn't a proper Git conversion. I still have some vague notes here detailing approximately what we need, some of these are out of date. The "Split up and convert" section is somewhat accurate though: http://www.mediawiki.org/wiki/Git_conversion No SVN to Git tool does exactly what we need due to our messy history. I came to the conclusion that it was probably easiest to filter the SVN dump (to e.g. fix up branch paths) before feeding the history to one of these tools. Of course even if we come up with a perfect conversion it's pretty much useless if Wikimedia doesn't want to use it for its main repositories. So getting a yes/no on whether this is wanted by WM before you proceed with something would prevent you/others from wasting their time on this.

On Tue, Mar 22, 2011 at 10:25 AM, Siebrand Mazeland <s.mazeland(a)xs4all.nl> wrote: I've actually come to partially agree with you since the last time we discussed this. Really, the extension repository *should* remain in Subversion as it is. I would, however, like to move phase3 to git. Then i18n can just be two commits, instead of 400+ -Chad

On Tue, Mar 22, 2011 at 10:44 AM, Roan Kattouw <roan.kattouw(a)gmail.com> wrote: Perhaps in the long run. I think in the short-run it'd be more pain-free and perhaps a useful experiment to just move phase3 to git. Then we can see how we feel about moving the rest over (or if we hate it and want to go back) -Chad

Your objections seem to be based on the assumption that you would need to have push access to all repositories, but I think that's the point of DCVS, you can just fork them, and then people can pull your changes in themselves (or using a tool). Pull requests could even be generated when things are out of sync. I think it's quite possible this could make i18n/L10n work easier, not more difficult. - Trevor On Mar 22, 2011, at 7:25 AM, Siebrand Mazeland wrote:

On 22.03.2011, 18:08 Trevor wrote: You seem to miss Siebrand's point: curerently, all localisation updates take one commit per day. Splitting stuff to separate repos will result in up to 400 commits per day that will also need to be pushed and reintegrated - an epic waste of time and common sense. Or localisation will simply lie aside in forks and people will miss them when checking out from the "official" source. -- Best regards, Max Semenik ([[User:MaxSem]])

On 22-03-11 16:38 Trevor Parscal <tparscal(a)wikimedia.org> wrote: IMO that a bridge too far. My question is "Why should we make this happen?", and more specifically, what do our various stakeholders (which groups?) gain or lose in case MediaWiki development would shift from Subversion to Git? Only if the gain in the analysis would be greater than the loss, it makes sense to me look look further into a move to Git. On 22-03-11 16:08 Trevor Parscal <tparscal(a)wikimedia.org> wrote: Yes, sir, indeed. Getting the L10n updates (as well as the i18n updates) into the code as soon as possible is of paramount importance to the success MediaWiki has in its i18n and L10n efforts. Having to wait until possibly active repo maintainers pull updates is unacceptable. This would kill translator motivation, and take us back years. Siebrand

"Chad" <innocentkiller(a)gmail.com> wrote in message news:AANLkTikrRE_3O+pYCjdX2+QiL6zT3tPohQuCodwE_SrU@mail.gmail.com... To my mind, this is one of the most important points. We have built up a very comprehensive infrastructure for code review in SVN, and there is a lot of manhours behind that work; and just as many hours associated with setting up a replacement system for git. Who is going to put that infrastructure in place, in amongst all the many other priorities we have at the moment? Moving to a VCS which makes it "easier to review stuff" **in principle** is going to be of no use whatsoever if it sends the **practical implementation** of that review process back to the stone age. --HM

On 23/03/11 12:05, Rob Lanphier wrote: What proportion of a reviewer's time do you suppose is spent battling with Subversion? I thought most of it was just spent reading code. If you want someone to dig a hole faster, you don't buy them a nicer-looking shovel. I think we have to look at the benefits of Git carefully, and to weigh it against the costs, both of conversion and ongoing. I think our focus at the moment should be on deployment of extensions and core features from the 1.17 branch to Wikimedia. We have heard on several occasions that it is the delay between code commit and deployment, and the difficulty in getting things deployed, which is disheartening for developers who come to us from the Wikimedia community. I'm not so concerned about the backlog of trunk reviews. We cleared it before, so we can clear it again. -- Tim Starling

2011/3/23 Tim Starling <tstarling(a)wikimedia.org>rg>: This post has convinced me that planning a move to Git at this time would be premature. I still believe it would be better than SVN, but as Tim points out we have much more serious issues to address first. I disagree, however, that the backlog of trunk reviews is not concerning. It means that we still haven't come up with a good process for reliable and quick (as in time between commit and review) code review. I have briefly told the list about my ideas for this in the past, maybe I should revive that thread some time. I also believe that, once we have a process where every commit is reviewed within a reasonable timespan (ideally at most a week), getting deployment closer to trunk and getting it to stay there will actually be fairly easy to do. We used to have such a process once: a few years ago, Brion used to spend every Monday catching up on code review, reverting broken things, and deploying something resembling HEAD. At some point this broke down due to Brion's lack of scalability, and in the years that have passed our SVN activity has grown to a level where I don't believe code review is a one-person, one-day-a-week job any more. But that's not necessarily a problem: we can scale it by adding people and giving them the time and management they need. But that wasn't really ever done in a permanent fashion. Roan Kattouw (Catrope) P.S.: The final paragraph is not meant to suggest that Brion was the one and only code reviewer back in those days. Other people also reviewed code, and I don't mean to marginalize their contributions, I just wanted to point out that Brion was the driving force behind regular review and deployment happening.

2011/3/23 Marcin Cieslak <saper(a)saper.info>fo>: I was personally going for something more like assigning reviewers to MW components, paths, time slots, or some combination thereof. But if we're gonna have that discussion for real, let's have it in its on thread :) Roan Kattouw (Catrope)

On 11-03-23 06:36 AM, Marcin Cieslak wrote: I'd actually like to eliminate legacy skins altogether. They show up and throw a thorn into skin improvements repeatedly. Perhaps after coming up with some replacements that fit the requirements of the people that actually use them. I haven't got any statistics yet on how many active users are actually using these skins. The fact that we'd discuss a bug like that not because CologneBlue and the other legacy skins have a unique feature or layout they were created to offer and we're trying to improve it, but because CologneBlue and the other legacy skins lay out the ui in a pre-SkinTemplate era way of layout we're maintaining and trying to backport modern improvements into is a good example of why I want to get rid of them. It's like trying to backport modern features to REL1_11. ~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://daniel.friesen.name]

On 3/22/11 6:05 PM, Rob Lanphier wrote: At the risk of being impolite -- our code review tool is not that nice. (I don't expect that anyone who worked on it would even disagree with me here.) It happens to be our home grown tool, and it uses a framework that more of us are familiar with. But it's not such an overwhelming asset that we should consider staying on SVN because of it. In 2011 there are lots of code review frameworks out there to choose from. I have to confess this is my main interest in at least re-examining our source control situation. Git doesn't necessarily make pre-commit code review easier, but as a side effect it will allow us to consider other options. If you don't believe me about pre-commit code review, ask any of your friends who work for (or who have worked for) Google. Even people who were very skeptical will usually say that it is has been a huge benefit. -- Neil Kandalgaonkar (| <neilk(a)wikimedia.org>

On 3/23/11 2:13 PM, Robert Leverington wrote: Perhaps I was unclear. My point was that I personally would like to *not* use our current code review system. By that I mean both the general paradigm we have, and the tools we are using. I'm asserting that: - There are other software frameworks for code review with more and better features. - There are other code review paradigms that are better for team health, other than "let's spend several months on the backlog every time we want to release". -- Neil Kandalgaonkar (| <neilk(a)wikimedia.org>

On Thu, Mar 24, 2011 at 7:07 AM, Neil Kandalgaonkar &lt;neilk(a)wikimedia.org&gt; wrote: I don't believe anyone said it had to a home grown solution, but rather, that we needed a soultion that work before any transfer could really take place, Perhaps some of theses CR frameworks could be listed on the git pages on wiki. On Thu, Mar 24, 2011 at 7:13 AM, Robert Leverington &lt;robert(a)rhl.me.uk&gt; wrote: There is a bug for that already in bugzilla! On Thu, Mar 24, 2011 at 7:19 AM, Neil Kandalgaonkar &lt;neilk(a)wikimedia.org&gt; wrote: I'm sure that discussion is always welcome, but perhaps that should be discussed in a non SVN vs <insert (D)VCS of choice> thread. No matter what, we will always end up with a backlog of unreviewed code because so few people people that actually spend their time doing it unless they are dragged over from their projects to do it.

The only one I know and like is Gerrit. -Chad On Mar 23, 2011 8:43 PM, "Platonides" &lt;Platonides(a)gmail.com&gt; wrote:

On Fri, Mar 25, 2011 at 11:56 PM, Mark A. Hershberger &lt;mhershberger(a)wikimedia.org&gt; wrote: I see no reason to grandfather in current code. TBH, the list of fixmes is appalling and we should make a sprint at saying "fix this or it'll be reverted in 24 hours" for every last one of them. -Chad

On 26/03/11 05:48, Daniel Friesen wrote: If it is unclear: it either need a clarification or deserve a reversion. We already have enough lines hiding in the fog, read to jump at you when you get out of the path. Do we even have fixmes for the last two cases? Anyway for tests, they might be required just to make sure other developers using the feature will use it as intended. There are always funny corner cases to handle, specially with PHP. -- Ashar Voultoiz

Bryan Tong Minh wrote: $ time make noparser Tests: 823, Assertions: 9512, Failures: 8, Incomplete: 42, Skipped: 3. make: *** [noparser] Error 1 real 0m45.697s user 0m10.389s sys 0m1.523s I have mysql tmpdir set to a tmpfs filesystem (mysql doesn't support in-memory tables with BLOBs). Using a different hardware, a cold cache and creating the temporary tables on disk, it may take a few minutes, but not an hour. On the other hand, running phpunit parser tests can take that long. Whereas the good old parserTests.php takes ~44s, too. All the other time is db overhead droping and duplicating tables, inserting articles and waiting for the db answer. I tested performing a new mysql connection instead of dropping each table separatedly, but it was slower. A change that could improve perfomance would be to insert everything on a main temporary table, and clone that with its content for each parser test. Or we could try to remove the db dependency altogether for parser tests.

Roan Kattouw wrote: You mentioned reverting broken code. Mark proposes reverting *unreviewed* code. We are generally polite by marking fixme the code from others, and avoiding reverting as much as possible. I agree with the proposal of reverting after a few days with an "important fixme". But reverting new revisions because noone reviewed it, seems going too far (at least at this moment). It would make much more sense to draft some process where you have to review the previous revision of the files you are changing. However, that would forbid fast fixes (eg. fixing the whitespace of the previous commit) without fully reviewing it, which is also undesirable (the revision keeps unreviewed, and with the wrong whitespace).

On Fri, Mar 25, 2011 at 8:56 PM, Mark A. Hershberger < mhershberger(a)wikimedia.org&gt; wrote: tl;dr summary: The biggest single improvement that can be made is to *ship code faster*. When new code comes in, there's basically a few things it can do: * break in an obvious and visible way * break in some circumstances, which might not be obvious on first look * mostly work, but be inefficient or have other negative side effects that need fixing * mostly work, but cause HORRIBLE DATA CORRUPTION that's not noticed for some time * work pretty well Because we're afraid of letting hard-to-find bugs go through, we're holding *everything* back for too long in the hopes that we'll somehow develop the ability to find hard-to-find bugs easily. As a result, the code paths don't get exercised until a giant last-minute review-and-push comes through months later, and finding the actual source of the bugs becomes even *more* difficult because you have 6 months of changes to search all at once instead of a few days. Ship sooner -> fail faster -> fix quickly. Update the live site no less frequently than weekly; update test sites more frequently than that. Make sure those test sites include things that developers are actually dogfooding. Encourage other testers to run on trunk and report issues. A smaller, but still relevant issue is to see if we can change how we think about review priorities: something that changes the core of the parser or how pages get saves might well cause HORRIBLE DATA CORRUPTION, but changes in UI code probably won't. Changes in UI code might cause an XSS vulnerability, however... so when thinking about how much attention code needs, we should be considering the module rather than laying down blanket policies. Some more explicit reviewer module 'ownership' could indeed be helpful -- especially if we have a more explicit review process for 'big changes', but even for less formal review. As far as I can see, the main reason that people think code review That isn't specific to git; the same methodology works in SVN or CVS or whatever where you're reviewing patches submitted through email, bug tracker systems, etc. The advantage git has here is that your intermediate work is easier to keep and share within the revision control system, as opposed to having to keep your work *outside* the version control system until it's been approved by someone else. IMO that's a big advantage, but you can still do review-first with SVN, and we always have for patches submitted through bugzilla or the mailing list by non-committers. If review and application of submitted patches can be made consistent and reasonably speedy, that would again be a big improvement without requiring a toolset change: getting more good stuff through, with no danger of it breaking things _before_ approval & merging. So, while Subversion doesn't support review-first, it can incorporate This is actually a lot harder than it might sound; even in only a week, trimming out dependency on dependency on dependency can be extremely difficult, especially if some change involved lots of giant whitespace cleanup or variable renames or other things that play hell with patch resolution. Reverting generically questionable code should probably happen a lot faster than after a week. -- brion

On 26/03/11 14:56, Mark A. Hershberger wrote: It's no mystery. After the 1.17 deployment, the team that was doing code review was disassembled. If you want code review to happen faster, then getting people to work on it would be a good start. Find someone to review it? If the experienced developers on the WMF payroll aren't assigned to code review, then under your proposal, the only option for avoiding a revert will be to get someone with no clue about anything to rubber-stamp the code. However, volunteer developers aren't always the most capable people at navigating bureaucracy. In practice, a lot of people would commit code, have it reverted, and leave. If the code review manpower is there, we can be friendly and encouraging to our developers, not threaten them with a revert unless they can make at least one developer be their friend within seven days. The WMF really is central in this, because we have a policy of hiring as many experienced developers as possible from the volunteer community. So that is where the expertise is concentrated. By definition, our volunteer developers have lives outside of MediaWiki. We have to fit in with their schedules. I don't think we should give them a kick in the teeth just because they committed something on Sunday and have to go to school on Monday. If a commit is insecure, or changes interfaces in a way that will be disruptive to other developers, or breaks key functionality, then sure, we should revert it right away. There's no need to wait 24 hours. But I don't think we need to be issuing death sentences for typos in comments. A "fixme" status just means that there is something wrong with the commit, however minor, it doesn't mean that any urgent action is required. Your proposal seems to be based on the idea that review under Git is many times better than review with CodeReview and Subversion. I don't think that's true, I think it's very slightly better. Whether you use Git or Subversion, you still need people with brains reading code. -- Tim Starling

Tim Starling wrote: You're one of the most senior developers and you're a Wikimedia employee, yet some of your writing comes off as though you're on the outside. Yes, Wikimedia needs to devote more manpower to code review. This has been pretty apparent for quite some time. The central question at this point seems to be: what's the hold-up? Long ago I lost track of who's in charge of what, but I'm told there is some sort of hierarchy in place in the tech department. Who's empowered to start assigning people to review code in a reasonable timeframe? Like Aryeh, I find this entire thread a bit baffling. MZMcBride

Tim Starling wrote: I agree. But that only works for extensions since: * They are self-contained * They are relatively small * They are not deployment blockers And still they are harder to fix months later when the author has moved on (think in the poolcounterd bug). I don't think that would work as well for core MediaWiki, albeit it may be feasible for not-so-big features with a kill switch. Large commits changing many files would need a branch to be reviewable in a set. However, our problem with branches is that it removes almost all peer review and testing, and merges are likely to introduce subtle bugs. The late review drawbacks are also there. How do you propose to fix it? The committer deferring its own revision? It may be worth making a list of review requests at mediawiki.

On Fri, Mar 25, 2011 at 11:56 PM, Mark A. Hershberger &lt;mhershberger(a)wikimedia.org&gt; wrote: I don't think that's the right way to put it. In a properly-functioning review-then-commit system, it should be easy to get code reviewed. The advantage of reviewing the code first is that psychologically, it's much easier to say "Fix these minor things and then I'll approve it" than to say "Fix these minor things or else I'll revert it". The first gives positive incentives, while the second gives negative incentives, and people appreciate positive incentives a lot more. In a review-first system, you're going to routinely have reviewers asking that the patch author write better comments or conform to style guidelines or simplify the logic a bit before they give approval -- or even restructure the changes entirely. In a commit-first system, reviewers are going to be reluctant to revert code that works, even if it has some minor deficiencies, so committers have little incentive to fix minor code issues. Code quality suffers as a result. This is a terrible idea. Review needs to be something that everyone is guaranteed to get without effort on their part. You cannot design a code review system on the theory that code authors are supposed to somehow get their code reviewed when no one is formally required or expected to review it. I'm all for giving people incentives to do the right thing, but incentives are pointless if the person being incentivized has no way to do what you're trying to get them to do. Incentives have to be placed on the code *reviewers*, because they're the only ones who can decide to review a given patch. Conveniently, almost all the code reviewers happen to be employed by Wikimedia, so the incentive can be the good old conventional "your boss told you to". If, as Tim says, Wikimedia developers were un-assigned from code review after the 1.17 deployment, *that* is the problem that needs to be fixed. We need a managerial decision that all relatively experienced developers employed by Wikimedia need to set aside their other work to do as much code review as necessary to keep current. If commits are not, as a general rule, consistently reviewed within two or three days, the system is broken. I don't know why this isn't clear to everyone yet.

On 24/03/11 06:47, MZMcBride wrote: I have answered to this message in a new one to create a new thread. -- Ashar Voultoiz

On 23/03/11 04:24, Rob Lanphier wrote: The article seems quite biased. "Here’s the part where you’re just going to have to take my word for it. "Mercurial is better than Subversion." The tone is quite different to one of the first things I read about Mercurial: "Oops! Mercurial cut off your arm! "Don't randomly try stuff to see if it'll magically fix it. Remember what you stand to lose, and set down the chainsaw while you still have one good arm." https://developer.mozilla.org/en/Mercurial_basics The main argument is that merging is easy so you can branch without the slightest worry. I think this is an exaggeration. Interfaces change, and when they change, developers change all the references to those interfaces in the code which they can see in their working copy. The greater the time difference in the branch points, the more likely it is that your new code will stop working. As the branch point gap grows, merging becomes more a task of understanding the interface changes and rewriting the code, than just repeating the edits and copying in the new code. I'm not talking about the interfaces between core and extensions, which are reasonably stable. I'm mainly talking mainly about the interfaces which operate within and between core modules. These change all the time. The problem of changing interfaces is most severe when developers are working on different features within the same region of core code. Doing regular reintegration merges from trunk to development branches doesn't help, it just means that you get the interface changes one at a time, instead of in batches. Having a short path to trunk means that the maximum amount of code is visible to the developers who are doing the interface changes, so it avoids the duplication of effort that occurs when branch maintainers have to understand and account for every interface change that comes through. If we split up the extensions directory, each extension having its own repository, then this will discourage developers from updating the extensions in bulk. This affects both interface changes and general code maintenance. I'm sure translatewiki.net can set up a script to do the necessary 400 commits per day, but I'm not sure if every developer who wants to fix unused variables or change a core/extension interface will want to do the same. I don't know enough about Git to know if these things are really an argument against it. This is just a comment on the ideas in this thread. -- Tim Starling

On Tue, Mar 22, 2011 at 7:46 PM, Tim Starling &lt;tstarling(a)wikimedia.org&gt; wrote: That doesn't mean it's wrong, and no one implied it was objective. Those quotes apply to any version control system, or for that matter, any system (randomly trying stuff and praying for magic rarely seems like good advice). The main guidance that relates to SVN vs Mercurial was the fact that Mercurial doesn't leave conflict markers. Git leaves conflict markers just like Subversion. There is also a couple of bits of guidance about Mercurial Queues, which is a very popular add-on to Mercurial that behaves in ways that are pretty specific to Mercurial (and is a chainsaw and a handgun all in one). The Git equivalent is Quilt, but most Git users don't use Quilt, because Git has some core functionality (rebase, ability to delete branches) that makes such an add-on less interesting. Git rebase is a beast of its own, and there are many arguments pro and con about its use: Pro: http://darwinweb.net/articles/the-case-for-git-rebase Con: http://changelog.complete.org/archives/586-rebase-considered-harmful Fervent Mercurial advocates are also a fine source of more "con" material for the git rebase option. We will probably need to adopt some guidelines about the use of rebase assuming we move to Git. Yes, merging is hard. Subversion is particularly bad at it. Git and Mercurial are both much, much better. That doesn't mean they are flawless, but they tend to work pretty well. I get that there are many unsupported, volatile interfaces between and within components. However, if we're changing things around so much that it really negates the benefit that Git or some other DVCS brings us, then that's another conversation. We can still have the rough equivalent of the big happy trunk where everything goes in prior to review. On the flip side, I'm not sure if every developer who wants to make a clone of a single extension on Github or Gitorious wants to use up their quota getting the source of every single extension. Being able to have public clone for pushing/pulling on a hosting service is a large benefit of DVCS for our workflow; it means that no one even has to ask us for permission before effectively operating as a normal contributor. It would be unfortunate if we set up our repository in such a way as to deter people from doing this. We don't need to switch to the one extension per repo model right away, though. We could throw all of the extensions into a single repository at first, and then split it later if we run into this or other similar problems. I'm not so hellbent on deploying Git that I would push a hasty deployment over reasonable, unmitigated objections. It's fair to wait until some time after we're done with 1.17, and wait until we've figured out how this will work with Translatewiki. Rob

Daniel Friesen wrote: On which OS? Seems easier to keep home at the same version as the server, rsync from server to home, and then update before committing. (...) Good point.

On Tue, Mar 22, 2011 at 10:46 PM, Tim Starling &lt;tstarling(a)wikimedia.org&gt; wrote: My experience with Mercurial is that if you type the wrong commands, it likes to destroy data. For instance, when doing an hg up with conflicts once, it opened up some kind of three-way diff in vim that I had no idea how to use, and so I exited. This resulted in my working copy (or parts of it) being lost, since apparently it defaulted to assuming that I was okay with whatever default merging it had done, so it threw out the rest. I also once lost commits under similar circumstances when doing hg rebase. I'm pretty sure you can configure it to be safer, but it's one of the major reasons I dislike Mercurial. (I was able to recover my lost data from filesystem backups.) git, on the other hand, never destroys committed data. Barring bugs (which I don't recall ever running into), the only command that destroys data is git gc, and that normally only destroys things that have been disconnected for a number of days. If you do a rebase, for instance, the old commits are no longer accessible from normal commands like "git log", but they'll stick around for some period of time, so you can recover them if needed (although the process is a bit arcane if you don't know the commit id's). There are also no git commands I've run into that will do anything nasty to your working copy without asking you, except obvious ones like git reset --hard. In the event of update conflicts, for instance, git adds conflict markers just like Subversion. In practice, this is generally not true. Realistically, most patches change a relatively small amount of code and don't cause merge conflicts even if you keep them out of trunk for quite a long time. For instance, I maintain dozens of patches to the proprietary forum software vBulletin for the website I run. I store them all in git, and to upgrade I do a git rebase. Even on a major version upgrade, I only have to update a few of the patches, and the updates are small and can be done mindlessly. It's really very little effort. Even a commit that touches a huge amount of code (like my conversion of named entity references to numeric) will only conflict with a small percentage of patches. Of course, you have to be more careful with changing interfaces around when people use branches a lot. But in practice, you spend very little of your time resolving merge conflicts, relative to doing actual development work. It's not a significant disadvantage in practice. Experienced Subversion users just expect it to be, since merging in Subversion is horrible and they assume that's how it has to be. (Disclaimer: merges in Subversion are evidently so horrible that I never actually learned how to do them, so I can't give a good breakdown of why exactly DVCS merging is so much better. I can just say that I've never found it to be a problem at all while using a DVCS, but everyone complains about it with Subversion.) I mean, the DVCS model was popularized by the Linux kernel. It's hard to think of individual codebases that large, or with that much developer activity. In recent years it's over 9,000 commits per release changing several hundred thousand lines of code, which works out to several thousand LOC changed a day. But merging is not a big problem for them -- they spend their time doing development, not wrestling with version control. I've thought about this a bit. We want bulk code changes to extensions to be easy, but it would also be nice if it were easier to host extensions "officially" to get translations, distribution, and help from established developers. We also don't want anyone to have to check out all extensions just to get at trunk. Localization, on the other hand, is entirely separate from development, and has very different needs -- it doesn't need code review, and someone looking at the revision history for the whole repository doesn't want to see localization updates. (Especially in extensions, where often you have to scroll through pages of l10n updates to get to the code changes.) Unfortunately, git's submodule feature is pretty crippled. It basically works like SVN externals, as I understand it: the larger repository just has markers saying where the submodules are, but their actual history is entirely separate. We could probably write a script to commit changes to all extensions at once, but it's certainly a less ideal solution. If we moved to git, I'd tentatively say something like * Separate out the version control of localization entirely. Translations are already coordinated centrally on translatewiki.net, where the wiki itself maintains all the actual history and permissions, so the SVN checkin right now is really a needless formality that keeps translations less up-to-date and spams revision logs. Keep the English messages with the code in git, and have the other messages available for checkout in a different format via our own script. This checkout should always grab the latest translatewiki.net messages, without the need for periodic commits. (I assume translatewiki.net already does automatic syntax checks and so on.) Of course, the tarballs would package all languages. * Keep the core code in one repository, each extension in a separate repository, and have an additional repository with all of them as submodules. Or maybe have extensions all be submodules of core (you can check out only a subset of submodules if you want). * Developers who want to make mass changes to extensions are probably already doing them by script (at least I always do), so something like "for EXTENSION in extensions/*; do cd $EXTENSION; git commit -a -m 'Boilerplate message'; cd ..; done" shouldn't be an exceptional burden. If it comes up often enough, we can write a script to help out. * We should take the opportunity to liberalize our policies for extension hosting. Anyone should be able to add an extension, and get commit access only to that extension. MediaWiki developers would get commit access to all hosted extensions, and hooking into our localization system should be as simple as making sure you have a properly-formatted ExtensionName.i18n.php file. If any human involvement is needed, it should only be basic sanity checks. * Code review should migrate to an off-the-shelf tool like Gerrit. I don't think it's a good idea at all for us to reinvent the code-review wheel. To date we've done it poorly. This is all assuming that we retain our current basic development model, namely commit-then-review with a centrally-controlled group of people with commit access. One step at a time. On Tue, Mar 22, 2011 at 11:16 PM, Tim Starling &lt;tstarling(a)wikimedia.org&gt; wrote: I don't think moving to git will make code review very much easier in the short term. It would probably disrupt code review considerably, in fact, because people would have to get used to the new system. So I definitely think code review needs to be worked out before we overhaul anything. And that doesn't mean clearing out backlogs, it means not letting them accumulate in the first place -- like scaps once a month at the very minimum, and preferably at least once a week. On Wed, Mar 23, 2011 at 2:51 PM, Diederik van Liere &lt;dvanliere(a)gmail.com&gt; wrote: They gave three reasons: 1) git's Windows support isn't as good as Mercurial's. I don't know how much merit that has these days, so it bears investigation. I have the impression that the majority of MediaWiki developers use non-Windows platforms for development, so as long as it works well enough, I don't know if this should be a big deal. 2) Python developers preferred Mercurial when surveyed. Informally, I'm pretty certain that most MediaWiki developers with a preference prefer git. 3) Mercurial is written in Python, and Python developers want to use stuff written in Python. Not really relevant to us, even those of us who like Python a lot. :) (FWIW, despite being a big Python fan, I'm a bit perturbed that Mercurial often prints out a Python stack trace when it dies instead of a proper error message . . .) GNOME also surveyed available options, and they decided to go with git: <http://blogs.gnome.org/newren/2009/01/03/gnome-dvcs-survey-results/> Although of course, (1) would be a bit of a nonissue for them. On Wed, Mar 23, 2011 at 3:41 PM, Rob Lanphier &lt;robla(a)wikimedia.org&gt; wrote: I don't see why. Rebase can never be used on publicly-visible repositories -- anyone who tries to pull from the repo both before and after the rebase will get errors, since the current upstream HEAD is not a descendant of the old upstream HEAD. So rebasing is only relevant to what developers do in their own private branches, before they push them to the central public repository. What we'd need is policies on *merging*. Do we encourage people to submit clean merges with an empty merge commit so the development history is preserved, or encourage rebasing so that the development history is linear and easier to analyze (e.g., bisect)? Whatever policies we adopt, people can always rebase in their private repos as much as they want, if they like. I guess we could discourage it, but I don't see why, as long as it doesn't cause bugs. No we can't. Any clone of the repository will have all history. If you want to split out extensions at a later date, you're not going to save much space, since they'll still be cloned with all the rest of the history. To really get rid of them, you'd have to create a whole new repository, forcing everyone to do a fresh clone and seriously hampering git's ability to merge any uncommitted work from before you broke up the repo. If we want to split off some things into their own repos, the time to do that is when we switch to git, not afterward.

On Fri, Mar 25, 2011 at 2:12 AM, Aryeh Gregor &lt;Simetrical+wikilist(a)gmail.com&gt; wrote: I find this, together with Roan's mail in the other thread on how to actually make CodeReview easier, a very sensible idea. We should first fix our "management" problems (i.e. getting code review done) before we fix our technical problems (making branching/merging suck less). Bryan

On Tue, Mar 22, 2011 at 16:33, Max Semenik &lt;maxsem.wiki(a)gmail.com&gt; wrote: I think you're missing the point that there's no reason why 400 commits should be harder than 1 in this case. When he makes a commit now he ends up stat()-ing 400 files, but he doesn't notice because it's all abstracted away. Similarly he could make 400 commits by issuing one command, just like he does today. And what does "pushed and reintegrated" mean? He'd presumably push to the canonical upstream, just like he does now. (Or he could push somewhere else if people would like that, pulling from that should also be trivial).

On 22 March 2011 18:00, Ævar Arnfjörð Bjarmason &lt;avarab(a)gmail.com&gt; wrote: Thanks to Ævar I realised that we're all missing the point and assuming things which are never spoken out. These silent assumptions must be spoken out to get everyone on the same line. For twn it's not the commits which is hard, it's all the other things: 1) picking up new projects when they are committed (Mediawiki-svn list) - add few lines to our config file - initial import of messages 2) following changes (Mediawiki-svn list) - import new/changed - run fuzzy if needed - bully developers if needed (Code review, IRC) Now, when we move to git how do we keep the workflow as simple as it is now? Where will the repos be? Will they all share user accounts? Will everyone be able to commit everywhere? How is the standard repo location & file layout enforced? What will we offer to people who check out all extensions from svn and want to update them all in one command? What about only a subset of extensions (extensions used in twn)? What about only the checkout of i18n files (extensions translated in twn)? How do we know when repos are created or deleted? How do we know which repos are the official upstreams and not just clones of extensions developed elsewhere? What will replace Mediawiki-commits list and code review? It's really hard to say what could be the real issues when there is no proposal how it would actually work. For example I don't think what avar said to me on IRC (below) has been stated here before, while I find it essential to judge the whole idea. avar> We're not going to move from a "central" SVN server to Git repositories scattered all over the place, just Git repositories hosted on some WM server. avar> So all the ssh keys etc. would already be set up, and getting a list of repos would be no harder than getting a list of extension dirs today. What other unspoken assumptions are there? -Niklas -- Niklas Laxström

Daniel Friesen wrote: I realised today that we are trying to adapt our layout to the DCVS. It should be the tool the one which should adapt to our own working. It is a good time to split in repos if we want to, but we shouldn't split folders just because a version system does not support it. We should file it as a bug upstream and wait for any of the several alternatives to fix it. Have you seen Ryan Lane work? The users could add their ssh key and automatically get commit access to its own extension with that. Google has some scripts like that for dealing with multiple repositories (and multiple vcs).

Toolserver has a social problem, not a technological one. They have the ability to use SVN, or a source control system of their choosing, yet they don't. This thread is discussing a perceived problem with a tool we are already successfully using. Let's focus on one issue at a time. - Ryan Lane

2011/3/22 Gerard Meijssen &lt;gerard.meijssen(a)gmail.com&gt;om>: There seems to be this misconception with the TranslateWiki people that using Git automatically means that integrating localization updates will somehow be more difficult. This is not the case. In fact, it won't even be appreciably different from the way we currently handle localization updates in SVN. While Git *supports* the workflow of pushing freshly written code to some branch and merging it into trunk after it has been reviewed, it does not *require* it. It's trivial to set things up so that TWN can commit and push their localization updates directly into trunk, without a need for review or merging or whatever. Having LocalisationUpdate continue to pull those updates from trunk and apply them to the live site is equally trivial. The only thing that may be different, depending on what our workflow ends up being, is that messages that have been added in some branch that hasn't been merged to trunk yet will not automatically be picked up by TWN for translation. This is technically already the case, but branches aren't very common at this time and will likely become more common with Git. If we end up with a first-review-then-merge-to-trunk workflow, messages wouldn't be available for translation on TWN until after the commits that introduced them have been reviewed and merged to trunk, so TWN will be behind the curve a little bit. But I'm not convinced that's necessarily bad: it'll hopefully prevent poorly organized or poorly translatable messages from making their way into trunk, thereby making sure translators never even see those. Roan Kattouw (Catrope)

On 22/03/11 20:26, Brion Vibber wrote: Tools are almost never an issue. We could as well use Bugzilla as a patch/review queue and have ONE release manager to apply them in a CVS tree. I have seen developers using MS Word to track code, they even managed to merge their code this way. The issue we have is a proper workflow and, I believe, the lack of a roadmap. Do we really need all the language messages in core? They are probably "useless" for day to day code hacking. Most developers probably use the English messages only anyway. The only things we have to do is to fill up the Messages.inc metadata file, and the English message. Nowadays, I do not even bother to translate my own messages to my native language (which is French). So, to me, messages serves no need for the developers. They are only useful for live site and MediaWiki releases. For the live site, we could just pull messages from whatever system is used (gettext .po, rosetta, git, ftp site). This can be done every week, day or hour. Then comes the need to release a MediaWiki release. This mean you have to sync both projects (code + l10n) and this is not possible while you keep having new messages added in MediaWiki or messages parameters being changed. That is what I meant by a workflow. We have actors, tasks and responsibilities and eventually end up with a product. I have said it already: we need a rough roadmap to release a new version. One of the first steps would be: - obviously 'no new features' step . Followed by... - 'no new UI messages' - 'no parameter changes' - 'no messages changes' This will give time to translators to finish up the messages translations for the release. The lack of roadmap for 1.18 is probably the same cause that makes our code review queue filled again. It looks a bit like a wiki sandbox, some code (myself included) should never have been sent to trunk. To conclude, the good point, is that we have 1.17 feature frozen, deployed on live site and it might even get released before this summer. Given the situation back in September 2010: this is a huge accomplishment :-) -- Ashar "hashar" Voultoiz

Roan Kattouw wrote: What workflow would we be using? It's fine that git supports many ways to be used but the important one is how would mw be used, and probably each one is thinking that it would be used in a slightly different way. I'm not a proficient git user. I see benefits from a DVCS. Automatically merging/umerging a revision to 1.17/wmf with a checkbox at CR would be cool, for instance. Stacking several commits before one push also looks nice. How would that be presented to the reviewer? As a single diff, as several ones? Note that this could already be done with svn (if they are easy diffs). Even with some kind of review-before-merge approach we would still need a trunk from which to work usually, and the stable reviewed branch would advance behind it. (Do git patch queue work like that?) Backing out changesets would be easier, and they could be continued in a forked branch, but I'm not sure that looks nice. Robla says that it will mean less reviewing problems. It may decrease them, but there will still be "errors noticed just after pushing" (ie. separate changes that should be just one logical change). The benefits for working with branches would just be those derived from better merging. We can already review a branch (although nobody likes to), or perform a review by files. And still, having a DVCS won't avoid silly merging errors. A DCVS also complicates some things. There is an entry barrier to pass. Revision number such as r12345 is really easy (still, I'd like having a summary attached to revision names), and we use them everywhere* . Talking about revision 0fda45694 is ugly. Mercurial also has revision numbers, but they follow the revisions at the working copy, not those at the master repo. * Which is good. I am proud for example on how we reference the relevant commits on bugzilla, so the path from bug to fix is trivial. On some projects, that is hard, and you have to end up comparing the bug closing date with near commits.