Well of course, Where would we be if we let the world end without a new(ish) mediawiki release? -Peachey

On Sun, Feb 13, 2011 at 8:11 PM, Mark A. Hershberger <mhershberger(a)wikimedia.org> wrote: On the other hand, in Linux, it can be hard to get patches reviewed and accepted in a timely fashion, because there's no clear chain of command unless Linus personally intervenes. I think Mozilla is an excellent model to follow, in that they have a well-defined review process (not "everyone can object to design decisions at the eleventh hour" like Linux) and make sure that all patches get reviewed in a timely fashion (at least as far as I've seen). I've submitted two patches to Mozilla, and I got immediate feedback and review on both of them from developers responsible for the relevant areas. What I think is important is that a) there's a formal process that ensures all submitted code gets reviewed, and b) this process is basically the same for everyone (with no group of people with "commit access" who get to jump the queue). Without (b), code by new contributors will too easily slip through the cracks. Mozilla has everyone submit code on Bugzilla, which is awkward but works -- even core developers have to file a bug, submit a patch there, and get review from another qualified coder, essentially the same as anyone. (Obviously with exceptions like backing out build breakage and so on.) Mozilla does have people with commit access, but they're just the ones tasked with the chore of checking in code once it's been reviewed -- they aren't allowed to just check stuff in without going through the review process first. If we switched to git, perhaps we should take a look at Gerrit for a review tool. I've heard good things about it, but never used it myself. Whatever we use, I think it should really be "review then commit", not "commit then review". Cherry-picking from trunk to a branch is possibly better than reverting things in trunk (not sure), but in the medium term it would be much better to get rid of commit access status entirely -- once we're sure we have a properly-functioning review process that can keep up with changes.

2011/2/13 Bryan Tong Minh <bryan.tongminh(a)gmail.com>om>: +1 If we want to move to continuous integration (and I think the consensus is we do, considering the mess we've made for ourselves by deploying 9 months worth of commits and not knowing which of the ~15,000 new revisions killed the cluster the other day), our first step should be to get closer to continuous integration, i.e. bring deployment closer to trunk. By the time we deploy 1.17, trunk will already be more than two months ahead. Of course this is because we needed time to stabilize 1.17, which in turn was caused by the amount of new code in it. Stabilizing and deploying 1.18wmf1 should take considerably less time and allow us to get much closer to a continuous integration model. Roan Kattouw (Catrope)

2011/2/14 Mark A. Hershberger <mhershberger(a)wikimedia.org>rg>: That works for me too. wmfN branches can't be tagged from release branches, they have a different structure so you need to run a script that branches off the various parts, but otherwise this sounds good. Roan Kattouw (Catrope)

On 14/02/11 15:11, Roan Kattouw wrote: What about stopping adding new code in trunk until the branch is stable/live? Most new features / refactoring can probably held by their authors for two or three months. -- Ashar Voultoiz

On 16/02/2011 07:53, Chad wrote: +1 Further arguments on this issue are in [1] under "Don’t freeze the trunk for long periods" where freezing is associated with a significant loss of contribution activity, even beyond the time that the trunk was frozen. - Markus [1] http://www.codesimplicity.com/post/open-source-community-simplified/

Unless I'm mistaken no one has actually suggested doing that. -- Jeroen De Dauw http://www.bn2vs.com Don't panic. Don't be evil. --

On 11-02-13 06:15 PM, Brion Vibber wrote: On extensions there's another idea not mentioned on that page. Right now the distinction there is everything in one git repo vs. use git submodules. The dilemma there is this: - In one repo everyone would need the whole history of all extensions to check out one extension, naturally this is not what we desire so we want to split it. - however if we split it then you no longer can check out all the extensions for mass development, bulk translation, etc... purposes. - so we consider having a dedicated repo which simply has git submodules to all the different extensions so you can easily check them out - unfortunately git submodules aren't like svn externals (whether svn's externals behavior could be considered a good thing all the time is a side topic) a commit id is part of the package, so you can't say "I want the latest commit from this repo", you have to say "I want <x> commit from this repo" and to get the same behavior you have to do some hacky things committing a change to the repo hosting submodule references every time the remote repo adds a new commit -- ((Although on the other hand, this could be interesting for extension branching... if we make rel1_16 style branches in the git repo they can easily target a specific commit, and "backporting" things as compatible for that rev simply means going into that branch and updating what commit the submodule refers to)) Naturally both options are not ideal. We can't have all extensions in one repo, nor is making automated commits to a repo every time a remote repo is updated an ideal. However a key thing to note is that git submodules aren't anything really special. Sure, they're integrated into git, but the only real special feature about them is that you can target a specific commit id... and heck, we don't even want that feature, that's the whole reason it's problematic. Git submodules don't optimize cloning the repo, it doesn't store any of the remote data in the repo, so whether you use a submodule or not the actual task of getting those extension repos is still that of cloning each and every one of those remote repos individually. The only difference submodules make is that now instead of `git clone <x>` or `git pull <x>` over and over and over. You just run `git submodule <update,etc...>` and it runs the clone or pull for every repo sequentially without you having to explicitly do it. And the result is essentially the same, you end up with a bunch of git repos in subdirectories, only with the extra ability to do a batch update of them all. (Though there is a negative that when using submodules these are attached to a specific commit id and not a branch, so you can no longer decide you want to `git pull` only one of the extensions instead of all of them at once) Hence I think there is another option. Instead of (ab)using git's submodules to reference extension repos, we could instead write a quick script that talks to a simple database or api (heck it could be a tab delimited text file committed to the git repo where the script is) which gives a list of all extension names, the url for the git repo, and optionally could even list what commit id was last used at the branch point for a release (in other words serve the same purpose as branching extensions does), while perhaps also making it easy to update what version of extensions is marked as compatible with certain releases. As a bonus; We'll retain the ability to git pull individual repos without conflicting with submodules. We can write some extra quick scripting to give us info like "These extensions are running off master, while these are running of commits tagged as compatible with a specific release". And other features we can't offer just using submodules. We also can also manage extension groups and dependencies like "AbuseFilter requires AntiSpoof" and set it up so that if you individually "./mwextension checkout AbuseFilter" it implicitly checks out AntiSpoof for you too. Naturally it would make sense to write it in php, since MediaWiki already requires php it's the language most likely to be relevant to those checking things out. Although you could argue for users who check out onto their own machine and not onto the machine they run MW and may not have php installed. In that case you could have a case for writing it in python, since by comparison python is probably one of the most commonly installed scripting languages, and imho it's comparatively easier to install than php or ruby, or compiling something yourself in a cross-compatible way. Though on the other hand, this almost sounds like something we might make a maintenance script. Ohh... if the translatewiki guys are looking for a dummy for streamlining support for extensions based in git in preparation for a git migration if we do so, I'd be happy to offer monaco-port up as a existing extension (well, skin) using git that could be used as a test for streamlining git support. ;) having monaco-port get proper i18n while it's still not up to a level I believe I want to commit it into svn yet wouldn't be a bad thing. ~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://daniel.friesen.name]

If I am not mistaken then mercurial has better support for highly modularized open source software projects. You can use a  mercurial subrepository (which is very similar to svn external and git submodule). According to their manual: "Subrepositories is a feature that allows you to treat a collection of repositories as a group. This will allow you to clone, commit to, push, and pull projects and their associated libraries as a group." see: http://mercurial.selenic.com/wiki/Subrepository http://mercurial.selenic.com/wiki/NestedRepositories just my 2 cents. On Mon, Feb 14, 2011 at 2:18 AM, Siebrand Mazeland &lt;s.mazeland(a)xs4all.nl&gt; wrote: -- <a href="http://about.me/diederik">Check out my about.me profile!</a>

On Feb 14, 2011, at 9:42 AM, Mark A. Hershberger wrote: I'm not sure how much worse it could really get? The workflow for review and merging is pretty bad as it is right now. - Trevor

On Mon, Feb 14, 2011 at 2:18 AM, Siebrand Mazeland &lt;s.mazeland(a)xs4all.nl&gt; wrote: What's the problem with doing a hundred or more commits to different repositories? It be be trivial to script, if that's the problem. I don't see that fragmentation in a filesystem sense is relevant here. What disorganisation/disharmony are you talking about? The "decentralized" aspect will be in terms of who gets to change the code, not where it's put. In other words, I'd like to see anyone have equal standing to submit code, not have a separate "commit access" group. The code will still all be hosted by Wikimedia and made available centrally. A DVCS just makes it much easier to do development without having official approval by the upstream project. Checkouts are not just for developers. That's one of the pluses of a DVCS -- anyone can check out and maintain local patches in the version control, easily merging them with upstream changes. This is incredibly useful if you're maintaining local, site-specific hacks. I've done it with vBulletin for years (making my own git repository from their tarball releases). It also makes it very easy to submit your hacks for review upstream, if upstream is set up right. On Mon, Feb 14, 2011 at 9:57 AM, Diederik van Liere &lt;dvanliere(a)gmail.com&gt; wrote: I've used git a lot (I use it for everything I want to version) and Mercurial a fair bit (the W3C seems to like it), and I *strongly* prefer git. Major problems I have with Mercurial: 1) It doesn't support lots of useful functionality that's built into git unless you enable various extensions. This includes colored diffs, automatic paging for large output, and rebasing. 2) It makes it much too easy to delete stuff forever. I've had data loss once already due to getting confused by the merge UI during a rebase -- it kept no copies of my work. git always stores everything even after it's no longer accessible through normal means, like after a rebase or reset --hard. It only cleans up untouched stuff during periodic garbage collection. On the other hand, there's basically nothing I've seen in Mercurial where I've said "wow, I wish git had that". So for what my opinion is worth, we should be going for git, not Mercurial, despite the former's quirks and flaws.

Daniel Friesen wrote: Knowing how git just stores everything in files, it's likely that submodules are just a plain text list inside a file. So it would be easy to script an update to the list on a post-commit hook.

On 11-02-13 06:15 PM, Brion Vibber wrote: Don't forget about what git does for those of us using remote servers too. The ease of development between working on my monaco-port (in git for now) vs. making core changes (svn of course) is fairly noticeable. I do my development and testing on remote servers, and multiple ones at that (monaco-port is actually located in 4 or 5 locations; My local repo, on the dragonballencyclopedia's prototype and live site, another location still setup on commonjs (I used to use it for experimenting before I had other wikis to try it on) and in my trunktest for development in a 1.18 env). And I usually vary whether I work on a feature from the dragonballencyclopedia prototype or trunktest. Sometimes doing a little work on both since one is a 1.16 env and the other trunk and some features vary in support over that border. The last key piece of background, is I don't commit/push things to remote repos from these servers. I don't trust my own servers with any of my private keys used to commit to remote repos (sure, I have separate keys, and I could give it just the ability to push to the repo without accessing other servers... but I don't trust my own servers enough to give them the ability to commit something to Wikimedia's svn under my name). Hence any task of committing/pushing to a remote repo is done from my physical computer, not the server, despite 99% of the actual development being on the servers. So since I work on remote machines, but commit from my local machine I need to find a way to transfer the changes from the remote machine to the local machine before submitting them. This is completely trivial with git. I just make the actual commit on the server itself (since I don't need any special access for that). Then I pull from my machine to pull those changes from the server into my local copy. And finally I push those changes to the public repo. Only takes a few seconds. I even have a trivial shell script for monaco-port that pulls changes from all my remote repos into my local one, pushes that to my github repo, and then makes sure each of my servers using monaco-port are up to date. (It's really just a series of git pull/push commands) But when it comes to svn... I've taken to the habit of running `ssh <server> "cd /path/to/wiki; svn diff" | patch -p0`. Naturally I have to svn up both copies before that to avoid any mishaps. And if I added a new file, I have to do a separate explicit scp. And after that I have to go back to the remote server, and svn up it again so that it understands that those changes are no longer a dirty working copy. If I added a file sometimes I have to deal with a slight mess. And if I forget to svn up, have to deal with some blech from patch. etc... Naturally of course, git is also beautiful for my disorganized working copy. Right now trunktest has some uncommitted stuff I'm experimenting with. And I usually end up transferring my changes to my local copy, then reverting everything not relevant, and in the off case where two experiments used the same file, I have to manually edit one of those out (though I slightly streamlined that by piping svn diff <file> to a file, running svn revert <file>, editing the patch and deleting the change I don't want to commit, and then running the patch command; rather than manually undoing the code). Git changes that extremely, since there is an index I just explicitly say which parts I want to add to the index to commit. And it works beautifully for files where two experiments work on the same file, I can look through the file and explicitly say which diff chunks I actually DO want to commit without having to edit the file. And `git gui` really does make picking those diff chunks easier (yay for ssh -X too). ~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://daniel.friesen.name]