Hi everyone,
For a long time, we've been talking about migrating from Subversion to Git. It's time to start getting more serious about it.
First: the need to do this. There is pretty broad acceptance that we should move to a distributed version control system (DVCS). Our current Subversion-based version control system has served us well, but we're in need of a more suitable version control system for our development effort. Our community is very distributed, with many parallel efforts and needs to integrate many different feature efforts. While we've developed lots of coping mechanisms, we sure could use a system that's well suited to more fluid branching and merging.
There has been resistance to this in the past, and there still may be some resistance. However, I think we've worn everyone down. :)
Next: the selection of Git over other DVCSs. Over the past couple of years, other systems have been mentioned (Bzr, Hg), but there hasn't been any evidence (at least on this mailing list) for anything other than mild support for the alternatives. As you might have seen, our Ops folks have already moved to Git[1], and while they're right in the middle of the tough part of the learning curve, they seem to be adjusting just fine. The complaints seem to be of the "I really need to get used to that" variety rather than the "why are we doing this again?" variety. So, given the momentum that Git has, the ample discussion we've had on the subject, and the fact that Ops is already planning to support Git, this seems to be a settled question.
So now, the questions shift from "if?" to "when?" and "how?".
When? After some amount of arm twisting by Erik and Brion (*hugz*), I've agreed to float a plan that has us making the migration by the end of the year. This is completely contingent on our ability to get 1.19 deployed in a rapid fashion (which, if we can get through the code review queue at our current rate, could be done in November). Until we have a more fleshed out plan, though, "end of the year" purely a guess, and subject to change (partly based on any ensuing conversation after this mail). However, assuming we can clear the technical hurdles, there's not any procedural issues I can see getting in the way of a rapid transition.
How? Lots of unsorted pieces. There are still decisions we need to make: * Code review tool: barring unforeseen complications, we're planning to use Gerrit. We need to make sure it'll be a suitable replacement for our existing tool * How do we break up the repository? One big repo? Extensions each get their own? We need to sort all of that out.
A draft plan is available here: http://www.mediawiki.org/wiki/Git_conversion
Rob
[1] ...or so I've read on Slashdot, so it must be true: http://hardware.slashdot.org/story/11/09/21/0531246/wikimedia-foundation-rel...
+∞**
This is not going to be easy, but nothing worth doing ever is. I've been using git for personal projects for a while, and would agree that the issues that have come about are more to do with learning than regret.
- Trevor
On Thu, Sep 22, 2011 at 4:06 PM, Rob Lanphier robla@wikimedia.org wrote:
Hi everyone,
For a long time, we've been talking about migrating from Subversion to Git. It's time to start getting more serious about it.
First: the need to do this. There is pretty broad acceptance that we should move to a distributed version control system (DVCS). Our current Subversion-based version control system has served us well, but we're in need of a more suitable version control system for our development effort. Our community is very distributed, with many parallel efforts and needs to integrate many different feature efforts. While we've developed lots of coping mechanisms, we sure could use a system that's well suited to more fluid branching and merging.
There has been resistance to this in the past, and there still may be some resistance. However, I think we've worn everyone down. :)
Next: the selection of Git over other DVCSs. Over the past couple of years, other systems have been mentioned (Bzr, Hg), but there hasn't been any evidence (at least on this mailing list) for anything other than mild support for the alternatives. As you might have seen, our Ops folks have already moved to Git[1], and while they're right in the middle of the tough part of the learning curve, they seem to be adjusting just fine. The complaints seem to be of the "I really need to get used to that" variety rather than the "why are we doing this again?" variety. So, given the momentum that Git has, the ample discussion we've had on the subject, and the fact that Ops is already planning to support Git, this seems to be a settled question.
So now, the questions shift from "if?" to "when?" and "how?".
When? After some amount of arm twisting by Erik and Brion (*hugz*), I've agreed to float a plan that has us making the migration by the end of the year. This is completely contingent on our ability to get 1.19 deployed in a rapid fashion (which, if we can get through the code review queue at our current rate, could be done in November). Until we have a more fleshed out plan, though, "end of the year" purely a guess, and subject to change (partly based on any ensuing conversation after this mail). However, assuming we can clear the technical hurdles, there's not any procedural issues I can see getting in the way of a rapid transition.
How? Lots of unsorted pieces. There are still decisions we need to make:
- Code review tool: barring unforeseen complications, we're planning
to use Gerrit. We need to make sure it'll be a suitable replacement for our existing tool
- How do we break up the repository? One big repo? Extensions each
get their own? We need to sort all of that out.
A draft plan is available here: http://www.mediawiki.org/wiki/Git_conversion
Rob
[1] ...or so I've read on Slashdot, so it must be true:
http://hardware.slashdot.org/story/11/09/21/0531246/wikimedia-foundation-rel...
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Yay!
I've volunteered to do a quick intro-to-our-scary-git-future session at the New Orleans hackathon; I'll see if I can lay out a nice workflow demonstration from a few different perspectives:
* staff or very active volunteer developer who's doing a lot of core or high-priority extension work all the time * reviewers monitoring incoming stuff * extension maintainers working on their own and sharing their code * ad-hoc patch submissions * larger feature/refactoring submissions * batch updates such as localization maintainers * deployment branch management * using the VCS as a deployment source * how things can interact with Bugzilla etc
-- brion
On Thu, Sep 22, 2011 at 6:29 PM, Brion Vibber brion@pobox.com wrote:
Yay!
I've volunteered to do a quick intro-to-our-scary-git-future session at the New Orleans hackathon; I'll see if I can lay out a nice workflow demonstration from a few different perspectives:
- staff or very active volunteer developer who's doing a lot of core or
high-priority extension work all the time
- reviewers monitoring incoming stuff
- extension maintainers working on their own and sharing their code
- ad-hoc patch submissions
- larger feature/refactoring submissions
- batch updates such as localization maintainers
- deployment branch management
- using the VCS as a deployment source
- how things can interact with Bugzilla etc
I also had some ideas on how to integrate Labs with a git process that I'd like to discuss at the hack-a-thon:
http://www.mediawiki.org/wiki/WMF_Projects/Wikimedia_Labs/Development_Proces...
Please edit the above mercilessly if you feel it makes no sense ;).
- Ryan
This is awesome. Seconding Trevor on our move to git.... +∞**
Brion - thanks for jumping in to do our-scary-git-future session as a repeat at tech days :-)
Alolita
On Thu, Sep 22, 2011 at 6:41 PM, Ryan Lane rlane32@gmail.com wrote:
On Thu, Sep 22, 2011 at 6:29 PM, Brion Vibber brion@pobox.com wrote:
Yay!
I've volunteered to do a quick intro-to-our-scary-git-future session at
the
New Orleans hackathon; I'll see if I can lay out a nice workflow demonstration from a few different perspectives:
- staff or very active volunteer developer who's doing a lot of core or
high-priority extension work all the time
- reviewers monitoring incoming stuff
- extension maintainers working on their own and sharing their code
- ad-hoc patch submissions
- larger feature/refactoring submissions
- batch updates such as localization maintainers
- deployment branch management
- using the VCS as a deployment source
- how things can interact with Bugzilla etc
I also had some ideas on how to integrate Labs with a git process that I'd like to discuss at the hack-a-thon:
http://www.mediawiki.org/wiki/WMF_Projects/Wikimedia_Labs/Development_Proces...
Please edit the above mercilessly if you feel it makes no sense ;).
- Ryan
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
There is no scary git future. A future with SVN, now that's scary....
On 9/22/11 10:22 PM, Alolita Sharma wrote:
This is awesome. Seconding Trevor on our move to git.... +∞**
Brion - thanks for jumping in to do our-scary-git-future session as a repeat at tech days :-)
Alolita
On Thu, Sep 22, 2011 at 6:41 PM, Ryan Lanerlane32@gmail.com wrote:
On Thu, Sep 22, 2011 at 6:29 PM, Brion Vibberbrion@pobox.com wrote:
Yay!
I've volunteered to do a quick intro-to-our-scary-git-future session at
the
New Orleans hackathon;
Alolita Sharma alolita.sharma@gmail.com writes:
This is awesome. Seconding Trevor on our move to git.... +∞**
+100!
This is great news.
Congrats for this ongoing effort, and good luck for the move :)
Has the svn:externals problem been solved? What flow would need to follow people currently using sparse checkouts. There have been patches in git ml for sparse clones, but they hadn't been applied, last time I checked.
On Fri, Sep 23, 2011 at 12:14 PM, Platonides Platonides@gmail.com wrote:
Has the svn:externals problem been solved? What flow would need to follow people currently using sparse checkouts. There have been patches in git ml for sparse clones, but they hadn't been applied, last time I checked.
Another thing: I currently use SVN's trick with embedded checkouts where the content of /extensions in a trunk/phase3 checkout is replaced with a checkout of trunk/extensions. This way, extensions are located at the canonical path relative to core and I'm able to commit into phase3 and extensions at the same time.
Does Git support anything like that? MW_INSTALL_PATH is not really convenient when you have to use trunk and several branches at the same time.
On 23/09/11 10:39, Max Semenik wrote: <snip>
Another thing: I currently use SVN's trick with embedded checkouts where the content of /extensions in a trunk/phase3 checkout is replaced with a checkout of trunk/extensions. This way, extensions are located at the canonical path relative to core and I'm able to commit into phase3 and extensions at the same time.
Does Git support anything like that? MW_INSTALL_PATH is not really convenient when you have to use trunk and several branches at the same time.
Maybe with git submodules ? The /trunk/extensions could be set to have all extensions git repositories as submodules.
We still need to find out how we will organize the git repositories though.
On 11-09-23 09:51 AM, Ashar Voultoiz wrote:
On 23/09/11 10:39, Max Semenik wrote:
<snip> > Another thing: I currently use SVN's trick with embedded checkouts where the > content of /extensions in a trunk/phase3 checkout is replaced with a > checkout of trunk/extensions. This way, extensions are located at the > canonical path relative to core and I'm able to commit into phase3 and > extensions at the same time. > > Does Git support anything like that? MW_INSTALL_PATH is not really > convenient when you have to use trunk and several branches at the same time. Maybe with git submodules ? The /trunk/extensions could be set to have all extensions git repositories as submodules.
We still need to find out how we will organize the git repositories though.
Git submodules are tied to specific commit id's, so I wouldn't really use it.
But you can put a git repo inside a directory that's part of another git repo perfectly fine even without submodules (though without an ignore it might try to /helpfully/ provide it as an option to add as a submobule within the gui while committing). We will probably include a .gitignore into gore that will let us have /extensions/README but ignore all other things inside /extensions/, same thing for our other dynamic folders like /images/. So you shouldn't have to worry about something like that.
Does svn actually update disconnected svn directories that just happen to be in a subdirectory? If it does do that then I do admit we might want to provide a handy script to batch upgrade... well, actually extension wise I've been thinking of that for extensions for awhile. ---- Oh... wait, are you talking about a root commit where you make one commit that spans phase3 and extensions? No, I can almost assure you that core and extensions are going to be in separate repos so commits will need to be made separately to them (though we may want to include helpers to make it easier to do a commit to multiple extensions en-masse).
~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://daniel.friesen.name]
Daniel Friesen wrote:
Does svn actually update disconnected svn directories that just happen to be in a subdirectory? If it does do that then I do admit we might want to provide a handy script to batch upgrade... well, actually extension wise I've been thinking of that for extensions for awhile.
I have heard of people doing that, and I assume that's what MaxSem means by 'svn trick', although I have been unable to make it myself. So if you can share the instructions, I would appreciate it. :)
On 24 September 2011 00:49, Platonides Platonides@gmail.com wrote:
Daniel Friesen wrote:
Does svn actually update disconnected svn directories that just happen to be in a subdirectory? If it does do that then I do admit we might want to provide a handy script to batch upgrade... well, actually extension wise I've been thinking of that for extensions for awhile.
I have heard of people doing that, and I assume that's what MaxSem means by 'svn trick', although I have been unable to make it myself. So if you can share the instructions, I would appreciate it. :)
rm -rf extensions svn co ..... extensions
svn status should now show S extensions and you have all extensions. If you don't want them all, use --depth empty and svn up them individually.
On 24/09/11 09:22, Niklas Laxström wrote:
rm -rf extensions svn co ..... extensions
svn status should now show S extensions and you have all extensions. If you don't want them all, use --depth empty and svn up them individually.
We really need to have this tip somewhere on MediaWiki.org :-D
Niklas Laxström wrote:
On 24 September 2011 00:49, Platonides wrote:
Daniel Friesen wrote:
Does svn actually update disconnected svn directories that just happen to be in a subdirectory? If it does do that then I do admit we might want to provide a handy script to batch upgrade... well, actually extension wise I've been thinking of that for extensions for awhile.
I have heard of people doing that, and I assume that's what MaxSem means by 'svn trick', although I have been unable to make it myself. So if you can share the instructions, I would appreciate it. :)
rm -rf extensions svn co ..... extensions
svn status should now show S extensions and you have all extensions. If you don't want them all, use --depth empty and svn up them individually.
Yes. I have done so, both with extensions a sparse checkout and with the folders their own checkouts inside extensions folder. What I mean is that a phase3 update doesn't seem to also update extensions.
The trick was using svn switch:
svn checkout http://svn.wikimedia.org/svnroot/mediawiki/trunk/phase3 cd phase3 svn switch http://svn.wikimedia.org/svnroot/mediawiki/trunk/extensions extensions/ --depth immediates cd extensions/ svn checkout http://svn.wikimedia.org/svnroot/mediawiki/trunk/extensions/SportsTeam --depth infinity cd ../.. svn up phase3 U phase3/extensions/SportsTeams/SportsTeams.i18n.php U phase3/includes/parser/Parser.php Updated to revision 100758.
On Fri, Sep 23, 2011 at 1:06 AM, Rob Lanphier robla@wikimedia.org wrote:
- Code review tool: barring unforeseen complications, we're planning
to use Gerrit. We need to make sure it'll be a suitable replacement for our existing tool
I've talked to some of the ops folks a bit, and we've already agreed that inline display of diffs is something we really need to add to Gerrit. I've filed a feature request to this end: http://code.google.com/p/gerrit/issues/detail?id=1137 . As promised in the bug report, I will work on this during the NOLA hacakthon if no one beats me to it.
Roan
I've talked to some of the ops folks a bit, and we've already agreed that inline display of diffs is something we really need to add to Gerrit. I've filed a feature request to this end: http://code.google.com/p/gerrit/issues/detail?id=1137
It's worth pointing out that Google Code will let each of us indicate interest in this feature by clicking the little star next to the title.
-Ian
I've talked to some of the ops folks a bit, and we've already agreed that inline display of diffs is something we really need to add to Gerrit. I've filed a feature request to this end: http://code.google.com/p/gerrit/issues/detail?id=1137
It's worth pointing out that Google Code will let each of us indicate interest in this feature by clicking the little star next to the title.
If people are voting for bugs, please vote for this one as well:
http://code.google.com/p/gerrit/issues/detail?id=1124
It'll make the gerrit/labs access process less of a pain.
- Ryan
Rob Lanphier <robla <at> wikimedia.org> writes:
So now, the questions shift from "if?" to "when?" and "how?".
How? Lots of unsorted pieces. There are still decisions we need to make:
- How do we break up the repository? One big repo? Extensions each
get their own? We need to sort all of that out.
Currently, svn.wikimedia.org also hosts other repositories, most notably the pywikipedia repository. Is the plan to keep svn.wikimedia.org and svn-based code review online after the switch, or will the pywikipedia repository also have to switch?
If so, we could use some of the expertise in terms of converting the repository.
From experience, I can tell the pywikipedia repository does not play well with
git-svn.
Best,
Merlijn
On Tue, Oct 4, 2011 at 9:21 AM, Merlijn van Deen valhallasw@arctus.nl wrote:
Currently, svn.wikimedia.org also hosts other repositories, most notably the pywikipedia repository. Is the plan to keep svn.wikimedia.org and svn-based code review online after the switch, or will the pywikipedia repository also have to switch?
We're not in a big hurry to shut down our svn services, and pywikipedia is not really tangled up in the MediaWiki codebase, so I don't imagine we'll be rushing to convert it to git. I imagine there will come a day when we want to get out of the svn business, but we have a few months (at least) to figure out our strategy there. In the short term, pywikipedia in svn can peacefully coexist with MediaWiki + extensions in git.
Rob
We also want to move over the generic Wikimedia repot.
--tomasz
On Wed, Oct 5, 2011 at 7:41 AM, Rob Lanphier robla@wikimedia.org wrote:
On Tue, Oct 4, 2011 at 9:21 AM, Merlijn van Deen valhallasw@arctus.nl wrote:
Currently, svn.wikimedia.org also hosts other repositories, most notably the pywikipedia repository. Is the plan to keep svn.wikimedia.org and svn-based code review online after the switch, or will the pywikipedia repository also have to switch?
We're not in a big hurry to shut down our svn services, and pywikipedia is not really tangled up in the MediaWiki codebase, so I don't imagine we'll be rushing to convert it to git. I imagine there will come a day when we want to get out of the svn business, but we have a few months (at least) to figure out our strategy there. In the short term, pywikipedia in svn can peacefully coexist with MediaWiki + extensions in git.
Rob
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Thanks for bringing this up Tomasz - I meant to send a reply about this earlier this week but it fell off my radar.
For those that don't know, the Wikimedia repo holds WMF-related resources that are not directly MediaWiki related. The fundraising team and fundraising analytics relies on this repo extensively. It would be a huge win for us if this repo was migrated to Git along with the Mediawiki repo.
On Thu, Oct 6, 2011 at 12:46 PM, Tomasz Finc tfinc@wikimedia.org wrote:
We also want to move over the generic Wikimedia repot.
--tomasz
On Wed, Oct 5, 2011 at 7:41 AM, Rob Lanphier robla@wikimedia.org wrote:
On Tue, Oct 4, 2011 at 9:21 AM, Merlijn van Deen valhallasw@arctus.nl
wrote:
Currently, svn.wikimedia.org also hosts other repositories, most
notably the
pywikipedia repository. Is the plan to keep svn.wikimedia.org and
svn-based code
review online after the switch, or will the pywikipedia repository also
have to
switch?
We're not in a big hurry to shut down our svn services, and pywikipedia is not really tangled up in the MediaWiki codebase, so I don't imagine we'll be rushing to convert it to git. I imagine there will come a day when we want to get out of the svn business, but we have a few months (at least) to figure out our strategy there. In the short term, pywikipedia in svn can peacefully coexist with MediaWiki + extensions in git.
Rob
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On 07/10/11 23:13, Arthur Richards wrote:
Thanks for bringing this up Tomasz - I meant to send a reply about this earlier this week but it fell off my radar.
For those that don't know, the Wikimedia repo holds WMF-related resources that are not directly MediaWiki related. The fundraising team and fundraising analytics relies on this repo extensively. It would be a huge win for us if this repo was migrated to Git along with the Mediawiki repo.
Maybe it could be the first one to migrate to git ? It seems easier to handle since: - there is less history - probably no awkward branching - most users are probably all in the WMF office.
That could provide useful feedback before migrating the MW repo.
Maybe it [Wikimedia] could be the first one to migrate to git ? It seems easier to
Normally I'd be into this, but I'm nervous about the timing. As I understand it, the conversion is set to begin in November, which is when the fundraiser will be starting. Using a repo that is predominantly used for fundraising as the guinea pig during the fundraiser makes me more than a little nervous.
In fact, now that I think about it, I'm nervous about doing a git migration on MediaWiki repo during the fundraiser as well.
Skimming over the plan on the wiki page, I don't see anything about a contingency or failsafe plan, which is something I'd really like to know!
On Tue, Oct 11, 2011 at 10:48 AM, Arthur Richards arichards@wikimedia.org wrote:
Skimming over the plan on the wiki page, I don't see anything about a contingency or failsafe plan, which is something I'd really like to know!
Just go back to using SVN :)
wikitech-l@lists.wikimedia.org