Hi all,
As some of you are probably aware, we've got a test repository converting phase3 to git up and running on gerrit. You should be able to clone by
`git clone https://gerrit.wikimedia.org/r/p/test/mediawiki/core.git%60
Couple of caveats (things I'm gonna try and fix): * Permissions aren't sorted yet, so it's only supporting anonymous clones, no pushing yet. * The revision graph is crazy. svn:mergeinfo is unreliable and we're pretty much unable to build a cohesive history without a *lot* of manual labor. Right now I'm thinking of just dropping the mergeinfo so the branches look like linear graphs cherry picking from master. Not perfect, but less annoying than now.
But yay progress! Clone it. Try it out, see what works (and what doesn't). I'll try to get the permissions sorted later today so we can go ahead and try some test pushes (and merges).
Things we still need to do: * Make a git-setup like we've done for the other git repos. This will setup your environment/hooks/etc for you. * Figure out our WMF branching/deployment strategy, since we're very SVN- centric right now with this.
Thanks for any input, guys.
-Chad
On Tue, Dec 13, 2011 at 11:44 AM, Chad innocentkiller@gmail.com wrote:
Couple of caveats (things I'm gonna try and fix):
- Permissions aren't sorted yet, so it's only supporting anonymous clones,
no pushing yet.
- The revision graph is crazy. svn:mergeinfo is unreliable and we're pretty
much unable to build a cohesive history without a *lot* of manual labor. Right now I'm thinking of just dropping the mergeinfo so the branches look like linear graphs cherry picking from master. Not perfect, but less annoying than now.
Also there's two stupid commits at the head of master due to my mistake when initially pushing the repo. That won't happen again on subsequent tests or the real conversion.
-Chad
Hi Chad,
Reposurgeon (http://catb.org/~esr/reposurgeon/ ) might be a useful tool to help fix the svn history.
Best,
Diederik
On Tue, Dec 13, 2011 at 11:47 AM, Chad innocentkiller@gmail.com wrote:
On Tue, Dec 13, 2011 at 11:44 AM, Chad innocentkiller@gmail.com wrote:
Couple of caveats (things I'm gonna try and fix):
- Permissions aren't sorted yet, so it's only supporting anonymous
clones,
no pushing yet.
- The revision graph is crazy. svn:mergeinfo is unreliable and we're
pretty
much unable to build a cohesive history without a *lot* of manual labor.
Right
now I'm thinking of just dropping the mergeinfo so the branches look
like linear
graphs cherry picking from master. Not perfect, but less annoying than
now.
Also there's two stupid commits at the head of master due to my mistake when initially pushing the repo. That won't happen again on subsequent tests or the real conversion.
-Chad
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On 13 December 2011 17:02, Diederik van Liere dvanliere@gmail.com wrote:
Reposurgeon (http://catb.org/~esr/reposurgeon/ ) might be a useful tool to help fix the svn history.
In particular, Eric Raymond (who develops it) is interested in large and troublesome repos that teach him new and interesting things. If ours is particularly gnarly, we might turn up new bugs :-D
- d.
On 13/12/11 20:54, David Gerard wrote:
On 13 December 2011 17:02, Diederik van Liere dvanliere@gmail.com wrote:
Reposurgeon (http://catb.org/~esr/reposurgeon/ ) might be a useful tool to help fix the svn history.
In particular, Eric Raymond (who develops it) is interested in large and troublesome repos that teach him new and interesting things. If ours is particularly gnarly, we might turn up new bugs :-D
- d.
Thanks for the name, I had remembered seeing a "crazy" guy that wanted to port hard histories but didn't remember who he was. (to my defense, those blog entries don't name the author :) )
The blog post: http://esr.ibiblio.org/?p=3839
On Tue, Dec 13, 2011 at 8:44 AM, Chad innocentkiller@gmail.com wrote:
Hi all,
As some of you are probably aware, we've got a test repository converting phase3 to git up and running on gerrit. You should be able to clone by
`git clone https://gerrit.wikimedia.org/r/p/test/mediawiki/core.git%60
\o/
- The revision graph is crazy. svn:mergeinfo is unreliable and we're pretty
much unable to build a cohesive history without a *lot* of manual labor. Right now I'm thinking of just dropping the mergeinfo so the branches look like linear graphs cherry picking from master. Not perfect, but less annoying than now.
I'd tend to agree. Subideal but at least manageable, especially since we've usually been pretty good about saying which revs we merged in the commit summaries.
But yay progress! Clone it. Try it out, see what works (and what doesn't). I'll try to get the permissions sorted later today so we can go ahead and try some test pushes (and merges).
It's taking a few more minutes than I expect for the initial clone (though that's usually not performance-critical); of course since there's a lot more data in there than a SVN checkout I don't mind so much.
Final checkout sizes: * SVN trunk/phase3 only: 145M * git core, all branches with full history since a 2003 code reorg: 198M
Niiiiiiice!
Release branches from 1.1 to 1.18 are present... no individual release tags visible though. Did we figure out how to get those done?
History (including 'git blame' view) seems to have transferred most contributors' identities over nicely, using @users.mediawiki.org fake addresses and adding realnames.
As viewed in gitk, branch points off of master are visible though as noted the various merges don't preserve much data. Oh wells!
Things we still need to do:
- Make a git-setup like we've done for the other git repos. This will
setup your environment/hooks/etc for you.
- Figure out our WMF branching/deployment strategy, since we're very SVN-
centric right now with this.
... and extensions. Whee! ;)
WMF branching/deployment should actually be a lot more git-friendly, since we can pull in the entire REL1_18 or REL1_19 or whatever into the deployment branch in a single command instead of cherry-picking every one. The key is making sure we've signed off of them so we know what's going into release, naturally. :)
Merging individual fixes into the release branch would first be done by cherry-picking, most likely.
-- brion
On Tue, Dec 13, 2011 at 11:44 AM, Chad innocentkiller@gmail.com wrote:
- Permissions aren't sorted yet, so it's only supporting anonymous clones,
no pushing yet.
This has been done. If you pull the lastest from master you'll find a git-setup file and you should be able to push your changes with `git push-for-review`
Right now the group who can approve merges to master is small, but I can hand this out liberally while we're figuring things out, just ping me on IRC if you want it.
-Chad
On Thu, Dec 15, 2011 at 9:43 AM, Chad innocentkiller@gmail.com wrote:
On Tue, Dec 13, 2011 at 11:44 AM, Chad innocentkiller@gmail.com wrote:
- Permissions aren't sorted yet, so it's only supporting anonymous clones,
no pushing yet.
This has been done. If you pull the lastest from master you'll find a git-setup file and you should be able to push your changes with `git push-for-review`
Whoops, need to clarify: that's if you have an LDAP/SVN account. If you do, you'll need to re-clone from the other[0].
-Chad
[0] ssh://<user>@gerrit.wikimedia.org:29418/test/mediawiki/core
On Thu, Dec 15, 2011 at 9:43 AM, Chad innocentkiller@gmail.com wrote: that's if you have an LDAP/SVN account. If you do, you'll need to re-clone from the other[0].
[0] ssh://<user>@gerrit.wikimedia.org:29418/test/mediawiki/core
Couldn't get this to work on Windows 7. I did the following steps: 1. copied my svn.wikimedia.org:22 session to gerrit.wikimedia.org:29418 in PuTTY 2. In commandline: git clone ssh://siebrand@gerrit.wikimedia.org:29418/test/mediawiki/core mwcore
Result: <pop-up with: PuTTY Fatal Error: Disconnected. No supported authentication methods available (server sent: public key) Cloning into mwcore... fatal: The remote end hung up unexpectedly
Any suggestions on how to get this working?
Siebrand
On Thu, Dec 15, 2011 at 10:18 AM, Siebrand Mazeland s.mazeland@xs4all.nl wrote:
On Thu, Dec 15, 2011 at 9:43 AM, Chad innocentkiller@gmail.com wrote: that's if you have an LDAP/SVN account. If you do, you'll need to re-clone from the other[0].
[0] ssh://<user>@gerrit.wikimedia.org:29418/test/mediawiki/core
Couldn't get this to work on Windows 7. I did the following steps:
- copied my svn.wikimedia.org:22 session to gerrit.wikimedia.org:29418 in
PuTTY 2. In commandline: git clone ssh://siebrand@gerrit.wikimedia.org:29418/test/mediawiki/core mwcore
Same result on *nix, I'm assuming this means we need somebody to add us to gerrit from svn?
You need a labs account for this to work. If you'd like one, email me with the following information:
1. Your preferred wiki user name. This will also be your gerrit/git user name. If you want your git username to be your real name, then make this your real name. 2. Your svn account name 3. Your preferred email address
We're in closed beta, but all svn account holders are welcome to join at this time.
- Ryan
On Thu, Dec 15, 2011 at 9:01 AM, OQ overlordq@gmail.com wrote:
On Thu, Dec 15, 2011 at 10:18 AM, Siebrand Mazeland s.mazeland@xs4all.nl wrote:
On Thu, Dec 15, 2011 at 9:43 AM, Chad innocentkiller@gmail.com wrote: that's if you have an LDAP/SVN account. If you do, you'll need to re-clone from the other[0].
[0] ssh://<user>@gerrit.wikimedia.org:29418/test/mediawiki/core
Couldn't get this to work on Windows 7. I did the following steps:
- copied my svn.wikimedia.org:22 session to gerrit.wikimedia.org:29418 in
PuTTY 2. In commandline: git clone ssh://siebrand@gerrit.wikimedia.org:29418/test/mediawiki/core mwcore
Same result on *nix, I'm assuming this means we need somebody to add us to gerrit from svn?
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
I talked with Chad today and together we redid:
* https://www.mediawiki.org/wiki/Git * https://www.mediawiki.org/wiki/Git/Conversion - the schedule for Git conversion * https://www.mediawiki.org/wiki/Git/Workflow - describing the workflow for gerrit, git-review, and any other relevant tools we use * https://www.mediawiki.org/wiki/Git/Guide - Basic guide to using git. Trying not to duplicate docs elsewhere on the web -- keep this appropriate for our users and link generously to outside resources
Hope this helps. You can expect more communication from Chad on this next week, as well as a blog post.
Thanks for sharing these Sumana. One thing I haven't really seen mentioned anywhere is what's going to happen with the WIkimedia repository (rather than the MediaWiki repository) and how that figures into the roadmap/conversion schedule. Chad or Sumana, can you shed some light on this?
On Wed, Feb 1, 2012 at 2:30 PM, Sumana Harihareswara sumanah@wikimedia.orgwrote:
I talked with Chad today and together we redid:
- https://www.mediawiki.org/wiki/Git
- https://www.mediawiki.org/wiki/Git/Conversion - the schedule for Git
conversion
- https://www.mediawiki.org/wiki/Git/Workflow - describing the workflow
for gerrit, git-review, and any other relevant tools we use
- https://www.mediawiki.org/wiki/Git/Guide - Basic guide to using git.
Trying not to duplicate docs elsewhere on the web -- keep this appropriate for our users and link generously to outside resources
Hope this helps. You can expect more communication from Chad on this next week, as well as a blog post.
-- Sumana Harihareswara Volunteer Development Coordinator Wikimedia Foundation
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Wed, Feb 1, 2012 at 8:15 PM, Arthur Richards arichards@wikimedia.org wrote:
Thanks for sharing these Sumana. One thing I haven't really seen mentioned anywhere is what's going to happen with the WIkimedia repository (rather than the MediaWiki repository) and how that figures into the roadmap/conversion schedule. Chad or Sumana, can you shed some light on this?
I totally want to apologize for delaying in responding on this. Somehow it got past my inbox and I missed it until I was looking for another e-mail.
The main point I want to reiterate for everyone is that there is no deadline. MediaWiki and its extensions are on a pretty firm migration path right now and we've got dates pencilled in for when we'd like to finish converting. Converting these parts of the repository are going to be the hardest anyway, so once we've cleared that hump hopefully it'll be trivial to migrate the remaining projects.
I know fundraising is on a different schedule than everyone else, so that's perfectly ok and we can look to a time after the main migrations are done that will work well for you guys too.
I've mentioned this number in a couple of places, but I think I'll say it here too so everyone's clear: I'd like to have the svn repository completely read-only by this time next year (very roughly). If you've got a project that you're maintaining in svn (other than core or extensions), now's the time to start thinking about where you want it to eventually end up. If you're wanting to move to git with the rest of us, the WMF git repo will obviously welcome any existing svn project. If you're wanting to stick with svn or go another route--let me know, I'll be happy to work with you to figure something out.
-Chad
On 02/10/2012 08:10 AM, Chad wrote:
On Wed, Feb 1, 2012 at 8:15 PM, Arthur Richards arichards@wikimedia.org wrote:
Thanks for sharing these Sumana. One thing I haven't really seen mentioned anywhere is what's going to happen with the WIkimedia repository (rather than the MediaWiki repository) and how that figures into the roadmap/conversion schedule. Chad or Sumana, can you shed some light on this?
I totally want to apologize for delaying in responding on this. Somehow it got past my inbox and I missed it until I was looking for another e-mail.
The main point I want to reiterate for everyone is that there is no deadline. MediaWiki and its extensions are on a pretty firm migration path right now and we've got dates pencilled in for when we'd like to finish converting. Converting these parts of the repository are going to be the hardest anyway, so once we've cleared that hump hopefully it'll be trivial to migrate the remaining projects.
I know fundraising is on a different schedule than everyone else, so that's perfectly ok and we can look to a time after the main migrations are done that will work well for you guys too.
I've mentioned this number in a couple of places, but I think I'll say it here too so everyone's clear: I'd like to have the svn repository completely read-only by this time next year (very roughly). If you've got a project that you're maintaining in svn (other than core or extensions), now's the time to start thinking about where you want it to eventually end up. If you're wanting to move to git with the rest of us, the WMF git repo will obviously welcome any existing svn project. If you're wanting to stick with svn or go another route--let me know, I'll be happy to work with you to figure something out.
-Chad
(CC'ing Merlijn as he's been my liaison to pywikipediabot on this topic.)
Thanks for the timeline on that, Chad. I've added that mid-2013 hope to
https://www.mediawiki.org/wiki/Git/Conversion#Affected_development_projects
I want that to be a list of all the projects that live on svn.wikimedia.org (fundraising, Wikimedia analytics, mwdumper, etc.) so we can systematically reach out to them and help them figure out what to do.
I'm also collecting open git migration questions today at
https://www.mediawiki.org/wiki/Talk:Git/Conversion/issues#Topics_for_Chad.27...
so that Chad and I can work on an email and a blog post about 8 hours from now. Feel free to add your questions. (Example: "what about gerrit? when will it be not a UX nightmare?")
I might be missing something obvious - but when should/can extension developers start working on git vs. svn?
Should we wait for the conversion of all extensions or will the extensions area of the repo be available for use beforehand for any new extensions?
-greg aka varnent
On Feb 10, 2012, at 8:50 AM, Sumana Harihareswara wrote:
On 02/10/2012 08:10 AM, Chad wrote:
On Wed, Feb 1, 2012 at 8:15 PM, Arthur Richards arichards@wikimedia.org wrote:
Thanks for sharing these Sumana. One thing I haven't really seen mentioned anywhere is what's going to happen with the WIkimedia repository (rather than the MediaWiki repository) and how that figures into the roadmap/conversion schedule. Chad or Sumana, can you shed some light on this?
I totally want to apologize for delaying in responding on this. Somehow it got past my inbox and I missed it until I was looking for another e-mail.
The main point I want to reiterate for everyone is that there is no deadline. MediaWiki and its extensions are on a pretty firm migration path right now and we've got dates pencilled in for when we'd like to finish converting. Converting these parts of the repository are going to be the hardest anyway, so once we've cleared that hump hopefully it'll be trivial to migrate the remaining projects.
I know fundraising is on a different schedule than everyone else, so that's perfectly ok and we can look to a time after the main migrations are done that will work well for you guys too.
I've mentioned this number in a couple of places, but I think I'll say it here too so everyone's clear: I'd like to have the svn repository completely read-only by this time next year (very roughly). If you've got a project that you're maintaining in svn (other than core or extensions), now's the time to start thinking about where you want it to eventually end up. If you're wanting to move to git with the rest of us, the WMF git repo will obviously welcome any existing svn project. If you're wanting to stick with svn or go another route--let me know, I'll be happy to work with you to figure something out.
-Chad
(CC'ing Merlijn as he's been my liaison to pywikipediabot on this topic.)
Thanks for the timeline on that, Chad. I've added that mid-2013 hope to
https://www.mediawiki.org/wiki/Git/Conversion#Affected_development_projects
I want that to be a list of all the projects that live on svn.wikimedia.org (fundraising, Wikimedia analytics, mwdumper, etc.) so we can systematically reach out to them and help them figure out what to do.
I'm also collecting open git migration questions today at
https://www.mediawiki.org/wiki/Talk:Git/Conversion/issues#Topics_for_Chad.27...
so that Chad and I can work on an email and a blog post about 8 hours from now. Feel free to add your questions. (Example: "what about gerrit? when will it be not a UX nightmare?")
-- Sumana Harihareswara Volunteer Development Coordinator Wikimedia Foundation
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Feb 10, 2012, at 8:50 AM, Sumana Harihareswara wrote:
On 02/10/2012 08:10 AM, Chad wrote:
On Wed, Feb 1, 2012 at 8:15 PM, Arthur Richards arichards@wikimedia.org wrote:
Thanks for sharing these Sumana. One thing I haven't really seen mentioned anywhere is what's going to happen with the WIkimedia repository (rather than the MediaWiki repository) and how that figures into the roadmap/conversion schedule. Chad or Sumana, can you shed some light on this?
I totally want to apologize for delaying in responding on this. Somehow it got past my inbox and I missed it until I was looking for another e-mail.
The main point I want to reiterate for everyone is that there is no deadline. MediaWiki and its extensions are on a pretty firm migration path right now and we've got dates pencilled in for when we'd like to finish converting. Converting these parts of the repository are going to be the hardest anyway, so once we've cleared that hump hopefully it'll be trivial to migrate the remaining projects.
I know fundraising is on a different schedule than everyone else, so that's perfectly ok and we can look to a time after the main migrations are done that will work well for you guys too.
I've mentioned this number in a couple of places, but I think I'll say it here too so everyone's clear: I'd like to have the svn repository completely read-only by this time next year (very roughly). If you've got a project that you're maintaining in svn (other than core or extensions), now's the time to start thinking about where you want it to eventually end up. If you're wanting to move to git with the rest of us, the WMF git repo will obviously welcome any existing svn project. If you're wanting to stick with svn or go another route--let me know, I'll be happy to work with you to figure something out.
-Chad
(CC'ing Merlijn as he's been my liaison to pywikipediabot on this topic.)
Thanks for the timeline on that, Chad. I've added that mid-2013 hope to
https://www.mediawiki.org/wiki/Git/Conversion#Affected_development_projects
I want that to be a list of all the projects that live on svn.wikimedia.org (fundraising, Wikimedia analytics, mwdumper, etc.) so we can systematically reach out to them and help them figure out what to do.
I'm also collecting open git migration questions today at
https://www.mediawiki.org/wiki/Talk:Git/Conversion/issues#Topics_for_Chad.27...
so that Chad and I can work on an email and a blog post about 8 hours from now. Feel free to add your questions. (Example: "what about gerrit? when will it be not a UX nightmare?")
-- Sumana Harihareswara Volunteer Development Coordinator Wikimedia Foundation
On 02/10/2012 06:12 PM, Gregory Varnum wrote:
I might be missing something obvious - but when should/can extension
developers start working on git vs. svn?
Should we wait for the conversion of all extensions or will the
extensions area of the repo be available for use beforehand for any new extensions?
-greg aka varnent
Good question, Greg!
We had a meeting today to lay out some of the open questions and the answers. Really raw notes are at
https://www.mediawiki.org/wiki/Talk:Git/Conversion/issues#Topics_for_Chad.27...
It was so long that the email and blog post are going to happen early next week instead. But here are some key points:
* Please stop creating new extensions in Subversion, right now. It makes Chad's migration work more error-prone. If you want to create a new extension in git, ask Chad and he'll help you.
* Extension developers can start learning git now and will have a choice to move to git starting in March.
* As far as Chad knows, none of the gerrit bugs listed in https://labsconsole.wikimedia.org/wiki/Gerrit_bugs_that_matter are bad enough to block the migration. What, *specifically*, do people hate about gerrit? He needs to know specifics to make decisions that might delay the migration.
* In Chad's view, we can't use Phabricator as it is, mainly because it doesn't have LDAP support (and possibly doesn't have per-project permissions -- this needs investigating), and it is unlikely that our community can get those back-end features added and tested, to a quality comparable to gerrit's support for those features, by March. However, three months after MW core switches to git-and-gerrit, we'll have a serious tools overview and evaluate gerrit, git-review, gitweb, and all the other parts of our code management infrastructure, and revisit the gerrit decision. At that time we might find that Phabricator is a better fit, and git repos are more portable so switching would be feasible.
* There will be gerrit training documents available before MW core moves, and some kind of screen-sharing training by or before Monday, Feb. 27th.
* Chad is trying to get the real git repo up and running ASAP so people can start doing their real work there. He and RobLa believe we should encourage people to consider SVN indefinitely slushed. If you are going to do large things, they prefer you start doing them in git.
Please feel free to reply with questions in this thread, or start a new topic on that talk page. Thanks.
On Fri, Feb 10, 2012 at 8:51 PM, Sumana Harihareswara sumanah@wikimedia.org wrote:
- Chad is trying to get the real git repo up and running ASAP so people
can start doing their real work there. He and RobLa believe we should encourage people to consider SVN indefinitely slushed. If you are going to do large things, they prefer you start doing them in git.
Just a quick note before I head off to bed for the night. I've dropped the latest conversion of core with all tags into gerrit. You can browse it via gitweb at[0] and clone via [1] for LDAP users or [2] for anonymous users.
-Chad
[0] https://gerrit.wikimedia.org/r/gitweb?p=test/mediawiki/core2.git;a=summary [1] ssh://<user>@gerrit.wikimedia.org/test/mediawiki/core2.git [2] https://gerrit.wikimedia.org/r/p/test/mediawiki/core2.git
On Fri, Feb 10, 2012 at 8:31 PM, Chad innocentkiller@gmail.com wrote:
[1] ssh://<user>@gerrit.wikimedia.org/test/mediawiki/core2.git
I think this one is actually on port 29418.
This will work: ssh://<user>@gerrit.wikimedia.org:29418/test/mediawiki/core2.git
Or, add the following to ~/.ssh/config (the "User" line is only necessary if different from your local username) Host gerrit.wikimedia.org User <user> Port 29418
and then this will work: git clone gerrit.wikimedia.org:test/mediawiki/core2.git
~Rusty
On 11 February 2012 03:51, Sumana Harihareswara sumanah@wikimedia.org wrote:
We had a meeting today to lay out some of the open questions and the answers. Really raw notes are at
https://www.mediawiki.org/wiki/Talk:Git/Conversion/issues#Topics_for_Chad.27...
It was so long that the email and blog post are going to happen early next week instead. But here are some key points:
- Please stop creating new extensions in Subversion, right now. It
makes Chad's migration work more error-prone. If you want to create a new extension in git, ask Chad and he'll help you.
Since you already are suggesting people to start git repositories, we should prepare the i18n support right now. Translatewiki.net is not ready to handle random git repositories all over the world (nor it necessarily wants to do that). MediaWIki has had very good i18n and l10n including extensions - I don't want to lose that. So we need to plan it now, not when we have hundreds of extensions migrated to git and every one doing the things differently.
If our commit access is restricted (committing updated translations, but also fixing i18n issues) we cannot do our job properly, which means that we will start dropping support for stuff that doesn't handle i18n adequately.
Which reminds me, does LocalisationUpdate support git? Or Extension Distributor?
- As far as Chad knows, none of the gerrit bugs listed in
https://labsconsole.wikimedia.org/wiki/Gerrit_bugs_that_matter are bad enough to block the migration. What, *specifically*, do people hate about gerrit? He needs to know specifics to make decisions that might delay the migration.
I read through almost all of the commit mails. If I need to spend any more time on it or do more clicking around my head will probably explode. This means that I want commit emails with links to review tool, review tool should load fast, display all diffs inline (not hidden behind links or anything) and I should be able to filter commits by path or author. In addition I want to be able to tag commits (for l10n team to review or keeping list of commits I want to deploy to the live site). Any failure to meet these requirements is highly likely to be a blocker for me. Oh and something that signs that this commit has followups that fixes problems in it.
- Chad is trying to get the real git repo up and running ASAP so people
can start doing their real work there. He and RobLa believe we should encourage people to consider SVN indefinitely slushed. If you are going to do large things, they prefer you start doing them in git.
We have been in a slush most of this year already and it has affected the work of our team. We have already started to ignore the slush and will probably do it again. On the other hand we probably will migrate our extensions to git among the first ones.
Are extensions used by WMF forced to have the gated trunk model?
How much space do the git repos require? Right now we spend under 500M to support the i18n of three branches of mediawiki and all extensions.
-Niklas
On Sat, Feb 11, 2012 at 7:20 PM, Niklas Laxström niklas.laxstrom@gmail.com wrote:
Which reminds me, does LocalisationUpdate support git? Or Extension Distributor?
AFAIK No and No, For the latter I believe Tim (As to my understanding when I last saw that discussed) is going to set up a RO Git -> SVN mirror/gateway for that functionality, And since I saw that discussed there has also been several end users comment in the channel about wanting a mirror of phase3 as well.
Has the work been done, so the cluster is ready to support the base being managed in Git compared to SVN?
Hi Niklas
Comments inline...
On Sat, Feb 11, 2012 at 1:20 AM, Niklas Laxström niklas.laxstrom@gmail.com wrote:
Since you already are suggesting people to start git repositories, we should prepare the i18n support right now. Translatewiki.net is not ready to handle random git repositories all over the world (nor it necessarily wants to do that). MediaWIki has had very good i18n and l10n including extensions - I don't want to lose that. So we need to plan it now, not when we have hundreds of extensions migrated to git and every one doing the things differently.
Fair enough. I think we should handle new extension directories on a case-by-case basis.
If our commit access is restricted (committing updated translations, but also fixing i18n issues) we cannot do our job properly, which means that we will start dropping support for stuff that doesn't handle i18n adequately.
Which reminds me, does LocalisationUpdate support git?
Not yet: https://bugzilla.wikimedia.org/show_bug.cgi?id=34137
Or Extension Distributor?
Not yet: https://bugzilla.wikimedia.org/show_bug.cgi?id=27812
- As far as Chad knows, none of the gerrit bugs listed in
https://labsconsole.wikimedia.org/wiki/Gerrit_bugs_that_matter are bad enough to block the migration. What, *specifically*, do people hate about gerrit? He needs to know specifics to make decisions that might delay the migration.
I read through almost all of the commit mails. If I need to spend any more time on it or do more clicking around my head will probably explode. This means that I want commit emails with links to review tool, review tool should load fast, display all diffs inline (not hidden behind links or anything) and I should be able to filter commits by path or author. In addition I want to be able to tag commits (for l10n team to review or keeping list of commits I want to deploy to the live site). Any failure to meet these requirements is highly likely to be a blocker for me. Oh and something that signs that this commit has followups that fixes problems in it.
I copied the issues you raised to the "Gerrit bugs that matter" page. I also added a tracking bug to make sure we triage that list: https://bugzilla.wikimedia.org/show_bug.cgi?id=34349
- Chad is trying to get the real git repo up and running ASAP so people
can start doing their real work there. He and RobLa believe we should encourage people to consider SVN indefinitely slushed. If you are going to do large things, they prefer you start doing them in git.
We have been in a slush most of this year already and it has affected the work of our team. We have already started to ignore the slush and will probably do it again.
This comment really bears scrutiny. You realize that Chad isn't going to be able to fix the Gerrit/Git issues above if we're mired in a sea of code review, right? Or fixing bugs introduced because some deprecated feature had to be completely removed *right now*, despite the fact that it wasn't harming anyone to keep it around? It's not like your code is actually going to get deployed if you slow down the process this way.
I think rather than ignore Brion's email here: http://thread.gmane.org/gmane.science.linguistics.wikipedia.technical/58622/...
...you should respond to it.
On the other hand we probably will migrate our extensions to git among the first ones.
Are extensions used by WMF forced to have the gated trunk model?
This is a good question. I don't see how we can avoid it. Gerrit doesn't work with post-commit review, and we make a point of reviewing the extensions that are going onto the cluster.
How much space do the git repos require? Right now we spend under 500M to support the i18n of three branches of mediawiki and all extensions.
My understanding is that the core repo is about 100M with the full revision history. I don't know what it looks like with all extensions, but I have to believe it'll be manageable.
Rob
On Sun, Feb 12, 2012 at 1:40 AM, Rob Lanphier robla@wikimedia.org wrote:
How much space do the git repos require? Right now we spend under 500M to support the i18n of three branches of mediawiki and all extensions.
My understanding is that the core repo is about 100M with the full revision history. I don't know what it looks like with all extensions, but I have to believe it'll be manageable.
The reason the core repo is so large is because it's tracking the history of 19 branches and 161 tags ;-)
-Chad
On Sun, Feb 12, 2012 at 7:40 AM, Rob Lanphier robla@wikimedia.org wrote:
Which reminds me, does LocalisationUpdate support git?
Not yet: https://bugzilla.wikimedia.org/show_bug.cgi?id=34137
That bug is not about LocalisationUpdate. LU is not a TranslateWiki tool, it's a WMF-side extension for pulling in up-to-date translations (which happen to come from TWN, but it doesn't care about that). See https://www.mediawiki.org/wiki/Extension:LocalisationUpdate for documentation (details are slightly outdated, but the main points are accurate).
LocalisationUpdate has SVN support built-in, but WMF stopped using that a long time ago due to performance issues. Instead, the wrapper script that we use to run the LU update on the cluster updates a local checkout of the SVN repository, then passes the path as a command-line parameter to the LU update script. This wrapper script is managed by puppet, and is in the puppet git repository [1].
I read through almost all of the commit mails. If I need to spend any more time on it or do more clicking around my head will probably explode. This means that I want commit emails with links to review tool,
These exist. Gerrit sends notifications when an event occurs (creation of a new change, comment/review on an existing change, new version of an existing change, change merged), and it sends these notifications to all users that have commented on the change, as well as the author. I believe that Mark B gets e-mail notifications for all new changes in the puppet repository (I think he said this once, not sure), so that would seem to mean that setting up a mailing list that all notifications go to is possible.
These notification e-mails have links to the change in Gerrit, and also include a description of who did what (e.g. "Mark Bergsma submitted this change and it was merged") and, for comment notifications, the text of the comment.
review tool should load fast, display all diffs inline (not hidden behind links or anything)
That's a problem. I also want a full diff (you can't get that with Gerrit currently, only per-file diffs) that's displayed inline (can't do this either, diffs are displayed in new tabs). The only workaround that exists is that clicking the (kind of hidden) "Diff All Unified" button will load all of the per-file diffs at once, i.e. it will open a new tab for each file in the diff. I find this sort of bearable, but it's really not nce.
This is the first thing I tried to fix once I got my hands on a working dev install of Gerrit, but since I don't have that much Java experience, I didn't really get anywhere.
and I should be able to filter commits by path or author.
Mark does post-commit (after the fact) review of puppet changes sometimes, and he's sort of indicated that that doesn't work very nicely; Mark, could you pitch in here and tell us what works for that and what's lacking?
Per-author filtering of commits is implemented for sure, and you can also filter for unmerged commits by a given author. There are no path filtering features that I know of, at least for unmerged commits. I'm pretty sure Gitweb can do path filtering for commits that have already been merged into the mainline.
In addition I want to be able to tag commits (for l10n team to review or keeping list of commits I want to deploy to the live site).
There is some notion of a "topic", which is a kind of tag that the committer applies when they submit their commit into Gerrit. The topic is usually the name of the branch the committer used locally, or a bug number, and it's used to group related changes. But you can't change the topic after submission AFAIK, and you can't have multiple topics for one revision. So it's not real tagging as it exists in CodeReview.
Any failure to meet these requirements is highly likely to be a blocker for me. Oh and something that signs that this commit has followups that fixes problems in it.
Follow-up marking is also lacking, as far as I know. The way I've been doing it is to include the change number or Gerrit URL of the followed-up revision in the commit summary, but that doesn't list the follow-up in the reverse direction (i.e. if I commit B as a follow-up to A, B will say it's a follow-up to A, but A will not say it was followed up on in B). It's true that follow-ups should be less common because commits can be amended while they're being reviewed (so there will be no follow-up commits for things that are caught during review), but of course reviewers aren't perfect, so if something passes review (and unit tests!) but is still broken in some way, it'll still need a separate follow-up commit.
I copied the issues you raised to the "Gerrit bugs that matter" page. I also added a tracking bug to make sure we triage that list: https://bugzilla.wikimedia.org/show_bug.cgi?id=34349
I've edited that page a bit to add the notes I added above. Specifically, having notification e-mails is not a "Gerrit bug that matters", it's a feature that has already been implemented. Please only add things to the bugs list after you've verified they're not actually already in Gerrit :)
Roan
On Sun, Feb 12, 2012 at 11:36 PM, Roan Kattouw roan.kattouw@gmail.com wrote:
LocalisationUpdate has SVN support built-in, but WMF stopped using that a long time ago due to performance issues. Instead, the wrapper script that we use to run the LU update on the cluster updates a local checkout of the SVN repository, then passes the path as a command-line parameter to the LU update script. This wrapper script is managed by puppet, and is in the puppet git repository [1].
Now with URL:
[1] https://gerrit.wikimedia.org/r/gitweb?p=operations/puppet.git;a=blob_plain;f...
P.S.: Gitweb pages load slowly, and for some reason they seem to disable the bfcache in Firefox. These issues should be looked into as well.
These exist. Gerrit sends notifications when an event occurs (creation of a new change, comment/review on an existing change, new version of an existing change, change merged), and it sends these notifications to all users that have commented on the change, as well as the author. I believe that Mark B gets e-mail notifications for all new changes in the puppet repository (I think he said this once, not sure), so that would seem to mean that setting up a mailing list that all notifications go to is possible.
Yep. This is possible. In fact, we do this with our internal operations list for puppet. It also lets you subscribe to repositories with a number of options. That said, the emails are both a little spammy, and a little terse.
These notification e-mails have links to the change in Gerrit, and also include a description of who did what (e.g. "Mark Bergsma submitted this change and it was merged") and, for comment notifications, the text of the comment.
Yeah, this is why it's a little terse. A full inline diff in the email would rock. I checked, this is a code change :(.
Mark does post-commit (after the fact) review of puppet changes sometimes, and he's sort of indicated that that doesn't work very nicely; Mark, could you pitch in here and tell us what works for that and what's lacking?
There's no tagging, like we have in CodeReview. You can only comment - can't -1 an already merged change (which makes sense). We really need to add freeform tagging support.
Per-author filtering of commits is implemented for sure, and you can also filter for unmerged commits by a given author. There are no path filtering features that I know of, at least for unmerged commits. I'm pretty sure Gitweb can do path filtering for commits that have already been merged into the mainline.
Here's the full search capabilities. It looks like a lot of what we need is there, but you need to know the search commands:
http://gerrit.googlecode.com/svn/documentation/2.1.6/user-search.html
- Ryan
Thanks for the response, Chad.
I know fundraising is on a different schedule than everyone else, so that's
perfectly ok and we can look to a time after the main migrations are done that will work well for you guys too.
The Wikimedia repository is used by others besides just fundraising, although the fundraising team is probably its heaviest user. I know that some folks from Wikimedia Sweden are using it, as is Ryan Faulkner for community analytics, and I believe Nimish has been using at as well.
That said, I imagine timing of a switchover will be most important to coordinate with the fundraising team. I'm no longer the engineering lead for fundraising (I've moved to the mobile team), so I suggest coordinating with Katie Horn.
read-only by this time next year (very roughly). If you've got a project
that you're maintaining in svn (other than core or extensions), now's the time to start thinking about where you want it to eventually end up. If you're wanting to move to git with the rest of us, the WMF git repo will obviously welcome any existing svn project. If you're wanting to stick with svn or go another route--let me know, I'll be happy to work with you to figure something out.
IMO, it would be best to have the Wikimedia repo also completely converted to Git. I imagine it will be a big logistical headache attempting to maintain both SVN and Git repositories. But thank you for being willing to be flexible :)
On Sat, Feb 11, 2012 at 5:11 AM, Arthur Richards arichards@wikimedia.org wrote:
The Wikimedia repository is used by others besides just fundraising, although the fundraising team is probably its heaviest user. I know that some folks from Wikimedia Sweden are using it, as is Ryan Faulkner for community analytics, and I believe Nimish has been using at as well.
The proper course of action here is probably to split these things up into multiple repositories. That way each of these parties could move to git on their own terms.
Roan
wikitech-l@lists.wikimedia.org