Hi everyone,
This email is going to briefly describe the old SVN workflow, and then use that as a baseline to describe what we should do for Git. I haven't had a chance to coordinate this mail with Chad (or anyone else), so I'll reserve the right for him to completely contradict me here. This is meant to provoke a discussion about how we're really going to use Git, and to establish a plan for taking advantage of the new workflow to move to much more frequent deployments.
In the old world, we had this:
trunk ├── REL1_17 │ └── 1.17wmf1 (branched from REL1_17) ├── REL1_18 │ └── 1.18wmf1 (branched from REL1_18) └── REL1_19 └── 1.19wmf1 (branched from REL1_19)
Tarball releases would come out of the respective REL1_xx branches, and deployments would come out of the 1.xxwmf1 branches. REL1_xx branches have all extensions, and 1.xxwmf1 branches have only Wikimedia production code. Each would be a relatively long lived branch (6-18 months) into which critical fixes and priority features would be merged from trunk.
Looking ahead to deployments, there's a couple of different ways to go about this:
One plan would be to have a "wmf" branch that does not trail far behind the master. The extensions we deploy to the cluster can be included as submodules for that given branch. The process for deployment at that point will be "merge from master" or "update submodule reference" on the wmf branch. Then on fenari, you will git pull and git submodule update before scapping like you're currently used to. The downside of this approach is that there's not an obvious way to have multiple production branches in play (heterogeneous deploy). Seems solvable (e.g wmf1, wmf2, etc), but that also seems messy.
Another possible plan would be to have something *somewhat* closer to what we have today, with new branches off of trunk for each deployment, and deployments happening as frequently as weekly.
master ├── 1.20wmf01 ├── 1.20wmf02 ├── 1.20wmf03 ... ├── 1.20wmf11 ├── 1.20wmf12 ├── REL1_20 ├── 1.21wmf01 ├── 1.21wmf02 ├── 1.21wmf03 ...
This is how I was envisioning the process working, and just didn't get a chance to sync up with Chad to find out what the issues of this approach would be.
Since we don't have an imminent deployment coming from Git, we have a little time to figure this situation out.
Regardless of the branching strategy, the goal would be to start as early as April with much more frequent deployments to production. The deployment plan would look something like this: * Deploy 1.20wmf01 to test2 real soon now (say, no later than April 16). * Deploy 1.20wmf01 to mediawiki.org a couple deploy days after that ("deploy day" meaning Monday through Thursday) * Let simmer for some short-ish amount of time (TBD) * Roll out 1.20wmf01 to more wikis, eventually making it to all of them
Given the way APC caches and other caching works, I suspect we can't get away with having more than two simultaneous versions out on the production cluster, but we could conceivably have a situation where, for example, a deploy day or two after rolling out 1.20wmf01 out to the last of the wikis, we then roll out 1.20wmf02 out to test2.
This topic is partially covered here: https://www.mediawiki.org/wiki/Git/Workflow#Who_can_review.3F_Gerrit_project...
...but I imagine we'll probably need to revise that based on this conversation and perhaps break this out into a separate page.
There's a few of us that plan to meet in a couple of weeks to formalize something here, but perhaps we can get this all hammered out on-list prior to that.
Thoughts on this process? Rob
I kinda like the first model (it feels more organic), but the second has the advantage that you've got a previous branch to revert to quickly if needed. That's a plus for deployment work.
-- brion On Mar 22, 2012 12:44 PM, "Rob Lanphier" robla@wikimedia.org wrote:
Hi everyone,
This email is going to briefly describe the old SVN workflow, and then use that as a baseline to describe what we should do for Git. I haven't had a chance to coordinate this mail with Chad (or anyone else), so I'll reserve the right for him to completely contradict me here. This is meant to provoke a discussion about how we're really going to use Git, and to establish a plan for taking advantage of the new workflow to move to much more frequent deployments.
In the old world, we had this:
trunk ├── REL1_17 │ └── 1.17wmf1 (branched from REL1_17) ├── REL1_18 │ └── 1.18wmf1 (branched from REL1_18) └── REL1_19 └── 1.19wmf1 (branched from REL1_19)
Tarball releases would come out of the respective REL1_xx branches, and deployments would come out of the 1.xxwmf1 branches. REL1_xx branches have all extensions, and 1.xxwmf1 branches have only Wikimedia production code. Each would be a relatively long lived branch (6-18 months) into which critical fixes and priority features would be merged from trunk.
Looking ahead to deployments, there's a couple of different ways to go about this:
One plan would be to have a "wmf" branch that does not trail far behind the master. The extensions we deploy to the cluster can be included as submodules for that given branch. The process for deployment at that point will be "merge from master" or "update submodule reference" on the wmf branch. Then on fenari, you will git pull and git submodule update before scapping like you're currently used to. The downside of this approach is that there's not an obvious way to have multiple production branches in play (heterogeneous deploy). Seems solvable (e.g wmf1, wmf2, etc), but that also seems messy.
Another possible plan would be to have something *somewhat* closer to what we have today, with new branches off of trunk for each deployment, and deployments happening as frequently as weekly.
master ├── 1.20wmf01 ├── 1.20wmf02 ├── 1.20wmf03 ... ├── 1.20wmf11 ├── 1.20wmf12 ├── REL1_20 ├── 1.21wmf01 ├── 1.21wmf02 ├── 1.21wmf03 ...
This is how I was envisioning the process working, and just didn't get a chance to sync up with Chad to find out what the issues of this approach would be.
Since we don't have an imminent deployment coming from Git, we have a little time to figure this situation out.
Regardless of the branching strategy, the goal would be to start as early as April with much more frequent deployments to production. The deployment plan would look something like this:
- Deploy 1.20wmf01 to test2 real soon now (say, no later than April 16).
- Deploy 1.20wmf01 to mediawiki.org a couple deploy days after that
("deploy day" meaning Monday through Thursday)
- Let simmer for some short-ish amount of time (TBD)
- Roll out 1.20wmf01 to more wikis, eventually making it to all of them
Given the way APC caches and other caching works, I suspect we can't get away with having more than two simultaneous versions out on the production cluster, but we could conceivably have a situation where, for example, a deploy day or two after rolling out 1.20wmf01 out to the last of the wikis, we then roll out 1.20wmf02 out to test2.
This topic is partially covered here:
https://www.mediawiki.org/wiki/Git/Workflow#Who_can_review.3F_Gerrit_project...
...but I imagine we'll probably need to revise that based on this conversation and perhaps break this out into a separate page.
There's a few of us that plan to meet in a couple of weeks to formalize something here, but perhaps we can get this all hammered out on-list prior to that.
Thoughts on this process? Rob
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Thu, Mar 22, 2012 at 3:57 PM, Brion Vibber bvibber@wikimedia.org wrote:
I kinda like the first model (it feels more organic), but the second has the advantage that you've got a previous branch to revert to quickly if needed. That's a plus for deployment work.
-- brion
I kinda like the idea of both. We need a wmf-branch (specific hacks for us, ability to trail master when need be), but having tags is super useful too.
Why not just have the branch but make a new tag before any scap? Best of both worlds ;-)
-Chad
On Fri, Mar 23, 2012 at 7:33 AM, Chad innocentkiller@gmail.com wrote:
On Thu, Mar 22, 2012 at 3:57 PM, Brion Vibber bvibber@wikimedia.org wrote:
I kinda like the first model (it feels more organic), but the second has the advantage that you've got a previous branch to revert to quickly if needed. That's a plus for deployment work.
-- brion
I kinda like the idea of both. We need a wmf-branch (specific hacks for us, ability to trail master when need be), but having tags is super useful too.
Well, we need two wmf-branches when we're mid-deploy, which is going to be far more frequently in the new regime. When mediawiki.org, meta, commons, and nlwiki are on 1.20wmf03 for a week, while enwiki and others are on 1.20wmf02, we'll need to be mindful of what we might backport to both branches.
Now, in this new model, backporting in general should be far, far less frequent, and only done for the most urgent bugfixes.
Why not just have the branch but make a new tag before any scap? Best of both worlds ;-)
There are almost certainly going to be times when the wmf branch isn't linear (e.g. a security fix needs to be deployed to all wikis, even though we're not ready to push all wikis to the tip of the wmf branch)
Rob
On Fri, Mar 23, 2012 at 12:57 PM, Rob Lanphier robla@wikimedia.org wrote:
On Fri, Mar 23, 2012 at 7:33 AM, Chad innocentkiller@gmail.com wrote:
On Thu, Mar 22, 2012 at 3:57 PM, Brion Vibber bvibber@wikimedia.org wrote:
I kinda like the first model (it feels more organic), but the second has the advantage that you've got a previous branch to revert to quickly if needed. That's a plus for deployment work.
-- brion
I kinda like the idea of both. We need a wmf-branch (specific hacks for us, ability to trail master when need be), but having tags is super useful too.
Well, we need two wmf-branches when we're mid-deploy, which is going to be far more frequently in the new regime. When mediawiki.org, meta, commons, and nlwiki are on 1.20wmf03 for a week, while enwiki and others are on 1.20wmf02, we'll need to be mindful of what we might backport to both branches.
Then a branch for each one we're deploying. No harm in having wmf-1.20 and wmf-1.19 at the same time.
Now, in this new model, backporting in general should be far, far less frequent, and only done for the most urgent bugfixes.
Agreed. We want to keep them as close to master as possible as it is.
Why not just have the branch but make a new tag before any scap? Best of both worlds ;-)
There are almost certainly going to be times when the wmf branch isn't linear (e.g. a security fix needs to be deployed to all wikis, even though we're not ready to push all wikis to the tip of the wmf branch)
I disagree here. The wmf branches should always be linear, and you should never merge to it without accepting that it may get scapped.
-Chad
On Fri, Mar 23, 2012 at 10:03 AM, Chad innocentkiller@gmail.com wrote:
I disagree here. The wmf branches should always be linear, and you should never merge to it without accepting that it may get scapped.
Actually, you should never merge to master without accepting that it may get scapped. Why have two layers here?
Rob
On Fri, Mar 23, 2012 at 1:11 PM, Rob Lanphier robla@wikimedia.org wrote:
On Fri, Mar 23, 2012 at 10:03 AM, Chad innocentkiller@gmail.com wrote:
I disagree here. The wmf branches should always be linear, and you should never merge to it without accepting that it may get scapped.
Actually, you should never merge to master without accepting that it may get scapped. Why have two layers here?
Well true, but I still don't see an instance where you'd merge to master but *not* be willing to have it deployed.
-Chad
On Fri, Mar 23, 2012 at 10:13 AM, Chad innocentkiller@gmail.com wrote:
On Fri, Mar 23, 2012 at 1:11 PM, Rob Lanphier robla@wikimedia.org wrote:
On Fri, Mar 23, 2012 at 10:03 AM, Chad innocentkiller@gmail.com wrote:
I disagree here. The wmf branches should always be linear, and you should never merge to it without accepting that it may get scapped.
Actually, you should never merge to master without accepting that it may get scapped. Why have two layers here?
Well true, but I still don't see an instance where you'd merge to master but *not* be willing to have it deployed.
Chad and I discussed this, and I think we figure out where we were talking past each other. I'll let him explain in more detail when he gets a chance, but the short answer is we'll probably be doing something roughly like I described as the second plan in my original email, with the addition of liberal use of tagging. We could even *try* to get away with tagging instead of branching, and then only branch when necessary.
Rob
Rob Lanphier robla@wikimedia.org wrote:
On Fri, Mar 23, 2012 at 10:13 AM, Chad innocentkiller@gmail.com wrote:
On Fri, Mar 23, 2012 at 1:11 PM, Rob Lanphier robla@wikimedia.org wrote:
On Fri, Mar 23, 2012 at 10:03 AM, Chad innocentkiller@gmail.com wrote:
I disagree here. The wmf branches should always be linear, and you should never merge to it without accepting that it may get scapped.
Actually, you should never merge to master without accepting that it may get scapped. Why have two layers here?
Well true, but I still don't see an instance where you'd merge to master but *not* be willing to have it deployed.
Chad and I discussed this, and I think we figure out where we were talking past each other. I'll let him explain in more detail when he gets a chance, but the short answer is we'll probably be doing something roughly like I described as the second plan in my original email, with the addition of liberal use of tagging. We could even *try* to get away with tagging instead of branching, and then only branch when necessary.
How "git committers" (i.e. mere mortals) will be able to propose changes to be deployed in wmf production (a.k.a. "1.19wmf1" tag in [[Special:Code]]?
If we plan to use gerrit for that (submitting commits for review in the "wmf" branches by the means of refs/for/wmf-something), I would ask for one thing - can we full go end-to-end and test the workflow with actual commands?
I have noticed potential issues with gerrit+git combination vs pure git workflow:
* It seems I can't have a longer running branch with my changes on some topic - those commits cannot be easily submitted to gerrit (since my branch will likely contain merges from master and we don't want't them to be resubmitted via the dependency tree)
* It seems necessary to change "Change-Id" everywhere for "git cherry-pick" to work without -n - it seems that cherry-pick does not invoke our commit-msg hook when committing. Neither does "git merge".
Those things limit possibilities in moving commits between branches (official or local ones), so I think it would be good to review the process in the test tree using some actual commands (and with some test users that don't have privilege to cross the gated trunk involved).
More on the second case in separate thread.
//Saper
On Mar 22, 2012 3:44 PM, "Rob Lanphier" robla@wikimedia.org wrote:
Another possible plan would be to have something *somewhat* closer to what we have today, with new branches off of trunk for each deployment, and deployments happening as frequently as weekly.
master ├── 1.20wmf01 ├── 1.20wmf02 ├── 1.20wmf03 ... ├── 1.20wmf11 ├── 1.20wmf12 ├── REL1_20 ├── 1.21wmf01 ├── 1.21wmf02 ├── 1.21wmf03 ...
This is how I was envisioning the process working, and just didn't get a chance to sync up with Chad to find out what the issues of this approach would be.
This seems like the best approach to me, because it meshes with het deploy better, as described by others on this thread. However, I'd recommend using a naming convention with a prefix, e.g. wmf/1.20/01 , so we can handle permissions on these deployment branches in a managable way.
Roan
On Sun, Mar 25, 2012 at 7:05 AM, Roan Kattouw roan.kattouw@gmail.com wrote:
On Mar 22, 2012 3:44 PM, "Rob Lanphier" robla@wikimedia.org wrote:
Another possible plan would be to have something *somewhat* closer to what we have today, with new branches off of trunk for each deployment, and deployments happening as frequently as weekly.
master ├── 1.20wmf01 ├── 1.20wmf02 ├── 1.20wmf03 ... ├── 1.20wmf11 ├── 1.20wmf12 ├── REL1_20 ├── 1.21wmf01 ├── 1.21wmf02 ├── 1.21wmf03 ...
This is how I was envisioning the process working, and just didn't get a chance to sync up with Chad to find out what the issues of this approach would be.
This seems like the best approach to me, because it meshes with het deploy better, as described by others on this thread. However, I'd recommend using a naming convention with a prefix, e.g. wmf/1.20/01 , so we can handle permissions on these deployment branches in a managable way.
*nod* Makes sense. And Rob and I totally agree actually with how we should be handling deployments. We should branch from master liberally and often. When it comes time to do an actual scap, you should also tag your release so we have a clearly repeatable state if we need to roll-back or figure out what broke.
Ideally, scap would become something like: `mw-deploy <tag> <wiki1,wiki2>`
-Chad
On Thu, Mar 22, 2012 at 12:44 PM, Rob Lanphier robla@wikimedia.org wrote:
Regardless of the branching strategy, the goal would be to start as early as April with much more frequent deployments to production. The deployment plan would look something like this:
- Deploy 1.20wmf01 to test2 real soon now (say, no later than April 16).
- Deploy 1.20wmf01 to mediawiki.org a couple deploy days after that
("deploy day" meaning Monday through Thursday)
- Let simmer for some short-ish amount of time (TBD)
- Roll out 1.20wmf01 to more wikis, eventually making it to all of them
Now that April is upon us, is there a more concrete timeline for getting deployment branches into Git and using them for deployment? As much as I love making patches between a Git repo and the deployment branch, I'm very eager for the migration as I expect it will make deployments much easier and less time consuming.
Also, what is the impact that this migration strategy will have on extension deployments? The mobile team makes code deployments very frequently and it would be helpful to be able to plan ahead around the migration timeline.
On Mon, Apr 2, 2012 at 9:23 PM, Arthur Richards arichards@wikimedia.org wrote:
Now that April is upon us, is there a more concrete timeline for getting deployment branches into Git and using them for deployment? As much as I love making patches between a Git repo and the deployment branch, I'm very eager for the migration as I expect it will make deployments much easier and less time consuming.
We've got a meeting Wednesday to work on this. There's some rough consensus already, so it should just be a matter finishing off the details.
Also, what is the impact that this migration strategy will have on extension deployments? The mobile team makes code deployments very frequently and it would be helpful to be able to plan ahead around the migration timeline.
The plan right now is to use extensions as submodules on the wmf branch. Updating your extension in deployment is just a matter of updating the referenced commit and then running `git submodule update` on fenari.
-Chad
Le 03/04/12 03:34, Chad a écrit :
The plan right now is to use extensions as submodules on the wmf branch. Updating your extension in deployment is just a matter of updating the referenced commit and then running `git submodule update` on fenari.
Which, as a side effect, means that extensions would have to either: - make sure master is production grade Or: - maintain a deployment branch
On Tue, Apr 3, 2012 at 1:20 AM, Antoine Musso hashar+wmf@free.fr wrote:
Le 03/04/12 03:34, Chad a écrit :
The plan right now is to use extensions as submodules on the wmf branch. Updating your extension in deployment is just a matter of updating the referenced commit and then running `git submodule update` on fenari.
Which, as a side effect, means that extensions would have to either: - make sure master is production grade Or: - maintain a deployment branch
Well we should all be submitting production grade code. Especially with pre-commit review ;-)
More seriously though: just as it is in SVN, you are expected to keep deployed code runnable at all times. This has never ever changed. If you need to develop some experimental feature that you don't want to deploy, that's what branches are for (and was a major reason we switched to begin with).
Regarding deployment branches...I'd really prefer this to not happen, since master should be the "this is the current stable version that I want people to use."
-Chad
On Thu, Mar 22, 2012 at 12:44 PM, Rob Lanphier robla@wikimedia.org wrote:
There's a few of us that plan to meet in a couple of weeks to formalize something here, but perhaps we can get this all hammered out on-list prior to that.
...and we've now done that. While it's not truly final, this is a last call before we start to implement this strategy.
Our plan, starting next week, is to have an initial 3 week deployment window, followed immediately by repeating 2 week windows.
The plan for the first window: * Week of April 9 - deploy to test2, and then to mediawiki.org * Week of April 16 - deploy to all non-Wikipedia sites (commons, Wiktionary, Wikisource, Wikinews, Wikibooks, Wikiquote, Wikiversity) * Week of April 23 - Monday: deploy to English Wikipedia, Wednesday: deploy to other Wikipedias
The deployment calendar we plan to use for subsequent deployments * Week 1: Monday: deploy to test2 and mediawiki.org * Week 1: Wednesday: non-Wikipedia projects * Week 2: Monday: English Wikipedia * Week 2: Wednesday: all other Wikipedias
We plan to branch once for each deployment cycle (soon every 2 weeks). We will branch off master, make a snapshot, immediately deploy to test2 and mediawiki.org. We'll use that branch for hot fixes for as long as that branch is in production. A branch will typically be in production for 4 weeks; the first 2 weeks as the "new" branch, and then another 2 weeks as the "old" branch.
As Roan pointed out earlier, the branch name needs to start with "wmf/", and suggested a naming scheme "wmf/1.xx/yy", where "yy" is the deployment number. The only thing that sucks about that is that it's easily confused with "1.xx.yy" release tarball versions (e.g. 1.18.1). Perhaps we can still do something like wmf/1.20wmf01.
Deployments will be core plus extensions, barring any exceptions. There are two reasons for exceptions:
* Extensions that need to be deployed on a slower schedule (if any). We're going to figure out how to migrate these extensions off of master and onto a development branch, so that master is available for deployment every two weeks. * Extensions that need to be deployed on a faster schedule (e.g. MobileFrontend). The teams responsible for these deployments will also be responsible for making sure they deploy their code to both of the appropriate production branches, since there will almost always be two in play.
Does this all make sense to everyone?
Rob
wikitech-l@lists.wikimedia.org