Hi.
Is there a status update about more regular code deployments to Wikimedia wikis? I know it's been discussed endlessly, but I was under the impression that it was a real goal going forward. Is that still the case?
MZMcBride
MZMcBride wrote:
Is there a status update about more regular code deployments to Wikimedia wikis? I know it's been discussed endlessly, but I was under the impression that it was a real goal going forward. Is that still the case?
Hmm, I guess not. Perhaps we'll soon see a return to the "before Wikimania" meme (shortly followed by the "after the fundraiser" meme).
MZMcBride
On Sun, May 29, 2011 at 9:15 AM, MZMcBride z@mzmcbride.com wrote:
MZMcBride wrote:
Is there a status update about more regular code deployments to Wikimedia wikis? I know it's been discussed endlessly, but I was under the impression that it was a real goal going forward. Is that still the case?
Hmm, I guess not. Perhaps we'll soon see a return to the "before Wikimania" meme (shortly followed by the "after the fundraiser" meme).
We updated the MediaWiki roadmap in Berlin: http://www.mediawiki.org/wiki/MediaWiki_roadmap
The tl;dr version: * 1.17 - May (we still might make it...we're very, very close) * 1.18 - July, with tarball release in August * 1.19 - October, with tarball release in November * 1.20 - January 2012, with tarball release in February
That's a very rough outline, and I put confidence numbers in there (e.g. 30% for 1.18) to emphasize that these are pretty rough projections.
As stated above: 1.17 is very, very close to being complete, and the 1.18 review queue is starting to look much better. Compare: 1.18: http://toolserver.org/~robla/crstats/crstats.118all.html 1.17: http://toolserver.org/~robla/crstats/crstats.117all.html
As you can see, there's still a lot of review to do for a 1.18 release, but it's not as out of control as 1.17 was. A 1.18 deployment in July is plausible, though comparing the graphs and mapping them to our actual 1.17 deployment, we're probably looking at August.
Rob
Rob Lanphier wrote:
We updated the MediaWiki roadmap in Berlin: http://www.mediawiki.org/wiki/MediaWiki_roadmap
The tl;dr version:
- 1.17 - May (we still might make it...we're very, very close)
- 1.18 - July, with tarball release in August
- 1.19 - October, with tarball release in November
- 1.20 - January 2012, with tarball release in February
Thank you for replying and thank you for updating the roadmap.
I think the main question in my message got lost, though. When I said "more regular" code deployments, I meant less than or equal to a month. Perhaps even weekly.
At one point or another, it was a goal to get back to that kind of more regular code deployments to the live sites (particularly given the issues that resulted from the large 1.17 upgrade). Is that still a goal?
MZMcBride
Based on the schedule given, I would deduct that it's no longer our goal to release as often as we may have once believed was ideal - and sometimes even achieved.
There's been tons of discussion about what is an ideal release schedule for us (probably literally if you printed it out at 12pt on standard copy paper). At this point I think we are balancing between fighting the bad habits that got us into the 1.17 rut, and trying to give our engineering team enough time to do significant work (rather than spending 1 day a week pushing code).
It's my expectation that this will always fluctuate, never reach anything resembling a stable rhythm, and that whether that's good or bad, that is and will continue to be reality.
If you feel that releasing frequently is important, as a non-reviewer without shell access and no official power to release anything, this must be frustrating. Maybe we could instead of bickering about when code should be released, come up with and support new roles that volunteers can take on that would accelerate the code review and release process.
Right now, volunteers can effectively slow it by committing code - not that that's in any way a bad thing of course, volunteer code contribution is incredibly valuable. But if volunteers can also help in other ways to push that code along the pipeline in productive ways, that would likely take much of the frustration out of this situation for everyone.
Any ideas?
- Trevor
On Tue, May 31, 2011 at 12:12 AM, MZMcBride z@mzmcbride.com wrote:
Rob Lanphier wrote:
We updated the MediaWiki roadmap in Berlin: http://www.mediawiki.org/wiki/MediaWiki_roadmap
The tl;dr version:
- 1.17 - May (we still might make it...we're very, very close)
- 1.18 - July, with tarball release in August
- 1.19 - October, with tarball release in November
- 1.20 - January 2012, with tarball release in February
Thank you for replying and thank you for updating the roadmap.
I think the main question in my message got lost, though. When I said "more regular" code deployments, I meant less than or equal to a month. Perhaps even weekly.
At one point or another, it was a goal to get back to that kind of more regular code deployments to the live sites (particularly given the issues that resulted from the large 1.17 upgrade). Is that still a goal?
MZMcBride
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On 5/31/11 10:03 AM, Trevor Parscal wrote:
It's my expectation that this will always fluctuate, never reach anything resembling a stable rhythm, and that whether that's good or bad, that is and will continue to be reality.
I understand how you feel, but that is unnecessarily pessimistic.
A lot of other projects of equal or greater size and complexity manage to release much more frequently. And there are just as many voices tugging in different directions.
So we know that the solution exists. We just have not taken the right steps to achieve that solution.
Any ideas?
Well, AFAIK our bottleneck is reviewing. I'm unaware of any concrete steps that have been taken to get a scalable reviewing system happening. I'm not saying nothing's been done, it's just that I've never seen anyone say "okay, here's the plan".
Maybe this sounds like I'm volunteering for it, and I guess I would, but it seems to me that the current users who can push have to come up with a system that *they* trust. Whatever bright idea I have is going to go down in flames if Tim et al. aren't fully committed to it and adequate sacrifices are made in other areas to adjust to a new system. If pushing more frequently isn't a priority for these users (or they are unable to find the time to fix the current system) then I don't know what to do either.
Are we all in deadlock or something? Are the users who can push waiting from some proposals/work from the rest of the community?
Followup -- I've conflated "pushing" and "releasing" in my post, but anyway I believe it's mostly the same circle of users who do both.
On 5/31/11 10:35 AM, Neil Kandalgaonkar wrote:
On 5/31/11 10:03 AM, Trevor Parscal wrote:
It's my expectation that this will always fluctuate, never reach anything resembling a stable rhythm, and that whether that's good or bad, that is and will continue to be reality.
I understand how you feel, but that is unnecessarily pessimistic.
A lot of other projects of equal or greater size and complexity manage to release much more frequently. And there are just as many voices tugging in different directions.
So we know that the solution exists. We just have not taken the right steps to achieve that solution.
Any ideas?
Well, AFAIK our bottleneck is reviewing. I'm unaware of any concrete steps that have been taken to get a scalable reviewing system happening. I'm not saying nothing's been done, it's just that I've never seen anyone say "okay, here's the plan".
Maybe this sounds like I'm volunteering for it, and I guess I would, but it seems to me that the current users who can push have to come up with a system that *they* trust. Whatever bright idea I have is going to go down in flames if Tim et al. aren't fully committed to it and adequate sacrifices are made in other areas to adjust to a new system. If pushing more frequently isn't a priority for these users (or they are unable to find the time to fix the current system) then I don't know what to do either.
Are we all in deadlock or something? Are the users who can push waiting from some proposals/work from the rest of the community?
Well, AFAIK our bottleneck is reviewing.
I don't think that's the only bottleneck. An unwillingness to release code is a sign of a lack of trust. Reviewed code is a source of trust. But there are other sources of trust: o code reviews o test cases (a bug isn't fixed until there's a test case which ensures that it won't come back.) o readable code. o beta testing that actually gets tested.
On 5/31/11, Trevor Parscal tparscal@wikimedia.org wrote:
There's been tons of discussion about what is an ideal release schedule for us (probably literally if you printed it out at 12pt on standard copy paper). At this point I think we are balancing between fighting the bad habits that got us into the 1.17 rut, and trying to give our engineering team enough time to do significant work (rather than spending 1 day a week pushing code).
One issue is that "time spent deploying" does not appear to be independent of "time between deployments." If it always took a day to deploy new code, that would be a good argument in favor of more time between deployments. But in practice, the more time that passes without pushing code to the site, the longer it takes to complete the deployment. Spending a week deploying every other month doesn't really save much time over spending a day every week.
-- Alex
On Tue, May 31, 2011 at 2:50 PM, Alex "Mr.Z-man" mrzmanwiki@gmail.comwrote:
On 5/31/11, Trevor Parscal tparscal@wikimedia.org wrote:
There's been tons of discussion about what is an ideal release schedule
for
us (probably literally if you printed it out at 12pt on standard copy paper). At this point I think we are balancing between fighting the bad habits that got us into the 1.17 rut, and trying to give our engineering team enough time to do significant work (rather than spending 1 day a
week
pushing code).
One issue is that "time spent deploying" does not appear to be independent of "time between deployments." If it always took a day to deploy new code, that would be a good argument in favor of more time between deployments. But in practice, the more time that passes without pushing code to the site, the longer it takes to complete the deployment. Spending a week deploying every other month doesn't really save much time over spending a day every week.
I'd actually consider it pretty ideal to spend a day every week deploying updates, gathering feedback, and making fixes. Rather than an argument in favor of longer cycles, I'd consider that an argument in favor of shorter cycles!
-- brion
Trevor Parscal wrote:
If you feel that releasing frequently is important, as a non-reviewer without shell access and no official power to release anything, this must be frustrating. Maybe we could instead of bickering about when code should be released, come up with and support new roles that volunteers can take on that would accelerate the code review and release process.
Yes, it's incredibly frustrating to the point that volunteer developers have walked away from MediaWiki code development. This is the "wiki project"; things are supposed to be fast! When someone makes a patch or a commit and the best we can say is "it might get looked at this year with a fifty percent confidence interval," that's not okay.
Code deployments to the live sites need to happen more often than quarterly. There are almost no benefits to such a long gap between deployments and the detriments are clear.
Is there a problem with saying hell or high water, we won't go more than a month without a deployment? I realize Wikimedia is a bit different, but taking a page out of the rest of the business world's book, you set a deadline and then it just fucking gets met. No excuses, no questions. The problem seems to be finding anyone to lay down the damn law.
MZMcBride
On Tue, May 31, 2011 at 3:20 PM, MZMcBride z@mzmcbride.com wrote:
Yes, it's incredibly frustrating to the point that volunteer developers have walked away from MediaWiki code development. This is the "wiki project"; things are supposed to be fast! When someone makes a patch or a commit and the best we can say is "it might get looked at this year with a fifty percent confidence interval," that's not okay.
Code deployments to the live sites need to happen more often than quarterly. There are almost no benefits to such a long gap between deployments and the detriments are clear.
Is there a problem with saying hell or high water, we won't go more than a month without a deployment? I realize Wikimedia is a bit different, but taking a page out of the rest of the business world's book, you set a deadline and then it just fucking gets met. No excuses, no questions. The problem seems to be finding anyone to lay down the damn law.
Sing it, brother! We're getting *some* stuff through quicker within the deployment branches, but not nearly as much as we ought to.
Whatever the overall release branching frequency, there's also an update cycle within that in getting fixes & independent features (usually extensions) pushed out to the wild, and that's the cycle that's often most important to users (because that's where their most immediate problems get fixed) & developers (because users are where we get feedback).
Fixes need to be hitting production within days at longest, and more features than just UploadWizard and ArticleFeedback should be able to make it out on their own schedule without being forced to wait for a total infrastructure update.
-- brion
On 5/31/11 3:20 PM, MZMcBride wrote:
taking a page out of the rest of the business world's book, you set a deadline and then it just fucking gets met. No excuses, no questions.
I think you have an optimistic view of how businesses actually work. :)
But, in any case, in a business, there is a Plan that everyone is trying to follow and in theory, deviations from that Plan are avoided. In our environment we want to be responsive to the schedule of a volunteer developer, who may be completely unaware or uninterested in our plans.
Perhaps the answer is that we have to give the volunteer developers some obvious pathway to harmonizing their and our priorities. Like, if you're working on files and multimedia, you should be emailing Bryan, me, or maybe Tim or Russell. Could it be that simple?
The problem seems to be finding anyone to lay down the damn law.
Well, it's not like wiki pages happen by someone cracking a whip either. That said, we would benefit from some urgency towards correcting the problem.
Neil Kandalgaonkar wrote:
On 5/31/11 3:20 PM, MZMcBride wrote:
taking a page out of the rest of the business world's book, you set a deadline and then it just fucking gets met. No excuses, no questions.
I think you have an optimistic view of how businesses actually work. :)
It's funny that you say that. When there was a hard deadline for the UploadWizard being deployed (November 30?), what happened? Was the deadline met?
It's not a matter of Wikimedia having the ability to set and meet deadlines; they've pretty clearly demonstrated that when something is important enough, it's not an issue. In this case, it's a matter of finding/creating the _willingness_ to set/meet the deadlines (and the lack of resulting consequences if/when deadlines are missed).
I occasionally show my face at wiki-meetups, and invariably tech comes up for discussion (MediaWiki/tech being at the center of nearly everything Wikimedia). My view (and perhaps others share it) is that Wikimedia code development can be summed up like this currently: there are so many cooks in the kitchen and yet dinner is always late.
Perhaps the answer is that we have to give the volunteer developers some obvious pathway to harmonizing their and our priorities. Like, if you're working on files and multimedia, you should be emailing Bryan, me, or maybe Tim or Russell. Could it be that simple?
I don't think being more explicit/clear about what people are working on could ever be a bad thing. Will it help alleviate whichever problem it is you're thinking about? I don't know. But I can't imagine a wiki page making it clear who works on what is going to hurt anything. It can only really help. :-)
MZMcBride
On Tue, May 31, 2011 at 6:47 PM, MZMcBride z@mzmcbride.com wrote:
I occasionally show my face at wiki-meetups, and invariably tech comes up for discussion (MediaWiki/tech being at the center of nearly everything Wikimedia). My view (and perhaps others share it) is that Wikimedia code development can be summed up like this currently: there are so many cooks in the kitchen and yet dinner is always late.
Maybe people should stop adding so many courses to dinner ;-)
-Chad
"Neil Kandalgaonkar" neilk@wikimedia.org wrote in message news:4DE56CF9.9090608@wikimedia.org...
On 5/31/11 3:20 PM, MZMcBride wrote:
taking a page out of the rest of the business world's book, you set a deadline and then it just fucking gets met. No excuses, no questions.
I think you have an optimistic view of how businesses actually work. :)
The only modification needed to bring that sentence in line with business reality is adding "it just fucking gets met **or someone's head rolls**.
But, in any case, in a business, there is a Plan that everyone is trying to follow and in theory, deviations from that Plan are avoided. In our environment we want to be responsive to the schedule of a volunteer developer, who may be completely unaware or uninterested in our plans.
Plenty of businesses work on a rolling-development model, probably more businesses than have totally static specs. The difference between that and WMF, and even between WMF and other non-businesses like Linux and Mozilla, is that if a release is mandated by some higher power and something is holding up that release, **whatever it is gets steamrollered out of the way**. If there is a clear roadmap that says that any feature that's not debugged and ready-to-go by Wednesday morning, by the first Tuesday of the month, by the 32nd of June, whenever, then it gets reverted, no one is going to complain when lo and behold, such features get reverted. *Everyone* is going to complain if the 32nd of June becomes the 32nd of December before the feature even makes it onto the cluster.
Perhaps the answer is that we have to give the volunteer developers some obvious pathway to harmonizing their and our priorities. Like, if you're working on files and multimedia, you should be emailing Bryan, me, or maybe Tim or Russell. Could it be that simple?
The problem seems to be finding anyone to lay down the damn law.
Well, it's not like wiki pages happen by someone cracking a whip either. That said, we would benefit from some urgency towards correcting the problem.
Wiki pages don't need a whip, and nor does MediaWiki. Wiki pages are *missing* several layers of delays and checkpoints in the volunteer-writes-something-to-volunteer-sees-it-in-use chain. MediaWiki is currently like a wiki with FlaggedRevisions turned on on every articlespace page, but with all the admins on the wiki working on other things such that no one gets round to cleaning out the Special:OldReviewedPages list more than once every *nine months*. That would kill a wiki community stone dead *pretty much instantly*.
"Brion Vibber" brion@pobox.com wrote in message news:BANLkTinZzef=oRVuw9dWiCVAHYBUWXniqg@mail.gmail.com...
Sing it, brother! We're getting *some* stuff through quicker within the deployment branches, but not nearly as much as we ought to.
The fact that some things are *not* getting stuck in the CR quagmire is part of the *problem*, not the solution. The upper levels of the developer hierarchy ***obviously know*** that the mainline CR process is substantially broken, BECAUSE ***THEY'RE NOT USING IT*** FOR THINGS THAT ARE IMPORTANT TO THEM. The unavoidable implication of that observation is that the work of the volunteer developers *DOESN'T* matter to them. Whether or not that implication is correct (I have confidence that it's not) is irrelevant, the fact that it's there is what's doing horrible damage to the MediaWiki community.
--HM
On Tue, May 31, 2011 at 4:31 PM, Happy-melon happy-melon@live.com wrote:
"Brion Vibber" brion@pobox.com wrote in message news:BANLkTinZzef=oRVuw9dWiCVAHYBUWXniqg@mail.gmail.com...
Sing it, brother! We're getting *some* stuff through quicker within the deployment branches, but not nearly as much as we ought to.
The fact that some things are *not* getting stuck in the CR quagmire is part of the *problem*, not the solution. The upper levels of the developer hierarchy ***obviously know*** that the mainline CR process is substantially broken, BECAUSE ***THEY'RE NOT USING IT*** FOR THINGS THAT ARE IMPORTANT TO THEM.
That's not correct -- the same code review tools are being used to look over and comment on those things, AND THEN THE DAMN THING ACTUALLY GETS MERGED TO THE DEPLOYMENT BRANCH BY SOMEBODY AND PUSHED TO PRODUCTION BY SOMEBODY.
It's that step 2 that's missing for everything else until the accumulated release pressure forces a system-wide update.
The unavoidable implication of that observation is that the work of
the volunteer developers *DOESN'T* matter to them. Whether or not that implication is correct (I have confidence that it's not) is irrelevant, the fact that it's there is what's doing horrible damage to the MediaWiki community.
-- brion
On Wed, Jun 1, 2011 at 1:43 AM, Brion Vibber brion@pobox.com wrote:
substantially broken, BECAUSE ***THEY'RE NOT USING IT*** FOR THINGS THAT ARE IMPORTANT TO THEM.
That's not correct -- the same code review tools are being used to look over and comment on those things, AND THEN THE DAMN THING ACTUALLY GETS MERGED TO THE DEPLOYMENT BRANCH BY SOMEBODY AND PUSHED TO PRODUCTION BY SOMEBODY.
Alright, we've had fun with all caps for a while, so let's focus on solutions now.
MZ says we need to move to a system with regular deployments, where regular is more than monthly and ideally weekly or bi-weekly. I completely agree with this, per the "spend a day per week or a week per month" argument. The discussion then turns into conflating releases and deployments, followed by some back and forth yelling about deadlines and CR processes.
So let me sketch how I see us getting there.
1. Get 1.18 reviewed 2a. Get 1.18 deployed and released 2b. At the same time, continue efforts to have code review catch up with SVN HEAD 3. Once we have reviewed everything up to HEAD (or up to a revision less than a week older than HEAD), deploy it to Wikimedia 4. Keep reviewing so the CR backlog doesn't grow, and deploy HEAD again in 1-2 weeks. Rinse and repeat.
There are a few caveats here, of course: * 2a and 2b largely compete for the time of the same developers * 3 (initial HEAD deployment) will probably be a bit hairy because it's a lot of code the first time. After that it's only 1-2 weeks' worth each time * For 4, we need to figure out a way to make CR actually happen in a regular and timely fashion
So a lot of it comes down to implementing a proper code review process, both for the one-off catch-up sprint (to 1.18, then to HEAD; we've done one of these before with 1.17) and for the day-to-day keep-up work. I think we may also need to make it clearer that reviewers (who, typically, are paid developers working on many other things) need to spend one-fifth of their time (i.e. a day per week for full-time employees) on code review. This is basically the "admins aren't watching Special:Unreviewedpages at all" problem that Happy-melon mentioned.
So mostly, I think our immediate problems are: * no agreed-upon strategy to tackle the problem (what do y'all think about mine?) * not enough man-hours spent on keeping up with new commits ** ...presumably because employed people aren't given the time to work on it and/or made to work on it ** ...and because some of our reviewers are college students working part-time (fortunately I'll graduate soon and have more time :D)
Thoughts?
Roan Kattouw (Catrope)
On Wed, Jun 1, 2011 at 7:58 AM, Roan Kattouw roan.kattouw@gmail.com wrote:
Alright, we've had fun with all caps for a while, so let's focus on solutions now.
Ok, I'm turning in my caps lock key. ;)
So let me sketch how I see us getting there.
- Get 1.18 reviewed
2a. Get 1.18 deployed and released 2b. At the same time, continue efforts to have code review catch up with SVN HEAD 3. Once we have reviewed everything up to HEAD (or up to a revision less than a week older than HEAD), deploy it to Wikimedia 4. Keep reviewing so the CR backlog doesn't grow, and deploy HEAD again in 1-2 weeks. Rinse and repeat.
There are a few caveats here, of course:
- 2a and 2b largely compete for the time of the same developers
That's one of the reasons actually assigning time for it should be helpful: currently we have zero staff dedicated full-time to getting code ready for deployment and deployed. We have several staff & contractors with it on their list of duties, but no actual requirement that it get done on a regular basis. With no regular short-term review & deployment schedule (even for fixes), we all find other things to do and get out of the habit.
And let's not forget that a big part of what review does is to help develop the reviewee's coding skills! By pointing out things that need fixing and things to watch out for, we help all our coders become better coders. And by pushing review closer to the original code-writing, we'll make that feedback loop tighter & more effective.
- 3 (initial HEAD deployment) will probably be a bit hairy because
it's a lot of code the first time. After that it's only 1-2 weeks' worth each time
- For 4, we need to figure out a way to make CR actually happen in a
regular and timely fashion
+1
So a lot of it comes down to implementing a proper code review process, both for the one-off catch-up sprint (to 1.18, then to HEAD; we've done one of these before with 1.17) and for the day-to-day keep-up work. I think we may also need to make it clearer that reviewers (who, typically, are paid developers working on many other things) need to spend one-fifth of their time (i.e. a day per week for full-time employees) on code review. This is basically the "admins aren't watching Special:Unreviewedpages at all" problem that Happy-melon mentioned.
*nod*
-- brion
On Wed, Jun 1, 2011 at 7:42 PM, Brion Vibber brion@pobox.com wrote:
That's one of the reasons actually assigning time for it should be helpful: currently we have zero staff dedicated full-time to getting code ready for deployment and deployed. We have several staff & contractors with it on their list of duties, but no actual requirement that it get done on a regular basis. With no regular short-term review & deployment schedule (even for fixes), we all find other things to do and get out of the habit.
It would be nice to have someone doing this full-time yes, but even without that, I agree that having regular short-term schedules helps enforcing discipline.
However, I guess some people are afraid that assigning someone to this full-time will waste their other skills and possibly make them unhappy, and I kind of agree. However, dedicating one FTE spread among multiple people should work too, as long as it's not more than, say, 3. I would quite like to be one of these people once I'm able to work full-time.
And let's not forget that a big part of what review does is to help develop the reviewee's coding skills! By pointing out things that need fixing and things to watch out for, we help all our coders become better coders. And by pushing review closer to the original code-writing, we'll make that feedback loop tighter & more effective.
Absolutely! Another thing that happens is that people's coding skills improve to the point where they can *review things themselves* . This is how we build our reviewer base: hold our existing devs to high standards, and wait for them to grow into it.
I'm saying this because what you say in the first paragraph can be (and at one point was) used as an argument to hire a "code reviewer", i.e. hire a new person specifically for code review. IMO this is doomed to fail because code review is pretty much the opposite of an entry-level position. We can train people to be reviewers, but only if they already have significant dev experience and have shown potential, not if we just grabbed them off the proverbial street and they're just getting their feet wet with MediaWiki.
Roan Kattouw (Catrope)
Roan Kattouw wrote:
So mostly, I think our immediate problems are:
- no agreed-upon strategy to tackle the problem (what do y'all think
about mine?)
- not enough man-hours spent on keeping up with new commits
** ...presumably because employed people aren't given the time to work on it and/or made to work on it
I think your analysis and implementation strategy are both fine. The issue seems to be, plainly, that nobody is forced to do code review in order to reduce the backlog. When they are (think of the period directly before the 1.17 deployment), it gets done. This is largely what I was talking about yesterday with deadlines and being forced to meet obligations.
This topic has come up several times and each time, it seems like the people with any kind of power are conspicuously absent. Where is Danese? Is she subscribed to this list? Is she reading it? I'm not sure how it's really possible for this issue to keep coming up and for the situation to not ever be improved. We're not talking about a new Apollo program; this is mostly PHP code development. What's the issue here and how can it be resolved once and for all so that we can stop having these repetitive threads?
** ...and because some of our reviewers are college students working part-time (fortunately I'll graduate soon and have more time :D)
Not to be cynical, but there's no doubt in my mind that when you have more free time, you'll be tasked to work on further "high priority" (but very low impact) tasks like ArticleFeedback (or whatever the new pet project is in a few months).
MZMcBride
MZMcBride wrote:
This topic has come up several times and each time, it seems like the people with any kind of power are conspicuously absent. Where is Danese? Is she subscribed to this list? Is she reading it? I'm not sure how it's really possible for this issue to keep coming up and for the situation to not ever be improved. We're not talking about a new Apollo program; this is mostly PHP code development. What's the issue here and how can it be resolved once and for all so that we can stop having these repetitive threads?
I think she is, as she has previously sent emails here (although none this year). I suspect she doesn't actively follows it, though.
On Thu, Jun 2, 2011 at 12:52 AM, MZMcBride z@mzmcbride.com wrote:
Not to be cynical, but there's no doubt in my mind that when you have more free time, you'll be tasked to work on further "high priority" (but very low impact) tasks like ArticleFeedback (or whatever the new pet project is in a few months).
Basically part of what I'm saying is that we need to ingrain the notion that short-cycle code review and regular deployments are a critical responsibility of our tech dept, and that this means that certain people (e.g. Brion, Tim, me) will simply have to spend 1/5, 1/4 or even 1/3 or whatever on reviewing and deploying code in general, and that this means they're that much less available for project work or whatever else is competing for their time.
Roan Kattouw (Catrope)
On Wed, Jun 1, 2011 at 6:52 PM, MZMcBride z@mzmcbride.com wrote:
This topic has come up several times and each time, it seems like the people with any kind of power are conspicuously absent. Where is Danese? Is she subscribed to this list? Is she reading it?
As I read it, EPMs are directly beneath Danese and presumably have some say in how tech resources are used (thus "Manager"):
http://wikimediafoundation.org/wiki/Engineering_Program_Manager
Software EPMs listed at http://wikimediafoundation.org/wiki/Staff#Technology (excluding Usability) are Alolita Sharma, Tomasz Finc, and Rob Lanphier. Robla is the EPM in charge of General Engineering, which as I understand it is the group that includes most of the staff developers participating in this discussion. He's consistently commented on all of these threads, and is clearly reading them. So I'm pretty sure the issue isn't being ignored.
On Wed, Jun 1, 2011 at 5:57 PM, Aryeh Gregor Simetrical+wikilist@gmail.comwrote:
On Wed, Jun 1, 2011 at 6:52 PM, MZMcBride z@mzmcbride.com wrote:
This topic has come up several times and each time, it seems like the
people
with any kind of power are conspicuously absent. Where is Danese? Is she subscribed to this list? Is she reading it?
As I read it, EPMs are directly beneath Danese and presumably have some say in how tech resources are used (thus "Manager"):
http://wikimediafoundation.org/wiki/Engineering_Program_Manager
Software EPMs listed at http://wikimediafoundation.org/wiki/Staff#Technology (excluding Usability) are Alolita Sharma, Tomasz Finc, and Rob Lanphier. Robla is the EPM in charge of General Engineering, which as I understand it is the group that includes most of the staff developers participating in this discussion. He's consistently commented on all of these threads, and is clearly reading them. So I'm pretty sure the issue isn't being ignored.
Indeed, this conversation is as active live in the office as it is on the list, and the folks most directly in the line of fire are commenting.
-- brion
Aryeh Gregor wrote:
On Wed, Jun 1, 2011 at 6:52 PM, MZMcBride z@mzmcbride.com wrote:
This topic has come up several times and each time, it seems like the people with any kind of power are conspicuously absent. Where is Danese? Is she subscribed to this list? Is she reading it?
As I read it, EPMs are directly beneath Danese and presumably have some say in how tech resources are used (thus "Manager"):
http://wikimediafoundation.org/wiki/Engineering_Program_Manager
Software EPMs listed at http://wikimediafoundation.org/wiki/Staff#Technology (excluding Usability) are Alolita Sharma, Tomasz Finc, and Rob Lanphier. Robla is the EPM in charge of General Engineering, which as I understand it is the group that includes most of the staff developers participating in this discussion. He's consistently commented on all of these threads, and is clearly reading them. So I'm pretty sure the issue isn't being ignored.
Yes, RobLa definitely deserves props for being active on this list (as do many other staffers).
I'm still trying to understand the nature of this problem. I think that's what's bothering me the most at the moment. It's frustrating that I still can't quite figure out exactly what the issue is [with code deployment being so slow] and why it isn't being resolved (even if I don't hold out much hope for an actual resolution being implemented). To me, the solution seems a lot more obvious than it does to others, a point that I'm still struggling with in my head.
MZMcBride
On 6/1/11 8:50 PM, MZMcBride wrote:
To me, the solution seems a lot more obvious than it does to others, a point that I'm still struggling with in my head.
Your solution, as you've described it in the past, comprises "people do code review or orf wit' dere heads".
I know of no professional developer who has dignity who will work under those conditions. So it's untenable.
I propose we stop focusing on the "do this or you're fired" style thinking and instead move towards more constructive process.
----- Original Message -----
From: "Brandon Harris" bharris@wikimedia.org
On 6/1/11 8:50 PM, MZMcBride wrote:
To me, the solution seems a lot more obvious than it does to others, a point that I'm still struggling with in my head.
Your solution, as you've described it in the past, comprises "people do code review or orf wit' dere heads".
I know of no professional developer who has dignity who will work under those conditions. So it's untenable.
I propose we stop focusing on the "do this or you're fired" style thinking and instead move towards more constructive process.
Yeah; Aspergers is a bitch, isn't it? ;-)
Cheers, -- jra
On Thu, Jun 2, 2011 at 12:10 AM, Brandon Harris bharris@wikimedia.org wrote:
Your solution, as you've described it in the past, comprises "people do code review or orf wit' dere heads".
I know of no professional developer who has dignity who will work under those conditions. So it's untenable.
Wow, I picked the wrong career path. A whole profession where you're never asked to do anything that you don't want to do and there's no repercussions for not finishing assigned tasks on time. That sounds awesome!
-- Alex Z.
Στις 02-06-2011, ημέρα Πεμ, και ώρα 08:31 -0400, ο/η Alex "Mr.Z-man" έγραψε:
On Thu, Jun 2, 2011 at 12:10 AM, Brandon Harris bharris@wikimedia.org wrote:
Your solution, as you've described it in the past, comprises "people docode review or orf wit' dere heads".
I know of no professional developer who has dignity who will work underthose conditions. So it's untenable.
Wow, I picked the wrong career path. A whole profession where you're never asked to do anything that you don't want to do and there's no repercussions for not finishing assigned tasks on time. That sounds awesome!
-- Alex Z.
*sigh*
If code review is simply added to the list of tasks that must all be done in the same time period, it's not going to succeed. We need to make sure that time is allocated to it, which means less time spent on other things, and reprioritizing those other things appropriately, *and sticking to that*. That last bit is the hard part.
Most pgroammers that I've known aren't willing to put up with "here's more things on your plate, get them done in the same time frame or walk" for very long. It's also true that generally people are hired for jobs with given job descriptions, and shops that tend to pile on substantial tasks not in those descriptions without checking in with folks first, wind up with morale issues at best and people quitting at worst.
We ought to be able to avoid both of those outcomes. So why not do so?
Ariel
On Thu, Jun 2, 2011 at 9:14 AM, Ariel T. Glenn ariel@wikimedia.org wrote:
Στις 02-06-2011, ημέρα Πεμ, και ώρα 08:31 -0400, ο/η Alex "Mr.Z-man" έγραψε:
On Thu, Jun 2, 2011 at 12:10 AM, Brandon Harris bharris@wikimedia.org wrote:
Your solution, as you've described it in the past, comprises "people do code review or orf wit' dere heads".
I know of no professional developer who has dignity who will work under those conditions. So it's untenable.
Wow, I picked the wrong career path. A whole profession where you're never asked to do anything that you don't want to do and there's no repercussions for not finishing assigned tasks on time. That sounds awesome!
-- Alex Z.
*sigh*
If code review is simply added to the list of tasks that must all be done in the same time period, it's not going to succeed. We need to make sure that time is allocated to it, which means less time spent on other things, and reprioritizing those other things appropriately, *and sticking to that*. That last bit is the hard part.
Most pgroammers that I've known aren't willing to put up with "here's more things on your plate, get them done in the same time frame or walk" for very long. It's also true that generally people are hired for jobs with given job descriptions, and shops that tend to pile on substantial tasks not in those descriptions without checking in with folks first, wind up with morale issues at best and people quitting at worst.
We ought to be able to avoid both of those outcomes. So why not do so?
Ariel
I agree about reprioritizing. Right now, CR appears to be near the bottom of Wikimedia's technical priorities. Sort of a "if you run out of other things to do" sort of thing. Then no one does it and it becomes urgent when it's time to do a release. Really, sticking to it shouldn't be the hard part, as long as the change comes from the top down (i.e. from the people setting the priorities). If people want to do code review, but are told that everything else is a higher priority, then it's doomed to fail. The hard part will be, initially, determining what things that are currently high priorities shouldn't be. Cutting anything is going to bother some people.
If CR isn't part of the job description for new hires, that's another thing that can be fixed by the people in charge.
-- Alex Z.
Well, see, here's the problem: programmers are artists. We'd prefer that they be able to be solely technicians or engineers, but we don't know how to do that. Consequently, you have situations where one programmer is ten times more productive than another equally experienced or qualified programmer. And given that programming is a creative environment, people need an environment free of pressure. Pressure shuts down creativity. It doesn't work for professionals, and it really REALLY doesn't work for volunteers.
Of course, that said, people need to do stuff that earns them a salary. Keeping those two opposing interests balanced is what makes managing programmers difficult, rewarding when done well, and, Alex, not easily summarized in your sarcastic sound bite. The reason Brandon sent out his email was to gently correct MZMcBride's management technique of "do this or you're fired". You then proposed the same thing, which is kinda... ironic.
Managing programmers is difficult. If you think you can do a better job than the professionals, you NEED to give it a try. You might be successful! Or you might learn from your failure. ________________________________________ From: wikitech-l-bounces@lists.wikimedia.org [wikitech-l-bounces@lists.wikimedia.org] on behalf of Alex "Mr.Z-man" [mrzmanwiki@gmail.com]
Wow, I picked the wrong career path. A whole profession where you're never asked to do anything that you don't want to do and there's no repercussions for not finishing assigned tasks on time. That sounds awesome!
On Thu, Jun 2, 2011 at 9:41 AM, Russell N. Nelson - rnnelson rnnelson@clarkson.edu wrote:
Well, see, here's the problem: programmers are artists. We'd prefer that they be able to be solely technicians or engineers, but we don't know how to do that. Consequently, you have situations where one programmer is ten times more productive than another equally experienced or qualified programmer. And given that programming is a creative environment, people need an environment free of pressure. Pressure shuts down creativity. It doesn't work for professionals, and it really REALLY doesn't work for volunteers.
Of course, that said, people need to do stuff that earns them a salary. Keeping those two opposing interests balanced is what makes managing programmers difficult, rewarding when done well, and, Alex, not easily summarized in your sarcastic sound bite. The reason Brandon sent out his email was to gently correct MZMcBride's management technique of "do this or you're fired". You then proposed the same thing, which is kinda... ironic.
Managing programmers is difficult. If you think you can do a better job than the professionals, you NEED to give it a try. You might be successful! Or you might learn from your failure.
This still just seems like a sense of entitlement. If programers want to be treated like artists, let them be. Have their pay be tied to output and quality. If they want to earn a salary, then it should be like the rest of the professional world with deadlines and repercussions for not meeting them (they don't necessarily have to be fired). But trying to have the best of both worlds sounds like a recipe for disaster. But I have a few friends and relatives who are programmers, one of them does work in a typical professional situation in a large company, with deadlines and such. She's been with the same company for years. None of the others have exhibited anything like this "I'll only work in a pressure-free environment" attitude. They treat it like engineering. There's design work involved, but the design is to achieve some specified function, not just design for art's sake.
I've never personally worked in a setting where the main activity is programming. I have worked for an architecture firm. Architects are expected to literally be artists and engineers. They have to design buildings that look good and meet some function. However, they often have strict deadlines and there can be serious penalties (for the firm at least) for not meeting them. Not once did I hear any of this sort of complaining about how enforcing a simple deadline makes people want to quit.
Could people be more creative if they have unlimited time? Probably. But "creative" does not always mean "productive." Sometimes it's better to just get something done than to find an innovative or elegant solution. Such as, when the people paying your salary are concerned that you're taking too long. Or when not doing some task because you don't like it or have more interesting things to do (code review) negatively affects the final product (loss of volunteer contributions).
-- Alex Z.
On Wed, Jun 1, 2011 at 11:50 PM, MZMcBride z@mzmcbride.com wrote:
I'm still trying to understand the nature of this problem. I think that's what's bothering me the most at the moment. It's frustrating that I still can't quite figure out exactly what the issue is [with code deployment being so slow] and why it isn't being resolved (even if I don't hold out much hope for an actual resolution being implemented). To me, the solution seems a lot more obvious than it does to others, a point that I'm still struggling with in my head.
As far as I can tell, the people in charge have just prioritized other projects over reviewing volunteers' code. Presumably that's because they think those projects are more important than reviewing volunteers' code. It seems pretty straightforward to me.
On Thu, Jun 2, 2011 at 12:10 AM, Brandon Harris bharris@wikimedia.org wrote:
Your solution, as you've described it in the past, comprises "people do code review or orf wit' dere heads".
I know of no professional developer who has dignity who will work under those conditions. So it's untenable.
I propose we stop focusing on the "do this or you're fired" style thinking and instead move towards more constructive process.
Um, how is "please spend today reducing the code review backlog" any different from "please spend today working on further issues in ArticleFeedback" or whatever? Developers are assigned to particular tasks by managers. You can't just assign them to any old task as though they're automatons, but I don't see what exactly is so untenable about assigning people to do code review. Wikimedia did it for 1.17. Tons of other organizations that produce open-source projects assign their employees to spend part of their time reviewing code: Mozilla, Google (for things like Chromium and WebKit), Apple (WebKit), etc., etc.
So I really don't get what you're saying here. Superficially, you seem to be saying that managers cannot assign developers to do specific things at all, but you can't possibly mean that. Do you feel there's some reason it's less practical to assign people to review code than to write it?
On Thu, Jun 2, 2011 at 12:35 AM, Tim Starling tstarling@wikimedia.org wrote:
In case anyone's wondering what I think of this, I was pretty blunt last time around:
http://lists.wikimedia.org/pipermail/wikitech-l/2011-April/052893.html
To be any more blunt than that, I'd have to press the caps lock key ;)
Do you think this should be the policy for all code? Because I'm noticing a pretty decent number of commits being deployed right now:
http://svn.wikimedia.org/viewvc/mediawiki/branches/wmf/1.17wmf1/?view=log
Some of them are things like "Merge ArticleFeedback to trunk state". Do you feel that projects like ArticleFeedback should also be deployed only once every few months? Or are they different for some reason?
On 03/06/11 08:49, Aryeh Gregor wrote:
Do you think this should be the policy for all code? Because I'm noticing a pretty decent number of commits being deployed right now:
http://svn.wikimedia.org/viewvc/mediawiki/branches/wmf/1.17wmf1/?view=log
Some of them are things like "Merge ArticleFeedback to trunk state". Do you feel that projects like ArticleFeedback should also be deployed only once every few months? Or are they different for some reason?
No, I think extension updates can be backported more regularly, as long as the changes are tested, and the potential impact is limited.
Core changes are typically complex and interdependent, and can cause problems for extensions. Large projects are committed to core in a low-quality state, and it takes time to stabilise them, despite the efforts of development branch reviewers. That's why major core updates require more testing.
-- Tim Starling
On Thu, Jun 2, 2011 at 10:22 PM, Tim Starling tstarling@wikimedia.org wrote:
No, I think extension updates can be backported more regularly, as long as the changes are tested, and the potential impact is limited.
What about core updates that are tested and have limited potential impact? Like, say, most of the updates we'd get to core in any given week? There isn't much of a hard distinction between core and extensions.
Core changes are typically complex and interdependent, and can cause problems for extensions. Large projects are committed to core in a low-quality state, and it takes time to stabilise them, despite the efforts of development branch reviewers. That's why major core updates require more testing.
This would argue equally well for not committing large projects to trunk all at once, or only committing them hidden behind off-by-default preferences. It doesn't mean you can't regularly deploy the simple changes that are unlikely to cause significant problems, and where the problems can be easily fixed when they're found.
On 31/05/11 17:12, MZMcBride wrote:
I think the main question in my message got lost, though. When I said "more regular" code deployments, I meant less than or equal to a month. Perhaps even weekly.
In case anyone's wondering what I think of this, I was pretty blunt last time around:
http://lists.wikimedia.org/pipermail/wikitech-l/2011-April/052893.html
To be any more blunt than that, I'd have to press the caps lock key ;)
-- Tim Starling
Tim Starling wrote:
On 31/05/11 17:12, MZMcBride wrote:
I think the main question in my message got lost, though. When I said "more regular" code deployments, I meant less than or equal to a month. Perhaps even weekly.
In case anyone's wondering what I think of this, I was pretty blunt last time around:
http://lists.wikimedia.org/pipermail/wikitech-l/2011-April/052893.html
To be any more blunt than that, I'd have to press the caps lock key ;)
I don't understand how someone who's spent the last eight years working on wiki (Hawaiian for "fast") software thinks it's acceptable for there to be a delay of several months between code being committed and code being deployed.
I re-read your post from April (a few times) and Brion's reply. The larger dynamics of this situation (slower code deployment) are becoming clearer to me, so I suppose it's time for me to back off of this. I think the system you've proposed (and effectively implemented) is anti-wiki, actively harmful to volunteer development efforts, and will result in a worse final product, but that's just the way it goes, I guess.
I still respect and admire all of the work you do.
MZMcBride
In case anyone's wondering what I think of this, I was pretty blunt last time around:
http://lists.wikimedia.org/pipermail/wikitech-l/2011-April/052893.html
To be any more blunt than that, I'd have to press the caps lock key ;)
This post seems to be a much more optimistic view of the 1.17 deployment than I remember. Wasn't it rolled back twice? I also remember getting reports of random issues for weeks after the deployment as well. People were asking for us to do a rollback a couple weeks after the deployment even.
1.17 still doesn't have a tarball release. It is *very* late. It doesn't seem to me that this process is working.
I agree with the testing process, but I disagree with the deployment process. We can apply the testing process to a continuous integration deployment model. From an ops perspective, this is much nicer; if something is going wrong, we know fairly quickly what caused it to go wrong.
- Ryan
On 03/06/11 09:46, Ryan Lane wrote:
In case anyone's wondering what I think of this, I was pretty blunt last time around:
http://lists.wikimedia.org/pipermail/wikitech-l/2011-April/052893.html
To be any more blunt than that, I'd have to press the caps lock key ;)
This post seems to be a much more optimistic view of the 1.17 deployment than I remember. Wasn't it rolled back twice? I also remember getting reports of random issues for weeks after the deployment as well. People were asking for us to do a rollback a couple weeks after the deployment even.
The first deployment caused disruption because we tried to deploy it to all wikis at once, hopefully we're not going to try that again.
The second deployment caused disruption because it was done while I was asleep, and the people involved decided to spend a long time trying to debug the problem while the site mostly down, instead of immediately reverting and isolating it offline. Hopefully we won't try that again either.
Despite these "learning experiences", I think it went a lot better than previous deployments. The number of bugs that needed fixing was relatively small.
1.17 still doesn't have a tarball release. It is *very* late. It doesn't seem to me that this process is working.
We have a beta release, do you mean a stable release? That's mostly due to priorities, not process. Also the Berlin meeting stopped pretty much all work on it for a week, and there were a couple of security issues reported that we'd like to get fixed before 1.17.0 is released.
I agree with the testing process, but I disagree with the deployment process. We can apply the testing process to a continuous integration deployment model. From an ops perspective, this is much nicer; if something is going wrong, we know fairly quickly what caused it to go wrong.
I'm not sure what you're proposing here, in terms of how branches would be used and what sort of testing would be done. Can you elaborate?
-- Tim Starling
wikitech-l@lists.wikimedia.org