Hi, our metrics are reflecting a sudden growth of unreviewed changes in Gerrit. Your attention and interpretations are welcome.
http://korma.wmflabs.org/browser/gerrit_review_queue.html
Look at the green line in the first graph, "Volume of open changesets". It shows the number of open changesets waiting for a review (WIP and -1 are excluded). In the last months, the number of changesets waiting for review has gone from 311 (June) to 529 (July) and 790 (August).
The increase is so high that first I thought the problem was in the metric, but Alvaro has reviewed the raw data and says that the metric is legit.
https://bugzilla.wikimedia.org/show_bug.cgi?id=70278
Trying to find an explanation, I went to the MediaWiki Core repository, and there are some inusual curves there as well:
http://korma.wmflabs.org/browser/repository.html?repository=gerrit.wikimedia...
However, not even these numbers alone would explain the increase.
Has there been an unusual activity (or lack of reviews) in Gerrit or are we missing an important detail in the metrics?
(pause)
In any case, in the past weeks I have been chasing old open reviews and in general I get the impression that developers/maintainers are too busy to review so many contributions. The tricky part is that many (most?) of those contributions come from the same developers/maintainers...
Simplifying a lot the picture, it looks as if we can't review code because we are busy writing code that won't be reviewed because we are busy writing code that won't be reviewed because... I'm sure this representation is unfair, but you get the point.
In many cases, unreviewed patches rotting mean real resources put into waste -- right? Maybe teams should prioritize the attention to open changesets at the expense of writing new code themselves? It's a real question and I won't pretend to have an answer. Different teams and repositories probably have different answers.
When I look at http://koti.kapsi.fi/~federico/crstats/extensions.txt , I get the impression that the answer may be simple: the handful devs who take care of most code review in MediaWiki have been doing something else in the last months. When you remove those few, the queue doesn't move that much.
Nemo
Could it be just... holidays?
Strainu
2014-09-09 15:26 GMT+03:00 Federico Leva (Nemo) nemowiki@gmail.com:
When I look at http://koti.kapsi.fi/~federico/crstats/extensions.txt , I get the impression that the answer may be simple: the handful devs who take care of most code review in MediaWiki have been doing something else in the last months. When you remove those few, the queue doesn't move that much.
Nemo
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Tue, Sep 9, 2014 at 2:30 PM, Strainu strainu10@gmail.com wrote:
Could it be just... holidays?
Could be... although the holidays reviewing code seem to come together with no-holidays uploading new patches.
Is the increase connected to any particular users (or type of users, new/experienced)? Any repositories?
There have been a few sweeping changes recently that resulted in a lot of search-and-replace changesets being submitted for many extensions. Perhaps some of these are stuck somewhere.
On Tue, Sep 9, 2014 at 3:08 PM, Bartosz Dziewoński matma.rex@gmail.com wrote:
Is the increase connected to any particular users (or type of users, new/experienced)? Any repositories?
I have mentioned the case of MediaWiki core. operations/puppet also had a steep curve this Summer:
http://korma.wmflabs.org/browser/repository.html?repository=gerrit.wikimedia...
But so far it is guesswork for me.
There have been a few sweeping changes recently that resulted in a lot of search-and-replace changesets being submitted for many extensions. Perhaps some of these are stuck somewhere.
Reedy alone seems to have plenty of patches waiting to be reviewed. Maybe these are the ones you are referring to? See for example
https://gerrit.wikimedia.org/r/#/q/owner:%22Reedy+%253Creedy%2540wikimedia.o...
http://korma.wmflabs.org/browser/gerrit_review_queue.html shows most increase is in August vs. May, of which about +550 "waiting for review" (those without any comment?) and +200 "pending". The easiest way to identify outliers is diffing this report among the two dates: https://www.mediawiki.org/w/index.php?title=Gerrit%2FReports%2FOpen_changesets_by_owner&diff=1128160&oldid=1018652 Devs having an increase of 15 mediawiki/.* patches or more (and sometimes over 50) are bawolff, deepali, Florianschmidtwelzow, jackmcbarn, MarkAHershberger, Paladox, Rohan013, Withoutaname. Usually they're worth looking into and resolving a way or another. :)
Nemo
On 09/09/14 03:06, Quim Gil wrote:
Hi, our metrics are reflecting a sudden growth of unreviewed changes in Gerrit. Your attention and interpretations are welcome.
http://korma.wmflabs.org/browser/gerrit_review_queue.html
Look at the green line in the first graph, "Volume of open changesets". It shows the number of open changesets waiting for a review (WIP and -1 are excluded). In the last months, the number of changesets waiting for review has gone from 311 (June) to 529 (July) and 790 (August).
The increase is so high that first I thought the problem was in the metric, but Alvaro has reviewed the raw data and says that the metric is legit.
I've noticed that when I submitted patches to review, it used to send out emails inviting people to review it. The last several patches I've submitted did not generates such emails (or at least if it did, I was not CC'ed on it).
I wrote it off as there being some mysterious rules for the emails to go out and my most recent patches didn't trigger them, but maybe something is wrong with the email system. That could explain the lack of attention to recent submissions.
Cheers,
Jeff
(Who is posting this through gmane.science.linguistics.wikipedia.technical and has no idea whether it will propagate to the list or not)
On Thu, Sep 11, 2014 at 9:15 PM, Jeff Janes jeff.janes@gmail.com wrote:
I've noticed that when I submitted patches to review, it used to send out emails inviting people to review it. The last several patches I've submitted did not generates such emails (or at least if it did, I was not CC'ed on it).
The way the emails work is that people get added if the change matches the configurations at https://www.mediawiki.org/wiki/Git/Reviewers. If you don't happen to match anyone's rules, you won't get any reviewers automatically added.
When that happens, you could check https://www.mediawiki.org/wiki/Developers/Maintainers to see if anyone is listed for relevant components. It's also usually helpful to look through the git log for the code you're changing, pick out some similar changesets, and see who authored and reviewed them.
Do we have any similar data which shows how neglected certain users are? I would be interesting in knowing for a given user what is the average time it takes for their patches to get attention. It would be good to give these developers more attention to keep them more engaged.
pseudo code:
users_patches = get_patches_for_user( $username ) avg_time = 0; foreach( $patches as $patch ) { time = today - patch.created avg_time += today - patch.created } avg_time /= len( $patches )
echo $username + ': ' + days(avg_time)
On Fri, Sep 12, 2014 at 6:44 AM, Brad Jorsch (Anomie) bjorsch@wikimedia.org wrote:
On Thu, Sep 11, 2014 at 9:15 PM, Jeff Janes jeff.janes@gmail.com wrote:
I've noticed that when I submitted patches to review, it used to send out emails inviting people to review it. The last several patches I've submitted did not generates such emails (or at least if it did, I was not CC'ed on it).
The way the emails work is that people get added if the change matches the configurations at https://www.mediawiki.org/wiki/Git/Reviewers. If you don't happen to match anyone's rules, you won't get any reviewers automatically added.
When that happens, you could check https://www.mediawiki.org/wiki/Developers/Maintainers to see if anyone is listed for relevant components. It's also usually helpful to look through the git log for the code you're changing, pick out some similar changesets, and see who authored and reviewed them.
-- Brad Jorsch (Anomie) Software Engineer Wikimedia Foundation _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
The good news is that the graph wasn't showing the metric I thought it was showing, and therefore we don't have such sudden growth of unreviewed changesets.
The not so good news is that now it's clear that we have a steady growth in our backlog of unreviewed changesets: 538 in August 2013, 1079 in August 2014. 100% increase in one year. These numbers might still change a bit based on the definition of "waiting for review", but I believe the shape of the lines, the relative growth of the Gerrit queue, will not change much.
http://korma.wmflabs.org/browser/gerrit_review_queue.html
On Saturday, September 13, 2014, Jon Robson jdlrobson@gmail.com wrote:
Do we have any similar data which shows how neglected certain users are?
We might look into that, but already now we have (in the URL above) a list of repositories sorted by median age of patchsets waiting to be reviewed. It is fair to assume that maintainers are not neglecting someone systematically, but it is more likely to find repositories that are less maintained, untidy, or with clearly more contributors than reviewers.
Unsurprisingly, quite often there is a relation between repositories at the top of the list and lack of maintainers declared at https://www.mediawiki.org/wiki/Developers/Maintainers
wikitech-l@lists.wikimedia.org