[teampractices] Hazards of back-estimating backlog story points

[Joel Aufrecht]

*tl;dr:* We assigned a default point value to all unestimated tasks in the
VE backlog, but exploration revealed that this was wrong.  Moral: the more
assumptions you make about your data, the more ways you need to inspect it
for the consequences of bad assumptions.

*Executive summary*
Lead-time analysis of VE revealed that the VE team puts almost all of its
effort into very old (2-3 + months) tasks.  By other measures, however, the
VE team puts a third or more of its effort into "interrupts", unplanned
work.  So either the interrupts are things that sit dormant and then erupt,
or something's funny in the numbers.  Further investigation revealed that
the assumption that all unpointed tasks are five points of effort conflicts
with real-life observation, in which many unpointed tasks are closed every
week with relatively trivial effort.  Not necessarily because they are
actually very low effort, but more because they are record-keeping issues
(duplicate, overtaken by events, fixed as a side effect, etc) than fresh
work.

*Details*
Because VE has around 5000 open tasks and only 800 estimated tasks, backlog
analysis is impossible without making assumptions about the open tasks.
Based on average point sizes for estimated tasks, we assigned a value of 5
to all unestimated tasks.  (VE doesn't use 5 in normal task estimation, so
all 5s are former nulls.)

I generated this lead-time chart, which shows the age of tasks that get
resolved every week.  Dark blue means the task is less than a few weeks old
on the day it gets resolved; light blue means the task is months old.  It's
scaled by value of tasks, not raw count.


​This suggests that the VE team has been spending 80% of effort recently
closing very old tasks.  This seems like a recent trend, but since almost
all data was loaded into Phabricator in late January, near the beginning of
the chart, tasks look artificially young on this scale for the first few
months. This next chart shows what fraction of VE team effort goes to
"interrupt" tasks, as opposed to planned work.  The line is total work
completed; the black area is interrupt work:

​The implication of the two charts together is that the VE team is working
on a third or more interrupts, but most of those interrupts are months
old.  Which is an odd thing, since most interrupts are actually a week or
two old.  So we dug into the task sizes for done tasks by week:


​The teal color in this chart is 5 point tasks, i.e., tasks that were null
points and that we assumed are actually a default of 5.  This chart says
that half of more of the effort each week has been on null-point tasks.  In
practice, however, the VE team is confident that this has not actually been
happening.  What is actually happening is that the VE team closes a number
of null-point tasks each week because they are duplicates, or incidentally
fixed by some other task, or otherwise more administrivia than substantive
primary development.  So the assumption that null-point tasks are really 5
points of work may hold true for the main backlog, but is not true for
tasks that actually get marked done.

So.  First, I identified a list of 470 resolved stories with zero points,
and the VE team will retroactively assign points to these, the majority of
which we expect to be 0 or 1 point.  Second, the VE team will have to start
making sure to assign points to each task as it gets closed.  Third, we
will have to reconsider our assumptions about how big the unpointed VE
backlog is.


*Joel Aufrecht*
Team Practices Group
Wikimedia Foundation
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/teampractices/attachments/20150804/6f8773bf/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: VE-histopoints.png
Type: image/png
Size: 21313 bytes
Desc: not available
URL: <https://lists.wikimedia.org/pipermail/teampractices/attachments/20150804/6f8773bf/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: VE-interrupt.png
Type: image/png
Size: 53026 bytes
Desc: not available
URL: <https://lists.wikimedia.org/pipermail/teampractices/attachments/20150804/6f8773bf/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: VE-leadtime.png
Type: image/png
Size: 22966 bytes
Desc: not available
URL: <https://lists.wikimedia.org/pipermail/teampractices/attachments/20150804/6f8773bf/attachment-0005.png>