On Thu, Jun 8, 2023 at 12:40 AM Physikerwelt <wiki(a)physikerwelt.de> wrote:
Hi all,
is there any research on common causes of Wikimedia production errors?
Based on recent examples, I plan to analyze and discuss how production
errors could be avoided. I am considering submitting a short paper on
that to the Wikidata workshop, with the deadline
Thursday, 20 July 2023
Website:
https://wikidataworkshop.github.io/2023/
However, there might be better suitable venues.
We (Release Engineering) file production-error tasks as part of the weekly
train and collect some data in the "train-stats" repo on GitLab[0].
Additionally, Timo Tijhof's "production excellence" blog posts and emails
to this list may be of interest to you[1].
The "train-stats" repo collects data for "software defect prediction"
based
on the use of "FixCaches" or "BugCaches."[2] Each week, we record
changes
that fix bugs (i.e., the change uses the git trailer `Bug: TXXX` and gets
backported to a currently deployed branch). The theory (per the paper
linked above) is that the more often a file needs a fix, the more likely it
is to cause future bugs. I have an extremely convoluted query to show the
list of commonly backported files[3].
Problems with this data:
- Many of these files are frequently touched files vs error-prone files
(e.g., "composer.json")
- Looking at the count of backports for each file means newer files are
less likely to be represented
- "Lower level" files may be overrepresented (although, that's probably to
be expected)
In 2013, a case study used data like this inside Google and found it to be
fairly accurate at predicting future bugs[4].
Also, in the case study, whenever a developer edited a file that was
present in their fixCache, researchers added a bot-generated note to the
patch in their code review tool.Their developers found this note unhelpful:
developers already knew these files were problematic—the warning just caused
confusion.
Based on that, in March 2020, we created the "Risky Change Template"[5]. My
thinking was: if developers already know what's risky, then they can flag
it in the train task for the week[6]. At the time, I hoped this would
reduce the total version deployment time (although I have no data on that).
I hope some of this helps!
– Tyler
[0]: <https://gitlab.wikimedia.org/repos/releng/train-stats
[1]: <
https://phabricator.wikimedia.org/phame/post/view/296/production_excellence…
[2]: <
https://people.csail.mit.edu/hunkim/images/3/37/Papers_kim_2007_bugcache.pdf
[3]: <
https://data.releng.team/train?sql=select%0D%0A++filename%2C%0D%0A++project…
[4]:
<https://doi.org/10.1109/ICSE.2013.6606583
[5]: <https://wikitech.wikimedia.org/wiki/Deployments/Risky_change_template
[6]:
<https://train-blockers.toolforge.org/> (here's an example this week: <
https://phabricator.wikimedia.org/T337526#8901982>)
> I am also open to collaboration on this
effort. If you are interested
> in a joint paper, drop me an email until the end of this week.
> All the best
> Moritz
> _______________________________________________
> Wikitech-l mailing list -- wikitech-l(a)lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-leave(a)lists.wikimedia.org
>
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/