We (Release Engineering) file production-error tasks as part of the weekly train and collect some data in the "train-stats" repo on GitLab[0]. Additionally, Timo Tijhof's "production excellence" blog posts and emails to this list may be of interest to you[1].
The "train-stats" repo collects data for "software defect prediction" based on the use of "FixCaches" or "BugCaches."[2] Each week, we record changes that fix bugs (i.e., the change uses the git trailer `Bug: TXXX` and gets backported to a currently deployed branch). The theory (per the paper linked above) is that the more often a file needs a fix, the more likely it is to cause future bugs. I have an extremely convoluted query to show the list of commonly backported files[3].
- "Lower level" files may be overrepresented (although, that's probably to be expected)
In 2013, a case study used data like this inside Google
and found it to be fairly accurate at predicting future bugs[
4].
Also, in the case study, whenever a developer edited a file that was present in their fixCache, researchers added a bot-generated note to the patch in their code review tool.Their developers found this note unhelpful: developers already knew these files were problematic
—the warning
just caused confusion.
Based on that, in March 2020, we created the "Risky Change Template"[
5].
My thinking was: if developers already know what's risky, then they can flag it in the train task for the week
[6].
At the time, I hoped this would reduce the total version deployment time (although I have no data on that).
I hope some of this helps!
– Tyler
[0]: <
https://gitlab.wikimedia.org/repos/releng/train-stats>