[Labs-l] Preventing future outages (was: [NFS outage] Tools is back)

[Legoktm]

On 06/19/2015 12:03 PM, Ryan Lane wrote:
> On Fri, Jun 19, 2015 at 3:53 AM, Gerard Meijssen
> <gerard.meijssen at gmail.com <mailto:gerard.meijssen at gmail.com>> wrote:
> 
>     Hoi,
>     Given that the proof of the pudding is in the eating... 
> 
> 
> One line metaphors may not be the best way to make a point. What in the
> world do you mean by this?

https://en.wiktionary.org/wiki/the_proof_of_the_pudding_is_in_the_eating

"You can only say something is a success after it has been tried out or
used."

I believe he is saying that there are incident reports with listed
actionables, but we continually keep having NFS failures. Here are the
incident reports that mention "NFS" in the title:

https://wikitech.wikimedia.org/wiki/Incident_documentation/20150331-LabsNFS-Filesystem-Switch
* https://phabricator.wikimedia.org/T94613 - open
* https://phabricator.wikimedia.org/T94608 - open
* 2 more actionables without bugs

https://wikitech.wikimedia.org/wiki/Incident_documentation/20150331-LabsNFS-Overload
* https://phabricator.wikimedia.org/T94606 - resolved
* https://phabricator.wikimedia.org/T94607 - open
* https://phabricator.wikimedia.org/T94609 - open

https://wikitech.wikimedia.org/wiki/Incident_documentation/20150401-LabsNFS-Overload
* https://phabricator.wikimedia.org/T94828 - open
* https://phabricator.wikimedia.org/T95293 - open

https://wikitech.wikimedia.org/wiki/Incident_documentation/20150413-LabsNFS
* https://phabricator.wikimedia.org/T94609 - open
* https://phabricator.wikimedia.org/T96045 - resolved
* https://phabricator.wikimedia.org/T96063 - stalled
* https://phabricator.wikimedia.org/T96065 - open

https://wikitech.wikimedia.org/wiki/Incident_documentation/20150607-LabsNFS-Outage
* https://phabricator.wikimedia.org/T101650 - open
* https://phabricator.wikimedia.org/T101742 - open
* https://phabricator.wikimedia.org/T101744 - open

So, 2 out of 14 fixed. Raw numbers don't tell the whole story of course,
but they aren't inspiring confidence either. Anyways, can I have some
pudding now?

-- Legoktm