[Labs-l] Preventing future outages (was: [NFS outage] Tools is back)

[Maximilian Doerr]

Can I have some too?  I would also like strawberries with whipped cream. :p

Thanks for that legotktm.

Cyberpower678
English Wikipedia Account Creation Team
Mailing List Moderator


-----Original Message-----
From: labs-l-bounces at lists.wikimedia.org [mailto:labs-l-bounces at lists.wikimedia.org] On Behalf Of Legoktm
Sent: Friday, June 19, 2015 8:55 PM
To: labs-l at lists.wikimedia.org
Subject: Re: [Labs-l] Preventing future outages (was: [NFS outage] Tools is back)

On 06/19/2015 12:03 PM, Ryan Lane wrote:
> On Fri, Jun 19, 2015 at 3:53 AM, Gerard Meijssen 
> <gerard.meijssen at gmail.com <mailto:gerard.meijssen at gmail.com>> wrote:
> 
>     Hoi,
>     Given that the proof of the pudding is in the eating... 
> 
> 
> One line metaphors may not be the best way to make a point. What in 
> the world do you mean by this?

https://en.wiktionary.org/wiki/the_proof_of_the_pudding_is_in_the_eating

"You can only say something is a success after it has been tried out or used."

I believe he is saying that there are incident reports with listed actionables, but we continually keep having NFS failures. Here are the incident reports that mention "NFS" in the title:

https://wikitech.wikimedia.org/wiki/Incident_documentation/20150331-LabsNFS-Filesystem-Switch
* https://phabricator.wikimedia.org/T94613 - open
* https://phabricator.wikimedia.org/T94608 - open
* 2 more actionables without bugs

https://wikitech.wikimedia.org/wiki/Incident_documentation/20150331-LabsNFS-Overload
* https://phabricator.wikimedia.org/T94606 - resolved
* https://phabricator.wikimedia.org/T94607 - open
* https://phabricator.wikimedia.org/T94609 - open

https://wikitech.wikimedia.org/wiki/Incident_documentation/20150401-LabsNFS-Overload
* https://phabricator.wikimedia.org/T94828 - open
* https://phabricator.wikimedia.org/T95293 - open

https://wikitech.wikimedia.org/wiki/Incident_documentation/20150413-LabsNFS
* https://phabricator.wikimedia.org/T94609 - open
* https://phabricator.wikimedia.org/T96045 - resolved
* https://phabricator.wikimedia.org/T96063 - stalled
* https://phabricator.wikimedia.org/T96065 - open

https://wikitech.wikimedia.org/wiki/Incident_documentation/20150607-LabsNFS-Outage
* https://phabricator.wikimedia.org/T101650 - open
* https://phabricator.wikimedia.org/T101742 - open
* https://phabricator.wikimedia.org/T101744 - open

So, 2 out of 14 fixed. Raw numbers don't tell the whole story of course, but they aren't inspiring confidence either. Anyways, can I have some pudding now?

-- Legoktm

_______________________________________________
Labs-l mailing list
Labs-l at lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/labs-l