Hello all,
between midnight and 11 o'clock (both time-valuies are estimates and UTC) willow was unusable, because the root-partion of willow had run full by a tool which flooded the user-log (I messaged the tool-runers seperatly already).
I cleaned some old audit-logs and moved the big user-log out of the way. I also rebooted willow to make sure that the space was freeed and that all crashed services run again.
I should now usable again. If not, please fill a bug. For details you can take a look at [1].
Sincerly, DaB.
[1] https://jira.toolserver.org/browse/MNT-1079
toolserver-l@lists.wikimedia.org