Hello all,
hemlock had another disc-problem and needed a emergency-reboot. It is back in action now (more or less) but it lost one of its partitions. This partition carries the user-store. A check-up (fsck) is running at the moment and it will take a while to complete (I guess the hole european night). Most things should work, but nothing that needs /mnt/user-store. I will send an update when the user-store is back.
Sincerely, DaB.
Hello again,
around 14 o'clock UTC the web-servers were down again because of hemlock's /aux0-partition vanished again. The problem was solved shortly after, and Nosy and I spent the evening to look where the problem is. As far as we see it is either the connection from hemlock to the SAN or hemlock's controller card. To let the web-servers work, we not mounted /aux0 in its normal place; the downside is that the user-store and the OSM-titles are not available at the moment. We will try to build-up a work-around tomorrow.
Sincerely, DaB.
toolserver-announce@lists.wikimedia.org