2009/7/17 Ariel T. Glenn ariel@wikimedia.org:
Well, the Sun patch didn't hold us. So we are going to do a gradual move of directories, some data, and the actual thumbnail regeneration over to another server, ms4. How much data we move depends on how fast ms1 can push it over to ms4, currently very slow indeed. Even removing files on ms1 is painfully slow. So...
So the ZFS bug came back?
The hard part is thinking of a file system that does snapshots as nicely, under any OS.
In order to get ms1 out of its hole, we're turning off uploads once again, so that it can free up space at a reasonable pace. Once we have enough free space to make ZFS happy, we'll turn it them back on. We expect that the rest of the migration of data and/or empty directories should take place over the next several days without further outages.
Honestly, a reboot will be quicker. I fully understand not even being able to, of course.
- d.