Hi all,
Etherpad [0], our real-time colaborative editing tool suffered an outage due to what we only know for now was database corruption. This was detected shortly after it happened 14:27 UTC and we (ops in charge of the service and the database) worked to reestablish the service.
As the service continued crashing despite our efforts, we decided to recover a database backup from 2016-06-22 01:00:01 UTC. The service is now back up and working since 16:11 UTC, but that means that you may have lost a day and a half of edits in the current available etherpad [0].
I understand that that may cause a lot of inconveniences, specially for the people at Wikimania. *We are now trying to recover more than that*, but as the corruption could come back, or not all could be recovered, and people need the service the plan is the following:
- Keep the current pads as is, not delete or add anything from now. You can continue using etherpad now as usual. - If possible, recover the last days of edits on a separate location. See [1] for progress if you are affected.
Sorry for the inconveniences. Please, more than ever, follow the recommendation we added at the beginning of every empty pad:
"Keep in mind as well that there is no guarantee that a pad's contents will always be available. A pad may be corrupted, deleted or similar. Please keep a copy of important data somewhere else as well"
The reason for this is that wiki content has proper HA and redundancy, etherpad does not.
Again, my most sincere apologies,
[0] https://etherpad.wikimedia.org/ [1] https://phabricator.wikimedia.org/T138516