Hi all,
Etherpad [0], our real-time colaborative editing tool suffered an outage due to what we only know for now was database corruption. This was detected shortly after it happened 14:27 UTC and we (ops in charge of the service and the database) worked to reestablish the service.
As the service continued crashing despite our efforts, we decided to recover a database backup from 2016-06-22 01:00:01 UTC. The service is now back up and working since 16:11 UTC, but that means that you may have lost a day and a half of edits in the current available etherpad [0].
I understand that that may cause a lot of inconveniences, specially for the people at Wikimania. *We are now trying to recover more than that*, but as the corruption could come back, or not all could be recovered, and people need the service the plan is the following:
- Keep the current pads as is, not delete or add anything from now. You can continue using etherpad now as usual. - If possible, recover the last days of edits on a separate location. See [1] for progress if you are affected.
Sorry for the inconveniences. Please, more than ever, follow the recommendation we added at the beginning of every empty pad:
"Keep in mind as well that there is no guarantee that a pad's contents will always be available. A pad may be corrupted, deleted or similar. Please keep a copy of important data somewhere else as well"
The reason for this is that wiki content has proper HA and redundancy, etherpad does not.
Again, my most sincere apologies,
[0] https://etherpad.wikimedia.org/ [1] https://phabricator.wikimedia.org/T138516
On Thu, Jun 23, 2016 at 7:02 PM, Jaime Crespo jcrespo@wikimedia.org wrote:
Hi all,
Etherpad [0], our real-time colaborative editing tool suffered an outage due to what we only know for now was database corruption. This
- If possible, recover the last days of edits on a separate location.
See [1] for progress if you are affected.
Thanks to Alex's incredible work to make it run again, the previous version of the etherpad database (a few minutes before the crash- around 13:30 UTC) was recovered, and it is available temporarily on:
https://etherpad-restore.wikimedia.org
If you want to recover some lost text, **you need to copy it manually from here and paste it into https://etherpad.wikimedia.org (the usual address)** We will **not** touch the current etherpad, as some of you have already added/recovered your texts.
The -restore url will be available for **a week** until it is deleted.
Please resend this information to anybody that may find this useful, so no important data is lost.
Regards,
Hi Jaime --
Thank you to you and the rest of the ops folks who rescued this system. We all really appreciate it.
-Toby
On Thu, Jun 23, 2016 at 5:56 PM, Jaime Crespo jcrespo@wikimedia.org wrote:
On Thu, Jun 23, 2016 at 7:02 PM, Jaime Crespo jcrespo@wikimedia.org wrote:
Hi all,
Etherpad [0], our real-time colaborative editing tool suffered an outage due to what we only know for now was database corruption. This
- If possible, recover the last days of edits on a separate location.
See [1] for progress if you are affected.
Thanks to Alex's incredible work to make it run again, the previous version of the etherpad database (a few minutes before the crash- around 13:30 UTC) was recovered, and it is available temporarily on:
https://etherpad-restore.wikimedia.org
If you want to recover some lost text, **you need to copy it manually from here and paste it into https://etherpad.wikimedia.org (the usual address)** We will **not** touch the current etherpad, as some of you have already added/recovered your texts.
The -restore url will be available for **a week** until it is deleted.
Please resend this information to anybody that may find this useful, so no important data is lost.
Regards,
Jaime Crespo http://wikimedia.org
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Fri, Jun 24, 2016 at 12:56 AM, Jaime Crespo jcrespo@wikimedia.org wrote:
The -restore url will be available for **a week** until it is deleted.
This was deleted today. Continue using https://etherpad.wikimedia.org as usual (but remember to backup anything important!).
Regards,
Jaime Crespo wrote:
Continue using https://etherpad.wikimedia.org as usual (but remember to backup anything important!).
The wikis should be, and generally are, the canonical source of information for our projects. One of the major advantages of using the wikis is that we then provide and distribute backups to mirrors and others.
MZMcBride
wikitech-l@lists.wikimedia.org