Forwarding news of data loss. :( Hopefully it will all be recoverable.
Pine ( https://meta.wikimedia.org/wiki/User:Pine )
---------- Forwarded message --------- From: Léa Lacroix lea.lacroix@wikimedia.de Date: Fri, Oct 12, 2018 at 2:46 PM Subject: [Wikidata] Data loss during the data centre switchover To: Discussion list for the Wikidata project. wikidata@lists.wikimedia.org
Hello all,
During the data centre switchover routine https://wikitech.wikimedia.org/wiki/Switch_Datacenter on October 10th, some unexpected problems occurred over the past days:
- For a few hours, a small part of the data was not accessible. Some items and lexemes seemed to have disappeared. - Some data may have been lost, including edits, preferences changed as well as user accounts created during a period or about 50 minutes (from 2018-09-13 09:08:17 UTC to 2018-09-13 09:58:26).
Part of the data has already been restored (edits and revisions. The rest (user accounts, preferences) will be restored at the beginning of next week.
If you edited Wikidata on September 13th, please check your contributions. If you encounter any problem in the next days, like items not reappearing or something missing, let me know.
If you're interested in technical details, you can have a look at the Phabricator ticket https://phabricator.wikimedia.org/T206743. Thanks for your understanding,
On Sat, Oct 13, 2018 at 4:38 AM Pine W wiki.pine@gmail.com wrote:
Forwarding news of data loss. :( Hopefully it will all be recoverable.
Hello,
We believe no data has been lost - the data inserted during the incident is still at our codfw DC (the active DC at the time), as there were no issues on that process.
At first and to quickly mitigate the impact of this incident we re-filled all the rows that were inserted using the binlogs from codfw for the most important tables, which might have left inconsistencies but not empty rows. At the same time, and in parallel, we are recloning eqiad DC hosts from codfw hosts (which have all the data, as there were no issues there) so all the data will be back in eqiad without inconsistencies as soon as we finish the recloning.
Cheers Manuel
---------- Forwarded message --------- From: Léa Lacroix lea.lacroix@wikimedia.de Date: Fri, Oct 12, 2018 at 2:46 PM Subject: [Wikidata] Data loss during the data centre switchover To: Discussion list for the Wikidata project. < wikidata@lists.wikimedia.org>
Hello all,
During the data centre switchover routine https://wikitech.wikimedia.org/wiki/Switch_Datacenter on October 10th, some unexpected problems occurred over the past days:
- For a few hours, a small part of the data was not accessible. Some
items and lexemes seemed to have disappeared.
- Some data may have been lost, including edits, preferences changed as
well as user accounts created during a period or about 50 minutes (from 2018-09-13 09:08:17 UTC to 2018-09-13 09:58:26).
Part of the data has already been restored (edits and revisions. The rest (user accounts, preferences) will be restored at the beginning of next week.
If you edited Wikidata on September 13th, please check your contributions. If you encounter any problem in the next days, like items not reappearing or something missing, let me know.
If you're interested in technical details, you can have a look at the Phabricator ticket https://phabricator.wikimedia.org/T206743. Thanks for your understanding, -- Léa Lacroix Project Manager Community Communication for Wikidata
Wikimedia Deutschland e.V. Tempelhofer Ufer 23-24 10963 Berlin www.wikimedia.de
Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.
Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207. _______________________________________________ Wikidata mailing list Wikidata@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Thanks for the info. I'm glad that all of the data will be recovered.
Pine ( https://meta.wikimedia.org/wiki/User:Pine )
On Sat, Oct 13, 2018 at 8:09 AM Manuel Arostegui marostegui@wikimedia.org wrote:
On Sat, Oct 13, 2018 at 4:38 AM Pine W wiki.pine@gmail.com wrote:
Forwarding news of data loss. :( Hopefully it will all be recoverable.
Hello,
We believe no data has been lost - the data inserted during the incident is still at our codfw DC (the active DC at the time), as there were no issues on that process.
At first and to quickly mitigate the impact of this incident we re-filled all the rows that were inserted using the binlogs from codfw for the most important tables, which might have left inconsistencies but not empty rows. At the same time, and in parallel, we are recloning eqiad DC hosts from codfw hosts (which have all the data, as there were no issues there) so all the data will be back in eqiad without inconsistencies as soon as we finish the recloning.
Cheers Manuel
---------- Forwarded message --------- From: Léa Lacroix lea.lacroix@wikimedia.de Date: Fri, Oct 12, 2018 at 2:46 PM Subject: [Wikidata] Data loss during the data centre switchover To: Discussion list for the Wikidata project. < wikidata@lists.wikimedia.org>
Hello all,
During the data centre switchover routine https://wikitech.wikimedia.org/wiki/Switch_Datacenter on October 10th, some unexpected problems occurred over the past days:
- For a few hours, a small part of the data was not accessible. Some
items and lexemes seemed to have disappeared.
- Some data may have been lost, including edits, preferences changed
as
well as user accounts created during a period or about 50 minutes
(from
2018-09-13 09:08:17 UTC to 2018-09-13 09:58:26).
Part of the data has already been restored (edits and revisions. The rest (user accounts, preferences) will be restored at the beginning of next week.
If you edited Wikidata on September 13th, please check your
contributions.
If you encounter any problem in the next days, like items not reappearing or something missing, let me know.
If you're interested in technical details, you can have a look at the Phabricator ticket https://phabricator.wikimedia.org/T206743. Thanks
for
your understanding,
Léa Lacroix Project Manager Community Communication for Wikidata
Wikimedia Deutschland e.V. Tempelhofer Ufer 23-24 10963 Berlin www.wikimedia.de
Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.
Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg
unter
der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207. _______________________________________________ Wikidata mailing list Wikidata@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
wikitech-l@lists.wikimedia.org