Hi everybody,
The upgrade day has been scheduled, we are going to migrate Hadoop to the Apache Bigtop distribution on February 9th, during the EU morning. This will require from 2 to 4 hours of Hadoop downtime, since the upgrade will be very delicate and complex.
I created https://phabricator.wikimedia.org/T273711 to track more precisely timings and updates, please use it to ask questions and to tell us if this impacts your work or important deadlines for your team (in case we'll try to find a different time window).
Since we are upgrading software that was released years ago, it may probably happen that right after the upgrade some tools/workflows/etc.. don't work as expected anymore. We have tested a wide variety of use cases in our testing environment, but some corner cases might have been missed. In case you notice something weird right after the upgrade, please let us know how to repro in the task, we'll follow up and hopefully fix promptly.
Thanks a lot for the support!
Luca
Hello all!
As of this moment the Hadoop migration is still ongoing. We have run into some hurdles and are past our scheduled downtime, but things seem to be proceeding ok. They are just taking longer than expected. We're trying to get the cluster back on for use as soon as possible, but I estimate we are only about halfway there. Once we get past the main migration phase, we still have some testing and verifying we need to do, so I expect the downtime to last several hours more.
Apologies for the unexpected delay! We're working on it. We'll try to keep you updated and let you know when the Hadoop cluster is safe to use again.
Thanks for your patience, - Andrew
On Wed, Feb 3, 2021 at 3:51 AM Luca Toscano ltoscano@wikimedia.org wrote:
Hi everybody,
The upgrade day has been scheduled, we are going to migrate Hadoop to the Apache Bigtop distribution on February 9th, during the EU morning. This will require from 2 to 4 hours of Hadoop downtime, since the upgrade will be very delicate and complex.
I created https://phabricator.wikimedia.org/T273711 to track more precisely timings and updates, please use it to ask questions and to tell us if this impacts your work or important deadlines for your team (in case we'll try to find a different time window).
Since we are upgrading software that was released years ago, it may probably happen that right after the upgrade some tools/workflows/etc.. don't work as expected anymore. We have tested a wide variety of use cases in our testing environment, but some corner cases might have been missed. In case you notice something weird right after the upgrade, please let us know how to repro in the task, we'll follow up and hopefully fix promptly.
Thanks a lot for the support!
Luca
Analytics-announce mailing list Analytics-announce@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics-announce
Hi everybody,
The upgrade is completed, we are resolving some bugs that are coming up with new versions etc.. Feel free to test and report to us if anything doesn't look right!
Thanks a lot!
Luca
On Tue, Feb 9, 2021 at 6:04 PM Andrew Otto otto@wikimedia.org wrote:
Hello all!
As of this moment the Hadoop migration is still ongoing. We have run into some hurdles and are past our scheduled downtime, but things seem to be proceeding ok. They are just taking longer than expected. We're trying to get the cluster back on for use as soon as possible, but I estimate we are only about halfway there. Once we get past the main migration phase, we still have some testing and verifying we need to do, so I expect the downtime to last several hours more.
Apologies for the unexpected delay! We're working on it. We'll try to keep you updated and let you know when the Hadoop cluster is safe to use again.
Thanks for your patience,
- Andrew
On Wed, Feb 3, 2021 at 3:51 AM Luca Toscano ltoscano@wikimedia.org wrote:
Hi everybody,
The upgrade day has been scheduled, we are going to migrate Hadoop to the Apache Bigtop distribution on February 9th, during the EU morning. This will require from 2 to 4 hours of Hadoop downtime, since the upgrade will be very delicate and complex.
I created https://phabricator.wikimedia.org/T273711 to track more precisely timings and updates, please use it to ask questions and to tell us if this impacts your work or important deadlines for your team (in case we'll try to find a different time window).
Since we are upgrading software that was released years ago, it may probably happen that right after the upgrade some tools/workflows/etc.. don't work as expected anymore. We have tested a wide variety of use cases in our testing environment, but some corner cases might have been missed. In case you notice something weird right after the upgrade, please let us know how to repro in the task, we'll follow up and hopefully fix promptly.
Thanks a lot for the support!
Luca
Analytics-announce mailing list Analytics-announce@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics-announce
-- Analytics-announce mailing list Analytics-announce@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics-announce
Hello! There are certainly still some remaining issues https://phabricator.wikimedia.org/T273711#6817038.
I think most relevant to users is that there is a large delay in Hive event tables being refined. I believe I've just found a workaround for this problem, so hopefully the data will start rolling in now. I'll check back in on this tomorrow.
Thanks all, -Andrew
On Tue, Feb 9, 2021 at 4:10 PM Luca Toscano ltoscano@wikimedia.org wrote:
Hi everybody,
The upgrade is completed, we are resolving some bugs that are coming up with new versions etc.. Feel free to test and report to us if anything doesn't look right!
Thanks a lot!
Luca
On Tue, Feb 9, 2021 at 6:04 PM Andrew Otto otto@wikimedia.org wrote:
Hello all!
As of this moment the Hadoop migration is still ongoing. We have run into some hurdles and are past our scheduled downtime, but things seem to be proceeding ok. They are just taking longer than expected. We're trying to get the cluster back on for use as soon as possible, but I estimate we are only about halfway there. Once we get past the main migration phase, we still have some testing and verifying we need to do, so I expect the downtime to last several hours more.
Apologies for the unexpected delay! We're working on it. We'll try to keep you updated and let you know when the Hadoop cluster is safe to use again.
Thanks for your patience,
- Andrew
On Wed, Feb 3, 2021 at 3:51 AM Luca Toscano ltoscano@wikimedia.org wrote:
Hi everybody,
The upgrade day has been scheduled, we are going to migrate Hadoop to the Apache Bigtop distribution on February 9th, during the EU morning. This will require from 2 to 4 hours of Hadoop downtime, since the upgrade will be very delicate and complex.
I created https://phabricator.wikimedia.org/T273711 to track more precisely timings and updates, please use it to ask questions and to tell us if this impacts your work or important deadlines for your team (in case we'll try to find a different time window).
Since we are upgrading software that was released years ago, it may probably happen that right after the upgrade some tools/workflows/etc.. don't work as expected anymore. We have tested a wide variety of use cases in our testing environment, but some corner cases might have been missed. In case you notice something weird right after the upgrade, please let us know how to repro in the task, we'll follow up and hopefully fix promptly.
Thanks a lot for the support!
Luca
Analytics-announce mailing list Analytics-announce@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics-announce
-- Analytics-announce mailing list Analytics-announce@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics-announce
-- Analytics-announce mailing list Analytics-announce@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics-announce
Hi everybody,
The summary of why the migration took so long is described in https://phabricator.wikimedia.org/T273711#6818136 for anybody interested :)
Luca
On Wed, Feb 10, 2021 at 1:13 AM Andrew Otto otto@wikimedia.org wrote:
Hello! There are certainly still some remaining issues https://phabricator.wikimedia.org/T273711#6817038.
I think most relevant to users is that there is a large delay in Hive event tables being refined. I believe I've just found a workaround for this problem, so hopefully the data will start rolling in now. I'll check back in on this tomorrow.
Thanks all, -Andrew
On Tue, Feb 9, 2021 at 4:10 PM Luca Toscano ltoscano@wikimedia.org wrote:
Hi everybody,
The upgrade is completed, we are resolving some bugs that are coming up with new versions etc.. Feel free to test and report to us if anything doesn't look right!
Thanks a lot!
Luca
On Tue, Feb 9, 2021 at 6:04 PM Andrew Otto otto@wikimedia.org wrote:
Hello all!
As of this moment the Hadoop migration is still ongoing. We have run into some hurdles and are past our scheduled downtime, but things seem to be proceeding ok. They are just taking longer than expected. We're trying to get the cluster back on for use as soon as possible, but I estimate we are only about halfway there. Once we get past the main migration phase, we still have some testing and verifying we need to do, so I expect the downtime to last several hours more.
Apologies for the unexpected delay! We're working on it. We'll try to keep you updated and let you know when the Hadoop cluster is safe to use again.
Thanks for your patience,
- Andrew
On Wed, Feb 3, 2021 at 3:51 AM Luca Toscano ltoscano@wikimedia.org wrote:
Hi everybody,
The upgrade day has been scheduled, we are going to migrate Hadoop to the Apache Bigtop distribution on February 9th, during the EU morning. This will require from 2 to 4 hours of Hadoop downtime, since the upgrade will be very delicate and complex.
I created https://phabricator.wikimedia.org/T273711 to track more precisely timings and updates, please use it to ask questions and to tell us if this impacts your work or important deadlines for your team (in case we'll try to find a different time window).
Since we are upgrading software that was released years ago, it may probably happen that right after the upgrade some tools/workflows/etc.. don't work as expected anymore. We have tested a wide variety of use cases in our testing environment, but some corner cases might have been missed. In case you notice something weird right after the upgrade, please let us know how to repro in the task, we'll follow up and hopefully fix promptly.
Thanks a lot for the support!
Luca
Analytics-announce mailing list Analytics-announce@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics-announce
-- Analytics-announce mailing list Analytics-announce@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics-announce
-- Analytics-announce mailing list Analytics-announce@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics-announce
-- Analytics-announce mailing list Analytics-announce@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics-announce
analytics-announce@lists.wikimedia.org