Hello!  There are certainly still some remaining issues. 

I think most relevant to users is that there is a large delay in Hive event tables being refined.  
I believe I've just found a workaround for this problem, so hopefully the data will start rolling in now.  I'll check back in on this tomorrow.

Thanks all,
-Andrew

On Tue, Feb 9, 2021 at 4:10 PM Luca Toscano <ltoscano@wikimedia.org> wrote:
Hi everybody,

The upgrade is completed, we are resolving some bugs that are coming up with new versions etc.. Feel free to test and report to us if anything doesn't look right!

Thanks a lot!

Luca


On Tue, Feb 9, 2021 at 6:04 PM Andrew Otto <otto@wikimedia.org> wrote:
Hello all!

As of this moment the Hadoop migration is still ongoing.  We have run into some hurdles and are past our scheduled downtime, but things seem to be proceeding ok.  They are just taking longer than expected.  We're trying to get the cluster back on for use as soon as possible, but I estimate we are only about halfway there.  Once we get past the main migration phase, we still have some testing and verifying we need to do, so I expect the downtime to last several hours more.

Apologies for the unexpected delay!  We're working on it.  We'll try to keep you updated and let you know when the Hadoop cluster is safe to use again.

Thanks for your patience,
- Andrew

On Wed, Feb 3, 2021 at 3:51 AM Luca Toscano <ltoscano@wikimedia.org> wrote:
Hi everybody,

The upgrade day has been scheduled, we are going to migrate Hadoop to the Apache Bigtop distribution on February 9th, during the EU morning. This will require from 2 to 4 hours of Hadoop downtime, since the upgrade will be very delicate and complex.

I created https://phabricator.wikimedia.org/T273711 to track more precisely timings and updates, please use it to ask questions and to tell us if this impacts your work or important deadlines for your team (in case we'll try to find a different time window).

Since we are upgrading software that was released years ago, it may probably happen that right after the upgrade some tools/workflows/etc.. don't work as expected anymore. We have tested a wide variety of use cases in our testing environment, but some corner cases might have been missed. In case you notice something weird right after the upgrade, please let us know how to repro in the task, we'll follow up and hopefully fix promptly.

Thanks a lot for the support!

Luca
--
Analytics-announce mailing list
Analytics-announce@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/analytics-announce
--
Analytics-announce mailing list
Analytics-announce@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/analytics-announce
--
Analytics-announce mailing list
Analytics-announce@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/analytics-announce