Hi all!
We'll be upgrading Spark to 2.4.4 in the Analytics (Hadoop) cluster on Tuesday November 5th. We don't expect any real downtime, but if you have regularly scheduled jobs that are built with Spark 2.3, you'll likely want to rebuild them to use Spark 2.4 after Tuesday.
You can follow along here: https://phabricator.wikimedia.org/T222253
- Andrew Otto (Systems Engineer) & Analytics Engineering Team