Hi Everyone,
The Data Engineering team is upgrading to Spark 3 and will no longer be
supporting Spark 2 jobs on the Hadoop cluster after March 31st, 2023. If
your team owns Spark 2 jobs in production, please plan for the time needed
to upgrade your jobs. For all future work use Spark 3.
You can find more information about the upgrade on:
https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Spark/Migrati….
Please add any missing jobs to the migration list on that page. If you need
help from the data engineering team you can reach out to Jackeline Argüello
<jarguello-ctr(a)wikimedia.org> or join us for the data engineering office
hours.
--
*Olja Dimitrijevic* (she/her)
Director of Data Engineering
Wikimedia Foundation <https://wikimediafoundation.org/>