Hello,
If you use wmfdata-python <https://github.com/wikimedia/wmfdata-python>,
please *upgrade it soon* to version 2.2.0 in order to allow its presto
support to keep working.
We have just deployed a new version of our conda-analytics
<https://wikitech.wikimedia.org/wiki/Data_Engineering/Systems/Conda>
environment, which has this new version of wmfdata installed by default,
so you can use conda-analytics-clone to make a new, custom environment
for yourself. Alternatively, you can update it within your existing
environments with:
pip install --upgrade
git+https://github.com/wikimedia/wmfdata-python.git@release
This upgrade is necessary because we are in the process of improving our
Presto <https://wikitech.wikimedia.org/wiki/Analytics/Systems/Presto>
services, for which we need to change the TLS certificates that secure
our client connections to the Presto cluster. Versions of wmfdata-python
prior to v2.2.0 were hard-coded
<https://github.com/wikimedia/wmfdata-python/commit/b7b5df4651c880ad6fc0980c…>
to use our Puppet based Certificate Authority.
We will implement a change
<https://gerrit.wikimedia.org/r/c/operations/puppet/+/709713> to the
Presto configuration to switch the certificates *around mid-January
2024*, at which point any versions of wmfdata-python prior to 2.2.0 will
cease to connect to Presto and will return an error. I will send further
updates nearer the time, with more precise dates.
Please do let me know if you have any queries or concerns about this change.
Kind regards,
Ben
--
*Ben Tullis*(he/him)
Senior Site Reliability Engineer
Wikimedia Foundation <https://wikimediafoundation.org/>
Hello,
You can ignore this email unless you use any of the Airflow
<https://wikitech.wikimedia.org/wiki/Data_Engineering/Systems/Airflow>
instances
<https://wikitech.wikimedia.org/wiki/Data_Engineering/Systems/Airflow/Instan…>
managed by the Data Platform Engineering team.
Since the recent Airflow upgrade to version 2.7.3 we discovered a
regression that affects all of our instances. It's a small bug
<https://github.com/apache/airflow/issues/36206> but it means that since
the upgrade users have been unable to add notes to any tasks. See:
T352534 <https://phabricator.wikimedia.org/T352534> for more detail.
In the short term, we have decided to implement a workaround, which is
to create an *admin:admin* user for each Airflow instance. You can use
this to log in if you wish to manage notes associated with your DAG run
tasks, the login link is at the top-right of the Airflow UI.
As the ability to access each Airflow instance is currently limited to
those with SSH access to the host, this change is not granting anyone
any additional rights that they do not already have. It's merely an
inconvenience, for which we apologise.
We have several longer-term options in mind, but I won't go into them here.
I have made a note of this configuration detail here, in case you would
like to refer to it again:
https://wikitech.wikimedia.org/wiki/Data_Engineering/Systems/Airflow#Authen…
Naturally, please feel free to get in touch if you have any queries or
concerns about this.
Kind regards,
Ben
--
*Ben Tullis*(he/him)
Senior Site Reliability Engineer
Wikimedia Foundation <https://wikimediafoundation.org/>