Hello!
Today I deployed a change that upgraded the anaconda-wmf
<https://wikitech.wikimedia.org/wiki/Analytics/Systems/Anaconda> release
across the analytics cluster. For most of you, this will be a no-op, as it
mostly contains improvements to the way conda environments are created, and
improves some integration with Spark in Jupyter notebooks.
This does update some dependencies in the anaconda-wmf base conda
environment, so if you see any weirdness with your existing conda
environments, let me know on https://phabricator.wikimedia.org/T224658.
Thanks! Stay tuned for more announcements about JupyterHub. :)
-Andrew Otto
SRE, Data Engineering
FYI, SRE has re-imaged bast1002.wikimedia.org. This means that if you use
this as your ssh bastion, you will get a warning about SSH key change.
If the key being offered you is the one in
https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/bast1002.wikimedi…,
you can accept the new key.
---------- Forwarded message ---------
From: Moritz Mühlenhoff <mmuhlenhoff(a)wikimedia.org>
Date: Wed, Feb 24, 2021 at 4:44 AM
Subject: Re: [Ops] Reimage of bast1002 tomorrow
To: Operations Engineers <ops(a)lists.wikimedia.org>
On Tue, Feb 23, 2021 at 11:25 AM Moritz Mühlenhoff
<mmuhlenhoff(a)wikimedia.org> wrote:
> I'm going to reimage bast1002 for an OS update to Buster tomorrow
> during the early European morning. Please use a different bastion
> during that time.
This is complete, you can use bast1002.wikimedia.org again.
You can fetch the updated fingerprint by running the
wmf-update-known-hosts-production script, or as a fallback updated
fingerprints are also at
https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/bast1002.wikimedi…
.
Cheers,
Moritz
_______________________________________________
Ops mailing list
Ops(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/ops
Hey all!
wmfdata-python <https://github.com/wikimedia/wmfdata-python> (a package
that streamlines access to private analytics data) has been updated to
version 1.1. Here's what's new:
- The new presto module supports querying the Data Lake using Presto
<https://wikitech.wikimedia.org/wiki/Analytics/Systems/Presto>.
- The spark module has been refactored to support local and custom
sessions.
- A new utils.get_dblist function provides easy access to wiki database
lists, which is particularly useful with mariadb.run.
- The hive.run_cli function now creates its temp files in standard
location, to avoid creating distracting new entries in the current working
directory.
Many thanks to:
- Andrew Otto and Adam Roses Wight for writing significant new code
- Mikhail Popov, Andrew Otto, and Luca Toscano for careful code review
As always, if you have questions or feedback about wmfdata-python, please
email Product Analytics at product-analytics(a)wikimedia.org.
--
Neil Shah-Quinn
senior data scientist, Product Analytics
<https://www.mediawiki.org/wiki/Product_Analytics>
Wikimedia Foundation <https://wikimediafoundation.org/>
Hi everybody,
I'd need to reboot stat1005 and stat1008 for kernel upgrades. The scheduled
maintenance window is: Monday Feb 22nd 9AM CET (so early EU morning).
Also added to
https://wikitech.wikimedia.org/wiki/Analytics/Systems/Maintenance_Schedule
As always, let me know if this is a problem for your work, in case we'll
schedule a different time window :)
Luca
Hi everybody,
I am back with reboots, please be patient with me :)
I am going to reboot stat1004 / stat1006 / stat1007 (only these three for
the moment) on Wednesday Feb 17 at 9AM CET for Linux Kernel upgrades.
Please let me know if this impacts your work, in case we'll find another
maintenance window :)
Scheduled maintenance also outlined in
https://wikitech.wikimedia.org/wiki/Analytics/Systems/Maintenance_Schedule
Luca (on behalf of the Data Engineering / Analytics team)
Hi everybody,
The upgrade day has been scheduled, we are going to migrate Hadoop to the
Apache Bigtop distribution on February 9th, during the EU morning. This
will require from 2 to 4 hours of Hadoop downtime, since the upgrade will
be very delicate and complex.
I created https://phabricator.wikimedia.org/T273711 to track more precisely
timings and updates, please use it to ask questions and to tell us if this
impacts your work or important deadlines for your team (in case we'll try
to find a different time window).
Since we are upgrading software that was released years ago, it may
probably happen that right after the upgrade some tools/workflows/etc..
don't work as expected anymore. We have tested a wide variety of use cases
in our testing environment, but some corner cases might have been missed.
In case you notice something weird right after the upgrade, please let us
know how to repro in the task, we'll follow up and hopefully fix promptly.
Thanks a lot for the support!
Luca
Hi all!
We just finished <https://phabricator.wikimedia.org/T269160> setting up an
internal instance of EventStreams called eventstreams-internal. This
instance is not public, but does expose all streams declared in stream
config*.
I've added documentation about how to access this here:
https://wikitech.wikimedia.org/wiki/Event_Platform/Instrumentation_How_To#I…
This instance isn't particularly useful for building any services (in
production you should just consume from Kafka), but it may be very useful
for debugging and troubleshooting events in production. EventStreams has a
GUI that will allow you to see events in Kafka as they flow in. In
production, this will allow you to see events right after they are emitted,
without having to wait a few hours for them to be ingested into Hive. You
can use this to make sure events you trigger in production make it through
EventGate into Kafka as you expect.
Big thanks to Marcel and Luca for their work on this! :)
- Andrew Otto
* i.e. those that use Event Platform, not legacy EventLogging events.
Hi everybody,
as described in https://phabricator.wikimedia.org/T263972 the Apache
Superset devs have deprecated the Druid datasource definitions, in favor of
SQLAlchemy with the so called "Druid tables". I have worked with Product
Analytics some months ago to migrate charts to the new format, but since
then the usage of Superset grew a lot and more users started to create
charts without using Druid tables (but using Druid datasources). Today I
have disabled the Druid datasources in Superset, but charts and dashboards
using them are still visible. The only downside is that when clicking on
the definition of an old datasource, the user will get a HTTP 404.
I added some documentation to help users in the migration:
https://wikitech.wikimedia.org/wiki/Analytics/Systems/Superset#Druid_dataso…https://wikitech.wikimedia.org/wiki/Analytics/Systems/Superset#Migrate_a_ch…
The migration is really easy but a little boring, since every chart needs
to be migrated manually with a simple procedure (see links above). The
Analytics team is preparing the work to upgrade to Superser 1.0 and this is
part of that process :)
Please ping the Analytics team if you encounter any issue, or if you need
some help in migrating over to Druid tables.
Tracking task: https://phabricator.wikimedia.org/T263972
Thanks a lot for the patience,
Luca (on behalf of the Analytics / Data Engineering team)
Hi everybody,
There is a new config on stat100x hosts for Jupyter that allows the tmp
directory to be shared from notebooks and OS, so there is only one kerberos
credential cache now. This means that if you have a valid ticket on
stat100x then it will not be necessary to also kinit again in a jupyter
terminal (the other way around holds true as well).
In order to get the new config you'll need to shutdown and start again any
running notebooks (not a simple restart), let us know if you encounter any
issues.
Thanks!
Luca (on behalf of the Analytics / Data Engineering team)
Hi all,
We have updated HDFS permissions for our ticket
https://phabricator.wikimedia.org/T270629, and the updated permissions have
revealed issues that may affect Superset dashboards and some of the batch
jobs. We are aware and working on resolving these issues currently.
If you'd like to report an issue, comment on the ticket linked above or
message us on #wikimedia-analytics on freenode.
Regards,
Razzi & Analytics