Hi everyone!
*tl;dr stop using notebook1001 by Monday April 2nd, use notebook1003
instead.*
*(If you don’t have production access, you can ignore this email.)*
As part of
https://phabricator.wikimedia.org/T183145, we’ve ordered new
hardware to replace the aging notebook1001. The new servers are ready to
go, so we need to schedule a deprecation timeline for notebook1001. That
timeline is Monday April 2nd. After that, your work on notebook1001 will
not longer be accessible. Instead you should use notebook1003 (or
notebook1004).
But there is good news too! Last week I rsynced everyone’s home
directories from notebook1001 over to notebook1003. I also upgraded the
default virtualenv your notebooks run from. Your notebook files should all
be accessible on notebook1003. However, the version of Python3 changed
from 3.4 to 3.5 during this upgrade. Dependencies that your notebook uses
that you installed on notebook1001 may not be available at first. You
might need to redo a pip install those dependencies into the new notebook
Python 3.5 virtualenv. (I can’t really give you explicit instructions to
do that, as I don’t know what you use for your notebooks.)
I’ll do a final rsync any newer files in home directories from notebook1001
on Monday April 2nd. If you’ve been working on notebook1001 since after
March 15th, this should get everything up to date on notebook1003 before
notebook1001 goes away. BUT! *Do not work on both notebook1001 and
notebook1003*! My final rsync will keep the most recently modified version
of files from either server.
OOooOo and there’s even more good news! I’ve made the notebooks able to
access system site packages, and installed a ton of useful packages
<https://github.com/wikimedia/puppet/blob/production/modules/statistics/manifests/packages.pp#L77-L98>
by default. pandas, scipy, requests, etc. If there’s something else you
think you might need, let us know. Or just pip install it into your
notebook.
Additionally, pyhive has been installed too, so you should be able to more
easily access Hive directly from a python notebook.
I’ve updated docs at
https://wikitech.wikimedia.org/wiki/SWAP#Usage, please
take a look.
If you have any questions, please don’t hesitate to ask, either here on or
phabricator:
https://phabricator.wikimedia.org/T183145.
- Andrew Otto & Analytics Engineering