I would love to migrate to stat1006 from stat1003, but stat1006 is unusably
slow right now while stat1003 is snappy. Connecting to analytics-store from
stat1006 times out sometimes, and even when it doesn't, simple DESCRIBE
queries take 5-15 seconds. I just talked to Adam Wight on IRC and he's
experiencing similar issues.
On Tue, Jul 18, 2017 at 10:31 AM, Andrew Otto <otto(a)wikimedia.org> wrote:
Hi all!
tl;dr: Stop using stat100[23] by September 1st.
We’re finally replacing stat1002 and stat1003. These boxes are out of
warranty, and are running Ubuntu Trusty, while most of the production fleet
is already on Debian Jessie or even Debian Stretch.
stat1005 is the new stat1002 replacement. If you have access to stat1002,
you also have access to stat1005. I’ve copied over home directories from
stat1002.
stat1006 is the new stat1003 replacement. If you have access to stat1003,
you also have access to stat1006. I’ve copied over home directories from
stat1003.
I have not migrated any personal cron jobs running on stat1002 or
stat1003. I need your help for this!
Both of these boxes are running Debian Stretch. As such, packages that
your work depends on may have upgraded. Please log into the new boxes and
try stuff out! If you find anything that doesn’t work, please let me know
by commenting on
https://phabricator.wikimedia.org/T152712.
Please be fully migrated to the new nodes by September 1st. This will
give us enough time to fully decommission stat1002 and stat1003 by the end
of this quarter.
I’ve only done a single rsync of home directories. If there is new data
on stat1002 or stat1003 that you want rsynced over, let me know on the
ticket.
A few notes:
- stat1002 used to have /a. This has been removed in favor of /srv. /a
no longer exists.
- Home directories are now much larger. You no longer need to create
personal directories in /srv.
- /tmp is still small, so please be careful. If you are running long jobs
that generate temporary data, please have those jobs write into your home
directory, rather than /tmp.
- We might implement user home directory quotas in the future.
Thanks all! I’ll send another email in about a months time to remind you
of the impending deadline of Sept 1.
-Andrew Otto
_______________________________________________
Ops mailing list
Ops(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/ops