-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi,
Due to the current high replag on thyme, I've switched sql-s1 to
rosemary. If your tools require a user database and you haven't updated
them to use sql-s1-user instead, you will find your databases are
missing. Once the replag is down to a reasonable level I'll switch it
back.
- river.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (NetBSD)
iEYEARECAAYFAk13HAsACgkQIXd7fCuc5vIvdQCffgfjFv9OKmvpoWNY/0+LiTqy
IeUAniODzwDbdQG7jcf3Jp/3awc40gLR
=MBKi
-----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi,
We're currently applying schema changes from WMF for a MediaWiki
upgrade. To reduce replication lag, we will first apply them to an
inactive server for each cluster, then switch users to that server and
apply them on the other one. The first part has started now, and we
will be switching in an hour or two.
During the second part, user databases will be unavailable until we
finish the changes and switch back. This may take several hours to
finish.
The exception is s2/s5, which only has one server; these clusters will
be lagged during the change.
- river.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (NetBSD)
iEYEARECAAYFAk11bNYACgkQIXd7fCuc5vIIowCgqKiJvPxtr+B19TRDrbJTy/70
fHQAn3CW69mS7tQPWrRxdMTKYnuFM5St
=DpI/
-----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi,
I've added yarrow as a login server and SGE execution host for all.q and
longrun. All SGE jobs are now eligible to run there; no action is
required from users.
This server is for SGE only, tools must not be started there manually.
- river.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (NetBSD)
iEYEARECAAYFAk11fv4ACgkQIXd7fCuc5vK24wCgjGrOQSwxLdndb3FeVbN/Dr4l
FbQAn1GMC00MN8vDklXQqFBgDVv1mS7J
=yQue
-----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi,
Tonight I will implement the mail forwarding changes described here:
<http://lists.wikimedia.org/pipermail/toolserver-announce/2011-February/0004…>.
There should be no interruption in service, except that mail to
user(a)toolserver.org might be delayed during the maintenance.
Start time: Monday, 7th March 0000h UTC
http://time.tcx.org.uk/utc/2011-03-07/00:00
End time: Monday, 7th Match 0100h UTC
http://time.tcx.org.uk/utc/2011-03-07/01:00
Sorry for the short notice.
- river.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (NetBSD)
iEYEARECAAYFAk1z3dgACgkQIXd7fCuc5vJtSwCgiZO5Mu3pMIOJrhcuYUekOHvw
awQAni555uAR0bEORIUV869VqSlS1hT/
=kRiq
-----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi,
During the maintenance on December 6th, 2010 I switched the Toolserver
SSH server from Sun SSH to OpenSSH. A difference in how OpenSSH uses
PAM to authenticate users meant that after the change, users were able
to log via SSH using their LDAP password, without using an SSH key.
This error has now been fixed.
If you have no LDAP password set, or if you have a strong password[0],
then this should not have affected you. However, if you had a weak or
easily guessable password set, or if your LDAP password could have been
compromised (e.g. if you wrote it down in plain text somewhere) then
it's possible someone could have used it to gain access to your account.
In that case, I suggest you immediately change your password (via
'passwd'), then review your home directory to ensure no unauthorised
changes have been made (e.g. new SSH keys added, or shell rc files
changed). If you have sensitive data such as SSH or PGP keys on the
Toolserver, you may wish to revoke them and issue new ones. (However,
storing that kind of data on the Toolserver is probably a bad idea in
any case.)
I'm very sorry for the inconvenience this issue might cause to users,
and I will be reviewing our authentication configuration to reduce the
chance of something like this happening in the future.
- river.
[0] Which is somewhat enforced by the LDAP password policy, but it's
still possible to set a weak password if you try hard enough.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (NetBSD)
iEYEARECAAYFAk1zvOgACgkQIXd7fCuc5vJpowCeMoLig31BAHnStWakKgeU/ZOr
pCYAoKMEF/6+yzzKGQNVYxXqJuhM2f63
=ykB1
-----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi,
Later tonight (around 12AM UTC[0]) I will restart the NFS server on
hemlock, which serves user-store. This will cause an interruption to
service that should last for under 1 minute.
- river.
[0] http://time.tcx.org.uk/utc/2011-03-05/00:00
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (NetBSD)
iEYEARECAAYFAk1xNqIACgkQIXd7fCuc5vL22QCeM2IXEqx0x3yRrcbXIbjVbCsW
GR0An1hYPDoUq58rs1G9fSxgTrT4P1h1
=4BKf
-----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi,
On the morning (UTC) of March 1 we will install some new hardware at the
Toolserver. Services will be affected as follows:
Service | Expected impact
----------------------------+--------------------------------
Databases: s3, s4, s6, s7 | User databases unavailable for
| < 10 minutes.
user-store filesystem | Unavailable for < 10 minutes.
Start time: Tuesday, 1st March, 1PM UTC (estimated)
http://time.tcx.org.uk/utc/2011/3/1/13/0
End time: Tuesday, 1st March, 4PM UTC (estimated)
http://time.tcx.org.uk/utc/2011/3/1/16/0
Details:
We will install new storage for databases and the user-store filesystem.
This requires installing HBAs in three servers: hemlock, cassia, and
hyacinth, which requires downtime. We will try to stagger the
installation of the two database servers so that those clusters
remaining accessible during the maintenance, but user databases will be
unavailable while hyacinth is down.
Before the maintenance on hemlock, we will forcibly unmount the
user-store filesystem. Any program accessing it at the time will
receive an I/O error.
- river.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (NetBSD)
iEYEARECAAYFAk1ro3AACgkQIXd7fCuc5vLhsQCfbKKphnxKxXCM3VNCJC+YOsz+
E5sAn3EBKJjF1PeU4WZCT5W9KFNuqGqM
=TGse
-----END PGP SIGNATURE-----