This Tuesday at 17:00 UTC we'll be switching over from our old opendj-based ldap servers to new openldap-based ldap servers.
If all goes well, this should be largely unnoticeable to end-users. Lots of things depend on ldap, though, so we may see some weird, unpredictable behaviors during the switchover.
During the transition, the old servers will be marked as read-only. For this reason I advise against doing any stateful work during the maintenance window. Specifically: account, project and instance creation on wikitech are likely to misfire in complicated and unpleasant ways.
Here are some other things which should not break, but require ldap and are therefore subject to the whims of fate:
- shell auth on all labs instances - sudo policies on all labs instances - public dns for the wmflabs.org domain - all cron jobs on tools - most of wikitech - user login to monitoring tools
Moritz, Coren and I will be available on IRC during the scheduled window to troubleshoot issues if and when they arise.
-Andrew
This switch-over is done, and we've confirmed that all services (that we can think of) are working. Caveats:
1) If your labs instance has stopped working (or you can't reach it), ping me or Coren on IRC and we'll have a look.
2) If you have any hard-coded references to neptunium or nembus (e.g. in a labs instance) please change them to seaborgium and serpens, respectively
3) We're leaving the old ldap servers up in read-only mode for a day or so. If everything is working for you now but in a few days things break, that's probably because you didn't do step 2. Act now!
Many thanks to Moritz for setting up the new ldap servers. And, thanks also to everyone who helped test!
-Andrew
On 12/3/15 12:07 PM, Andrew Bogott wrote:
This Tuesday at 17:00 UTC we'll be switching over from our old opendj-based ldap servers to new openldap-based ldap servers.
If all goes well, this should be largely unnoticeable to end-users. Lots of things depend on ldap, though, so we may see some weird, unpredictable behaviors during the switchover.
During the transition, the old servers will be marked as read-only. For this reason I advise against doing any stateful work during the maintenance window. Specifically: account, project and instance creation on wikitech are likely to misfire in complicated and unpleasant ways.
Here are some other things which should not break, but require ldap and are therefore subject to the whims of fate:
- shell auth on all labs instances
- sudo policies on all labs instances
- public dns for the wmflabs.org domain
- all cron jobs on tools
- most of wikitech
- user login to monitoring tools
Moritz, Coren and I will be available on IRC during the scheduled window to troubleshoot issues if and when they arise.
-Andrew
On 12/8/15 4:24 PM, Andrew Bogott wrote:
This switch-over is done, and we've confirmed that all services (that we can think of) are working. Caveats:
- If your labs instance has stopped working (or you can't reach it),
ping me or Coren on IRC and we'll have a look.
- If you have any hard-coded references to neptunium or nembus (e.g.
in a labs instance) please change them to seaborgium and serpens, respectively
...or, better yet, the actual ldap service names: ldap-labs.eqiad.wikimedia.org and ldap-labs.codfw.wikimedia.org.
In one hour we are going to turn off the old ldap servers. Here are some consequences you can expect from this:
- Anything that is using an ldap IP address cached from last week will either die or reset itself - Anything that contains literal references to neptunium or nembus will no longer work - Anything that contains literal references to ldap-eqiad.wikimedia.org or ldap-codfw.wikimedia.org is already being redirected to the new servers (but that results in cert mismatches in the case of tls or secure ldap.) This has been true since the 8th, and won't change. - Anything that is properly puppetized will be just fine.
Feel free to contact me or Moritz for advice about how to troubleshoot issues, should they arise.
-Andrew
On 12/8/15 4:24 PM, Andrew Bogott wrote:
This switch-over is done, and we've confirmed that all services (that we can think of) are working. Caveats:
- If your labs instance has stopped working (or you can't reach it),
ping me or Coren on IRC and we'll have a look.
- If you have any hard-coded references to neptunium or nembus (e.g.
in a labs instance) please change them to ldap-labs.eqiad.wikimedia.org and ldap-labs.codfw.wikimedia.org, respectively
- We're leaving the old ldap servers up in read-only mode for a day
or so. If everything is working for you now but in a few days things break, that's probably because you didn't do step 2. Act now!
Many thanks to Moritz for setting up the new ldap servers. And, thanks also to everyone who helped test!
-Andrew
On 12/3/15 12:07 PM, Andrew Bogott wrote:
This Tuesday at 17:00 UTC we'll be switching over from our old opendj-based ldap servers to new openldap-based ldap servers.
If all goes well, this should be largely unnoticeable to end-users. Lots of things depend on ldap, though, so we may see some weird, unpredictable behaviors during the switchover.
During the transition, the old servers will be marked as read-only. For this reason I advise against doing any stateful work during the maintenance window. Specifically: account, project and instance creation on wikitech are likely to misfire in complicated and unpleasant ways.
Here are some other things which should not break, but require ldap and are therefore subject to the whims of fate:
- shell auth on all labs instances
- sudo policies on all labs instances
- public dns for the wmflabs.org domain
- all cron jobs on tools
- most of wikitech
- user login to monitoring tools
Moritz, Coren and I will be available on IRC during the scheduled window to troubleshoot issues if and when they arise.
-Andrew
wikitech-l@lists.wikimedia.org