I suspect this will affect containers that run Debian Buster packages.  I see php7.3 and python3.7.
I’d suggest not even restarting web services on those runtimes until we have it fixed.  For anyone who has done so, we are working on it.  Any logs could be helpful.

Brooke Storm
Senior SRE
Wikimedia Cloud Services
bstorm@wikimedia.org
IRC: bstorm_

On Jan 13, 2020, at 9:06 AM, Brooke Storm <bstorm@wikimedia.org> wrote:

I see a problem with at least one container image (which has nothing to do with the new cluster, I can see it on the old cluster as well).  It looks like I’m going to be trying to fix that now.
(Magnus, this is probably what you are seeing as well).

Brooke Storm
Senior SRE
Wikimedia Cloud Services
IRC: bstorm_

On Jan 13, 2020, at 8:47 AM, Magnus Manske via Cloud <cloud@lists.wikimedia.org> wrote:

I switched a few "big ones" successfully, but ran into one that doesn't work:

glamtools https://tools.wmflabs.org/glamtools/ is 503 but `webservice status` says "Your webservice of type php7.3 is running".

On Thu, Jan 9, 2020 at 9:58 PM Bryan Davis <bd808@wikimedia.org> wrote:
I am happy to announce that a new and improved Kubernetes cluster is
now available for use by beta testers on an opt-in basis. A page has
been created on Wikitech [0] outlining the self-service migration
process.

Timeline:
* 2020-01-09: 2020 Kubernetes cluster available for beta testers on an
opt-in basis
* 2020-01-23: 2020 Kubernetes cluster general availability for
migration on an opt-in basis
* 2020-02-10: Automatic migration of remaining workloads from 2016
cluster to 2020 cluster by Toolforge admins

This new cluster has been a work in progress for more than a year
within the Wikimedia Cloud Services team, and a top priority project
for the past six months. About 35 tools, including
https://tools.wmflabs.org/admin/, are currently running on what we are
calling the "2020 Kubernetes cluster". This new cluster is running
Kubernetes v1.15.6 and Docker 19.03.4. It is also using a newer
authentication and authorization method (RBAC), a new ingress routing
service, and a different method of integrating with the Developer
account LDAP service. We have built a new tool [1] which makes the
state of the Kubernetes cluster more transparent and on par with the
information that we already expose for the grid engine cluster [2].

With a significant number of tools managed by Toolforge administrators
already migrated to the new cluster, we are fairly confident that the
basic features used by most Kubernetes tools are covered. It is likely
that a few outlying issues remain to be found as more tools move, but
we have confidence that we can address them quickly. This has led us
to propose a fairly short period of voluntary beta testing, followed
by a short general availability opt-in migration period, and finally a
complete migration of all remaining tools which will be done by the
Toolforge administration team for anyone who has not migrated their
self.

Please help with beta testing if you have some time and are willing to
get help on irc, Phabricator, and the cloud@lists.wikimedia.org
mailing list for early adopter issues you may encounter.

I want to publicly praise Brooke Storm and Arturo Borrero González for
the hours that they have put into reading docs, building proof of
concept clusters, and improving automation and processes to make the
2020 Kubernetes cluster possible. The Toolforge community can look
forward to more frequent and less disruptive software upgrades in this
cluster as a direct result of this work. We have some other feature
improvements in planning now that I think you will all be excited to
see and use later this year!

[0]: https://wikitech.wikimedia.org/wiki/News/2020_Kubernetes_cluster_migration
[1]: https://tools.wmflabs.org/k8s-status/
[2]: https://tools.wmflabs.org/sge-status/

Bryan (on behalf of the Toolforge admins and the Cloud Services team)
--
Bryan Davis              Technical Engagement      Wikimedia Foundation
Principal Software Engineer                               Boise, ID USA
[[m:User:BDavis_(WMF)]]                                      irc: bd808

_______________________________________________
Wikimedia Cloud Services announce mailing list
Cloud-announce@lists.wikimedia.org (formerly labs-announce@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce
_______________________________________________
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud
_______________________________________________
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud