Software that uses /data/scratch may see some disruption tomorrow when I merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/695447/ <https://gerrit.wikimedia.org/r/c/operations/puppet/+/695447/> around 20:00 UTC
In order to make the NFS it works on more resilient and healthy, the cluster it runs on is migrating to DRBD failover like Tools home and project uses. The current setup is quite broken.
If you are running something against scratch when the patch becomes active in your project, you may see some issues like a stale NFS handle. If that doesn’t resolve quickly, please let us know in Libera.chat: #wikimedia-cloud
Brooke Storm
Staff SRE
Wikimedia Cloud Services
bstorm(a)wikimedia.org
_______________________________________________
Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org
List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…
TL;DR:
* The #wikimedia-cloud IRC channel is moving from Freenode to Libera.Chat.
* Register an account on Libera.Chat and join us there!
There has been a lot of activity over the last 2-3 days related to
staffing changes on the Freenode IRC network [0]. The Wikimedia IRC
Group Contacts (GCs) [1] evaluated the situation and decided that
moving the Wikimedia IRC channels from Freenode to the brand new
Libera.Chat IRC network [2] would be the best course of action [3].
So, we are moving!
A new #wikimedia-cloud channel has been created on irc.libera.chat for
this Wikimedia sub-community to use. The old channel on Freenode still
exists and will be maintained at least until we can get all the bots
moved, our documentation updated on wikitech, and we see more folks on
the Libera channel than the Freenode one. Messages to our channel in
either IRC network, as well as the Telegram channel [4], will be seen
on all other channels.
There is a new subpage on meta [5] for information on how to create a
new account for yourself on Libera.Chat and other related information.
There is also a tracking task [6] that you can look at to see various
activities that the community hopes to take action on to complete the
migration.
One last thing: The #wmhack Freenode channel is bridged to
#wikimedia-hackathon on Libera.Chat. The new channel name will make it
easier for the GCs to help manage spam and other issues that come up
occasionally on IRC. Don't miss the fun of our 2021 virtual hackathon
from Friday, May 21st to Sunday, May 23rd! [7]
[0]: https://www.kline.sh/
[1]: https://meta.wikimedia.org/wiki/IRC/Group_Contacts
[2]: https://libera.chat/
[3]: https://meta.wikimedia.org/w/index.php?diff=21476411
[4]: https://t.me/wmcloudirc
[5]: https://meta.wikimedia.org/wiki/IRC/Migrating_to_Libera_Chat
[6]: https://phabricator.wikimedia.org/T283247
[7]: https://www.mediawiki.org/wiki/Wikimedia_Hackathon_2021
Bryan, on behalf of the WMCS team and the Cloud VPS and Toolforge admins
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
_______________________________________________
Wikimedia Cloud Services announce mailing list
To unsubscribe send an email to cloud-announce-leave(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org)
tl;dr: please respond with your use cases and concerns about secret
keys, passwords, etc on cloud-vps
Right now we have two not-very-good ways to distribute a secret key
within a cloud project:
1) copy the secret onto each VM by hand as you create it
2) create a puppetmaster in your project and and have it distribute the
secrets
Option #2 is pretty good for big, long-lived projects but involves a lot
of overhead. Option #1 is fine for projects with only one or two VMs
but scales terribly.
The WMCS team has been discussing the question of secrets distribution
for years, and we have a few different possible solutions in mind but no
favorite. Rather than rattle off those options here, we're trying to
rewind a bit and gather possible use-cases and user wishes in this area.
So: if this is a feature you've been missing, please respond with an
answer to this question:
"How would I make use of secrets on cloud-vps instances?"
If possible, please include thoughts about these points:
- Is it good enough to provide project-wide distribution, or do we need
finer-grained control, limiting secrets to particular users or instances?
- Is a web UI for managing secrets a requirement, or are command line
tools adequate? What if there were /only/ a web-ui and no command line?
- Would supporting secret management solve immediate issues on its own,
or is it only useful as a part of larger instrumentation tooling (e.g.
puppet, heat, or terraform integration)
Thank you! I welcome your thoughts on-list, but you're also welcome to
list thoughts or use-cases on the phabricator tracking task
https://phabricator.wikimedia.org/T283032
Hi,
Some of the jobs I submit to the grid take a long time (say, 30-60 minutes)
and I would like to check on their status without having to log back into
the labs.
I was hoping I could run shell_exec('qstat') in PHP and display its output
on a web page. While shell_exec() works with other commands—e.g. echo
shell_exec('whoami') correctly displays my tool's username on the
webpage—for reasons I cannot explain, the output of shell_exec('qstat') is
always blank. When I run qstat on the CLI at that same time, it does show
me the familiar table output of ongoing jobs.
Any idea why that is the case? Has anyone already created a solution for
fetching a list of active jobs from the grid and displaying it on a
web-based status page for your tool?
Thanks,
Huji
I'm getting errors regarding expired certificates for wmflabs.org and
wmcloud.org, e.g. https://wikispeech.wmflabs.org and
https://codesearch.wmcloud.org. Is this related to the maintenance later
today or has something gone wrong? Here's an example from curl:
$ curl https://codesearch.wmcloud.org
curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.haxx.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
Best,
*Sebastian Berlin*
Utvecklare/*Developer*
Wikimedia Sverige (WMSE)
E-post/*E-Mail*: sebastian.berlin(a)wikimedia.se
Telefon/*Phone*: (+46) 0707 - 92 03 84
Hello there,
We will be doing an upgrade to the CloudVPS edge network Thursday 2021-05-06 @
15:00 UTC that will likely impact user experience, including Toolforge.
We scheduled an 1h operation window. During that time, intermittent network
interruption, packet loss and other network problems are to be expected.
The edge network maintenance will affect how virtual machines (and Toolforge
tools) contact NFS, wiki-replicas, wikis API endpoints, and, in general, any
network traffic that flows leaving or entering the cloud (also known as
north-south traffic).
More information on the operation can be found in phabricator [0] and in
wikitech [1].
Regards.
[0]https://phabricator.wikimedia.org/T270704
[1]
https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/Enhanceme…
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
<tl;dr>: Your feedback as a technically-minded Wikimedian is welcome on
https://www.mediawiki.org/wiki/Developer_Advocacy/Developer_Portal/Content_…
Hi everyone,
last year[1] the Developer Advocacy team started to work on a single,
central entry point for developers and tech-minded people for
Wikimedia's technical documentation[2].
A central entry point ("developer portal") would cover common technical
use cases to allow existing and future technical contributors and
developers to find the information they need.
Each use case links to its most relevant documentation (i.e. to pages
on wikitech or mediawiki.org).
This is part of a larger initiative to implement an organization
strategy for key technical documents: Understand challenges about
finding and maintaining docs, identify key docs, and investigate ways
to improve our workflows around documentation.
So far we:
* Researched and reviewed existing documentation venues and pages
* Reviewed developer/documentation portals in the broader industry
* Interviewed several engineering teams around technical documentation
workflows, audiences, and key technical docs (the key themes from
these conversations are available, see the link above)
* Created an initial draft for the structure and content of the single
entry point
Now we would like to improve this initial content draft with your help.
1) Please take a look at the initial draft at
https://www.mediawiki.org/wiki/Developer_Advocacy/Developer_Portal/Content_…
2) Then, please help improve it by sharing your thoughts and feedback
until *May 25th* at
https://www.mediawiki.org/wiki/Developer_Advocacy/Developer_Portal/Content_…
Note that this is only a draft how to structure content.
It is not a design or layout proposal and it is not an implementation.
For additional future work, see also the Phabricator workboard[3].
Next steps include:
* A session at the remote Hackathon (May 22-23; [4])
* Incorporate content improvements, based on your feedback
* Check the documents linked from the single entry point proposal for
accuracy
* Investigate requirements for the technical implementation
* Investigate improvements of processes around technical documentation
(structure, locations, navigation, stewardship, etc).
If you want to learn more about the project, please see
https://www.mediawiki.org/wiki/Developer_Advocacy/Developer_Portal
Thanks to everybody who has provided their valuable input to get to
this stage, and thanks in advance to everyone who will!
Cheers,
andre
[1] https://lists.wikimedia.org/pipermail/wikitech-l/2020-August/093773.html
[2] https://www.mediawiki.org/wiki/Developer_Advocacy/Developer_Portal
[3] https://phabricator.wikimedia.org/tag/wikimedia-developer-portal/
[4] https://www.mediawiki.org/wiki/Wikimedia_Hackathon_2021
--
Andre Klapper (he/him) | Bugwrangler / Developer Advocate
https://blogs.gnome.org/aklapper/