Following several requests from users over the past eight years [0],
we are finally enabling access to ToolsDB's "public" databases (the
ones with a name ending with "_p") [1] from both Quarry [2] and
Superset [3].
The data stored in those databases have always been accessible to
every Toolforge user, but after this change they will become more
broadly accessible, as Quarry can be accessed by anyone with a
Wikimedia account, and saved queries in Quarry can be shared with
public links that require no login at all.
== This change is planned to go live on Monday, July 1st. ==
If you have any concerns or questions related to this change, please
leave a comment in the Phabricator task or one of its subtasks. [0]
Thanks to everyone for your patience and for keeping the task alive
over the years!
[0] https://phabricator.wikimedia.org/T151158
[1] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#Privileges_on_t…
[2] https://meta.wikimedia.org/wiki/Research:Quarry
[3] https://superset.wmcloud.org/
--
Francesco Negri (he/him) -- IRC: dhinus
Site Reliability Engineer, Cloud Services team
Wikimedia Foundation
Just now one of our hypervisors crashed and left several VMs shut down
for around 45 minutes. I have rebooted the host and service has been
restored.
The affected VMs were:
web01
tools-k8s-worker-nfs-55
tools-k8s-worker-nfs-24
toolsbeta-test-k8s-worker-nfs-2
opensearch2
traffic-acmechief01
section-ranker
integration-agent-docker-1047
xtools-prod08
xtools-dev06
statanalyser
bullseye
deployment-ms-fe04
deployment-ms-be07
signwriting-swis-2022
signwriting-swserver-2022
mwparserfromhtml
video-nfs-1
striker-docker-01
twl-nfs-1
teyora
suggestbot-02
edit-types
soweego
quarry-dbbackup-01
pki-db
Followup on this incident is at https://phabricator.wikimedia.org/T373740
Cross-posting this announcement from the wikitech-l mailing list. As I
understand it, this policy change is intended to be clarifying rather
than introducing new limits or hurdles for the existing API consumer
community.
Discussion should happen on wiki please as folks will not be
monitoring the cloud mailing lists for feedback.
Bryan
---------- Forwarded message ---------
From: Luca Martinelli [Sannita@WMF] <sannita(a)wikimedia.org>
Date: Thu, Aug 29, 2024 at 6:21 AM
Subject: [Wikitech-l] New draft text of WMF legal policy regarding use of APIs
To: <wikitech-l(a)lists.wikimedia.org>, <wikimedia-l(a)lists.wikimedia.org>
Hello all,
We have published a new draft text of WMF legal policy discussing the
use of its APIs. The full page is available on Meta.[1]
We are currently looking for feedback on this draft text, and you're
welcome to express your considerations on the text's talk page.[2]
This new document would be published at "API usage guidelines" on the
Foundation wiki, unless community comments suggest the text should be
appended on an existing page (for example, the User-Agent policy
page),[3] or a different wiki.
This page is available for community comment for a period of at least
two weeks or until new comments, questions, and suggestions have
concluded.
Thank you in advance for your consideration!
[1] https://meta.wikimedia.org/wiki/API_Policy_Update_2024
[2] https://meta.wikimedia.org/wiki/Talk:API_Policy_Update_2024
[3] https://foundation.wikimedia.org/wiki/Special:MyLanguage/Policy:User-Agent_…
--
Luca Martinelli [Sannita] (he/him)
Movement Communications Specialist
_______________________________________________
Wikitech-l mailing list -- wikitech-l(a)lists.wikimedia.org
To unsubscribe send an email to wikitech-l-leave(a)lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
--
Bryan Davis Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
I will be upgrading the cloud-vps openstack install on Thursday,
beginning around 16:00 UTC. Here's what to expect:
- Intermittent Horizon and API downtime (maybe an hour or two total)
- Inability to schedule new VMs (also for an hour or two)
Toolforge users will be unaffected by this outage. Existing, running
services and VMs on cloud-vps should also be unaffected.
This is the upgrade to version 2024.1 'caracal', tracked at
https://phabricator.wikimedia.org/T369044
-Andrew + the WMCS team
Due to a serious hardware failure[0] we've had to take much of our file
storage cluster offline. This should not affect any existing toolforge
or cloud-vps usage, but new file-intensive work (e.g. creating a big new
database or volume) may fail due to lack of space.
We hope to have things restored to full service in a couple of days but
right now we're just watching the suspect switch to see if it fails again.
Sorry for any inconvenience!
-Andrew
[0] https://phabricator.wikimedia.org/T371878
TL;DR:
* Nominations are open for the Toolforge standards committee at
<https://wikitech.wikimedia.org/wiki/Help_talk:Toolforge/Toolforge_standards…>
The Toolforge standards committee [0] (née Tool Labs standards
committee) was established in 2017 following a membership nomination
process in December 2016.[1] There have been no formal membership
changes in the subsequent 7.5 years. As one might imagine, a number of
the highly active Toolforge members who formed the original committee
have since drifted away from the project as their personal lives and
interests have changed. Those that are still active within the
movement seem to have drifted away from this particular functionary
role.
I feel both grateful to those who have served for so long and
responsible for not helping them transition away more gracefully prior
to now. It is long past time for us to thank Eranroz, Huji, Ladsgroup,
Matanya, Quiddity, and Zhuyifei1999 for their efforts and find a new
set of folks to carry on the work of the committee.
The committee's primary duties are to oversee the Right to fork
policy[2] and Abandoned tool policy[3]. The committee is also
encouraged to implement additional programs designed to promote and
improve the use of Toolforge for the benefit of the Wikimedia
movement.
Nominations are being collected on Wikitech.[4] Any Toolforge project
member with a Developer account not blocked in any Wikimedia technical
space is eligible for nomination. Self-nominations are acceptable.
Nominations by a third party should be counter-signed by the nominee
to indicate they would accept the responsibility and be willing to
sign the Volunteer NDA.[5] Nominations will remain open until at least
2024-08-26.
[0]: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Toolforge_standards_comm…
[1]: https://wikitech.wikimedia.org/wiki/Help_talk:Toolforge/Toolforge_standards…
[2]: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Right_to_fork_policy
[3]: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Abandoned_tool_policy
[4]: https://wikitech.wikimedia.org/wiki/Help_talk:Toolforge/Toolforge_standards…
[5]: https://wikitech.wikimedia.org/wiki/Volunteer_NDA
--
Bryan Davis Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
Hi!
I'm happy to announce that we have a stable API for the Toolforge platform!
You can now access and manage your tools using an API and build your own
wrappers around it.
This API is currently only available from within Toolforge, including from
within your own jobs and webservices (just make sure to use `--mount=all` so
the certs are available). We plan to make this API accessible from anywhere in
the future.
This is the same API that the toolforge clients use directly, so expect to be
able to do everything the clients can do. Webservice is not available yet, but
will be soon! (https://phabricator.wikimedia.org/T348755)
You can find the actual API docs here: https://api-docs.toolforge.org
And the wiki page with details on deprecation process, authentication, and a
python example here: https://wikitech.wikimedia.org/wiki/Help:Toolforge/API
Let us know if you find any issues or if you have any questions through the
usual channels.
The full list is under "Communication and support" on:
https://wikitech.wikimedia.org/wiki/Portal:Toolforge
Thanks!
David
--
David Caro
SRE - Cloud Services
Wikimedia Foundation <https://wikimediafoundation.org/>
PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3
"Imagine a world in which every single human being can freely share in the
sum of all knowledge. That's our commitment."
If you are a toolforge user and your workflow still relies on
tools-login.wmflabs.org, there are a few things to be aware of:
1) Until yesterday, tools-login.wmflabs.org pointed to a login host
running Debian10(buster), which is no longer a supported operation
system. I adjusted it to point to a newer bastion running Debian 12
(bookworm).
2) Some ssh key types (e.g. ssh-rsakeys created with puttygen on
Windows, RSA keys signed with SHA-1) are no longer supported by modern
operating systems for security reasons. If you are encountering a new
login failure with toolforge /and/ you were using the .wmflabs.org login
host, you probably need to generate modern keys (ECDSA) and upload them
to https://idm.wikimedia.org
3) The software available on the new bastion differs somewhat from that
on the old bastion. If you find yourself unable to work on the new
hosts, please open a phabricator ticketwith the #toolforge tagor respond
to this email with specifics.
4) The surprising nature of this change is due to the fact that WMCS has
not used the .wmflabs.org domain for many years; anything under that
domain is only working by accident and will probably be broken by us
without warning. Please update your workflows and login scripts to use
the new, preferred login name: login.toolforge.org.
5) If you urgently need the functions of the old login host, you can
reach it as login-buster.toolforge.org. Be warned, though, that that
host is not likely to exist for more than a few more weeks.
Sorry for any inconvenience caused!
-Andrew