Hi,
If you do not use the raw Toolforge APIs directly, you can ignore this
message.
As we are moving toward stable API versions for the three Toolforge APIs
(jobs, builds, envvars), we are further consolidating the API paths. The
following backward-incompatible changes will take effect next Tuesday July
23 at 12:00 UTC:
- The /api prefix will be dropped from jobs API, e.g.
/v1/tool/<toolname>/jobs/
instead of */api*/v1/tool/<toolname>/jobs/
- All api resources will be in plural instead of singular, e.g.
/v1/tool/{toolname}/build*s* instead of /v1/tool/{toolname}/build
The new API docs can be seen here:
- Builds API:
https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/blob/slavin…
- Envvars API:
https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/blob/slavi…
- Jobs API:
https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/blob/slavina/…
While these APIs are not officially supported (use at your own risk), we
are happy to assist if you need help or more time for the transition. Thank
you for your understanding as we work to improve our services.
Best,
Slavina on behalf of the Cloud Services team
--
Slavina Stefanova (she/her)
Software Engineer | Developer Experience
Wikimedia Foundation
Next week (starting 2024-07-22) I'll be updating Trove database
servers[0] to use more up-to-date VM images and guest agents. This will
involve a minute or two of downtime (in the best case) and possibly an
hour or more of downtime for troublesome (and unpredictable) edge cases.
The complete list of database servers can be found on the phabricator
task below[1]. If you would prefer to schedule upgrades for your
database for a particular time, please follow up with specifics on the
task and we will find a time that works for everyone. If I don't hear
otherwise, I'll perform the maintenance on your database at my convenience.
This maintenance will affect the following cloud-vps projects:
adiutor
baglama2
citefix
copypatrol
deployment-prep
discordbots
dumps
glamwikidashboard
hoiscript
huma
library-upgrader
linkwatcher
maps
mwoffliner
mwstake
osmit
pm20database
quarry
reading-web-staging
spacemedia
wikisp
wmcz-stats
xtools
Thank you!
-Andrew
[0] Trove is the software behind Horizons 'database' tab. Databases on
VMs that were setup up manually or via puppet are unaffected by this
maintenance.
[1] https://phabricator.wikimedia.org/T369723
Hi there,
The Toolforge Kubernetes system has been scheduled to be upgraded to version
1.25 [0] next Tuesday 2024-07-17 @ 09:00 UTC.
The operation window will last 2 hours, and during this time, some Toolforge
components will briefly and intermittently become unavailable.
Examples of things that can be affected include:
* running the different `toolforge` commands from bastions
* accessing the web URL of some tools
* the internal operations of some jobs, which may be restarted as containers are
moved to different worker nodes.
Please report [1] any problems you see in Toolforge after the upgrade.
regards.
[0] https://phabricator.wikimedia.org/T369172
[1] https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_communication
Hi,
If you do not use the raw Toolforge APIs directly, you can ignore this
message.
We are writing to inform you about upcoming changes to several API
endpoints. Most importantly, the* /api/v1/jobs/* endpoint, which is
currently the only one still in use, will be changing.
The general pattern for affected endpoints will shift from:
{jobs|builds|envvars}/api/v1/<resource>/
to:
{jobs|builds|envvars}/api/v1/*tool/{toolname}*/<resource>/
Key points:
- This change will take effect next *Thursday July 18 at 12:00 UTC*.
- If you are using the */api/v1/jobs/* endpoint, please update your
tools accordingly.
- While these APIs are not officially supported (use at your own risk),
we are happy to assist if you need help or more time for the transition.
We are also working on developing stable API versions. If you are
interested in the details or want to contribute to the discussion, you can
check out https://phabricator.wikimedia.org/T356974.
Thank you for your understanding as we work to improve our services.
Best,
Slavina on behalf of the Cloud Services team
--
Slavina Stefanova (she/her)
Software Engineer | Developer Experience
Wikimedia Foundation
Following several requests from users over the past eight years [0],
we are finally enabling access to ToolsDB's "public" databases (the
ones with a name ending with "_p") [1] from both Quarry [2] and
Superset [3].
The data stored in those databases have always been accessible to
every Toolforge user, but after this change they will become more
broadly accessible, as Quarry can be accessed by anyone with a
Wikimedia account, and saved queries in Quarry can be shared with
public links that require no login at all.
== This change is planned to go live on Monday, July 1st. ==
If you have any concerns or questions related to this change, please
leave a comment in the Phabricator task or one of its subtasks. [0]
Thanks to everyone for your patience and for keeping the task alive
over the years!
[0] https://phabricator.wikimedia.org/T151158
[1] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#Privileges_on_t…
[2] https://meta.wikimedia.org/wiki/Research:Quarry
[3] https://superset.wmcloud.org/
--
Francesco Negri (he/him) -- IRC: dhinus
Site Reliability Engineer, Cloud Services team
Wikimedia Foundation
Hi there,
tomorrow 2024-06-26 @ 08:30Z we will start enforcing new Kubernetes security
rules in Toolforge [0].
We have taken measures to eliminate any user impact, but this being a
potentially sensitive change, I wanted to send a heads up email.
In a nut-shell, pod-related kubernetes resources, like Deployment or CronJob
need to have a new set of security-related attributes correctly specified.
This is because we are introducing Kyverno policies as a replacement of the
deprecated PodSecurityPolicies (PSP) [1].
The new Kyverno policies have been deployed already, but are in 'Audit' mode.
What we will be doing tomorrow is setting them to 'Enforce', which is the final
step in this migration, before we can finally drop PSP [2].
Please, report [3] any disruption that you may see.
regards.
[0] https://phabricator.wikimedia.org/T368141
[1] https://phabricator.wikimedia.org/T279110
[2] https://phabricator.wikimedia.org/T364297
[3] https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_communication
Hello!
You may have been notified about Buster VM deprecation on projects where
you are not an admin.
This was in error.
Kindly ignore.
Thanks!
--
Seyram Komla Sapaty
Developer Advocate
Wikimedia Cloud Services
Hello Cloud users,
Just wanted to give a heads-up for an upcoming maintenance for our cloudelastic hosts <https://wikitech.wikimedia.org/wiki/Help:CirrusSearch_elasticsearch_replicas>. We will be migrating them to a new load balancer starting Thursday, Jun 20 at 1400 UTC . The service will be down through the maintenance, which is expected to last between 1 and 2 hours.
Fine details are available in this Phabricator task <https://phabricator.wikimedia.org/T367511> . If you have any questions about this, feel free to respond here or in IRC (see my nick below).
Best,
Brian King
SRE, Data Platform/Search Platform
Wikimedia Foundation
IRC: inflatador
Hi all!
This is to let you know that Toolforge continuous jobs now support internal
domain names!
This means you can now make a request from any job type to a continuous job
using the name of the continuous job directly, without having to know or
keep track of its internal ephemeral IP address.
To use it you need to specify `--port <number>` while creating
your continuous job. Once done your continuous job can now be reached from
other jobs with `https://<continuous-job-name>:<port>`.
Note: This can only be configured for continuous jobs.
Note: The name of the continuous job automatically becomes the internal
domain name and there is currently no way to specify a custom name.
Also, a reminder that you can find this and smaller user-facing updates
about
the Toolforge platform features here:
https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Changelog
Original task: https://phabricator.wikimedia.org/T348758
--
Ndibe Raymond Olisaemeka
Software Engineer - Technical Engagement
Wikimedia Foundation <https://wikimediafoundation.org/>
<https://wikimediafoundation.org>
Hi,
I've added a new set of flavors on Cloud VPS.[0] The new flavors are
almost identical to the existing `g3` flavors - the only difference is
that instances on the new `g4` flavors get scheduled on hypervisors
running more modern networking software.[1] When creating new
instances, either by hand or via some automated means, please use the
new flavors from now on. Using g3 flavors for new instances will be
disabled in the coming days.
We will be migrating existing VMs to the new flavors at some point in
the near future. This will most likely involve briefly shutting down
each VM as it's migrated from one network agent to the other. There
will be a dedicated announcement for that once the timeline on that is
more certain. As always, if your project needs some coordination
before restarting things or this is otherwise a problem, feel free to
contact us.[2]
Migrating a g3 instance to a g4 flavor (or the other way around)
without manual intervention would instead make that instance
inaccessible. For that reason instance resizing has temporarily been
disabled, if you need to resize an existing instance for whatever
reason before it's been migrated please contact us.
[0]: https://wikitech.wikimedia.org/wiki/Help:Cloud_VPS_Instances#Instance_infor…
[1]: https://phabricator.wikimedia.org/T364458
[2]: https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_communication
Taavi
--
Taavi Väänänen (he/him)
Site Reliability Engineer, Cloud Services
Wikimedia Foundation