Following several requests from users over the past eight years [0],
we are finally enabling access to ToolsDB's "public" databases (the
ones with a name ending with "_p") [1] from both Quarry [2] and
Superset [3].
The data stored in those databases have always been accessible to
every Toolforge user, but after this change they will become more
broadly accessible, as Quarry can be accessed by anyone with a
Wikimedia account, and saved queries in Quarry can be shared with
public links that require no login at all.
== This change is planned to go live on Monday, July 1st. ==
If you have any concerns or questions related to this change, please
leave a comment in the Phabricator task or one of its subtasks. [0]
Thanks to everyone for your patience and for keeping the task alive
over the years!
[0] https://phabricator.wikimedia.org/T151158
[1] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#Privileges_on_t…
[2] https://meta.wikimedia.org/wiki/Research:Quarry
[3] https://superset.wmcloud.org/
--
Francesco Negri (he/him) -- IRC: dhinus
Site Reliability Engineer, Cloud Services team
Wikimedia Foundation
_______________________________________________
Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org
List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…
Dear all,
Has anyone figured out how to use Toolforge Build Service for a tool (job +
webservice) written in golang?
I'm trying to switch qrank.toolforge.org to Toolforge Build Service, but
I'm struggling with it.
To debug this, I wrote a minimal job that just prints a hello message to
stdout:
https://github.com/brawer/wikidata-qrank/blob/main/cmd/hello/main.go
Because Toolforge is emulating Heroku, it appears necessary to tell
buildpack which binaries should get packaged into the container. The "//
+heroku install" comment on line 14 of the following go.mod file seems to
do the trick, together with listing the binaries again in a Heroku Procfile:
https://github.com/brawer/wikidata-qrank/blob/main/go.mod#L14https://github.com/brawer/wikidata-qrank/blob/main/Procfile
When running the following command on login.toolforge.org, the project
seems to be built just fine:
$ toolforge build start https://github.com/brawer/wikidata-qrank
Some interesting log message from build service:
[step-build] 2024-04-30T18:13:31.170521374Z Building packages:
[step-build] 2024-04-30T18:13:31.170585858Z - ./cmd/hello
[step-build] 2024-04-30T18:13:31.170599159Z - ./cmd/qrank-builder
[step-build] 2024-04-30T18:13:31.170608711Z - ./cmd/webserver
[step-build] 2024-04-30T18:13:44.563334902Z
[step-build] 2024-04-30T18:13:44.563404532Z [Setting launch table]
[step-build] 2024-04-30T18:13:44.563417703Z Detected processes:
[step-build] 2024-04-30T18:13:44.563427021Z - hello: hello
[step-build] 2024-04-30T18:13:44.563437573Z - qrank-builder: qrank-builder
[step-build] 2024-04-30T18:13:44.563446463Z - webserver: webserver
[step-build] 2024-04-30T18:13:44.563488376Z - web: hello
[step-build] 2024-04-30T18:13:44.577068596Z
[step-build] 2024-04-30T18:13:44.577163104Z [Discovering process types]
[step-build] 2024-04-30T18:13:44.579933448Z Procfile declares types -> web,
qrank-builder, hello
The "web: hello" in the launch table looks very suspicious; it's not what
my Procfile states (see link above). But at least the build seems to have
been successful:
$ toolforge build show
Build ID: qrank-buildpacks-pipelinerun-vhcrt
Start Time: 2024-04-30T18:13:03Z
End Time: 2024-04-30T18:14:00Z
Status: ok
Message: Tasks Completed: 1 (Failed: 0, Cancelled 0), Skipped: 0
Parameters:
Source URL: https://github.com/brawer/wikidata-qrank
Ref: N/A
Envvars: N/A
Destination Image: tools-harbor.wmcloud.org/tool-qrank/tool-qrank:latest
Now, I'd be happy to just run the "hello" job and see its output log.
However, when I do this:
$ toolforge jobs run --command hello --image tool-qrank/tool-qrank:latest
--mount=all --filelog
The job runs for about five minutes (?!?!), and then:
$ cat hello.out
ERROR: failed to launch: bash exec: argument list too long
So clearly something is off... but what? How to run a golang tool on
Toolforge when using Build Service?
Thanks for any help,
— Sascha
On Monday I'll be upgrading the database that backs toolforge's Harbor
service. Any attempts to download new container images during this
window will fail.
Existing, running tools will not be affected, and periodically scheduled
jobs should also be fine, but any brand new jobs may fail, depending on
what is or isn't cache at the moment.
The outage should be brief; if you see a cryptic failure around the
scheduled time just take a 10 minute break and try again.
-Andrew
_______________________________________________
Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org
List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…
There is currently an issue with Toolforge's image registry (Harbor) due to
its database being out of space.
This affects the Toolforge build service, meaning no new builds can be
started, and no images can currently be pulled from the registry. Currently
running tools should be unaffected.
We are actively working on the issue and will update you here. Sorry for
the downtime!
--
Slavina Stefanova (she/her)
Software Engineer | Developer Experience
Wikimedia Foundation
_______________________________________________
Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org
List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…
Next week (starting 2024-07-22) I'll be updating Trove database
servers[0] to use more up-to-date VM images and guest agents. This will
involve a minute or two of downtime (in the best case) and possibly an
hour or more of downtime for troublesome (and unpredictable) edge cases.
The complete list of database servers can be found on the phabricator
task below[1]. If you would prefer to schedule upgrades for your
database for a particular time, please follow up with specifics on the
task and we will find a time that works for everyone. If I don't hear
otherwise, I'll perform the maintenance on your database at my convenience.
This maintenance will affect the following cloud-vps projects:
adiutor
baglama2
citefix
copypatrol
deployment-prep
discordbots
dumps
glamwikidashboard
hoiscript
huma
library-upgrader
linkwatcher
maps
mwoffliner
mwstake
osmit
pm20database
quarry
reading-web-staging
spacemedia
wikisp
wmcz-stats
xtools
Thank you!
-Andrew
[0] Trove is the software behind Horizons 'database' tab. Databases on
VMs that were setup up manually or via puppet are unaffected by this
maintenance.
[1] https://phabricator.wikimedia.org/T369723
_______________________________________________
Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org
List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…
Hello,
Just got this email, but the referenced site doesn't exist (
idm.wikipedia.org). Wondering if this was sent by mistake?
Thanks,
-Travis
On Mon, Jul 22, 2024 at 12:14 AM Komla Sapaty <ssapaty(a)wikimedia.org> wrote:
> Hello Audiodude
>
> We are reaching out to inform you about an important upcoming change
> regarding your Wikitech account.
> As part of our ongoing efforts to streamline account management and
> improve security, we will be
> migrating all old developer Wikitech accounts to the MediaWiki account
> system (also referred to as global account or Wikimedia global account).
> This means that in the future you will be able to access Wikitech using
> your MediaWiki/Wikipedia account.
>
> ==What You Need to Do==
> To ensure a smooth transition, we request that you manually link your
> Wikitech account to
> your MediaWiki account before September 16th. Please follow these steps:
> *Log in to idm.wikipedia.org with the same username and password you
> would use on wikitech.[0]
> *Follow the instructions provided to link your Wikitech (LDAP) account to
> your MediaWiki account.
>
> ==Why This Is Important==
> Currently, Wikitech accounts are stored in LDAP, which has been
> historically used for managing
> technical community accounts across Wikimedia projects. We are now
> consolidating these accounts
> under the MediaWiki account system.
> This change will help us better manage the Wikitech wiki, which also
> contains essential technical
> documentation.
>
> ==Migration Details==
> *Linked Accounts: If you link your accounts by the deadline, your Wikitech
> account will be migrated
> to your Mediawiki account, retaining your full edit history.
> *Unlinked Accounts: If you do not link your accounts by the deadline, your
> Wikitech account will be
> migrated to a default scheme (e.g., wikitech_OLDWIKITECHUSERNAME). Your
> old edits will still be visible,
> but new edits will be made under a new MediaWiki account with no immediate
> link to your past edits.
>
> Your prompt action will help us ensure a seamless transition.
> If you have already linked your accounts, no further action is required.
> If you have any questions or need assistance, please do not hesitate to
> contact us[1]
> We are also maintaining a FAQ page here[2] that hopefully answers some
> questions you may have.
>
> [0]: https://idm.wikipedia.org
> [1]:
> https://wikitech.wikimedia.org/wiki/Portal:Toolforge/About_Toolforge#Commun…
> [2]:
> https://wikitech.wikimedia.org/wiki/News/Migrating_Wikitech_Account_to_SUL
>
> Thanks
> Komla Sapaty
> (Developer Experience)
>
>
>
Many of you have either received or are about to get an email from Komla
about connecting your developer account with an SUL account. The action
necessary is very simple, just log in to https://idm.wikimedia.org and
gill in the Wikimedia Global Account (SUL) field.
Please update this field as soon as you're able; otherwise Komla's email
will only be the first of many.
If you are curious about the very many technical details about this
project, you can read about it here:
https://wikitech.wikimedia.org/wiki/News/Migrating_Wikitech_Account_to_SUL
Thanks!
-Andrew (who is only repeating Komla because I'm less likely to land in
your spam filter).
_______________________________________________
Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org
List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…
Hi,
If you do not use the raw Toolforge APIs directly, you can ignore this
message.
As we are moving toward stable API versions for the three Toolforge APIs
(jobs, builds, envvars), we are further consolidating the API paths. The
following backward-incompatible changes will take effect next Tuesday July
23 at 12:00 UTC:
- The /api prefix will be dropped from jobs API, e.g.
/v1/tool/<toolname>/jobs/
instead of */api*/v1/tool/<toolname>/jobs/
- All api resources will be in plural instead of singular, e.g.
/v1/tool/{toolname}/build*s* instead of /v1/tool/{toolname}/build
The new API docs can be seen here:
- Builds API:
https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/blob/slavin…
- Envvars API:
https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/blob/slavi…
- Jobs API:
https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/blob/slavina/…
While these APIs are not officially supported (use at your own risk), we
are happy to assist if you need help or more time for the transition. Thank
you for your understanding as we work to improve our services.
Best,
Slavina on behalf of the Cloud Services team
--
Slavina Stefanova (she/her)
Software Engineer | Developer Experience
Wikimedia Foundation
_______________________________________________
Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org
List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…
Greetings,
This patch[0] sets the default configuration in Wikimedia CI to enable the
temporary accounts feature.[1] This means: PHPUnit, Selenium, API-Testing,
and QUnit tests run in an environment where temporary account autocreation
is enabled.
## Why?
Next week, the Trust and Safety Product team is deploying the temporary
accounts feature to testwiki.[2] Later this year, we will move forward with
deploying to other production wikis. By enabling temporary accounts in CI,
we increase the likelihood that new code works smoothly with the temporary
accounts feature.
## What do I need to do?
This time, nothing. All MediaWiki core, extension and skins tests pass,
whether temporary accounts are enabled or disabled.[3]
For new PHPUnit tests, please make use of `TempUserTestTrait` to disable
the temporary accounts feature for tests that make assertions specifically
about anonymous IP editing.
We also recommend that you enable temporary accounts in your local
development environment by adding `$wgAutoCreateTempUser['enabled'] = true;`
to your LocalSettings.php.
## Questions?
Please contact us[4] or file a task on Phab and tag it with “#Temporary
accounts”.
[0] https://gerrit.wikimedia.org/r/c/mediawiki/core/+/980947
[1]
https://www.mediawiki.org/wiki/Trust_and_Safety_Product/Temporary_Accounts/…,
https://www.mediawiki.org/wiki/Help:Temporary_accounts
[2] https://phabricator.wikimedia.org/T348895
[3] https://phabricator.wikimedia.org/T355879
[4]
https://www.mediawiki.org/wiki/Talk:Trust_and_Safety_Product/Temporary_Acco…
Cheers,
Kosta & Thalia, Trust and Safety Product
and
Szymon Grabarczuk (he/him)
Lead Movement Communications Specialist
(Product & Tech)
Wikimedia Foundation <https://wikimediafoundation.org/>
Hi there,
The Toolforge Kubernetes system has been scheduled to be upgraded to version
1.25 [0] next Tuesday 2024-07-17 @ 09:00 UTC.
The operation window will last 2 hours, and during this time, some Toolforge
components will briefly and intermittently become unavailable.
Examples of things that can be affected include:
* running the different `toolforge` commands from bastions
* accessing the web URL of some tools
* the internal operations of some jobs, which may be restarted as containers are
moved to different worker nodes.
Please report [1] any problems you see in Toolforge after the upgrade.
regards.
[0] https://phabricator.wikimedia.org/T369172
[1] https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_communication