Dear all,
Has anyone figured out how to use Toolforge Build Service for a tool (job +
webservice) written in golang?
I'm trying to switch qrank.toolforge.org to Toolforge Build Service, but
I'm struggling with it.
To debug this, I wrote a minimal job that just prints a hello message to
stdout:
https://github.com/brawer/wikidata-qrank/blob/main/cmd/hello/main.go
Because Toolforge is emulating Heroku, it appears necessary to tell
buildpack which binaries should get packaged into the container. The "//
+heroku install" comment on line 14 of the following go.mod file seems to
do the trick, together with listing the binaries again in a Heroku Procfile:
https://github.com/brawer/wikidata-qrank/blob/main/go.mod#L14https://github.com/brawer/wikidata-qrank/blob/main/Procfile
When running the following command on login.toolforge.org, the project
seems to be built just fine:
$ toolforge build start https://github.com/brawer/wikidata-qrank
Some interesting log message from build service:
[step-build] 2024-04-30T18:13:31.170521374Z Building packages:
[step-build] 2024-04-30T18:13:31.170585858Z - ./cmd/hello
[step-build] 2024-04-30T18:13:31.170599159Z - ./cmd/qrank-builder
[step-build] 2024-04-30T18:13:31.170608711Z - ./cmd/webserver
[step-build] 2024-04-30T18:13:44.563334902Z
[step-build] 2024-04-30T18:13:44.563404532Z [Setting launch table]
[step-build] 2024-04-30T18:13:44.563417703Z Detected processes:
[step-build] 2024-04-30T18:13:44.563427021Z - hello: hello
[step-build] 2024-04-30T18:13:44.563437573Z - qrank-builder: qrank-builder
[step-build] 2024-04-30T18:13:44.563446463Z - webserver: webserver
[step-build] 2024-04-30T18:13:44.563488376Z - web: hello
[step-build] 2024-04-30T18:13:44.577068596Z
[step-build] 2024-04-30T18:13:44.577163104Z [Discovering process types]
[step-build] 2024-04-30T18:13:44.579933448Z Procfile declares types -> web,
qrank-builder, hello
The "web: hello" in the launch table looks very suspicious; it's not what
my Procfile states (see link above). But at least the build seems to have
been successful:
$ toolforge build show
Build ID: qrank-buildpacks-pipelinerun-vhcrt
Start Time: 2024-04-30T18:13:03Z
End Time: 2024-04-30T18:14:00Z
Status: ok
Message: Tasks Completed: 1 (Failed: 0, Cancelled 0), Skipped: 0
Parameters:
Source URL: https://github.com/brawer/wikidata-qrank
Ref: N/A
Envvars: N/A
Destination Image: tools-harbor.wmcloud.org/tool-qrank/tool-qrank:latest
Now, I'd be happy to just run the "hello" job and see its output log.
However, when I do this:
$ toolforge jobs run --command hello --image tool-qrank/tool-qrank:latest
--mount=all --filelog
The job runs for about five minutes (?!?!), and then:
$ cat hello.out
ERROR: failed to launch: bash exec: argument list too long
So clearly something is off... but what? How to run a golang tool on
Toolforge when using Build Service?
Thanks for any help,
— Sascha
Hello all,
The next language community meeting is scheduled in a few weeks - May 31st
at 16:00 UTC. If you're interested, you can sign up on this wiki page: <
https://www.mediawiki.org/w/index.php?title=Wikimedia_Language_engineering/…
>.
This is a participant-driven meeting, where we share language-specific
updates related to various projects, collectively discuss technical issues
related to language wikis, and work together to find possible solutions.
For example, in the last meeting, the topics included the machine
translation service (MinT) and the languages and models it currently
supports, localization efforts from the Kiwix team, and technical
challenges with numerical sorting in files used on Bengali Wikisource.
Do you have any ideas for topics to share technical updates related to your
project? Any problems that you would like to bring for discussion during
the meeting? Do you need interpretation support from English to another
language? Please reach out to me at ssethi(a)wikimedia.org and add
agenda items to the document here: <
https://etherpad.wikimedia.org/p/language-community-meeting-may-2024>.
We look forward to your participation!
Cheers,
Jon, Mary, Oscar, Amir and Srishti
*Srishti Sethi*
Senior Developer Advocate
Wikimedia Foundation <https://wikimediafoundation.org/>
If you are a user of cloud-vps, please visit the following page and mark
any projects that are still in use:
https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2024_Purge
There are currently 95 unmarked projects on that page; any that remain
unclaimed will be deemed abandoned and subject to shutdown in June.
I'm attaching the names of those 95 projects at the end of this email.
If you see something on that list that you know to be important, please
make a note on the wiki page (even if you are not an admin) so we know
that somebody cares.
Thanks!
Unclaimed projects as of 2024-05-24:
adiutor
automation-framework
baglama2
canasta2_test
capacity-exchange
citefix
clouddb-services
codereview
commons-corruption-checker
download
duct
dump-references-processor
entity-detection
etytree
extdist
foundationmemory
gitlab-test
glamwikidashboard
gratitude
hoiscript
huggle
huwiki-dev
idm-dev
image-suggestion-api
imagebulk
impactvisualizer
integration
ipoidopensearch
isa
k8splay
language
ldap-dev
linkwatcher
lta-tracker
lutz
machine-learning
maps-experiments
math
matrix
media-streaming
mediawiki-vagrant
mix-n-match
mwcli
mwoffliner
mwv-apt
onfire
openvas
packaging
petscan
pixel
policy-test-project
puppet-dev
rcm
reading-web-staging
schematreerecommender
search
security-tools
service
shiny-r
signwriting
spi-tools
sre-sandbox
sso
superset
text-to-speech
teyora
tf-infra-test
traffic
video
videocuttool
visualeditor
vuessr
wcdo
webperf
wikibase-registry
wikicite
wikicommunityhealth
wikidata-dev
wikidata-history-query-service
wikidata-query
wikidocumentaries
wikidumpparse
wikifunctions
wikilabels
wikimania-mautic
wikipathways
wikispeech
wikitextexp
wildcat
wm-bot
wmcs-uptime
wmcz-stats
wmde-dashboards
wmdeanalytics
wmf-dumps-playground
_______________________________________________
Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org
List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…
How do I fix this? I followed the steps for debugging on
wt:Portal:Cloud_VPS/Admin/Runbooks/Cloud
VPS alert Puppet failure on
<https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Cloud_V…>
and got the same error. I couldn't find any instructions for it on that
page.
Full log:
sebastian-berlin-wmse@producer:~$ sudo run-puppet-agent
2024-05-22 09:28:12.274041 WARN puppetlabs.facter - locale environment
variables were bad; continuing with LANG=C LC_ALL=C
2024-05-22 09:28:13.088451 WARN puppetlabs.facter - locale environment
variables were bad; continuing with LANG=C LC_ALL=C
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER:
Server Error: Evaluation Error: Error while evaluating a Resource
Statement, Evaluation Error: Error while evaluating a Resource Statement,
Class[Memcached]: parameter 'memcached_user' expects a String value, got
Undef (file:
/srv/puppet_code/environments/production/modules/profile/manifests/simplelamp2.pp,
line: 32, column: 5) on node producer.wikispeech.eqiad1.wikimedia.cloud
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
*Sebastian Berlin*
Utvecklare/*Developer*
Wikimedia Sverige (WMSE)
E-post/*E-Mail*: sebastian.berlin(a)wikimedia.se
Telefon/*Phone*: (+46) 0707 - 92 03 84
---------- Forwarded message ---------
From: root <root(a)producer.wikispeech.eqiad1.wikimedia.cloud>
Date: Wed, 22 May 2024 at 10:15
Subject: [Cloud VPS alert][wikispeech] Puppet failure on
producer.wikispeech.eqiad1.wikimedia.cloud (172.16.0.200)
To: <sebastian.berlin(a)wikimedia.se>
Puppet is having issues on the "producer.wikispeech.eqiad1.wikimedia.cloud
(172.16.0.200)" instance in project
wikispeech in Wikimedia Cloud VPS.
Puppet is running with failures.
Working Puppet runs are needed to maintain instance security and logins.
As long as Puppet continues to fail, this system is in danger of becoming
unreachable.
You are receiving this email because you are listed as member for the
project that contains this instance. Please take steps to repair
this instance or contact a Cloud VPS admin for assistance.
If your host is expected to fail puppet runs and you want to disable this
alert, you can create a file under /.no-puppet-checks, that will skip the
checks.
You might find some help here:
https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Cloud_V…
For further support, visit #wikimedia-cloud on libera.chat or
<https://wikitech.wikimedia.org>
Some extra info follows:
---- Last run summary:
application:
converged_environment: production
initial_environment: production
run_mode: agent
changes:
total: 0
events:
failure: 0
success: 0
total: 0
resources:
changed: 0
corrective_change: 0
failed: 0
failed_to_restart: 0
out_of_sync: 0
restarted: 0
scheduled: 0
skipped: 0
total: 0
time:
fact_generation: 0.5950437020510435
last_run: 1716364257
plugin_sync: 0.8315697535872459
startup_time: 0.862627795
total: 4.96828438
version:
config: null
puppet: 7.23.0
---- Failed resources if any:
No failed resources.
--- Last run log:
ERR: Could not retrieve catalog from remote server: Error 500 on SERVER:
Server Error: Evaluation Error: Error while evaluating a Resource
Statement, Evaluation Error: Error while evaluating a Resource Statement,
Class[Memcached]: parameter 'memcached_user' expects a String value, got
Undef (file:
/srv/puppet_code/environments/production/modules/profile/manifests/simplelamp2.pp,
line: 32, column: 5) on node producer.wikispeech.eqiad1.wikimedia.cloud
WARNING: Not using cache on failed catalog
ERR: Could not retrieve catalog; skipping run
---- Exceptions that happened when running the script if any:
No exceptions happened.
Hello everyone,
I am reaching out on behalf of the Wikimedia Foundation Trust and Safety
Product <https://www.mediawiki.org/wiki/Trust_and_Safety_Product> team,
working on the Temporary Accounts
<https://www.mediawiki.org/wiki/Trust_and_Safety_Product/Temporary_Accounts>
project (formerly, IP Masking). We are making preparations for deployment
on testwiki.
We want to minimize the impact of temporary accounts on the ecosystem of
tools and features. Before deployment, we'll ask for stewards' confirmation
that critical workflows are not affected. Now, we would like to ask you to
check the impact on code you maintain. There may be breaking changes!
We do not have a firm deadline for deployment yet. We will let you know as
soon as we do. We would like to ask you to consider updating your tool's
code over the next weeks, if the tool works on any/all production cluster
wiki. If it only works on selected wikis and testwiki is not one of them,
you will have more time to update it. If you think it will take longer than
that to update your code, please let us know on our talk page
<https://www.mediawiki.org/wiki/Talk:Trust_and_Safety_Product/Temporary_Acco…>.
For a full explanation of the technical change, please read our
documentation
<https://www.mediawiki.org/wiki/Trust_and_Safety_Product/Temporary_Accounts/…>.
In particular, the section on how your code might need to be updated
<https://www.mediawiki.org/wiki/Trust_and_Safety_Product/Temporary_Accounts/…>.
Some examples of ways code might be affected (not exhaustive):
- A feature looks for the IP address in the user name (it will no longer
be found).
- A feature that relies on looking at the user’s ID to identify if the
user is registered or unregistered. Temporary accounts will not have a
user_id of 0.
- A feature works differently for unregistered vs registered users.
See more examples
<https://www.mediawiki.org/wiki/Trust_and_Safety_Product/Temporary_Accounts/…>.
We may have discovered your issue already and filed a Phabricator task
using the #temporary-accounts tag. You may wish to file a subtask of T337012
<https://phabricator.wikimedia.org/T337012>. We hope that this task will
help us all as the technical community coordinates on tools requiring
updates.
If you have any questions or would like to learn more about the project,
take a look at our project page and reach out to us on the talk page
<https://www.mediawiki.org/wiki/Talk:Trust_and_Safety_Product/Temporary_Acco…>.
Thank you!
Szymon Grabarczuk (he/him)
Lead Movement Communications Specialist
(Product & Tech)
Wikimedia Foundation <https://wikimediafoundation.org/>
We will be performing maintenance on the Cloud VPS network next
Tuesday (2024-05-21) starting at around 14:00 UTC. During this window
we will be replacing some software on the Cloud VPS network router.[0]
During the maintenance window there will be a brief period during
which Cloud VPS and any services hosted there (including Toolforge and
PAWS) will not have any external network connectivity. Based on tests
done in our staging environment the full outage should last for less
than a minute assuming there are no unexpected issues.
There is no action required on your side, unless your tools are not
resilient to unexpected network outages - in that case you may need to
manually restart those tools after the maintenance is complete.
[0]: https://phabricator.wikimedia.org/T364459
Taavi (+ the rest of the WMCS team)
--
Taavi Väänänen (he/him)
Site Reliability Engineer, Cloud Services
Wikimedia Foundation
_______________________________________________
Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org
List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…
Hi,
You can ignore this message if your tools do not query abuse filter tables in Wiki Replicas.
In case you missed the announcement in Tech News <https://meta.wikimedia.org/wiki/Tech/News/2024/16>, I'm letting you know about the upcoming removal of four fields from two database tables.
The fields "af_user" and "af_user_text" of the "abuse_filter" table and "afh_user" and "afh_user_text" of the "abuse_filter_history" table are being removed in favor of the new "af_actor" and "afh_actor" fields, respectively. The new fields are already in sync with the legacy ones and available for read operations.
If you run queries involving those fields, you have to update them. The queries will now be structured the same way as, e.g., queries with "rev_actor" of the "revision" table (i.e., by JOIN'ing the new fields with "actor.actor_id"). There are examples of minimalistic migration of each field in the Phabricator task <https://phabricator.wikimedia.org/T361996> that tracks the migration.
No deadline has been proposed for the removal, but it will probably have been done by the end of May unless someone asks for more time.
Thanks,
Matěj Suchánek