Debian Stretch's security support ends in mid 2022, and the Foundation's
OS policy already discourages use of existing Stretch machines. That
means that it's time for all project admins to start rebuilding your VMs
with Bullseye (or, if you must, Buster.)
Any webservices running in Kubernetes created in the last year or two
are most likely using Buster images already, so there's no action needed
for those. Older kubernetes jobs should be refreshed to use more modern
images whenever possible.
If you are still using the grid engine for webservices, we strongly
encourage you to migrate your jobs to Kubernetes. For other grid uses,
watch this space for future announcements about grid engine migration;
we don't yet have a solution prepared for that.
Details about the what and why for this process can be found here:
https://wikitech.wikimedia.org/wiki/News/Stretch_deprecation
Here is the deprecation timeline:
March 2021: Stretch VM creation disabled in most projects
July 6, 2021: Active support of Stretch ends, Stretch moves into LTS
<- You are Here ->
January 1st, 2022: Stretch VM creation disabled in all projects,
deprecation nagging begins in earnest. Stretch alternatives will be
available for tool migration in Toolforge
May 1, 2022: All active Stretch VMs will be shut down (but not deleted)
by WMCS admins. This includes Toolforge grid exec nodes.
June 30, 2022: LTS support for Debian Stretch ends, all Stretch VMs will
be deleted by WMCS admins
_______________________________________________
Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org
List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…
Hello everyone,
The eighth workshop on the topic of "How to interact with Wikidata via
Pywikibot" is coming up - it will take place on Friday, October 7th at
16:00 UTC. You can find more details on the workshop and a link to join
here: <
https://meta.wikimedia.org/wiki/Small_wiki_toolkits/Workshops#How_to_intera…>
[1].
This workshop will summarise how to access and save information to
Wikidata, including through page generators and queries*, using Pywikibot.
It will also cover the Wikidata conventions about the bot editing,
including the bot approvals process (*queries not included, please bring
your own!). This session will be run by Mike Peel.
To participate in this workshop, you would need basic familiarity with
Wikidata and Pywikibot installation.
We look forward to your participation!
Best,
Srishti
On behalf of the SWT Workshops Organization team
[1]
https://meta.wikimedia.org/wiki/Small_wiki_toolkits/Workshops#How_to_intera…
*Srishti Sethi*
Senior Developer Advocate
Wikimedia Foundation <https://wikimediafoundation.org/>
Yesterday, I reported some log messages in T318479 <https://phabricator.wikimedia.org/T318479>:
logs/django/django.log.2022-09-27:2022-09-28 16:43:39,873 [76e999afc82c10fb99b6c9bf76448d1a] INFO tools_app.middleware: IndexView()
logs/django/django.log.2022-09-27:2022-09-28 16:59:18,903 [76e999afc82c10fb99b6c9bf76448d1a] ERROR tools_app.redis: Redis ConnectionError: Error while reading from tools-redis.svc.eqiad.wmflabs:6379 : (110, 'Connection timed out')
logs/django/django.log.2022-09-27:2022-09-28 16:59:19,196 [76e999afc82c10fb99b6c9bf76448d1a] INFO tools_app.middleware: request took 0:15:39.323408
When I went to look at this again today, the messages were gone. After a bit of head-scratching, I discovered they were now in a .nfs file:
> (venv) spi-tools-dev [django] grep 76e999afc82c10fb99b6c9bf76448d1a .nfs0000000005f910c800000388
> 2022-09-28 16:43:39,873 [76e999afc82c10fb99b6c9bf76448d1a] INFO tools_app.middleware: IndexView()
> 2022-09-28 16:59:18,903 [76e999afc82c10fb99b6c9bf76448d1a] ERROR tools_app.redis: Redis ConnectionError: Error while reading from tools-redis.svc.eqiad.wmflabs:6379 : (110, 'Connection timed out')
> 2022-09-28 16:59:19,196 [76e999afc82c10fb99b6c9bf76448d1a] INFO tools_app.middleware: request took 0:15:39.323408
These log files are created by Python's TimedRotatingFileHandler. So it looks like something was holding the file open at the time it was rotated. In theory, I should be able to find what process has them open using lsof, but that doesn't work when I run it on tools-sgebastion-11:
> lsof .nfs0000000005f910c800000388
> lsof: WARNING: can't stat() tracefs file system /sys/kernel/debug/tracing
> Output information may be incomplete.
and if I shell into the krb instance, I just get:
> bash: lsof: command not found
So how do I figure out what's going on?
Hi all-
It’s time for the fourth edition of the Coolest Tool Award!
Tools play an essential role at Wikimedia, and so do the many volunteer
developers who experiment with new ideas, develop & maintain local & global
solutions and enhance the experience for Wikimedia communities.
We’d like to invite you all to nominate your favorite & most used tools and
help us celebrate the people who create them!
Please go to: https://meta.wikimedia.org/wiki/Coolest_Tool_Award
to recommend tools by October 12, 2022. You can nominate as many tools as
you want by filling out the form multiple times.
Thanks a lot for your ideas and recommendation(s)!
The award is organized & selected by the Coolest Tool Academy 2022
<https://meta.wikimedia.org/wiki/Coolest_Tool_Award/2022>[0]. We plan to
award the coolest tools in a variety of categories (see last year’s
categories <https://meta.wikimedia.org/wiki/Coolest_Tool_Award/2021>[1] for
an example). The award ceremony will take place virtually again this year
and we will provide more details about the specific logistics and dates
when we get closer to the event :-)
We’ll continue to spread the word over the next days and weeks, but any
help sharing this information with others is much appreciated :-)
[0]: https://meta.wikimedia.org/wiki/Coolest_Tool_Award/2022
[1]: https://meta.wikimedia.org/wiki/Coolest_Tool_Award/2021
Thanks-
Komla, for the Coolest Tool Academy 2022
--
Seyram Komla Sapaty
Developer Advocate
Wikimedia Cloud Services
Hello everyone,
Join us in congratulating 11 interns who successfully completed a project
via GSoC 2022 & Outreachy Round 24 with support and guidance from 20
mentors:
- Wikidata infobox rewrite in Lua for Commons by Lennard Hofmann.
Mentors: Mike Peel
- Edit request wizard by Ankit Gupta. Mentors: Enterprisey, SD0001
- Javascript build process and dependencies modernization for Wiki
Education Dashboard by Shashwat Khanna. Mentors: Sage Ross, Dhruvdutt Jadhav
- Custom picture selector improvements for Commons Android app by Ayan
Sarkar. Mentors: Nicolas Raoul, Aditya Srivastav
- Canasta CLI and Kubernetes template stack by Amal Paul. Mentors:
Jeffrey Wang, Yaron Koren
- WikidataComplete plugin extension for enabling data donations,
recommendations, and gamification by Lalit Suthar. Mentors: Andreas Both,
Dennis Diefenbach, Aleksandr Perevalov, Kunpeng GUO.
- Campaigns retention metrics dashboard by Nivas Ramisetty. Mentors: Jay
Prakash, Krishna Chaitanya Velaga, Alex Stinson
- Tracking support for specific namespaces for Programs & Events
Dashboard by Vaidehi Atpadkar. Mentors: Sage Ross
- Python library to work with html-dumps by Nazia Tasnim. Mentors:
Martin Gerlach, Isaac Johnson
- Bots for automatically identifying first and last author names for
Wikicite and Wikidata by Feliciss & Luis Roberto. Mentors: Mike Peel, Andy
Mabbett
You can learn more about these projects here:
<https://www.mediawiki.org/wiki/Google_Summer_of_Code/Past_projects#2022>
[1],
<https://www.mediawiki.org/wiki/Outreachy/Past_projects#Round_24> [2].
Cheers,
Wikimedia organization administrators (Srishti, Aisha, and Gopa)
[1] https://www.mediawiki.org/wiki/Google_Summer_of_Code/Past_projects#2022
[2] https://www.mediawiki.org/wiki/Outreachy/Past_projects#Round_24
*Srishti Sethi*
Senior Developer Advocate
Wikimedia Foundation <https://wikimediafoundation.org/>
TL;DR: If your tool makes use of Wikimedia Dumps, please restart your
tool today to avoid breakage next week.
---
As part of a hardware upgrades[0], the mount points hosting the dumps is
moving. In order to support this move we've needed to update the base
Kubernetes containers to know about the new mount points. For the vast
majority of use cases, a simple tool restart will pick up the new
container and ensure a smooth transition.
For rare non-conforming use cases, here are a few more details. The old
mountpoints were:
/mnt/nfs/dumps-labstore1006.wikimedia.org
/mnt/nfs/dumps-labstore1007.wikimedia.org
The new mountpoints are:
/mnt/nfs/dumps-clouddumps1001.wikimedia.org
/mnt/nfs/dumps-clouddumps1002.wikimedia.org
Currently all four mountpoints are present on toolforge nodes, but the
older mountpoints will be disabled next week. If your tool contains
references to ANY of the above paths, I encourage you to update your
code to use /public/dumps/ instead, which is a persistent server-neutral
link.
Tools using the old mountpoints or accessing dumps via old containers
will start breaking a week from today on Monday, October 3.
[0] https://phabricator.wikimedia.org/T309346
_______________________________________________
Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org
List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…
Is there an issue with the Toolforge redis server? Every so often, I get connection timeouts:
django.log.2022-09-10:2022-09-11 18:31:17,239 [0b5fd61ed67203bfe290ca99fcb52e1e] ERROR tools_app.redis: Redis ConnectionError: Error while reading from socket: (110, 'Connection timed out')
django.log.2022-09-11:2022-09-15 13:23:26,327 [d1b54ab741b3306abd6265a5adfe711d] ERROR tools_app.redis: Redis ConnectionError: Error while reading from socket: (110, 'Connection timed out')
django.log.2022-09-13:2022-09-15 13:12:51,447 [d46cdc3ce7e666e86a697d6319fc4753] ERROR tools_app.redis: Redis ConnectionError: Error while reading from socket: (110, 'Connection timed out')
django.log.2022-09-17:2022-09-19 02:03:30,663 [c3fc2d85621909d117c9d4fd638d1f9e] ERROR tools_app.redis: Redis ConnectionError: Error while reading from socket: (110, 'Connection timed out')
It's kind of annoying because the default timeout appears to be 15 minutes and my app hangs for that time.
I'm connecting to redis://tools-redis.svc.eqiad.wmflabs:6379/0 <redis://tools-redis.svc.eqiad.wmflabs:6379/0>
Hi all,
We've just configured a simple rate limiter on the Cloud VPS web proxy
that limits traffic from a single IP address to 100 requests/second. The
limit is designed to be high enough that it shouldn't affect most normal
workflows while also being low enough to properly block excess traffic.
This is in response to some misbehaving crawlers that were causing
instability in the proxy service. We applied a similar rate limit to
Toolforge earlier this year.
Please be in touch if this is causing issues for your project.
Thanks!
-- Taavi
_______________________________________________
Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org
List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…
Hello!
Thanks to everyone who took the time to provide their valuable feedback on
the Toolhub taxonomy[0]!
After a round of community feedback and input, we made the following
decisions about which categories and values to implement in the first
productionized version of the taxonomy.
=== Summary of Changes ===
1.
Exclude the proposed Programming language attribute.
2.
Exclude the proposed Platform attribute
3.
Revise the Tasks attribute values:
1.
Remove "Creating or uploading content"
2.
Add "Creating new content"
3.
Rename "Generating and recommending content" to "Recommending content"
4.
Revise the Content types attribute values:
1.
Add additional level of hierarchy to group content types and enable
both broad or specific values to be applied.
2.
Remove "Files".
3.
Split "Maps" and "Geographic Data"
Find more details of additional changes on the decision record log page[0]
=== Next Steps ===
The team will continue to observe and improve the taxonomy as the community
continues to use Toolhub.
We will monitor tags and community created lists to determine if certain
attributes would be useful or feasible in the future.
[0]: https://meta.wikimedia.org/wiki/Toolhub/Data_model#Taxonomy_v2 [1]:
https://meta.wikimedia.org/wiki/Toolhub/Decision_record#Taxonomy_v2
--
Seyram Komla Sapaty
Developer Advocate
Wikimedia Cloud Services
_______________________________________________
Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org
List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…
What: Diffusion git hosting moving to GitLab [0][1]
When: Tuesday 2022-09-06 between 15:00 - 23:00 UTC
Why: Unblocking sunsetting work for Differential/Diffusion [2]
What you can do: If you have not logged into
https://gitlab.wikimedia.org/ yet to attach your Developer account and
look around, now would be a great time!
Toolforge tool maintainers can use Striker
(<https://toolsadmin.wikimedia.org/>) to create git repositories for
each of their tools. Today these git repositories are hosted by
<https://phabricator.wikimedia.org/diffusion/>. Starting on 2022-09-06
new repositories will be hosted under
<https://gitlab.wikimedia.org/toolforge-repos> instead.
We will also be migrating the 474 existing Striker created
repositories from Diffusion to GitLab starting on Tuesday 2022-09-06.
This process will involve making each existing Diffusion repository
read-only, copying it to GitLab, and finally configuring the Diffusion
repo to be a read-only mirror of the GitLab repo. We hope that this
set of operations will be the least disruptive way to migrate the
repositories to the new hosting platform.
For tool maintainers with git repos that are migrating who *do not*
yet have their Developer account attached at
https://gitlab.wikimedia.org/, GitLab will send an email invitation to
join the new repo. Because of some quirks of the login process that we
are using for our GitLab service, the link in this email needs to be
used *after* you have attached your account in order to grant you
access [3].
[0]: https://phabricator.wikimedia.org/T296893
[1]: https://phabricator.wikimedia.org/T315706
[2]: https://phabricator.wikimedia.org/T191182
[3]: https://phabricator.wikimedia.org/T313366#8203450
Bryan, on behalf of the Toolforge administrators
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
_______________________________________________
Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org
List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…