(If you don’t work with pagelinks table, feel free to ignore this message)
Hello,
Here is an update and reminder on the previous announcement
<https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/…>
regarding normalization of links tables that was sent around a year ago.
As part of that work, soon the pl_namespace and pl_title columns of
pagelinks table will be dropped and you will need to use pl_target_id
joining with the linktarget table instead. This is basically identical to
the templatelinks normalization that happened a year ago.
Currently, MediaWiki writes to both data schemes of pagelinks for new rows
in all wikis except English Wikipedia and Wikimedia Commons (we will start
writing to these two wikis next week). We have started to backfill the data
with the new schema but it will take weeks to finish in large wikis.
So if you query this table directly or your tools do, You will need to
update them accordingly. I will write a reminder before dropping the old
columns once the data has been fully backfilled.
You can keep track of the general long-term work in T300222
<https://phabricator.wikimedia.org/T300222> and the specific work for
pagelinks in T299947 <https://phabricator.wikimedia.org/T299947>. You can
also read more on the reasoning in T222224
<https://phabricator.wikimedia.org/T222224> or the previous announcement
<https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/…>
.
Thank you,
--
*Amir Sarabadani (he/him)*
Staff Database Architect
Wikimedia Foundation <https://wikimediafoundation.org/>
Poking around on my debian bookworm instance, I found /usr/local/share/ca-certificates/wmf_ca_2017_2020.crt, which looks like an expired SSL certificate:
> Certificate:
> Data:
> Version: 3 (0x2)
> Serial Number:
> 9f:14:76:9e:ea:f4:18:c3
> Signature Algorithm: sha256WithRSAEncryption
> Issuer: C = US, ST = California, L = San Francisco, O = Wikimedia Foundation, OU = Operations, CN = WMF CA 2017-2020
> Validity
> Not Before: Jul 19 20:43:26 2017 GMT
> Not After : Jul 18 20:43:26 2020 GMT
> Subject: C = US, ST = California, L = San Francisco, O = Wikimedia Foundation, OU = Operations, CN = WMF CA 2017-2020
Does this do anything useful? Does it do any harm?
Trying this again without the screenshot, which got stuck in the filters.
> Begin forwarded message:
>
> From: Roy Smith <roy(a)panix.com>
> Subject: Installing elasticsearch on a VPS instance?
> Date: December 17, 2023 at 4:00:08 PM EST
> To: Wikimedia Cloud Services general discussion and support <cloud(a)lists.wikimedia.org>
>
> I want to install elasticsearch on a VPS instance. I’m guessing this involves puppet, but can’t figure out the details. The instructions at https://wikitech.wikimedia.org/wiki/Help:Puppet don’t match what I’m actually seeing in Horizon. I go to the Puppet Configuration tab for my instance and there’s no project/common/all sub-pages as described under Apply a puppet role to or change hiera config of an individual instance.
>
> https://horizon.wikimedia.org/project/instances/51660b10-1e33-4f49-a68f-e51…
Toolforge just now suffered a partial grid-engine outage. All grid
services should be back to normal as of this email; some k8s services
may misbehave for the next hour or two.
NFS misbehavior resulted in grid control mechanisms timing out, which
meant that no new jobs could be scheduled for the last 90 minutes or so.
We've rebooted the NFS server which has resolved the primary issues;
however, rebooting NFS is itself disruptive and may have caused other
jobs (both on the grid or in k8s) to fail.
We're currently rebooting all k8s worker nodes, which will take a couple
of hours to complete. During those reboots some jobs may fail or
experience surprise rescheduling.
Sorry for the outage! If your grid job was disrupted by this outage,
please take this as a sign to migrate your service off the grid! Details
about the grid shutdown can be found here:
https://wikitech.wikimedia.org/wiki/News/Toolforge_Grid_Engine_deprecation#…
-Andrew (+ Taavi who did most of the actual recovery work)
_______________________________________________
Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org
List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…
Hello,
We have communicated
<https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/…>
this change in August 2022 but here is a reminder that if you query
externallinks table in wikireplicas, you will need to rework your queries.
Changes are: el_to, el_index and el_index_60 fields will be dropped and you
need to query el_to_domain_index and el_to_path fields instead.
The data has been migrated in all wikis except English Wikipedia, Wikimedia
Commons and Wikidata, you can keep track of the data migration in T326314
<https://phabricator.wikimedia.org/T326314>.
In late June, we will stop updating the old fields and will start dropping
them
This is done to optimize storage of external links (drastically reducing
its size by removing duplication) and enabling user requested features such
as resolving two <https://phabricator.wikimedia.org/T14810> fifteen-year-old
tickets <https://phabricator.wikimedia.org/T17218> or easier aggregating of
external links domain [1], something that wasn’t possible until now.
[1] For example, If you want to get list of the top-linked websites, you
can query `SELECT el_to_domain_index, count(*) from externallinks group by
el_to_domain_index order by count(*) desc limit 50;`
Thank you and sorry for the inconvenience.
--
*Amir Sarabadani (he/him)*
Staff Database Architect
Wikimedia Foundation <https://wikimediafoundation.org/>
Hello everyone,
Wishathon is a new initiative that encourages collaboration across the
Wikimedia community to develop solutions for wishes collected through the
Community Wishlist Survey: <
https://meta.wikimedia.org/wiki/Community_Wishlist_Survey>. The inaugural
community edition will take place from *March 15th to 17th, 2024*. You can
learn more and sign up here: <
https://meta.wikimedia.org/wiki/Event:WishathonMarch2024>.
Since May 2022, the Community Tech team has organized five Wishathons <
https://meta.wikimedia.org/wiki/Community_Tech/Wishathon> internal to
Foundation staff to bring greater technical capacity to address open wishes
and promote collaboration on wishes within the product & technology teams.
Since the inception of the Wishathon, Foundation staff have helped
Community Tech complete work on 10 wishes that were otherwise stalling or
incomplete. This next edition of the Wishathon extends to Wikimedia
contributors as a next step in these efforts.
If you are interested in a project proposal as a user, developer, designer,
or product lead, you are welcome to join the event! If you wish to take
part in the Wishathon, you can register for the event on this wiki page: <
https://meta.wikimedia.org/wiki/Event:WishathonMarch2024>. Organizers will
send a message to your talk page with more details about the event in
January 2024 and post updates on this page.
Read the full announcement: <
https://diff.wikimedia.org/2023/12/11/introducing-wishathon-for-wikimedias-…>
Cheers,
Srishti
On behalf of the Wishathon organizing committee
(Harumi, TheresNoTime, Karolin, Mary, Sheila, Srishti)
*Srishti Sethi*
Senior Developer Advocate
Wikimedia Foundation <https://wikimediafoundation.org/>
Hello!
After our initial announcement of the Grid Engine shutdown timeline[0],
some of you raised concerns about losing your tools.
We want to address those apprehensions while hopefully providing
reassurance. No tools will be deleted until the grid engine shutdown date
on 14 February 2023. However, for tools with unreachable maintainers, an
outage will happen starting on 14 December 2023[1]. This is intended to
raise awareness for users or maintainers who have not otherwise been
reached. A list of these tools can be found here[2]. If you are a
maintainer or a user of a tool in this list, comment on the associated
phabricator ticket with migration plans or a request for more support. The
goal is to have a plan for all tools running on the grid. We want all
actively used tools to be migrated, and will help support users of critical
tools without a maintainer. Thanks for your help in identifying and
migrating those tools you maintain and depend on.
We acknowledge that the timeline might seem tight, and we want to clarify
that our approach is to make this process as seamless as possible. We have
been actively engaging with tool maintainers over the past year, and we
genuinely appreciate the efforts many of you have already made to migrate
your tools to Kubernetes.
We will continue to work closely with maintainers who might need additional
time or assistance.
If for any reason you have not received a phabricator ticket for your tool,
please reach out.
The phabricator ticket is a good place to communicate your needs and plans
for any remaining tools or jobs.
This will help us further organize and plan this process.
Our primary goal is to support you through this transition. If you have
further concerns about the deadline or if you need assistance with the
migration process, please don't hesitate to reach out to us. We are
available on IRC, Telegram, Phabricator[3], and through our other support
channels[4].
Do you still have concerns or questions? Please let us know. We want to do
this together with you, in a way which makes sense to everyone. We’re very
grateful for all the hard work you do, and our only goal here is to secure
the future of tools in the Wikimedia sphere, not to make your lives more
difficult.
Thank you!
[0]:
https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.…
[1]:
https://wikitech.wikimedia.org/wiki/News/Toolforge_Grid_Engine_deprecation#…
[2]:
https://wikitech.wikimedia.org/wiki/News/Toolforge_Grid_Engine_deprecation/…
[3]: https://phabricator.wikimedia.org/project/board/6135/
[4]:
https://wikitech.wikimedia.org/wiki/Portal:Toolforge/About_Toolforge#Commun…
--
Seyram Komla Sapaty
Developer Advocate
Wikimedia Cloud Services
_______________________________________________
Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org
List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…
On Dec 7, 2023 at 07:00:08, cloud-request(a)lists.wikimedia.org wrote:
> Hello Amory,
>
> We reached out directly on several occasions to maintainers via email,
> phabricator and announcements were made on the mailing list.
> What's the username you use and is this the same email address you use? Let
> me crosscheck this again.
Yes, I’m individually aware, but I was trying to note that, given I had not
been directly contacted (via assigned phab task, etc.) AFAICT, it appeared
possible some tools were being missed. For example, I am not listed on <
https://grid-deprecation.toolforge.org>.
~A
Hello, all!
Starting today we are kicking off the process to shut down Grid Engine and
we want to share the timeline with you.
== Background ==
WMCS made the Grid Engine available as a backend engine for hosting tools
on Toolforge - our Platform as a Service(PaaS) offering.
An additional backend engine, Kubernetes, was also made available on
Toolforge.
Over time, maintaining and securing the grid has proven to be difficult and
making it harder to provide support to the community in other ways because
a lot of man-hours of maintenance work is spent on this.
This is mainly due to the fact that there has been no new Grid Engine
releases (bug fixes, security patches, or otherwise) since 2016.[0]
Maintenance work on the grid continued because it was widely popular with
the community and the Kubernetes offering didn't yet have many grid-like
features that contributors came to love.
Once the Kubernetes platform could handle many of the workloads, we started
the grid deprecation process by asking maintainers to migrate off the
grid.[1]
Over the past year, we've been reaching out to our tool maintainers and
working with them to migrate their tools off the Grid to Kubernetes.
We have reached out directly to all maintainers with their phabricator
ticket IDs.
The latest updates to Build Service[2] have addressed many of the issues
that prevented tool maintainers from migrating.
== Initial Timeline ==
The detailed grid shutdown timeline is available on wiki.[3] The important
dates have been copied below.
* 14th December, 2023: Any maintainer who has not responded on phabricator
will have tools shutdown and crontabs commented out. Please plan to migrate
or tell us your plans on phabricator before that date.
* 14th February, 2024: The grid is completely shut down. All tools are
stopped.
If you need further clarification or help migrating your tool, don't
hesitate to reach out to us on IRC, Telegram, Phabricator[4] or via any of
our support channels.[5]
Thank you.
[0]: https://techblog.wikimedia.org/2022/03/14/toolforge-and-grid-engine/
[1]:
https://wikitech.wikimedia.org/wiki/News/Toolforge_Grid_Engine_deprecation
[2]: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Build_Service
[3]:
https://wikitech.wikimedia.org/wiki/News/Toolforge_Grid_Engine_deprecation#…
[4]: https://phabricator.wikimedia.org/project/profile/6135/
[5]:
https://wikitech.wikimedia.org/wiki/Portal:Toolforge/About_Toolforge#Commun…
--
Seyram Komla Sapaty
Developer Advocate
Wikimedia Cloud Services
_______________________________________________
Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org
List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…
Is there also the chance that those numbers are an undercount? I've got a
bunch of simple cron tasks but they're using jlocal, not jsub, and I don't
appear to have been contacted. The documentation vaguely alludes to this,
but I wonder if there are a number of other tools/maintainers who have also
fallen through the cracks.
~A