February 2021 - Cloud - lists.wikimedia.org

Re: [Cloud] Cloud Digest, Vol 110, Issue 17
by maxime delzenne 28 Feb '21

28 Feb '21

Hi Merlijn, Thanks a lot! I can actually remember reading the PLEASE PREFIX YOUR KEYS in the doc (link below), but didn't know how to do it. I will read again ;). https://wikitech.wikimedia.org/wiki/Help:Toolforge/Redis_for_Toolforge#Secu… Psemdel Le dim. 28 févr. 2021 à 13:02, <cloud-request(a)lists.wikimedia.org> a écrit : > Send Cloud mailing list submissions to > cloud(a)lists.wikimedia.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.wikimedia.org/mailman/listinfo/cloud > or, via email, send a message with subject or body 'help' to > cloud-request(a)lists.wikimedia.org > > You can reach the person managing the list at > cloud-owner(a)lists.wikimedia.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Cloud digest..." > > > Today's Topics: > > 1. Re: Does redis has a sync problem? (Merlijn van Deen (valhallasw)) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sat, 27 Feb 2021 13:56:48 +0100 > From: "Merlijn van Deen (valhallasw)" <valhallasw(a)arctus.nl> > To: Wikimedia Cloud Services general discussion and support > <cloud(a)lists.wikimedia.org> > Subject: Re: [Cloud] Does redis has a sync problem? > Message-ID: > < > CADJO0e+PvCeg5iX-KF5-8rOTigPkdaQ8Kor7KQzi4i8Zuz+JTw(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hi Maxime, > > Looking at the error logs, it sounds like there may be a clash between > celery workers of different tools. Specifically, the log shows that there > is a 'celery(a)cycling-init-bot.worker' ready, but it's a 'celery@algo-news' > worker that is using a different time zone. As the Redis server is shared, > clients should use a prefix to prevent this from happening. > > I'm not too familiar with Celery, but it seems the underlying code allows > you to set a KEY_PREFIX through the KOMBU_REDIS_PREFIX environment > variable: > > https://docs.celeryproject.org/projects/kombu/en/stable/_modules/kombu/tran… > > Hope this helps! > Merlijn > > On Sat, 27 Feb 2021 at 08:32, maxime delzenne <maxime.delzenne(a)gmail.com> > wrote: > > > Hi, > > > > Since yesterday evening (round about 22h CET), my celery send me > following > > error: > > > > [2021-02-27 07:26:44,782: INFO/MainProcess] Connected to > > redis://tools-redis.svc.eqiad.wmflabs:6379/0 > > [2021-02-27 07:26:44,867: INFO/MainProcess] mingle: searching for > neighbors > > [2021-02-27 07:26:45,924: INFO/MainProcess] mingle: sync with 5 nodes > > [2021-02-27 07:26:45,926: INFO/MainProcess] mingle: sync complete > > [2021-02-27 07:26:45,983: INFO/MainProcess] > > celery(a)cycling-init-bot.worker-56ddc85977-9sf2v ready. > > [2021-02-27 07:26:47,749: WARNING/MainProcess] Substantial drift from > > celery(a)algo-news.celery-6444f768f9-w89fb may mean clocks are out of > > sync. Current drift is > > 3600 seconds. [orig: 2021-02-27 07:26:47.748906 recv: 2021-02-27 > > 08:26:47.740345] > > > > I restarted, but no result. I use TIME_ZONE = 'UTC' in my Django > settings. > > > > Psemdel > > _______________________________________________ > > Wikimedia Cloud Services mailing list > > Cloud(a)lists.wikimedia.org (formerly labs-l(a)lists.wikimedia.org) > > https://lists.wikimedia.org/mailman/listinfo/cloud > > >

1 0

Does redis has a sync problem?
by maxime delzenne 27 Feb '21

27 Feb '21

Hi, Since yesterday evening (round about 22h CET), my celery send me following error: [2021-02-27 07:26:44,782: INFO/MainProcess] Connected to redis://tools-redis.svc.eqiad.wmflabs:6379/0 [2021-02-27 07:26:44,867: INFO/MainProcess] mingle: searching for neighbors [2021-02-27 07:26:45,924: INFO/MainProcess] mingle: sync with 5 nodes [2021-02-27 07:26:45,926: INFO/MainProcess] mingle: sync complete [2021-02-27 07:26:45,983: INFO/MainProcess] celery(a)cycling-init-bot.worker-56ddc85977-9sf2v ready. [2021-02-27 07:26:47,749: WARNING/MainProcess] Substantial drift from celery(a)algo-news.celery-6444f768f9-w89fb may mean clocks are out of sync. Current drift is 3600 seconds. [orig: 2021-02-27 07:26:47.748906 recv: 2021-02-27 08:26:47.740345] I restarted, but no result. I use TIME_ZONE = 'UTC' in my Django settings. Psemdel

2 1

[Cloud-announce] ssh session interruptions today
by Andrew Bogott 26 Feb '21

26 Feb '21

For security reasons we will be updating and rebooting many (possibly all) cloud-vps VMs today, over the weekend, and on Monday. The first step of this will be a reboot of all bastions (including the toolforge login hosts) in about an hour, at 18:30 UTC. That means that all ssh sessions to cloud-vps hosts will be interrupted as proxy sessions are terminated. After the tools bastion reboots, users of toolforge should not be further affected by these reboots. All other cloud-vps users can expect at least one more surprise reboot over the next few days. If you need that reboot to be scheduled at a particular time please contact me immediately. Sorry for the inconvenience! -Andrew + the WMCS team _______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce

1 0

Issue with jitsi - Anyone knows why?
by Amir Sarabadani 26 Feb '21

26 Feb '21

Hello, I usually wouldn't bother people with my issues but I'm sorta desperate here. This is about https://meet.wmcloud.org. WM Cloud jitsi instance. It's on a bigram VM and on docker <https://jitsi.github.io/handbook/docs/devops-guide/devops-guide-docker>. Users report that when using this "a session with three people today and it was rather poor. Bad grainy video from my side even though I have 100 Mbps both ways. After about 15 minutes the session froze and the other two dropped out." or "we were 4 persons and it was very unstable (no screen sharing). people's connection got lost, so they could often still hear but could not participate in the call anymore." and you can reproduce the issue too if you stay long enough in a meeting with another connection (your phone for example). I can't find any reason why this is happening. I wrote some of my investigations here <https://phabricator.wikimedia.org/T268393> but we checked the cloud's infra. CPU, memory, etc. all look fine as well as the network throughput. It doesn't happen with a new VM but quickly (after one meeting) builds up to have the same issues again. I added a regular restart of the docker containers (it even destroys them and recreates them again, also the docker service itself gets restarted) but nothing changes (maybe I should add a restart of the network manager too?). I assume the iptables being busy because of docker can contribute to the issue but not this much. I'm running out of ideas. If anyone has worked with such setup and feels comfortable debugging this, let me know and I give permission to check the VM. Thank you -- Amir (he/him)

1 0

Code of conduct committee call for new members
by Amir Sarabadani 26 Feb '21

26 Feb '21

Hello all, It's coming close to time for annual appointments of community members to serve on the Code of Conduct (CoC) committee. The Code of Conduct Committee is a team of five trusted individuals plus five auxiliary members with diverse affiliations responsible for general enforcement of the Code of conduct for Wikimedia technical spaces. Committee members are in charge of processing complaints, discussing with the parties affected, agreeing on resolutions, and following up on their enforcement. For more on their duties and roles, see https://www.mediawiki.org/wiki/Code_of_Conduct/Committee This is a call for community members interested in volunteering for appointment to this committee. Volunteers serving in this role should be experienced Wikimedians or have had experience serving in a similar position before. The current committee is doing the selection and will research and discuss candidates. Six weeks before the beginning of the next Committee term, meaning 9 April 2021, they will publish their candidate slate (a list of candidates) on-wiki. The community can provide feedback on these candidates, via private email to the group choosing the next Committee. The feedback period will be two weeks. The current Committee will then either finalize the slate, or update the candidate slate in response to concerns raised. If the candidate slate changes, there will be another two week feedback period covering the newly proposed members. After the selections are finalized, there will be a training period, after which the new Committee is appointed. The current Committee continues to serve until the feedback, selection, and training process is complete. If you are interested in serving on this committee or like to nominate a candidate, please write an email to techconductcandidates AT wikimedia.org with details of your experience on the projects, your thoughts on the code of conduct and the committee and what you hope to bring to the role and whether you have a preference in being auxiliary or constant member of the committee. The committee consists of five members plus five auxiliary members and they will serve for a year; all applications are appreciated and will be carefully considered. The deadline for applications is the end of day on 26 March 2021. Please feel free to pass this invitation along to any users who you think may be qualified and interested. Best, Amir on behalf of the CoC committee

1 0

Network change for VMs contacting NFS dumps
by Arturo Borrero Gonzalez 23 Feb '21

23 Feb '21

Hello, today 2021-02-23 in about ~30 minutes (16:00 UTC) we will change how virtual machine instances running in Cloud VPS contact NFS dump servers [0]. There is no action required on your side. We anticipate little to no impact as a result of the network changes. But in case you notice something is not properly working with dumps NFS in Cloud VPS (or Toolforge) please contact us [1] as soon as possible. The relevant phabricator ticket [2] is T272397. regards. [0] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Dumps [1] https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_Introduction#Commun… [2] https://phabricator.wikimedia.org/T272397 -- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation

1 0

Removing myself from Horizon projects
by Željko Filipin 22 Feb '21

22 Feb '21

Hi, I'm offboarding from my previous team[0]. As part of that process, I've noticed I'm a member in a few Horizon projects: - bastion - mediawiki-vagrant - search - tools I don't think I've ever used Horizon. For project security reasons, I would like to remove myself from all projects, since I'm not using them. I've failed to figure out how to do that. I've looked at Horizon[1] and Wikitech[2]. Can somebody point me in the right direction? Thanks, Željko -- 0: https://phabricator.wikimedia.org/T267313 1: https://horizon.wikimedia.org 2: https://wikitech.wikimedia.org/wiki/Help:Horizon_FAQ

3 4

tmpfs for tool account?
by Roy Smith 19 Feb '21

19 Feb '21

My personal account has an entry in /run/user: drwx------ 4 roysmith wikidev 80 Jan 14 14:04 /run/user/17980 but not my tool account. Is it possible to set one up for tool accounts?

1 1

X_REQUEST_ID?
by Roy Smith 16 Feb '21

16 Feb '21

I'm running a django app under toolforge. I see there's already a X_REQUEST_ID: e9ec1899b09244939d22c2387db3b75a header in the incoming request. That's totally cool, but I don't see it documented in the toolforge docs. is that something the toolforge infrastructure guarantees will be there?

2 2

Starting the Wikimedia Rust developers user group
by Kunal Mehta 12 Feb '21

12 Feb '21

Hi everyone, We are starting a user group for people interested in spreading the adoption of the Rust programming language[1] in the Wikimedia movement. If you're not familiar with it, Rust is a systems programming language that aims to provide the trifecta of safety, concurrency and speed. It's very fun to use (rated #1 favorite language in Stack Overflow's survey 5 years and counting) and has fantastic tooling. https://meta.wikimedia.org/wiki/Wikimedia_Rust_developers_user_group If you're already using Rust or looking to get started - please sign up! The current proposed goals of the user group are: * Develop a rich toolkit of Rust libraries and applications for working with Wikimedia and MediaWiki projects and APIs * Make Rust a first-class language in Wikimedia infrastructure like Toolforge * Encourage the usage of Rust where appropriate to build safer and better tools * Assist others who end up encountering Rust and need help (e.g. when upstreams adopt Rust) * Mentor new developers who are interested in contributing technically to the Wikimedia movement, including through programs like GSoC and Outreachy Have other ideas of what we could do? Please suggest them on the talk page! [1] https://www.wikidata.org/wiki/Q575650 [2] https://meta.wikimedia.org/wiki/Talk:Wikimedia_Rust_developers_user_group -- Legoktm

2 1

2024

2023

2022

2021

2020

2019

2018

2017

Cloud February 2021