Hi Merlijn,
Thanks a lot! I can actually remember reading the PLEASE PREFIX YOUR KEYS
in the doc (link below), but didn't know how to do it. I will read again ;).
https://wikitech.wikimedia.org/wiki/Help:Toolforge/Redis_for_Toolforge#Secu…
Psemdel
Le dim. 28 févr. 2021 à 13:02, <cloud-request(a)lists.wikimedia.org> a écrit :
> Send Cloud mailing list submissions to
> cloud(a)lists.wikimedia.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://lists.wikimedia.org/mailman/listinfo/cloud
> or, via email, send a message with subject or body 'help' to
> cloud-request(a)lists.wikimedia.org
>
> You can reach the person managing the list at
> cloud-owner(a)lists.wikimedia.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Cloud digest..."
>
>
> Today's Topics:
>
> 1. Re: Does redis has a sync problem? (Merlijn van Deen (valhallasw))
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Sat, 27 Feb 2021 13:56:48 +0100
> From: "Merlijn van Deen (valhallasw)" <valhallasw(a)arctus.nl>
> To: Wikimedia Cloud Services general discussion and support
> <cloud(a)lists.wikimedia.org>
> Subject: Re: [Cloud] Does redis has a sync problem?
> Message-ID:
> <
> CADJO0e+PvCeg5iX-KF5-8rOTigPkdaQ8Kor7KQzi4i8Zuz+JTw(a)mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Hi Maxime,
>
> Looking at the error logs, it sounds like there may be a clash between
> celery workers of different tools. Specifically, the log shows that there
> is a 'celery(a)cycling-init-bot.worker' ready, but it's a 'celery@algo-news'
> worker that is using a different time zone. As the Redis server is shared,
> clients should use a prefix to prevent this from happening.
>
> I'm not too familiar with Celery, but it seems the underlying code allows
> you to set a KEY_PREFIX through the KOMBU_REDIS_PREFIX environment
> variable:
>
> https://docs.celeryproject.org/projects/kombu/en/stable/_modules/kombu/tran…
>
> Hope this helps!
> Merlijn
>
> On Sat, 27 Feb 2021 at 08:32, maxime delzenne <maxime.delzenne(a)gmail.com>
> wrote:
>
> > Hi,
> >
> > Since yesterday evening (round about 22h CET), my celery send me
> following
> > error:
> >
> > [2021-02-27 07:26:44,782: INFO/MainProcess] Connected to
> > redis://tools-redis.svc.eqiad.wmflabs:6379/0
> > [2021-02-27 07:26:44,867: INFO/MainProcess] mingle: searching for
> neighbors
> > [2021-02-27 07:26:45,924: INFO/MainProcess] mingle: sync with 5 nodes
> > [2021-02-27 07:26:45,926: INFO/MainProcess] mingle: sync complete
> > [2021-02-27 07:26:45,983: INFO/MainProcess]
> > celery(a)cycling-init-bot.worker-56ddc85977-9sf2v ready.
> > [2021-02-27 07:26:47,749: WARNING/MainProcess] Substantial drift from
> > celery(a)algo-news.celery-6444f768f9-w89fb may mean clocks are out of
> > sync. Current drift is
> > 3600 seconds. [orig: 2021-02-27 07:26:47.748906 recv: 2021-02-27
> > 08:26:47.740345]
> >
> > I restarted, but no result. I use TIME_ZONE = 'UTC' in my Django
> settings.
> >
> > Psemdel
> > _______________________________________________
> > Wikimedia Cloud Services mailing list
> > Cloud(a)lists.wikimedia.org (formerly labs-l(a)lists.wikimedia.org)
> > https://lists.wikimedia.org/mailman/listinfo/cloud
> >
>
Hi,
Since yesterday evening (round about 22h CET), my celery send me following
error:
[2021-02-27 07:26:44,782: INFO/MainProcess] Connected to
redis://tools-redis.svc.eqiad.wmflabs:6379/0
[2021-02-27 07:26:44,867: INFO/MainProcess] mingle: searching for neighbors
[2021-02-27 07:26:45,924: INFO/MainProcess] mingle: sync with 5 nodes
[2021-02-27 07:26:45,926: INFO/MainProcess] mingle: sync complete
[2021-02-27 07:26:45,983: INFO/MainProcess]
celery(a)cycling-init-bot.worker-56ddc85977-9sf2v ready.
[2021-02-27 07:26:47,749: WARNING/MainProcess] Substantial drift from
celery(a)algo-news.celery-6444f768f9-w89fb may mean clocks are out of sync.
Current drift is
3600 seconds. [orig: 2021-02-27 07:26:47.748906 recv: 2021-02-27
08:26:47.740345]
I restarted, but no result. I use TIME_ZONE = 'UTC' in my Django settings.
Psemdel
For security reasons we will be updating and rebooting many (possibly
all) cloud-vps VMs today, over the weekend, and on Monday.
The first step of this will be a reboot of all bastions (including the
toolforge login hosts) in about an hour, at 18:30 UTC. That means that
all ssh sessions to cloud-vps hosts will be interrupted as proxy
sessions are terminated.
After the tools bastion reboots, users of toolforge should not be
further affected by these reboots. All other cloud-vps users can expect
at least one more surprise reboot over the next few days. If you need
that reboot to be scheduled at a particular time please contact me
immediately.
Sorry for the inconvenience!
-Andrew + the WMCS team
_______________________________________________
Wikimedia Cloud Services announce mailing list
Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce
Hello,
I usually wouldn't bother people with my issues but I'm sorta desperate
here. This is about https://meet.wmcloud.org. WM Cloud jitsi instance. It's
on a bigram VM and on docker
<https://jitsi.github.io/handbook/docs/devops-guide/devops-guide-docker>.
Users report that when using this "a session with three people today and it
was rather poor. Bad grainy video from my side even though I have 100 Mbps
both ways. After about 15 minutes the session froze and the other two
dropped out." or "we were 4 persons and it was very unstable (no screen
sharing). people's connection got lost, so they could often still hear but
could not participate in the call anymore." and you can reproduce the issue
too if you stay long enough in a meeting with another connection (your
phone for example).
I can't find any reason why this is happening. I wrote some of my
investigations here <https://phabricator.wikimedia.org/T268393> but we
checked the cloud's infra. CPU, memory, etc. all look fine as well as the
network throughput. It doesn't happen with a new VM but quickly (after one
meeting) builds up to have the same issues again. I added a regular restart
of the docker containers (it even destroys them and recreates them again,
also the docker service itself gets restarted) but nothing changes (maybe I
should add a restart of the network manager too?). I assume the iptables
being busy because of docker can contribute to the issue but not this much.
I'm running out of ideas. If anyone has worked with such setup and feels
comfortable debugging this, let me know and I give permission to check the
VM.
Thank you
--
Amir (he/him)
Hello all,
It's coming close to time for annual appointments of community members to
serve on the Code of Conduct (CoC) committee. The Code of Conduct Committee
is a team of five trusted individuals plus five auxiliary members with
diverse affiliations responsible for general enforcement of the Code of
conduct for Wikimedia technical spaces. Committee members are in charge of
processing complaints, discussing with the parties affected, agreeing on
resolutions, and following up on their enforcement. For more on their
duties and roles, see
https://www.mediawiki.org/wiki/Code_of_Conduct/Committee
This is a call for community members interested in volunteering for
appointment to this committee. Volunteers serving in this role should be
experienced Wikimedians or have had experience serving in a similar
position before.
The current committee is doing the selection and will research and discuss
candidates. Six weeks before the beginning of the next Committee term,
meaning 9 April 2021, they will publish their candidate slate (a list of
candidates) on-wiki. The community can provide feedback on these
candidates, via private email to the group choosing the next Committee. The
feedback period will be two weeks. The current Committee will then either
finalize the slate, or update the candidate slate in response to concerns
raised. If the candidate slate changes, there will be another two week
feedback period covering the newly proposed members. After the selections
are finalized, there will be a training period, after which the new
Committee is appointed. The current Committee continues to serve until the
feedback, selection, and training process is complete.
If you are interested in serving on this committee or like to nominate a
candidate, please write an email to techconductcandidates AT wikimedia.org
with details of your experience on the projects, your thoughts on the code
of conduct and the committee and what you hope to bring to the role and
whether you have a preference in being auxiliary or constant member of the
committee. The committee consists of five members plus five auxiliary
members and they will serve for a year; all applications are appreciated
and will be carefully considered. The deadline for applications is the end
of day on 26 March 2021.
Please feel free to pass this invitation along to any users who you think
may be qualified and interested.
Best,
Amir on behalf of the CoC committee
Hello,
today 2021-02-23 in about ~30 minutes (16:00 UTC) we will change how virtual
machine instances running in Cloud VPS contact NFS dump servers [0].
There is no action required on your side.
We anticipate little to no impact as a result of the network changes. But in
case you notice something is not properly working with dumps NFS in Cloud VPS
(or Toolforge) please contact us [1] as soon as possible. The relevant
phabricator ticket [2] is T272397.
regards.
[0] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Dumps
[1]
https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_Introduction#Commun…
[2] https://phabricator.wikimedia.org/T272397
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
Hi,
I'm offboarding from my previous team[0]. As part of that process, I've
noticed I'm a member in a few Horizon projects:
- bastion
- mediawiki-vagrant
- search
- tools
I don't think I've ever used Horizon. For project security reasons, I would
like to remove myself from all projects, since I'm not using them. I've
failed to figure out how to do that.
I've looked at Horizon[1] and Wikitech[2].
Can somebody point me in the right direction?
Thanks,
Željko
--
0: https://phabricator.wikimedia.org/T267313
1: https://horizon.wikimedia.org
2: https://wikitech.wikimedia.org/wiki/Help:Horizon_FAQ
My personal account has an entry in /run/user:
drwx------ 4 roysmith wikidev 80 Jan 14 14:04 /run/user/17980
but not my tool account. Is it possible to set one up for tool accounts?
I'm running a django app under toolforge. I see there's already a
X_REQUEST_ID: e9ec1899b09244939d22c2387db3b75a
header in the incoming request. That's totally cool, but I don't see it documented in the toolforge docs. is that something the toolforge infrastructure guarantees will be there?
Hi everyone,
We are starting a user group for people interested in spreading the
adoption of the Rust programming language[1] in the Wikimedia movement.
If you're not familiar with it, Rust is a systems programming language
that aims to provide the trifecta of safety, concurrency and speed. It's
very fun to use (rated #1 favorite language in Stack Overflow's survey 5
years and counting) and has fantastic tooling.
https://meta.wikimedia.org/wiki/Wikimedia_Rust_developers_user_group
If you're already using Rust or looking to get started - please sign up!
The current proposed goals of the user group are:
* Develop a rich toolkit of Rust libraries and applications for working
with Wikimedia and MediaWiki projects and APIs
* Make Rust a first-class language in Wikimedia infrastructure like
Toolforge
* Encourage the usage of Rust where appropriate to build safer and
better tools
* Assist others who end up encountering Rust and need help (e.g. when
upstreams adopt Rust)
* Mentor new developers who are interested in contributing technically
to the Wikimedia movement, including through programs like GSoC and
Outreachy
Have other ideas of what we could do? Please suggest them on the talk page!
[1] https://www.wikidata.org/wiki/Q575650
[2]
https://meta.wikimedia.org/wiki/Talk:Wikimedia_Rust_developers_user_group
-- Legoktm