Hi Merlijn,
Thanks a lot! I can actually remember reading the PLEASE PREFIX YOUR KEYS in the doc (link below), but didn't know how to do it. I will read again ;). https://wikitech.wikimedia.org/wiki/Help:Toolforge/Redis_for_Toolforge#Secur...
Psemdel
Le dim. 28 févr. 2021 à 13:02, cloud-request@lists.wikimedia.org a écrit :
Send Cloud mailing list submissions to cloud@lists.wikimedia.org
To subscribe or unsubscribe via the World Wide Web, visit https://lists.wikimedia.org/mailman/listinfo/cloud or, via email, send a message with subject or body 'help' to cloud-request@lists.wikimedia.org
You can reach the person managing the list at cloud-owner@lists.wikimedia.org
When replying, please edit your Subject line so it is more specific than "Re: Contents of Cloud digest..."
Today's Topics:
- Re: Does redis has a sync problem? (Merlijn van Deen (valhallasw))
Message: 1 Date: Sat, 27 Feb 2021 13:56:48 +0100 From: "Merlijn van Deen (valhallasw)" valhallasw@arctus.nl To: Wikimedia Cloud Services general discussion and support cloud@lists.wikimedia.org Subject: Re: [Cloud] Does redis has a sync problem? Message-ID: < CADJO0e+PvCeg5iX-KF5-8rOTigPkdaQ8Kor7KQzi4i8Zuz+JTw@mail.gmail.com> Content-Type: text/plain; charset="utf-8"
Hi Maxime,
Looking at the error logs, it sounds like there may be a clash between celery workers of different tools. Specifically, the log shows that there is a 'celery@cycling-init-bot.worker' ready, but it's a 'celery@algo-news' worker that is using a different time zone. As the Redis server is shared, clients should use a prefix to prevent this from happening.
I'm not too familiar with Celery, but it seems the underlying code allows you to set a KEY_PREFIX through the KOMBU_REDIS_PREFIX environment variable:
https://docs.celeryproject.org/projects/kombu/en/stable/_modules/kombu/trans...
Hope this helps! Merlijn
On Sat, 27 Feb 2021 at 08:32, maxime delzenne maxime.delzenne@gmail.com wrote:
Hi,
Since yesterday evening (round about 22h CET), my celery send me
following
error:
[2021-02-27 07:26:44,782: INFO/MainProcess] Connected to redis://tools-redis.svc.eqiad.wmflabs:6379/0 [2021-02-27 07:26:44,867: INFO/MainProcess] mingle: searching for
neighbors
[2021-02-27 07:26:45,924: INFO/MainProcess] mingle: sync with 5 nodes [2021-02-27 07:26:45,926: INFO/MainProcess] mingle: sync complete [2021-02-27 07:26:45,983: INFO/MainProcess] celery@cycling-init-bot.worker-56ddc85977-9sf2v ready. [2021-02-27 07:26:47,749: WARNING/MainProcess] Substantial drift from celery@algo-news.celery-6444f768f9-w89fb may mean clocks are out of sync. Current drift is 3600 seconds. [orig: 2021-02-27 07:26:47.748906 recv: 2021-02-27 08:26:47.740345]
I restarted, but no result. I use TIME_ZONE = 'UTC' in my Django
settings.
Psemdel _______________________________________________ Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud