Hi Maxime,

Looking at the error logs, it sounds like there may be a clash between celery workers of different tools. Specifically, the log shows that there is a 'celery@cycling-init-bot.worker' ready, but it's a 'celery@algo-news' worker that is using a different time zone. As the Redis server is shared, clients should use a prefix to prevent this from happening.

I'm not too familiar with Celery, but it seems the underlying code allows you to set a KEY_PREFIX through the KOMBU_REDIS_PREFIX environment variable: https://docs.celeryproject.org/projects/kombu/en/stable/_modules/kombu/transport/redis.html

Hope this helps!

Merlijn

On Sat, 27 Feb 2021 at 08:32, maxime delzenne <maxime.delzenne@gmail.com> wrote:

Hi,

Since yesterday evening (round about 22h CET), my celery send me following error:

[2021-02-27 07:26:44,782: INFO/MainProcess] Connected to redis://tools-redis.svc.eqiad.wmflabs:6379/0
[2021-02-27 07:26:44,867: INFO/MainProcess] mingle: searching for neighbors
[2021-02-27 07:26:45,924: INFO/MainProcess] mingle: sync with 5 nodes
[2021-02-27 07:26:45,926: INFO/MainProcess] mingle: sync complete
[2021-02-27 07:26:45,983: INFO/MainProcess] celery@cycling-init-bot.worker-56ddc85977-9sf2v ready.
[2021-02-27 07:26:47,749: WARNING/MainProcess] Substantial drift from celery@algo-news.celery-6444f768f9-w89fb may mean clocks are out of sync. Current drift is
3600 seconds. [orig: 2021-02-27 07:26:47.748906 recv: 2021-02-27 08:26:47.740345]

I restarted, but no result. I use TIME_ZONE = 'UTC' in my Django settings.

Psemdel
_______________________________________________
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud