Send Cloud mailing list submissions to
        cloud@lists.wikimedia.org

To subscribe or unsubscribe via the World Wide Web, visit
        https://lists.wikimedia.org/mailman/listinfo/cloud
or, via email, send a message with subject or body 'help' to
        cloud-request@lists.wikimedia.org

You can reach the person managing the list at
        cloud-owner@lists.wikimedia.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Cloud digest..."


Today's Topics:

   1. Re: Does redis has a sync problem? (Merlijn van Deen (valhallasw))


----------------------------------------------------------------------

Message: 1
Date: Sat, 27 Feb 2021 13:56:48 +0100
From: "Merlijn van Deen (valhallasw)" <valhallasw@arctus.nl>
To: Wikimedia Cloud Services general discussion and support
        <cloud@lists.wikimedia.org>
Subject: Re: [Cloud] Does redis has a sync problem?
Message-ID:
        <CADJO0e+PvCeg5iX-KF5-8rOTigPkdaQ8Kor7KQzi4i8Zuz+JTw@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi Maxime,

Looking at the error logs, it sounds like there may be a clash between
celery workers of different tools. Specifically, the log shows that there
is a 'celery@cycling-init-bot.worker' ready, but it's a 'celery@algo-news'
worker that is using a different time zone. As the Redis server is shared,
clients should use a prefix to prevent this from happening.

I'm not too familiar with Celery, but it seems the underlying code allows
you to set a KEY_PREFIX through the KOMBU_REDIS_PREFIX environment
variable:
https://docs.celeryproject.org/projects/kombu/en/stable/_modules/kombu/transport/redis.html

Hope this helps!
Merlijn

On Sat, 27 Feb 2021 at 08:32, maxime delzenne <maxime.delzenne@gmail.com>
wrote:

> Hi,
>
> Since yesterday evening (round about 22h CET), my celery send me following
> error:
>
> [2021-02-27 07:26:44,782: INFO/MainProcess] Connected to
> redis://tools-redis.svc.eqiad.wmflabs:6379/0
> [2021-02-27 07:26:44,867: INFO/MainProcess] mingle: searching for neighbors
> [2021-02-27 07:26:45,924: INFO/MainProcess] mingle: sync with 5 nodes
> [2021-02-27 07:26:45,926: INFO/MainProcess] mingle: sync complete
> [2021-02-27 07:26:45,983: INFO/MainProcess]
> celery@cycling-init-bot.worker-56ddc85977-9sf2v ready.
> [2021-02-27 07:26:47,749: WARNING/MainProcess] Substantial drift from
> celery@algo-news.celery-6444f768f9-w89fb may mean clocks are out of
> sync.  Current drift is
> 3600 seconds.  [orig: 2021-02-27 07:26:47.748906 recv: 2021-02-27
> 08:26:47.740345]
>
> I restarted, but no result. I use TIME_ZONE = 'UTC' in my Django settings.
>
> Psemdel
> _______________________________________________
> Wikimedia Cloud Services mailing list
> Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/cloud
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/cloud/attachments/20210227/ac030307/attachment-0001.htm>

------------------------------

Subject: Digest Footer

_______________________________________________
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud


------------------------------

End of Cloud Digest, Vol 110, Issue 17
**************************************