Nope, still down for me, and no update on the Phabricator ticket

---

De: WikiLovesMonuments <wikilovesmonuments-bounces@lists.wikimedia.org> em nome de effe iets anders <effeietsanders@gmail.com>
Enviado: quarta-feira, 11 de novembro de 2020 02:48
Para: Wiki Loves Monuments Photograph Competition <wikilovesmonuments@lists.wikimedia.org>
Assunto: Re: [Wiki Loves Monuments] Is Montage tool down?

 

I believe we're back up?

On Tue, Nov 10, 2020 at 11:50 AM Stephen LaPorte <stephen.laporte@gmail.com> wrote:

Hi all,

Toolforge is temporarily in read-only mode for planned maintenance. It may take an hour or longer before Montage is back running. 

I can send an email when we're back up, and please see below for additional info.

Thanks,

Stephen

---------- Forwarded message ---------
From: Brooke Storm <bstorm@wikimedia.org>
Date: Tue, Nov 10, 2020 at 7:52 AM
Subject: Re: [Cloud] [Cloud-announce] 2020-11-10 ToolsDB (User databases in Toolforge) read-only downtime
To: <Cloud-announce@lists.wikimedia.org>

This will be happening in around 10 minutes. ToolsDB will be read-only until we can get a consistent dump to rebuild replication.

Brooke Storm

Staff SRE

Wikimedia Cloud Services

bstorm@wikimedia.org

IRC: bstorm

On Nov 6, 2020, at 12:12 PM, Brooke Storm <bstorm@wikimedia.org> wrote:

The ToolsDB service suffered a breakage in replication on 2020-10-27. WMCS has tried to restore replication of data, but that has been unsuccessful so far including doing a dump to rebuild replication without downtime.

At this point, we have a new server waiting to become the replica, but to start the replication process, we need to set the database to read-only for a full dump. This could easily take more than an hour. During that entire time, the database will be read-only.

We will begin at 1600 UTC and finish when it is done. The database is quite large, but, with it in read-only mode, I hope the backup will not take terribly long.

Please see https://phabricator.wikimedia.org/T266587 for additional information. 

Brooke Storm

Staff SRE

Wikimedia Cloud Services

bstorm@wikimedia.org

IRC: bstorm

_______________________________________________
Wikimedia Cloud Services announce mailing list
Cloud-announce@lists.wikimedia.org (formerly labs-announce@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce

On Tue, Nov 10, 2020 at 11:21 AM Ciell Wikipedia <ciell.wikipedia@gmail.com> wrote:

Hi,

Thank you for reporting this: you are not the only one with this problem. I've alerted the developers of the tool already.

Vriendelijke groet,

Ciell

Op di 10 nov. 2020 om 20:17 schreef Éder Porto <eder.porto@hotmail.com>:

Hello to all,

Today, a few hours ago, the montage tool just stopped working, I can't even login. Is this happening with someone else? Shoul a ticket on Phabricator be open? A issue on gitHub? My error message is as folows:

<ExceptionInfo [sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (1290, 'The MariaDB server is running with the --read-only option so it cannot execute this statement') [SQL: u'UPDATE users SET last_active_date=%s WHERE users.id = %s'] [parameters: (datetime.datetime(2020, 11, 10, 19, 13, 33, 773317), 47501133L)]] (40 frames, last=Callpoint('defaulterrorhandler', 36, 'MySQLdb.connections', '/data/project/montage/www/python/venv/local/lib/python2.7/site-packages/MySQLdb/connections.py', 65, ' raise errorclass, errorvalue'))>

Thanks in advance for any comments,

Éder Porto | User:Ederporto

_______________________________________________
Wiki Loves Monuments mailing list
WikiLovesMonuments@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments
http://www.wikilovesmonuments.org

_______________________________________________
Wiki Loves Monuments mailing list
WikiLovesMonuments@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments
http://www.wikilovesmonuments.org

_______________________________________________
Wiki Loves Monuments mailing list
WikiLovesMonuments@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments
http://www.wikilovesmonuments.org

_______________________________________________
Wiki Loves Monuments mailing list
WikiLovesMonuments@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments
http://www.wikilovesmonuments.org