Good news: Montage is back up and running. 

They are still working on the Toolforge databases, but I hope it won't affect Montage at this point.

On Wed, Nov 11, 2020 at 7:15 PM Stephen LaPorte <stephen.laporte@gmail.com> wrote:
Hi everyone,

Montage is still down, and unfortunately we do not have a new estimate for when it will be back. We rely on a Toolforge databases for Montage, and Toolforge maintenance is taking longer than expected. The staff working on Toolforge still hope that it will be complete today, but it's hard to say how much longer it will take.

We will let you know as soon as it's back up. You can also follow along on Phabricator (https://phabricator.wikimedia.org/T266587), or add the Montage Status page (https://commons.wikimedia.org/wiki/Commons:Montage/Status) to your watchlist. Thanks for your patience, and I'm sorry this is interrupting the jury process!

Best,
Stephen

On Wed, Nov 11, 2020 at 4:43 AM Ciell Wikipedia <ciell.wikipedia@gmail.com> wrote:
It's not just Montage, it is with (all?) tools that use Toolforge.
According to the Montage talkpage this may take all through Wednesday to restore.

Vriendelijke groet,
Ciell


Op wo 11 nov. 2020 om 07:22 schreef Yaroslav Blanter <ymbalt@gmail.com>:
I also can not get it started.

Yaroslav

On Wed, Nov 11, 2020 at 7:08 AM Éder Porto <eder.porto@hotmail.com> wrote:
Nope, still down for me, and no update on the Phabricator ticket


De: WikiLovesMonuments <wikilovesmonuments-bounces@lists.wikimedia.org> em nome de effe iets anders <effeietsanders@gmail.com>
Enviado: quarta-feira, 11 de novembro de 2020 02:48
Para: Wiki Loves Monuments Photograph Competition <wikilovesmonuments@lists.wikimedia.org>
Assunto: Re: [Wiki Loves Monuments] Is Montage tool down?
 
I believe we're back up?

On Tue, Nov 10, 2020 at 11:50 AM Stephen LaPorte <stephen.laporte@gmail.com> wrote:
Hi all,

Toolforge is temporarily in read-only mode for planned maintenance. It may take an hour or longer before Montage is back running. 

I can send an email when we're back up, and please see below for additional info.

Thanks,
Stephen

---------- Forwarded message ---------
From: Brooke Storm <bstorm@wikimedia.org>
Date: Tue, Nov 10, 2020 at 7:52 AM
Subject: Re: [Cloud] [Cloud-announce] 2020-11-10 ToolsDB (User databases in Toolforge) read-only downtime
To: <Cloud-announce@lists.wikimedia.org>


This will be happening in around 10 minutes. ToolsDB will be read-only until we can get a consistent dump to rebuild replication.

Brooke Storm
Staff SRE
Wikimedia Cloud Services
IRC: bstorm

On Nov 6, 2020, at 12:12 PM, Brooke Storm <bstorm@wikimedia.org> wrote:

The ToolsDB service suffered a breakage in replication on 2020-10-27. WMCS has tried to restore replication of data, but that has been unsuccessful so far including doing a dump to rebuild replication without downtime.
At this point, we have a new server waiting to become the replica, but to start the replication process, we need to set the database to read-only for a full dump. This could easily take more than an hour. During that entire time, the database will be read-only.

We will begin at 1600 UTC and finish when it is done. The database is quite large, but, with it in read-only mode, I hope the backup will not take terribly long.

Please see https://phabricator.wikimedia.org/T266587 for additional information. 

Brooke Storm
Staff SRE
Wikimedia Cloud Services
IRC: bstorm


_______________________________________________
Wikimedia Cloud Services announce mailing list
Cloud-announce@lists.wikimedia.org (formerly labs-announce@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce

On Tue, Nov 10, 2020 at 11:21 AM Ciell Wikipedia <ciell.wikipedia@gmail.com> wrote:
Hi,

Thank you for reporting this: you are not the only one with this problem. I've alerted the developers of the tool already.

Vriendelijke groet,
Ciell


Op di 10 nov. 2020 om 20:17 schreef Éder Porto <eder.porto@hotmail.com>:
Hello to all,

Today, a few hours ago, the montage tool just stopped working, I can't even login. Is this happening with someone else? Shoul a ticket on Phabricator be open? A issue on gitHub? My error message is as folows:

<ExceptionInfo [sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (1290, 'The MariaDB server is running with the --read-only option so it cannot execute this statement') [SQL: u'UPDATE users SET last_active_date=%s WHERE users.id = %s'] [parameters: (datetime.datetime(2020, 11, 10, 19, 13, 33, 773317), 47501133L)]] (40 frames, last=Callpoint('defaulterrorhandler', 36, 'MySQLdb.connections', '/data/project/montage/www/python/venv/local/lib/python2.7/site-packages/MySQLdb/connections.py', 65, ' raise errorclass, errorvalue'))>

Thanks in advance for any comments,
Éder Porto | User:Ederporto
_______________________________________________
Wiki Loves Monuments mailing list
WikiLovesMonuments@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments
http://www.wikilovesmonuments.org
_______________________________________________
Wiki Loves Monuments mailing list
WikiLovesMonuments@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments
http://www.wikilovesmonuments.org
_______________________________________________
Wiki Loves Monuments mailing list
WikiLovesMonuments@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments
http://www.wikilovesmonuments.org
_______________________________________________
Wiki Loves Monuments mailing list
WikiLovesMonuments@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments
http://www.wikilovesmonuments.org
_______________________________________________
Wiki Loves Monuments mailing list
WikiLovesMonuments@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments
http://www.wikilovesmonuments.org
_______________________________________________
Wiki Loves Monuments mailing list
WikiLovesMonuments@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments
http://www.wikilovesmonuments.org