Hello to all,
Today, a few hours ago, the montage tool just stopped working, I can't even login. Is this happening with someone else? Shoul a ticket on Phabricator be open? A issue on gitHub? My error message is as folows:
<ExceptionInfo [sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (1290, 'The MariaDB server is running with the --read-only option so it cannot execute this statement') [SQL: u'UPDATE users SET last_active_date=%s WHERE users.id = %s'] [parameters: (datetime.datetime(2020, 11, 10, 19, 13, 33, 773317), 47501133L)]] (40 frames, last=Callpoint('defaulterrorhandler', 36, 'MySQLdb.connections', '/data/project/montage/www/python/venv/local/lib/python2.7/site-packages/MySQLdb/connections.py', 65, ' raise errorclass, errorvalue'))>
Thanks in advance for any comments, Éder Porto | User:Ederporto
Hi,
Thank you for reporting this: you are not the only one with this problem. I've alerted the developers of the tool already.
Vriendelijke groet, Ciell
Op di 10 nov. 2020 om 20:17 schreef Éder Porto eder.porto@hotmail.com:
Hello to all,
Today, a few hours ago, the montage tool just stopped working, I can't even login. Is this happening with someone else? Shoul a ticket on Phabricator be open? A issue on gitHub? My error message is as folows:
<ExceptionInfo [sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (1290, 'The MariaDB server is running with the --read-only option so it cannot execute this statement') [SQL: u'UPDATE users SET last_active_date=%s WHERE users.id = %s'] [parameters: (datetime.datetime(2020, 11, 10, 19, 13, 33, 773317), 47501133L)]] (40 frames, last=Callpoint('defaulterrorhandler', 36, 'MySQLdb.connections', '/data/project/montage/www/python/venv/local/lib/python2.7/site-packages/MySQLdb/connections.py', 65, ' raise errorclass, errorvalue'))>
Thanks in advance for any comments, Éder Porto | User:Ederporto _______________________________________________ Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
Hi all,
Toolforge is temporarily in read-only mode for planned maintenance. It may take an hour or longer before Montage is back running.
I can send an email when we're back up, and please see below for additional info.
Thanks, Stephen
---------- Forwarded message --------- From: Brooke Storm bstorm@wikimedia.org Date: Tue, Nov 10, 2020 at 7:52 AM Subject: Re: [Cloud] [Cloud-announce] 2020-11-10 ToolsDB (User databases in Toolforge) read-only downtime To: Cloud-announce@lists.wikimedia.org
This will be happening in around 10 minutes. ToolsDB will be read-only until we can get a consistent dump to rebuild replication.
Brooke Storm Staff SRE Wikimedia Cloud Services bstorm@wikimedia.org IRC: bstorm
On Nov 6, 2020, at 12:12 PM, Brooke Storm bstorm@wikimedia.org wrote:
The ToolsDB service suffered a breakage in replication on 2020-10-27. WMCS has tried to restore replication of data, but that has been unsuccessful so far including doing a dump to rebuild replication without downtime. At this point, we have a new server waiting to become the replica, but to start the replication process, we need to set the database to read-only for a full dump. This could easily take more than an hour. During that entire time, the database will be read-only.
We will begin at 1600 UTC and finish when it is done. The database is quite large, but, with it in read-only mode, I hope the backup will not take terribly long.
Please see https://phabricator.wikimedia.org/T266587 for additional information.
Brooke Storm Staff SRE Wikimedia Cloud Services bstorm@wikimedia.org IRC: bstorm
_______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce@lists.wikimedia.org (formerly labs-announce@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce
On Tue, Nov 10, 2020 at 11:21 AM Ciell Wikipedia ciell.wikipedia@gmail.com wrote:
Hi,
Thank you for reporting this: you are not the only one with this problem. I've alerted the developers of the tool already.
Vriendelijke groet, Ciell
Op di 10 nov. 2020 om 20:17 schreef Éder Porto eder.porto@hotmail.com:
Hello to all,
Today, a few hours ago, the montage tool just stopped working, I can't even login. Is this happening with someone else? Shoul a ticket on Phabricator be open? A issue on gitHub? My error message is as folows:
<ExceptionInfo [sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (1290, 'The MariaDB server is running with the --read-only option so it cannot execute this statement') [SQL: u'UPDATE users SET last_active_date=%s WHERE users.id = %s'] [parameters: (datetime.datetime(2020, 11, 10, 19, 13, 33, 773317), 47501133L)]] (40 frames, last=Callpoint('defaulterrorhandler', 36, 'MySQLdb.connections', '/data/project/montage/www/python/venv/local/lib/python2.7/site-packages/MySQLdb/connections.py', 65, ' raise errorclass, errorvalue'))>
Thanks in advance for any comments, Éder Porto | User:Ederporto _______________________________________________ Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
I believe we're back up?
On Tue, Nov 10, 2020 at 11:50 AM Stephen LaPorte stephen.laporte@gmail.com wrote:
Hi all,
Toolforge is temporarily in read-only mode for planned maintenance. It may take an hour or longer before Montage is back running.
I can send an email when we're back up, and please see below for additional info.
Thanks, Stephen
---------- Forwarded message --------- From: Brooke Storm bstorm@wikimedia.org Date: Tue, Nov 10, 2020 at 7:52 AM Subject: Re: [Cloud] [Cloud-announce] 2020-11-10 ToolsDB (User databases in Toolforge) read-only downtime To: Cloud-announce@lists.wikimedia.org
This will be happening in around 10 minutes. ToolsDB will be read-only until we can get a consistent dump to rebuild replication.
Brooke Storm Staff SRE Wikimedia Cloud Services bstorm@wikimedia.org IRC: bstorm
On Nov 6, 2020, at 12:12 PM, Brooke Storm bstorm@wikimedia.org wrote:
The ToolsDB service suffered a breakage in replication on 2020-10-27. WMCS has tried to restore replication of data, but that has been unsuccessful so far including doing a dump to rebuild replication without downtime. At this point, we have a new server waiting to become the replica, but to start the replication process, we need to set the database to read-only for a full dump. This could easily take more than an hour. During that entire time, the database will be read-only.
We will begin at 1600 UTC and finish when it is done. The database is quite large, but, with it in read-only mode, I hope the backup will not take terribly long.
Please see https://phabricator.wikimedia.org/T266587 for additional information.
Brooke Storm Staff SRE Wikimedia Cloud Services bstorm@wikimedia.org IRC: bstorm
Wikimedia Cloud Services announce mailing list Cloud-announce@lists.wikimedia.org (formerly labs-announce@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce
On Tue, Nov 10, 2020 at 11:21 AM Ciell Wikipedia < ciell.wikipedia@gmail.com> wrote:
Hi,
Thank you for reporting this: you are not the only one with this problem. I've alerted the developers of the tool already.
Vriendelijke groet, Ciell
Op di 10 nov. 2020 om 20:17 schreef Éder Porto eder.porto@hotmail.com:
Hello to all,
Today, a few hours ago, the montage tool just stopped working, I can't even login. Is this happening with someone else? Shoul a ticket on Phabricator be open? A issue on gitHub? My error message is as folows:
<ExceptionInfo [sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (1290, 'The MariaDB server is running with the --read-only option so it cannot execute this statement') [SQL: u'UPDATE users SET last_active_date=%s WHERE users.id = %s'] [parameters: (datetime.datetime(2020, 11, 10, 19, 13, 33, 773317), 47501133L)]] (40 frames, last=Callpoint('defaulterrorhandler', 36, 'MySQLdb.connections', '/data/project/montage/www/python/venv/local/lib/python2.7/site-packages/MySQLdb/connections.py', 65, ' raise errorclass, errorvalue'))>
Thanks in advance for any comments, Éder Porto | User:Ederporto _______________________________________________ Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
Nope, still down for me, and no update on the Phabricator ticket
[cid:67339ef0-94f0-4b54-90cb-4a1165a663f4]
________________________________ De: WikiLovesMonuments wikilovesmonuments-bounces@lists.wikimedia.org em nome de effe iets anders effeietsanders@gmail.com Enviado: quarta-feira, 11 de novembro de 2020 02:48 Para: Wiki Loves Monuments Photograph Competition wikilovesmonuments@lists.wikimedia.org Assunto: Re: [Wiki Loves Monuments] Is Montage tool down?
I believe we're back up?
On Tue, Nov 10, 2020 at 11:50 AM Stephen LaPorte <stephen.laporte@gmail.commailto:stephen.laporte@gmail.com> wrote: Hi all,
Toolforge is temporarily in read-only mode for planned maintenance. It may take an hour or longer before Montage is back running.
I can send an email when we're back up, and please see below for additional info.
Thanks, Stephen
---------- Forwarded message --------- From: Brooke Storm <bstorm@wikimedia.orgmailto:bstorm@wikimedia.org> Date: Tue, Nov 10, 2020 at 7:52 AM Subject: Re: [Cloud] [Cloud-announce] 2020-11-10 ToolsDB (User databases in Toolforge) read-only downtime To: <Cloud-announce@lists.wikimedia.orgmailto:Cloud-announce@lists.wikimedia.org>
This will be happening in around 10 minutes. ToolsDB will be read-only until we can get a consistent dump to rebuild replication.
Brooke Storm Staff SRE Wikimedia Cloud Services bstorm@wikimedia.orgmailto:bstorm@wikimedia.org IRC: bstorm
On Nov 6, 2020, at 12:12 PM, Brooke Storm <bstorm@wikimedia.orgmailto:bstorm@wikimedia.org> wrote:
The ToolsDB service suffered a breakage in replication on 2020-10-27. WMCS has tried to restore replication of data, but that has been unsuccessful so far including doing a dump to rebuild replication without downtime. At this point, we have a new server waiting to become the replica, but to start the replication process, we need to set the database to read-only for a full dump. This could easily take more than an hour. During that entire time, the database will be read-only.
We will begin at 1600 UTC and finish when it is done. The database is quite large, but, with it in read-only mode, I hope the backup will not take terribly long.
Please see https://phabricator.wikimedia.org/T266587 for additional information.
Brooke Storm Staff SRE Wikimedia Cloud Services bstorm@wikimedia.orgmailto:bstorm@wikimedia.org IRC: bstorm
_______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce@lists.wikimedia.orgmailto:Cloud-announce@lists.wikimedia.org (formerly labs-announce@lists.wikimedia.orgmailto:labs-announce@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce
On Tue, Nov 10, 2020 at 11:21 AM Ciell Wikipedia <ciell.wikipedia@gmail.commailto:ciell.wikipedia@gmail.com> wrote: Hi,
Thank you for reporting this: you are not the only one with this problem. I've alerted the developers of the tool already.
Vriendelijke groet, Ciell
Op di 10 nov. 2020 om 20:17 schreef Éder Porto <eder.porto@hotmail.commailto:eder.porto@hotmail.com>: Hello to all,
Today, a few hours ago, the montage tool just stopped working, I can't even login. Is this happening with someone else? Shoul a ticket on Phabricator be open? A issue on gitHub? My error message is as folows:
<ExceptionInfo [sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (1290, 'The MariaDB server is running with the --read-only option so it cannot execute this statement') [SQL: u'UPDATE users SET last_active_date=%s WHERE users.idhttp://users.id = %s'] [parameters: (datetime.datetime(2020, 11, 10, 19, 13, 33, 773317), 47501133L)]] (40 frames, last=Callpoint('defaulterrorhandler', 36, 'MySQLdb.connections', '/data/project/montage/www/python/venv/local/lib/python2.7/site-packages/MySQLdb/connections.py', 65, ' raise errorclass, errorvalue'))>
Thanks in advance for any comments, Éder Porto | User:Ederporto _______________________________________________ Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.orgmailto:WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org _______________________________________________ Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.orgmailto:WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org _______________________________________________ Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.orgmailto:WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
I also can not get it started.
Yaroslav
On Wed, Nov 11, 2020 at 7:08 AM Éder Porto eder.porto@hotmail.com wrote:
Nope, still down for me, and no update on the Phabricator ticket
*De:* WikiLovesMonuments wikilovesmonuments-bounces@lists.wikimedia.org em nome de effe iets anders effeietsanders@gmail.com *Enviado:* quarta-feira, 11 de novembro de 2020 02:48 *Para:* Wiki Loves Monuments Photograph Competition < wikilovesmonuments@lists.wikimedia.org> *Assunto:* Re: [Wiki Loves Monuments] Is Montage tool down?
I believe we're back up?
On Tue, Nov 10, 2020 at 11:50 AM Stephen LaPorte < stephen.laporte@gmail.com> wrote:
Hi all,
Toolforge is temporarily in read-only mode for planned maintenance. It may take an hour or longer before Montage is back running.
I can send an email when we're back up, and please see below for additional info.
Thanks, Stephen
---------- Forwarded message --------- From: *Brooke Storm* bstorm@wikimedia.org Date: Tue, Nov 10, 2020 at 7:52 AM Subject: Re: [Cloud] [Cloud-announce] 2020-11-10 ToolsDB (User databases in Toolforge) read-only downtime To: Cloud-announce@lists.wikimedia.org
This will be happening in around 10 minutes. ToolsDB will be read-only until we can get a consistent dump to rebuild replication.
Brooke Storm Staff SRE Wikimedia Cloud Services bstorm@wikimedia.org IRC: bstorm
On Nov 6, 2020, at 12:12 PM, Brooke Storm bstorm@wikimedia.org wrote:
The ToolsDB service suffered a breakage in replication on 2020-10-27. WMCS has tried to restore replication of data, but that has been unsuccessful so far including doing a dump to rebuild replication without downtime. At this point, we have a new server waiting to become the replica, but to start the replication process, we need to set the database to read-only for a full dump. This could easily take more than an hour. During that entire time, the database will be read-only.
We will begin at 1600 UTC and finish when it is done. The database is quite large, but, with it in read-only mode, I hope the backup will not take terribly long.
Please see https://phabricator.wikimedia.org/T266587 for additional information.
Brooke Storm Staff SRE Wikimedia Cloud Services bstorm@wikimedia.org IRC: bstorm
Wikimedia Cloud Services announce mailing list Cloud-announce@lists.wikimedia.org (formerly labs-announce@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce
On Tue, Nov 10, 2020 at 11:21 AM Ciell Wikipedia < ciell.wikipedia@gmail.com> wrote:
Hi,
Thank you for reporting this: you are not the only one with this problem. I've alerted the developers of the tool already.
Vriendelijke groet, Ciell
Op di 10 nov. 2020 om 20:17 schreef Éder Porto eder.porto@hotmail.com:
Hello to all,
Today, a few hours ago, the montage tool just stopped working, I can't even login. Is this happening with someone else? Shoul a ticket on Phabricator be open? A issue on gitHub? My error message is as folows:
<ExceptionInfo [sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (1290, 'The MariaDB server is running with the --read-only option so it cannot execute this statement') [SQL: u'UPDATE users SET last_active_date=%s WHERE users.id = %s'] [parameters: (datetime.datetime(2020, 11, 10, 19, 13, 33, 773317), 47501133L)]] (40 frames, last=Callpoint('defaulterrorhandler', 36, 'MySQLdb.connections', '/data/project/montage/www/python/venv/local/lib/python2.7/site-packages/MySQLdb/connections.py', 65, ' raise errorclass, errorvalue'))>
Thanks in advance for any comments, Éder Porto | User:Ederporto _______________________________________________ Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
It's not just Montage, it is with (all?) tools that use Toolforge. According to the Montage talkpage this may take all through Wednesday to restore. https://commons.wikimedia.org/wiki/Commons_talk:Montage#Montage_downtime_202...
Vriendelijke groet, Ciell
Op wo 11 nov. 2020 om 07:22 schreef Yaroslav Blanter ymbalt@gmail.com:
I also can not get it started.
Yaroslav
On Wed, Nov 11, 2020 at 7:08 AM Éder Porto eder.porto@hotmail.com wrote:
Nope, still down for me, and no update on the Phabricator ticket
*De:* WikiLovesMonuments wikilovesmonuments-bounces@lists.wikimedia.org em nome de effe iets anders effeietsanders@gmail.com *Enviado:* quarta-feira, 11 de novembro de 2020 02:48 *Para:* Wiki Loves Monuments Photograph Competition < wikilovesmonuments@lists.wikimedia.org> *Assunto:* Re: [Wiki Loves Monuments] Is Montage tool down?
I believe we're back up?
On Tue, Nov 10, 2020 at 11:50 AM Stephen LaPorte < stephen.laporte@gmail.com> wrote:
Hi all,
Toolforge is temporarily in read-only mode for planned maintenance. It may take an hour or longer before Montage is back running.
I can send an email when we're back up, and please see below for additional info.
Thanks, Stephen
---------- Forwarded message --------- From: *Brooke Storm* bstorm@wikimedia.org Date: Tue, Nov 10, 2020 at 7:52 AM Subject: Re: [Cloud] [Cloud-announce] 2020-11-10 ToolsDB (User databases in Toolforge) read-only downtime To: Cloud-announce@lists.wikimedia.org
This will be happening in around 10 minutes. ToolsDB will be read-only until we can get a consistent dump to rebuild replication.
Brooke Storm Staff SRE Wikimedia Cloud Services bstorm@wikimedia.org IRC: bstorm
On Nov 6, 2020, at 12:12 PM, Brooke Storm bstorm@wikimedia.org wrote:
The ToolsDB service suffered a breakage in replication on 2020-10-27. WMCS has tried to restore replication of data, but that has been unsuccessful so far including doing a dump to rebuild replication without downtime. At this point, we have a new server waiting to become the replica, but to start the replication process, we need to set the database to read-only for a full dump. This could easily take more than an hour. During that entire time, the database will be read-only.
We will begin at 1600 UTC and finish when it is done. The database is quite large, but, with it in read-only mode, I hope the backup will not take terribly long.
Please see https://phabricator.wikimedia.org/T266587 for additional information.
Brooke Storm Staff SRE Wikimedia Cloud Services bstorm@wikimedia.org IRC: bstorm
Wikimedia Cloud Services announce mailing list Cloud-announce@lists.wikimedia.org (formerly labs-announce@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce
On Tue, Nov 10, 2020 at 11:21 AM Ciell Wikipedia < ciell.wikipedia@gmail.com> wrote:
Hi,
Thank you for reporting this: you are not the only one with this problem. I've alerted the developers of the tool already.
Vriendelijke groet, Ciell
Op di 10 nov. 2020 om 20:17 schreef Éder Porto eder.porto@hotmail.com:
Hello to all,
Today, a few hours ago, the montage tool just stopped working, I can't even login. Is this happening with someone else? Shoul a ticket on Phabricator be open? A issue on gitHub? My error message is as folows:
<ExceptionInfo [sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (1290, 'The MariaDB server is running with the --read-only option so it cannot execute this statement') [SQL: u'UPDATE users SET last_active_date=%s WHERE users.id = %s'] [parameters: (datetime.datetime(2020, 11, 10, 19, 13, 33, 773317), 47501133L)]] (40 frames, last=Callpoint('defaulterrorhandler', 36, 'MySQLdb.connections', '/data/project/montage/www/python/venv/local/lib/python2.7/site-packages/MySQLdb/connections.py', 65, ' raise errorclass, errorvalue'))>
Thanks in advance for any comments, Éder Porto | User:Ederporto _______________________________________________ Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
Hi everyone,
Montage is still down, and unfortunately we do not have a new estimate for when it will be back. We rely on a Toolforge databases for Montage, and Toolforge maintenance is taking longer than expected. The staff working on Toolforge still hope that it will be complete today, but it's hard to say how much longer it will take.
We will let you know as soon as it's back up. You can also follow along on Phabricator (https://phabricator.wikimedia.org/T266587), or add the Montage Status page (https://commons.wikimedia.org/wiki/Commons:Montage/Status) to your watchlist. Thanks for your patience, and I'm sorry this is interrupting the jury process!
Best, Stephen
On Wed, Nov 11, 2020 at 4:43 AM Ciell Wikipedia ciell.wikipedia@gmail.com wrote:
It's not just Montage, it is with (all?) tools that use Toolforge. According to the Montage talkpage this may take all through Wednesday to restore.
https://commons.wikimedia.org/wiki/Commons_talk:Montage#Montage_downtime_202...
Vriendelijke groet, Ciell
Op wo 11 nov. 2020 om 07:22 schreef Yaroslav Blanter ymbalt@gmail.com:
I also can not get it started.
Yaroslav
On Wed, Nov 11, 2020 at 7:08 AM Éder Porto eder.porto@hotmail.com wrote:
Nope, still down for me, and no update on the Phabricator ticket
*De:* WikiLovesMonuments wikilovesmonuments-bounces@lists.wikimedia.org em nome de effe iets anders effeietsanders@gmail.com *Enviado:* quarta-feira, 11 de novembro de 2020 02:48 *Para:* Wiki Loves Monuments Photograph Competition < wikilovesmonuments@lists.wikimedia.org> *Assunto:* Re: [Wiki Loves Monuments] Is Montage tool down?
I believe we're back up?
On Tue, Nov 10, 2020 at 11:50 AM Stephen LaPorte < stephen.laporte@gmail.com> wrote:
Hi all,
Toolforge is temporarily in read-only mode for planned maintenance. It may take an hour or longer before Montage is back running.
I can send an email when we're back up, and please see below for additional info.
Thanks, Stephen
---------- Forwarded message --------- From: *Brooke Storm* bstorm@wikimedia.org Date: Tue, Nov 10, 2020 at 7:52 AM Subject: Re: [Cloud] [Cloud-announce] 2020-11-10 ToolsDB (User databases in Toolforge) read-only downtime To: Cloud-announce@lists.wikimedia.org
This will be happening in around 10 minutes. ToolsDB will be read-only until we can get a consistent dump to rebuild replication.
Brooke Storm Staff SRE Wikimedia Cloud Services bstorm@wikimedia.org IRC: bstorm
On Nov 6, 2020, at 12:12 PM, Brooke Storm bstorm@wikimedia.org wrote:
The ToolsDB service suffered a breakage in replication on 2020-10-27. WMCS has tried to restore replication of data, but that has been unsuccessful so far including doing a dump to rebuild replication without downtime. At this point, we have a new server waiting to become the replica, but to start the replication process, we need to set the database to read-only for a full dump. This could easily take more than an hour. During that entire time, the database will be read-only.
We will begin at 1600 UTC and finish when it is done. The database is quite large, but, with it in read-only mode, I hope the backup will not take terribly long.
Please see https://phabricator.wikimedia.org/T266587 for additional information.
Brooke Storm Staff SRE Wikimedia Cloud Services bstorm@wikimedia.org IRC: bstorm
Wikimedia Cloud Services announce mailing list Cloud-announce@lists.wikimedia.org (formerly labs-announce@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce
On Tue, Nov 10, 2020 at 11:21 AM Ciell Wikipedia < ciell.wikipedia@gmail.com> wrote:
Hi,
Thank you for reporting this: you are not the only one with this problem. I've alerted the developers of the tool already.
Vriendelijke groet, Ciell
Op di 10 nov. 2020 om 20:17 schreef Éder Porto eder.porto@hotmail.com:
Hello to all,
Today, a few hours ago, the montage tool just stopped working, I can't even login. Is this happening with someone else? Shoul a ticket on Phabricator be open? A issue on gitHub? My error message is as folows:
<ExceptionInfo [sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (1290, 'The MariaDB server is running with the --read-only option so it cannot execute this statement') [SQL: u'UPDATE users SET last_active_date=%s WHERE users.id = %s'] [parameters: (datetime.datetime(2020, 11, 10, 19, 13, 33, 773317), 47501133L)]] (40 frames, last=Callpoint('defaulterrorhandler', 36, 'MySQLdb.connections', '/data/project/montage/www/python/venv/local/lib/python2.7/site-packages/MySQLdb/connections.py', 65, ' raise errorclass, errorvalue'))>
Thanks in advance for any comments, Éder Porto | User:Ederporto _______________________________________________ Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
Good news: Montage is back up and running.
They are still working on the Toolforge databases, but I hope it won't affect Montage at this point.
On Wed, Nov 11, 2020 at 7:15 PM Stephen LaPorte stephen.laporte@gmail.com wrote:
Hi everyone,
Montage is still down, and unfortunately we do not have a new estimate for when it will be back. We rely on a Toolforge databases for Montage, and Toolforge maintenance is taking longer than expected. The staff working on Toolforge still hope that it will be complete today, but it's hard to say how much longer it will take.
We will let you know as soon as it's back up. You can also follow along on Phabricator (https://phabricator.wikimedia.org/T266587), or add the Montage Status page ( https://commons.wikimedia.org/wiki/Commons:Montage/Status) to your watchlist. Thanks for your patience, and I'm sorry this is interrupting the jury process!
Best, Stephen
On Wed, Nov 11, 2020 at 4:43 AM Ciell Wikipedia ciell.wikipedia@gmail.com wrote:
It's not just Montage, it is with (all?) tools that use Toolforge. According to the Montage talkpage this may take all through Wednesday to restore.
https://commons.wikimedia.org/wiki/Commons_talk:Montage#Montage_downtime_202...
Vriendelijke groet, Ciell
Op wo 11 nov. 2020 om 07:22 schreef Yaroslav Blanter ymbalt@gmail.com:
I also can not get it started.
Yaroslav
On Wed, Nov 11, 2020 at 7:08 AM Éder Porto eder.porto@hotmail.com wrote:
Nope, still down for me, and no update on the Phabricator ticket
*De:* WikiLovesMonuments < wikilovesmonuments-bounces@lists.wikimedia.org> em nome de effe iets anders effeietsanders@gmail.com *Enviado:* quarta-feira, 11 de novembro de 2020 02:48 *Para:* Wiki Loves Monuments Photograph Competition < wikilovesmonuments@lists.wikimedia.org> *Assunto:* Re: [Wiki Loves Monuments] Is Montage tool down?
I believe we're back up?
On Tue, Nov 10, 2020 at 11:50 AM Stephen LaPorte < stephen.laporte@gmail.com> wrote:
Hi all,
Toolforge is temporarily in read-only mode for planned maintenance. It may take an hour or longer before Montage is back running.
I can send an email when we're back up, and please see below for additional info.
Thanks, Stephen
---------- Forwarded message --------- From: *Brooke Storm* bstorm@wikimedia.org Date: Tue, Nov 10, 2020 at 7:52 AM Subject: Re: [Cloud] [Cloud-announce] 2020-11-10 ToolsDB (User databases in Toolforge) read-only downtime To: Cloud-announce@lists.wikimedia.org
This will be happening in around 10 minutes. ToolsDB will be read-only until we can get a consistent dump to rebuild replication.
Brooke Storm Staff SRE Wikimedia Cloud Services bstorm@wikimedia.org IRC: bstorm
On Nov 6, 2020, at 12:12 PM, Brooke Storm bstorm@wikimedia.org wrote:
The ToolsDB service suffered a breakage in replication on 2020-10-27. WMCS has tried to restore replication of data, but that has been unsuccessful so far including doing a dump to rebuild replication without downtime. At this point, we have a new server waiting to become the replica, but to start the replication process, we need to set the database to read-only for a full dump. This could easily take more than an hour. During that entire time, the database will be read-only.
We will begin at 1600 UTC and finish when it is done. The database is quite large, but, with it in read-only mode, I hope the backup will not take terribly long.
Please see https://phabricator.wikimedia.org/T266587 for additional information.
Brooke Storm Staff SRE Wikimedia Cloud Services bstorm@wikimedia.org IRC: bstorm
Wikimedia Cloud Services announce mailing list Cloud-announce@lists.wikimedia.org (formerly labs-announce@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce
On Tue, Nov 10, 2020 at 11:21 AM Ciell Wikipedia < ciell.wikipedia@gmail.com> wrote:
Hi,
Thank you for reporting this: you are not the only one with this problem. I've alerted the developers of the tool already.
Vriendelijke groet, Ciell
Op di 10 nov. 2020 om 20:17 schreef Éder Porto <eder.porto@hotmail.com
:
Hello to all,
Today, a few hours ago, the montage tool just stopped working, I can't even login. Is this happening with someone else? Shoul a ticket on Phabricator be open? A issue on gitHub? My error message is as folows:
<ExceptionInfo [sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (1290, 'The MariaDB server is running with the --read-only option so it cannot execute this statement') [SQL: u'UPDATE users SET last_active_date=%s WHERE users.id = %s'] [parameters: (datetime.datetime(2020, 11, 10, 19, 13, 33, 773317), 47501133L)]] (40 frames, last=Callpoint('defaulterrorhandler', 36, 'MySQLdb.connections', '/data/project/montage/www/python/venv/local/lib/python2.7/site-packages/MySQLdb/connections.py', 65, ' raise errorclass, errorvalue'))>
Thanks in advance for any comments, Éder Porto | User:Ederporto _______________________________________________ Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
Wiki Loves Monuments mailing list WikiLovesMonuments@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikilovesmonuments http://www.wikilovesmonuments.org
wikilovesmonuments@lists.wikimedia.org
