Database error
From Wikipedia, the free encyclopedia
Jump to: navigation, search A database query syntax error has occurred. This may indicate a bug in the software. The last attempted database query was:
(SQL query hidden)
from within function "ExternalStoreDB::store". MySQL returned error "1114: The table 'blobs' is full (10.0.2.161)".
Magnus Manske wrote:
Database error From Wikipedia, the free encyclopedia Jump to: navigation, search A database query syntax error has occurred. This may indicate a bug in the software. The last attempted database query was:
(SQL query hidden)from within function "ExternalStoreDB::store". MySQL returned error "1114: The table 'blobs' is full (10.0.2.161)".
Please report urgent system administration issues to IRC, specifically #wikimedia-tech on irc.freenode.net.
-- Tim Starling
Out of curiousity, when a technical problem shuts down all editing on a major wiki (as this did) are there any automated alerts? Is it likely to be noticed and addressed even if no one rushes to IRC?
I guess I am curious what is the normal delay between problem onset and problem recognition?
-Robert Rohde
Robert Rohde wrote:
Out of curiousity, when a technical problem shuts down all editing on a major wiki (as this did) are there any automated alerts? Is it likely to be noticed and addressed even if no one rushes to IRC?
I guess I am curious what is the normal delay between problem onset and problem recognition?
-Robert Rohde
The procedure is "a lot of people enter #wikimedia-tech complaining about it". There are automated alerts about servers going down or not having enough free space on disk, but not for 'saving an edit failed'. That would be tricky to do.
On 3/10/09 4:39 PM, Platonides wrote:
The procedure is "a lot of people enter #wikimedia-tech complaining about it". There are automated alerts about servers going down or not having enough free space on disk, but not for 'saving an edit failed'. That would be tricky to do.
Wouldn't be that tricky, but it's not our highest priority as the human alert system does an excellent job of this already. ;)
-- brion
On Wed, Mar 11, 2009 at 9:21 AM, Robert Rohde rarohde@gmail.com wrote:
Out of curiousity, when a technical problem shuts down all editing on a major wiki (as this did) are there any automated alerts? Is it likely to be noticed and addressed even if no one rushes to IRC?
I guess I am curious what is the normal delay between problem onset and problem recognition?
-Robert Rohde
I believe with this issue (Full MySQL table) that there is no easy way to automate the test..... maybe you could automatically query it every so often but even then that might not return reliable results.
2009/3/10 K. Peachey p858snake@yahoo.com.au:
On Wed, Mar 11, 2009 at 9:21 AM, Robert Rohde rarohde@gmail.com wrote:
Out of curiousity, when a technical problem shuts down all editing on a major wiki (as this did) are there any automated alerts? Is it likely to be noticed and addressed even if no one rushes to IRC?
I guess I am curious what is the normal delay between problem onset and problem recognition?
-Robert Rohde
I believe with this issue (Full MySQL table) that there is no easy way to automate the test..... maybe you could automatically query it every so often but even then that might not return reliable results.
A bot that edits the sandbox every few minutes would work, would it?
2009/3/10 K. Peachey p858snake@yahoo.com.au:
A bot that edits the sandbox every few minutes would work, would it?
Possibly, but i would bump it up to like every two hours. Plus since the MySQL is spread between multipul systems you would have make sure it checks the same one all the time.
If you're going to wait 2 hours, you might as well just wait for people to start complaining, that will be far quicker.
In case you didn't see the whole StatusBot fiasco on enwiki, I used to run a bot as a replacement to a replacement of [[User:StatusBot]]. The bot made 50k edis in a few months, and was soon shut down by Brion. A bot the edits the sandbox every few minutes would no way be approved.
On Mar 10, 2009, at 7:54 PM [Mar 10, 2009 ], Thomas Dalton wrote:
2009/3/10 K. Peachey p858snake@yahoo.com.au:
On Wed, Mar 11, 2009 at 9:21 AM, Robert Rohde rarohde@gmail.com wrote:
Out of curiousity, when a technical problem shuts down all editing on a major wiki (as this did) are there any automated alerts? Is it likely to be noticed and addressed even if no one rushes to IRC?
I guess I am curious what is the normal delay between problem onset and problem recognition?
-Robert Rohde
I believe with this issue (Full MySQL table) that there is no easy way to automate the test..... maybe you could automatically query it every so often but even then that might not return reliable results.
A bot that edits the sandbox every few minutes would work, would it?
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Yes, but whilst StatusBot and the proposed bot would have comparable edit statistics, the latter would have more of a reason for running than 'to update people's statuses'. It's not just about actions, it's about the justification for those actions.
- Chris
On Wed, Mar 11, 2009 at 2:11 AM, Soxred93 soxred93@gmail.com wrote:
In case you didn't see the whole StatusBot fiasco on enwiki, I used to run a bot as a replacement to a replacement of [[User:StatusBot]]. The bot made 50k edis in a few months, and was soon shut down by Brion. A bot the edits the sandbox every few minutes would no way be approved.
On Mar 10, 2009, at 7:54 PM [Mar 10, 2009 ], Thomas Dalton wrote:
2009/3/10 K. Peachey p858snake@yahoo.com.au:
On Wed, Mar 11, 2009 at 9:21 AM, Robert Rohde rarohde@gmail.com wrote:
Out of curiousity, when a technical problem shuts down all editing on a major wiki (as this did) are there any automated alerts? Is it likely to be noticed and addressed even if no one rushes to IRC?
I guess I am curious what is the normal delay between problem onset and problem recognition?
-Robert Rohde
I believe with this issue (Full MySQL table) that there is no easy way to automate the test..... maybe you could automatically query it every so often but even then that might not return reliable results.
A bot that edits the sandbox every few minutes would work, would it?
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Tue, Mar 10, 2009 at 4:43 PM, K. Peachey p858snake@yahoo.com.au wrote:
On Wed, Mar 11, 2009 at 9:21 AM, Robert Rohde rarohde@gmail.com wrote:
Out of curiousity, when a technical problem shuts down all editing on a major wiki (as this did) are there any automated alerts? Is it likely to be noticed and addressed even if no one rushes to IRC?
I guess I am curious what is the normal delay between problem onset and problem recognition?
-Robert Rohde
I believe with this issue (Full MySQL table) that there is no easy way to automate the test..... maybe you could automatically query it every so often but even then that might not return reliable results.
One could query count(*) from revisions (or some similar artifice, such as looking at the recent changes feed) and trigger an alert if it stops increasing.
Such things are probably totally unnecessary on enwiki, because there is no shortage of people to complain, but I could image it might be useful to have such an alert for smaller, non-English speaking wikis.
-Robert Rohde
What about replag? The bot would puke every time that replication stops.
On Mar 10, 2009, at 8:02 PM [Mar 10, 2009 ], Robert Rohde wrote:
On Tue, Mar 10, 2009 at 4:43 PM, K. Peachey p858snake@yahoo.com.au wrote:
On Wed, Mar 11, 2009 at 9:21 AM, Robert Rohde rarohde@gmail.com wrote:
Out of curiousity, when a technical problem shuts down all editing on a major wiki (as this did) are there any automated alerts? Is it likely to be noticed and addressed even if no one rushes to IRC?
I guess I am curious what is the normal delay between problem onset and problem recognition?
-Robert Rohde
I believe with this issue (Full MySQL table) that there is no easy way to automate the test..... maybe you could automatically query it every so often but even then that might not return reliable results.
One could query count(*) from revisions (or some similar artifice, such as looking at the recent changes feed) and trigger an alert if it stops increasing.
Such things are probably totally unnecessary on enwiki, because there is no shortage of people to complain, but I could image it might be useful to have such an alert for smaller, non-English speaking wikis.
-Robert Rohde
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Tue, Mar 10, 2009 at 7:15 PM, K. Peachey p858snake@yahoo.com.au wrote:
Not everyone knows how to use IRC, i would recommend that you recommend people report to bugzilla with urgent tags instead.
Sysadmins do not check Bugzilla constantly. They do check IRC constantly if you say their names in the right channel. Therefore, IRC is the preferred method of reporting urgent problems, and will remain so for the foreseeable future. If some people don't know how to use it, they can either find someone who does; learn quickly; or be content with their alert taking a while longer to reach the right people.
K. Peachey wrote:
Please report urgent system administration issues to IRC, specifically #wikimedia-tech on irc.freenode.net.
-- Tim Starling
Not everyone knows how to use IRC, i would recommend that you recommend people report to bugzilla with urgent tags instead.
That would just be a nuisance. By the time we check Bugzilla for new issues, urgent problems are already fixed due to IRC reports or automated alerts. We would have to waste time finding and closing all the relevant bugs. I would suggest either using IRC or waiting patiently.
-- Tim Starling
On Wed, Mar 11, 2009 at 5:59 PM, Tim Starling tstarling@wikimedia.org wrote:
K. Peachey wrote:
Please report urgent system administration issues to IRC, specifically #wikimedia-tech on irc.freenode.net.
-- Tim Starling
Not everyone knows how to use IRC, i would recommend that you recommend people report to bugzilla with urgent tags instead.
That would just be a nuisance. By the time we check Bugzilla for new issues, urgent problems are already fixed due to IRC reports or automated alerts. We would have to waste time finding and closing all the relevant bugs. I would suggest either using IRC or waiting patiently.
-- Tim Starling
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Not to mention, depending on the error it could be entirely possible that Bugzilla is down as well.
-Chad
Hoi, If this is true, you just provided the best argument why the setup of Bugzilla should be changed. You do not want to have the tools that report on the availability of your production servers to be dependent on those same production servers. Thanks, GerardM
2009/3/11 Chad innocentkiller@gmail.com
On Wed, Mar 11, 2009 at 5:59 PM, Tim Starling tstarling@wikimedia.org wrote:
K. Peachey wrote:
Please report urgent system administration issues to IRC, specifically #wikimedia-tech on irc.freenode.net.
-- Tim Starling
Not everyone knows how to use IRC, i would recommend that you recommend people report to bugzilla with urgent tags instead.
That would just be a nuisance. By the time we check Bugzilla for new issues, urgent problems are already fixed due to IRC reports or automated alerts. We would have to waste time finding and closing all the relevant bugs. I would suggest either using IRC or waiting patiently.
-- Tim Starling
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Not to mention, depending on the error it could be entirely possible that Bugzilla is down as well.
-Chad
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Wed, Mar 11, 2009 at 6:16 PM, Gerard Meijssen gerard.meijssen@gmail.com wrote:
Hoi, If this is true, you just provided the best argument why the setup of Bugzilla should be changed. You do not want to have the tools that report on the availability of your production servers to be dependent on those same production servers. Thanks, GerardM
2009/3/11 Chad innocentkiller@gmail.com
On Wed, Mar 11, 2009 at 5:59 PM, Tim Starling tstarling@wikimedia.org wrote:
K. Peachey wrote:
Please report urgent system administration issues to IRC, specifically #wikimedia-tech on irc.freenode.net.
-- Tim Starling
Not everyone knows how to use IRC, i would recommend that you recommend people report to bugzilla with urgent tags instead.
That would just be a nuisance. By the time we check Bugzilla for new issues, urgent problems are already fixed due to IRC reports or automated alerts. We would have to waste time finding and closing all the relevant bugs. I would suggest either using IRC or waiting patiently.
-- Tim Starling
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Not to mention, depending on the error it could be entirely possible that Bugzilla is down as well.
-Chad
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Which is one of several reasons we _don't_ handle urgent sysadmin stuff via Bugzilla. What's your point?
-Chad
Hoi, People _do_ post problems using Bugzilla.. What you are saying is that the _intention_ is not to use it for urgent sysadmin stuff. New bugzilla bugs _are_ posted on IRC in order to get people's attention. This real time attention negates your assertion that it is not treated as urgent. <grin> my point is that it is a de facto important tool for sysadmin work even when this is not intentional </grin> Thanks, GerardM
2009/3/11 Chad innocentkiller@gmail.com
On Wed, Mar 11, 2009 at 6:16 PM, Gerard Meijssen gerard.meijssen@gmail.com wrote:
Hoi, If this is true, you just provided the best argument why the setup of Bugzilla should be changed. You do not want to have the tools that report
on
the availability of your production servers to be dependent on those same production servers. Thanks, GerardM
2009/3/11 Chad innocentkiller@gmail.com
On Wed, Mar 11, 2009 at 5:59 PM, Tim Starling tstarling@wikimedia.org wrote:
K. Peachey wrote:
Please report urgent system administration issues to IRC,
specifically
#wikimedia-tech on irc.freenode.net.
-- Tim Starling
Not everyone knows how to use IRC, i would recommend that you recommend people report to bugzilla with urgent tags instead.
That would just be a nuisance. By the time we check Bugzilla for new issues, urgent problems are already fixed due to IRC reports or automated alerts. We would have to waste time finding and closing all the relevant bugs. I would suggest either using IRC or waiting
patiently.
-- Tim Starling
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Not to mention, depending on the error it could be entirely possible that Bugzilla is down as well.
-Chad
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Which is one of several reasons we _don't_ handle urgent sysadmin stuff via Bugzilla. What's your point?
-Chad
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Wed, Mar 11, 2009 at 6:38 PM, Gerard Meijssen gerard.meijssen@gmail.com wrote:
Hoi, People _do_ post problems using Bugzilla.. What you are saying is that the _intention_ is not to use it for urgent sysadmin stuff. New bugzilla bugs _are_ posted on IRC in order to get people's attention. This real time attention negates your assertion that it is not treated as urgent. <grin> my point is that it is a de facto important tool for sysadmin work even when this is not intentional </grin> Thanks, GerardM
Just because someone posts a bug doesn't mean that we are relying on that as a mechanism of notification. By and large, the urgent-opened bugs are quickly closed because the sysadmins are already working on the problem.
You're confusing people filing bugs with us relying on those bugs. They aren't the same thing.
-Chad
Gerard Meijssen wrote:
Hoi, People _do_ post problems using Bugzilla.. What you are saying is that the _intention_ is not to use it for urgent sysadmin stuff. New bugzilla bugs _are_ posted on IRC in order to get people's attention. This real time attention negates your assertion that it is not treated as urgent. <grin> my point is that it is a de facto important tool for sysadmin work even when this is not intentional </grin> Thanks, GerardM
Bug reports show at #mediawiki, so if nobody noticed and someone posts at bugzilla that xy.wikipedia.org editing is broken it is *likely* that some of the regulars will notice, verify and pass to #wikimedia-tech
On outages there are 4-5 bugs about "wikipedia is broken". Which means work to close them. All of us at #wikimedia-tech already knew about it. If you send people to bugzilla at least tell them to do a search before. And people joining at IRC, look at the topic before complaining, too.
Gerard Meijssen wrote:
2009/3/11 Chad innocentkiller@gmail.com wrote:
Not to mention, depending on the error it could be entirely possible that Bugzilla is down as well.
If this is true, you just provided the best argument why the setup of Bugzilla should be changed. You do not want to have the tools that report on the availability of your production servers to be dependent on those same production servers.
Bugzilla is not such a tool, as I already said. There is no need for it to be up when the site generally is down.
-- Tim Starling
On 3/11/09 3:41 PM, Tim Starling wrote:
Gerard Meijssen wrote:
2009/3/11 Chadinnocentkiller@gmail.com wrote:
Not to mention, depending on the error it could be entirely possible that Bugzilla is down as well.
If this is true, you just provided the best argument why the setup of Bugzilla should be changed. You do not want to have the tools that report on the availability of your production servers to be dependent on those same production servers.
Bugzilla is not such a tool, as I already said. There is no need for it to be up when the site generally is down.
Note that in nearly all cases, Bugzilla is not affected in any way by wiki site outages.
-- brion
Hoi, Brion thank you. Gerard
2009/3/12 Brion Vibber brion@wikimedia.org
On 3/11/09 3:41 PM, Tim Starling wrote:
Gerard Meijssen wrote:
2009/3/11 Chadinnocentkiller@gmail.com wrote:
Not to mention, depending on the error it could be entirely possible that Bugzilla is down as well.
If this is true, you just provided the best argument why the setup of Bugzilla should be changed. You do not want to have the tools that
report on
the availability of your production servers to be dependent on those
same
production servers.
Bugzilla is not such a tool, as I already said. There is no need for it to be up when the site generally is down.
Note that in nearly all cases, Bugzilla is not affected in any way by wiki site outages.
-- brion
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Wed, Mar 11, 2009 at 5:59 PM, Tim Starling tstarling@wikimedia.orgwrote:
I would suggest either using IRC or waiting patiently.
Or wikitech-l, in which case someone else will surely forward your message to IRC :).
(Unfortunately, it'll also spawn 30 posts about whether or not it was appropriate to post to wikitech-l.)
Or wikitech-l, in which case someone else will surely forward your message to IRC :).
I think you people should follow the trend and twitter about it. someone will definitely follow and notice!
On Tue, Mar 10, 2009 at 9:54 AM, Tim Starling tstarling@wikimedia.orgwrote:
Please report urgent system administration issues to IRC, specifically #wikimedia-tech on irc.freenode.net.
-- Tim Starling
Certainly IRC is the best way to inform developers and sys admins of a problem with Wikipedia.
As a user, I also like to know what's going on when there is a problem. But, I'm not regularly on IRC. If I'm at work where I sometimes look up things on Wikipedia (but don't edit), and see that Wikipedia is down, then I am curious what the problem is. However, IRC and other types of chat are forbidden at work. :( But, I am allowed to access gmail.
So, I very much appreciate it when people send messages to the mailing list about such problems.
-Aude
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
2009/3/11 Aude audevivere@gmail.com:
On Tue, Mar 10, 2009 at 9:54 AM, Tim Starling tstarling@wikimedia.orgwrote:
Please report urgent system administration issues to IRC, specifically #wikimedia-tech on irc.freenode.net.
-- Tim Starling
Certainly IRC is the best way to inform developers and sys admins of a problem with Wikipedia.
As a user, I also like to know what's going on when there is a problem. But, I'm not regularly on IRC. If I'm at work where I sometimes look up things on Wikipedia (but don't edit), and see that Wikipedia is down, then I am curious what the problem is. However, IRC and other types of chat are forbidden at work. :( But, I am allowed to access gmail.
So, I very much appreciate it when people send messages to the mailing list about such problems.
The admin log on http://wikitech.wikimedia.org/view/Server_admin_log sometimes has some decent clues as to what is wrong and what is being done to fix it.
Hello,
just for clarity, "table is full" was the edge case which consisted of very narrow index, slightly too low MAX_ROWS specification and lots of rows. In 4.1 and earlier versions MySQL was using 4-byte data pointers by default, and to bypass that one has to specify maximum rows and average row size. The specifications we had were good for data area, but bad for index area, which was a function of max_rows and index width.
As index was very very narrow, a 2-byte internal index area pointer was picked, which allowed just 64k 1024-byte blocks (thats right, total 64M for indexing per table), which restricted us to ~8 million rows per table. Of course, that was way above the one million we specified :)
This is quite unpopular situation (I've never seen anyone hit that before, except on our sites :) - and if we had any other additional index on that table or had wider field or more fields in it, the index pointer would be 4 or 6 bytes, thus allowing bazillion rows.
Now, the bad part was that we saw this before, and we knew what causes this, and still didn't change definition (or simply remove it, we run 5.0 for ES, and it doesn't need this specification, this is legacy thing).
Still, no need to believe that we ran out of diskspace.
BR,
wikitech-l@lists.wikimedia.org