Hi all.
I remember something about this posted aprox a year ago, but now we have new software versions so... There's something strange when I try to import dumps.
I've downloaded latest version of mwdumper.jar. I get a full .7z dump (for research purposes) of, let's say, svwiki.
Well, I create a new database, build the 29 tables with the tables.sql script (but remove the InnoDB type of tables, due to some compatibility problems in our MySQL, I think it doesn't mind).
Ok, I launch mwdumper.jar with the appropiate parameters, things go on and finally obtain:
373,142 pages (228.901/sec), 2,659,777 revs (1,631.616/sec)
As it should be. But when I log on in svwiki database in MySQL it says:
mysql> select count(page_id) from page; +----------------+ | count(page_id) | +----------------+ | 30000 | +----------------+ 1 row in set (0.00 sec)
mysql> select count(rev_id) from revision; +---------------+ | count(rev_id) | +---------------+ | 722180 | +---------------+ 1 row in set (0.00 sec)
Far lower results.
Shouldn't I get the same figures? Anyone could help?
When I fix this problem, I've built a functional script to recover per user statistics of contributions in several periods of time (months, weeks, etc...).
Thanks everybody.
Felipe.
______________________________________________ LLama Gratis a cualquier PC del Mundo. Llamadas a fijos y móviles desde 1 céntimo por minuto. http://es.voice.yahoo.com
I apologise for replying myself...
I've done exactly the same with a tiny dump (amwiki) and it worked perfectly (100% match between mwdumper and mysql results).
Is there any problem with bigger dumps?
--- Felipe Ortega glimmer_phoenix@yahoo.es escribió:
Hi all.
I remember something about this posted aprox a year ago, but now we have new software versions so... There's something strange when I try to import dumps.
I've downloaded latest version of mwdumper.jar. I get a full .7z dump (for research purposes) of, let's say, svwiki.
Well, I create a new database, build the 29 tables with the tables.sql script (but remove the InnoDB type of tables, due to some compatibility problems in our MySQL, I think it doesn't mind).
Ok, I launch mwdumper.jar with the appropiate parameters, things go on and finally obtain:
373,142 pages (228.901/sec), 2,659,777 revs (1,631.616/sec)
As it should be. But when I log on in svwiki database in MySQL it says:
mysql> select count(page_id) from page; +----------------+ | count(page_id) | +----------------+ | 30000 | +----------------+ 1 row in set (0.00 sec)
mysql> select count(rev_id) from revision; +---------------+ | count(rev_id) | +---------------+ | 722180 | +---------------+ 1 row in set (0.00 sec)
Far lower results.
Shouldn't I get the same figures? Anyone could help?
When I fix this problem, I've built a functional script to recover per user statistics of contributions in several periods of time (months, weeks, etc...).
Thanks everybody.
Felipe.
______________________________________________ LLama Gratis a cualquier PC del Mundo. Llamadas a fijos y móviles desde 1 céntimo por minuto. http://es.voice.yahoo.com _______________________________________________ Wikitech-l mailing list Wikitech-l@wikimedia.org
http://mail.wikipedia.org/mailman/listinfo/wikitech-l
______________________________________________ LLama Gratis a cualquier PC del Mundo. Llamadas a fijos y móviles desde 1 céntimo por minuto. http://es.voice.yahoo.com
Felipe Ortega wrote: [snip]
As it should be. But when I log on in svwiki database in MySQL it says:
mysql> select count(page_id) from page; +----------------+ | count(page_id) | +----------------+ | 30000 |
MySQL encountered an error during the import. Please make a note of error output from MySQL in order to debug it.
-- brion vibber (brion @ pobox.com)
Thanks, Brion. You're right, there's an error in the import process.
I've repeated it with the latest dump of eswiki, because it fails at a previous point (around page 3000). I think it's the same problem:
process output: [...] 3,647 pages (5.754/sec), 302,000 revs (476.442/sec) ERROR 1114 at line 7969: The table 'text' is full 3,653 pages (5.711/sec), 303,000 revs (473.708/sec) 3,666 pages (5.711/sec), 304,000 revs (473.587/sec) 3,682 pages (5.726/sec), 305,000 revs (474.344/sec) 3,691 pages (5.732/sec), 306,000 revs (475.239/sec) 3,710 pages (5.754/sec), 307,000 revs (476.113/sec) 3,721 pages (5.762/sec), 308,000 revs (476.971/sec) [...]
We use MySQL 4.0.24_Debian-10sarge2-log.
Felipe.
--- Brion Vibber brion@pobox.com escribió:
Felipe Ortega wrote: [snip]
As it should be. But when I log on in svwiki
database
in MySQL it says:
mysql> select count(page_id) from page; +----------------+ | count(page_id) | +----------------+ | 30000 |
MySQL encountered an error during the import. Please make a note of error output from MySQL in order to debug it.
-- brion vibber (brion @ pobox.com) _______________________________________________ Wikitech-l mailing list Wikitech-l@wikimedia.org
http://mail.wikipedia.org/mailman/listinfo/wikitech-l
______________________________________________ LLama Gratis a cualquier PC del Mundo. Llamadas a fijos y móviles desde 1 céntimo por minuto. http://es.voice.yahoo.com
Felipe Ortega wrote:
Thanks, Brion. You're right, there's an error in the import process.
I've repeated it with the latest dump of eswiki, because it fails at a previous point (around page 3000). I think it's the same problem:
process output: [...] 3,647 pages (5.754/sec), 302,000 revs (476.442/sec) ERROR 1114 at line 7969: The table 'text' is full
Couple of thigns to check: 1) Is the file butting against a file size limit? If you're using MyISAM tables sometimes there's a 2-gig or 4-gig or something limit on file sizes.
This may be adjustable in the MySQL configuration; check dev.mysql.com for details (I don't know the var name offhand).
If using InnoDB tables, check if the transaction log is backed up or something. (Probably not very likely on a server that's not being used for other things.)
-- brion vibber (brion @ pobox.com)
Thanks again, Brion. You hit the nail.
I'll explain it briefly (perhaps would be a good idea to update mwdumper wiki page, or the data-dump page with this hint, to help other people):
When the error raises, table text status is this:
mysql> show table status like 'text' \G *************************** 1. row *************************** Name: text Type: MyISAM Row_format: Dynamic Rows: 727224 Avg_row_length: 5905 Data_length: 4294928748 Max_data_length: 4294967295 Index_length: 6968320 Data_free: 0 Auto_increment: 2777159 Create_time: 2006-10-21 19:16:28 Update_time: 2006-10-21 19:37:22 Check_time: NULL Create_options:
dev.mysql.com says this about the 'full table error':
"You are using a MyISAM table and the space required for the table exceeds what is allowed by the internal pointer size. If you don't specify the MAX_ROWS table option when you create a table, MySQL uses the myisam_data_pointer_size system variable. From MySQL 5.0.6 on, the default value is 6 bytes, which is enough to allow 256TB of data. Before MySQL 5.0.6, the default value is 4 bytes, which is enough to allow only 4GB of data."
So it is in our case. Moreover, we find this about that variable:
"myisam_data_pointer_size
The default pointer size in bytes, to be used by CREATE TABLE for MyISAM tables when no MAX_ROWS option is specified. This variable cannot be less than 2 or larger than 7. The default value is 6 (4 before MySQL 5.0.6). This variable was added in MySQL 4.1.2"
We are using MySQL 4.02, so no magic is available.
Here you can learn how to fix it:
http://jeremy.zawodny.com/blog/archives/000796.html
Using an ALTER TABLE:
alter table text max_rows = 200000000000 avg_row_length = 50;
Felipe.
--- Brion Vibber brion@pobox.com escribió:
Felipe Ortega wrote:
Thanks, Brion. You're right, there's an error in
the
import process.
I've repeated it with the latest dump of eswiki, because it fails at a previous point (around page 3000). I think it's the same problem:
process output: [...] 3,647 pages (5.754/sec), 302,000 revs
(476.442/sec)
ERROR 1114 at line 7969: The table 'text' is full
Couple of thigns to check:
- Is the file butting against a file size limit? If
you're using MyISAM tables sometimes there's a 2-gig or 4-gig or something limit on file sizes.
This may be adjustable in the MySQL configuration; check dev.mysql.com for details (I don't know the var name offhand).
If using InnoDB tables, check if the transaction log is backed up or something. (Probably not very likely on a server that's not being used for other things.)
-- brion vibber (brion @ pobox.com) _______________________________________________ Wikitech-l mailing list Wikitech-l@wikimedia.org
http://mail.wikipedia.org/mailman/listinfo/wikitech-l
______________________________________________ LLama Gratis a cualquier PC del Mundo. Llamadas a fijos y móviles desde 1 céntimo por minuto. http://es.voice.yahoo.com
wikitech-l@lists.wikimedia.org