Hello, everyone.
As Manuel mentioned some time ago, I'm from Wikimedia Israel, and we're interested in trying the OpenZIM tools for a static version of the Hebrew Wikipedia we are working on, for inclusion in the Israeli government's One Laptop Per Child (OLPC) project.
So far, I've been able to compile and run all the tools, but am having some trouble creating ZIM files: I've dumped a locally-loaded Mediawiki installation to HTML using these instructions http://openzim.org/Wiki2html, but when I try to run the builZimFileFromDirectory.pl script, I get silly postgresql errors about failing to connect using "kiwix" user. I should probably create a user, or a database, or both, but I've never used postgresql (I use mysql), and despite trying some combinations of commands to create a 'kiwix' user, with and without a password, couldn't get the DBI::connect call to succeed. I am sure you are all familiar with postgresql and can give me a quick pointer on how to proceed. I'm using a modern unstable Debian box, with a default packager-supplied installation of postgresql and the related Perl DBD module.
Thanks in advance,
Asaf Bartov -- Asaf Bartov asaf@forum2.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi Asaf,
Asaf Bartov a écrit :
So far, I've been able to compile and run all the tools, but am having some trouble creating ZIM files: I've dumped a locally-loaded Mediawiki installation to HTML using these instructions http://openzim.org/Wiki2html, but when I try to run the builZimFileFromDirectory.pl script, I get silly postgresql errors about failing to connect using "kiwix" user. I should probably create a user, or a database, or both, but I've never used postgresql (I use mysql), and despite trying some combinations of commands to create a 'kiwix' user, with and without a password, couldn't get the DBI::connect call to succeed.
You have right, this seems to be the problem... I'm not sure to remember how I did that, I'm also not a psql expert:
By me, /etc/postgresql/8.3/main/pg_hba.conf looks like
# Database administrative login by UNIX sockets local all all trust
# TYPE DATABASE USER CIDR-ADDRESS METHOD
# "local" is for Unix domain socket connections only local all all trust # IPv4 local connections: host all all 127.0.0.1/32 trust # IPv6 local connections: host all all ::1/128 trust
To create a user you need: $createuser kiwix
Avoid password.
Maybe you can only do that with the user "postgres".
If Tommi reads you email, he will certainly be able to answer more properly to your quest.
Regards Emmanuel
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Asaf Bartov a écrit :
So far, I've been able to compile and run all the tools, but am having some trouble creating ZIM files: I've dumped a locally-loaded Mediawiki installation to HTML using these instructions http://openzim.org/Wiki2html, but when I try to run the builZimFileFromDirectory.pl script, I get silly postgresql errors about failing to connect using "kiwix" user.
I have commited to Kiwix svn a new version of builZimFileFromDirectory.pl allowing to specify the dbUser and dbPassword on the command line.
Emmanuel
Hello, Emmanuel, and everyone.
I'm still trying to create a ZIM file of the Hebrew Wikipedia dump. I've made some progress -- the dump is complete, but the buildZimFileFromDirectory.pl script is still failing:
The command line I used was:
asaf@abartov-deb:~/dev/kiwix/dumping_tools/scripts$ *./buildZimFileFromDirectory.pl --htmlPath=/home/asaf/dev/hewiki_dump --welcomePage=./index.html*
The output is:
NOTICE: CREATE TABLE will create implicit sequence "article_aid_seq" for serial column "article.aid" NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "article_pkey" for table "article" NOTICE: CREATE TABLE will create implicit sequence "category_cid_seq" for serial column "category.cid" NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "category_pkey" for table "category" NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "categoryarticles_pkey" for table "categoryarticles" NOTICE: CREATE TABLE will create implicit sequence "zimfile_zid_seq" for serial column "zimfile.zid" NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "zimfile_pkey" for table "zimfile" NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "zimdata_pkey" for table "zimdata" NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "zimarticles_pkey" for table "zimarticles" NOTICE: CREATE TABLE will create implicit sequence "indexarticle_xid_seq" for serial column "indexarticle.xid" NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "indexarticle_pkey" for table "indexarticle" NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "words_pkey" for table "words" NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "trivialwords_pkey" for table "trivialwords" Use of uninitialized value in concatenation (.) or string at ../classes//Kiwix/ZimIndexer.pm line 805. Use of uninitialized value $welcomePage in concatenation (.) or string at ../classes//Kiwix/ZimIndexer.pm line 746. DBD::Pg::db do failed: ERROR: invalid input syntax for integer: "" at ../classes//Kiwix/ZimIndexer.pm line 385. ERROR: invalid input syntax for integer: ""
The last few lines in all.log are:
[2009-06-30 07:05:43,997] builZimFileFromDirectory.pl - Rewriting url in /home/asaf/dev/hewiki_dump/articles/�×/� /22/Portal~�×� _�_����¢�×__79ba.html [2009-06-30 07:05:44,010] builZimFileFromDirectory.pl - Adding to DB /home/asaf/dev/hewiki_dump/articles/�×/� /22/Portal~�×� _�_����¢�×__79ba.html [2009-06-30 07:05:44,243] builZimFileFromDirectory.pl - Rewriting url in /home/asaf/dev/hewiki_dump/articles/�/�¨/� /��¨� ���_���.html [2009-06-30 07:05:44,305] builZimFileFromDirectory.pl - Adding to DB /home/asaf/dev/hewiki_dump/articles/�/�¨/� /��¨� ���_���.html [2009-06-30 07:05:44,547] builZimFileFromDirectory.pl - Rewriting url in /home/asaf/dev/hewiki_dump/articles/a/c/h/�§���¥~Achdut.JPG_bd38.html [2009-06-30 07:05:44,567] builZimFileFromDirectory.pl - Adding to DB /home/asaf/dev/hewiki_dump/articles/a/c/h/�§���¥~Achdut.JPG_bd38.html [2009-06-30 07:05:44,797] builZimFileFromDirectory.pl - Rewriting url in /home/asaf/dev/hewiki_dump/articles/�/�/�/�×�� ��×~������¤26����.html [2009-06-30 07:05:44,812] builZimFileFromDirectory.pl - Adding to DB /home/asaf/dev/hewiki_dump/articles/�/�/�/�×�� ��×~������¤26����.html [2009-06-30 07:05:45,007] builZimFileFromDirectory.pl - Rewriting url in /home/asaf/dev/hewiki_dump/articles/t/r/i/�§���¥~Triumph_of_the_Will_-_Congress_Hall.jpg_2aad.html [2009-06-30 07:05:45,024] builZimFileFromDirectory.pl - Adding to DB /home/asaf/dev/hewiki_dump/articles/t/r/i/�§���¥~Triumph_of_the_Will_-_Congress_Hall.jpg_2aad.html [2009-06-30 07:05:45,249] builZimFileFromDirectory.pl - Adding to DB /home/asaf/dev/hewiki_dump/articles/�×/�/�×/�×��×��×_������×.html [2009-06-30 07:05:45,509] builZimFileFromDirectory.pl - Rewriting url in /home/asaf/dev/hewiki_dump/articles/�/�/�/������ [2009-06-30 07:05:45,578] builZimFileFromDirectory.pl - Adding to DB /home/asaf/dev/hewiki_dump/articles/�/�/�/������
Any idea what is wrong? Any more diagnostics I can provide?
Thanks in advance,
Asaf
On Mon, Apr 20, 2009 at 11:11 PM, Emmanuel Engelhart <emmanuel@engelhart.org
wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Asaf Bartov a écrit :
So far, I've been able to compile and run all the tools, but am having
some
trouble creating ZIM files: I've dumped a locally-loaded Mediawiki installation to HTML using these instructions <
http://openzim.org/Wiki2html%3E,
but when I try to run the builZimFileFromDirectory.pl script, I get silly postgresql errors about failing to connect using "kiwix" user.
I have commited to Kiwix svn a new version of builZimFileFromDirectory.pl allowing to specify the dbUser and dbPassword on the command line.
Emmanuel -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAkns1vEACgkQn3IpJRpNWtNliwCeKyaPwXWdCr5rclkuYvqgCa2m wdMAn2F+sgUByhQT5g4o6XoD7WJeIeIL =USFE -----END PGP SIGNATURE-----
On Montag 20 April 2009 20:38:08 Asaf Bartov wrote:
Hello, everyone.
As Manuel mentioned some time ago, I'm from Wikimedia Israel, and we're interested in trying the OpenZIM tools for a static version of the Hebrew Wikipedia we are working on, for inclusion in the Israeli government's One Laptop Per Child (OLPC) project.
So far, I've been able to compile and run all the tools, but am having some trouble creating ZIM files: I've dumped a locally-loaded Mediawiki installation to HTML using these instructions http://openzim.org/Wiki2html, but when I try to run the builZimFileFromDirectory.pl script, I get silly postgresql errors about failing to connect using "kiwix" user. I should probably create a user, or a database, or both, but I've never used postgresql (I use mysql), and despite trying some combinations of commands to create a 'kiwix' user, with and without a password, couldn't get the DBI::connect call to succeed. I am sure you are all familiar with postgresql and can give me a quick pointer on how to proceed. I'm using a modern unstable Debian box, with a default packager-supplied installation of postgresql and the related Perl DBD module.
Thanks in advance,
Asaf Bartov
Asaf Bartov asaf@forum2.org
Hi Asaf,
postgresql is very flexible, how to authenticate users. This makes it a little more difficult. But not really that difficult.
Postgresql has a configuration file pg_hba.conf (on debian found at /etc/postgresql/8.3/main/pg_hba.conf). There you can find a line: local all all ident sameuser
Replace this "ident sameuser" with "trust": local all all trust
After restarting postgresql (/etc/init.d/postgresql restart) you are able to connect with every db-user you want without using passwords. Run createuser -U postgresql kiwix
This connects to the database using user postgresql (which is the standard admin user) and creates a user kiwix. It will ask you, if that user should have the right to create new users. Kiwix does not need that right (but it won't hurt, so it makes actually no difference). Now you should be able to connect using user kiwix.
Tommi