Hello,
We are having issues launching a local copy of wikidata, when we use the 'importDump.php' tool, below the issues that we are facing. If somebody has an idea of how we could solve this, please let me know. We are also considering professional services to get fixes for this being released in case somebody is professionally consulting around wikibase.
Thanks, Miquel
Here the issues: if I try to load the full dump, the error I get is: root@4fc8cc9b76b3:/var/www/html/maintenance# php importDump.php --conf ../LocalSettings.php ../images/wikidatawiki-20191101-pages-articles-multistream.xml.bz2 Warning: XMLReader::read(): uploadsource://d0cd78c216b067ffdd60946c258db6a7:45: parser error : Extra content at the end of the document in /var/www/html/includes/import/WikiImporter.php on line 646 Warning: XMLReader::read(): </siteinfo> in /var/www/html/includes/import/WikiImporter.php on line 646 Warning: XMLReader::read(): ^ in /var/www/html/includes/import/WikiImporter.php on line 646 Done! You might want to run rebuildrecentchanges.php to regenerate RecentChanges,
If I try to load a partial dump, the warnings that I get (which I think those mean nothing is loading) are: root@4fc8cc9b76b3:/var/www/html/maintenance# php importDump.php --conf ../LocalSettings.php ../images/wikidatawiki-20191020-pages-meta-current1.xml-p1p235321.bz2 Revision 1033865598 using content model wikibase-item cannot be stored on "Q15" on this wiki, since that model is not supported on that page. Revision 1034542603 using content model wikibase-item cannot be stored on "Q17" on this wiki, since that model is not supported on that page. Revision 1032554298 using content model wikibase-item cannot be stored on "Q18" on this wiki, since that model is not supported on that page. Revision 1032534215 using content model wikibase-item cannot be stored on "Q20" on this wiki, since that model is not supported on that page. Revision 1026713626 using content model wikibase-item cannot be stored on "Q21" on this wiki, since that model is not supported on that page. Revision 1023703278 using content model wikibase-item cannot be stored on "Q22" on this wiki, since that model is not supported on that page. Revision 1032815802 using content model wikibase-item cannot be stored on "Q25" on this wiki, since that model is not supported on that page. Revision 1032910600 using content model wikibase-item cannot be stored on "Q26" on this wiki, since that model is not supported on that page.
Hi all
We resolved this on the Wikibase telegram chat.
For anyone finding this email thread, here is a rough log of the chat....
---------------
Miquel Farre I just wrote to the wikidata-tech mail list asking for the question regarding the load of dumps https://t.me/c/1478172663/1845 if somebody here is professionally consulting around wikidata/wikibase and would be able to help writing fixes to those problems, we could study how to sponsor it. Please let me know.
Addshore I just scrolled up and it looks like it is a namespace issue Wikidata has namespace 0 (the main namespace) being wikidaa items,. On a default install of wikibase the item namespace is 120, and 0 is a wikitext namespace Either you'll have to configure your wikibase to be the same as Wikidata, or run something through the dump to change the namespace. I'm not sure there is anything built into the import process for XML for converting namespaces, I'll have a quick look, could be a good feature thoigh
Miquel Farre regarding the namespace issue, can I just add this config: https://www.mediawiki.org/wiki/Wikibase/Installation/Advanced_configuration#... chaning $baseNs = 0 ?
Addshore Yes, so if you look in the example settings you can find the rather verbose example of how to do this https://github.com/wikimedia/mediawiki-extensions-Wikibase/blob/master/repo/... defining $baseNs in local settings will not work, but defining those other things after you load the default settings will work changing this on an already existing wikibase might have unexpected consequences, and is not advised, but doing this on an empty one should be fine :)
Miquel Farre Adam, this is working, I can load a dump 🙂
On Tue, 19 Nov 2019 at 09:39, Miquel Àngel Farré miquel.farre@gmail.com wrote:
Hello,
We are having issues launching a local copy of wikidata, when we use the 'importDump.php' tool, below the issues that we are facing. If somebody has an idea of how we could solve this, please let me know. We are also considering professional services to get fixes for this being released in case somebody is professionally consulting around wikibase.
Thanks, Miquel
Here the issues: if I try to load the full dump, the error I get is: root@4fc8cc9b76b3:/var/www/html/maintenance# php importDump.php --conf ../LocalSettings.php ../images/wikidatawiki-20191101-pages-articles-multistream.xml.bz2 Warning: XMLReader::read(): uploadsource://d0cd78c216b067ffdd60946c258db6a7:45: parser error : Extra content at the end of the document in /var/www/html/includes/import/WikiImporter.php on line 646 Warning: XMLReader::read(): </siteinfo> in /var/www/html/includes/import/WikiImporter.php on line 646 Warning: XMLReader::read(): ^ in /var/www/html/includes/import/WikiImporter.php on line 646 Done! You might want to run rebuildrecentchanges.php to regenerate RecentChanges,
If I try to load a partial dump, the warnings that I get (which I think those mean nothing is loading) are: root@4fc8cc9b76b3:/var/www/html/maintenance# php importDump.php --conf ../LocalSettings.php ../images/wikidatawiki-20191020-pages-meta-current1.xml-p1p235321.bz2 Revision 1033865598 using content model wikibase-item cannot be stored on "Q15" on this wiki, since that model is not supported on that page. Revision 1034542603 using content model wikibase-item cannot be stored on "Q17" on this wiki, since that model is not supported on that page. Revision 1032554298 using content model wikibase-item cannot be stored on "Q18" on this wiki, since that model is not supported on that page. Revision 1032534215 using content model wikibase-item cannot be stored on "Q20" on this wiki, since that model is not supported on that page. Revision 1026713626 using content model wikibase-item cannot be stored on "Q21" on this wiki, since that model is not supported on that page. Revision 1023703278 using content model wikibase-item cannot be stored on "Q22" on this wiki, since that model is not supported on that page. Revision 1032815802 using content model wikibase-item cannot be stored on "Q25" on this wiki, since that model is not supported on that page. Revision 1032910600 using content model wikibase-item cannot be stored on "Q26" on this wiki, since that model is not supported on that page. _______________________________________________ Wikidata-tech mailing list Wikidata-tech@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-tech
Is this the recommended way to set up a local copy of Wikidata? (If not, what is the recommended way?)
peter
On 11/19/19 10:37 AM, Addshore wrote:
Hi all
We resolved this on the Wikibase telegram chat.
For anyone finding this email thread, here is a rough log of the chat....
Miquel Farre I just wrote to the wikidata-tech mail list asking for the question regarding the load of dumps https://t.me/c/1478172663/1845 if somebody here is professionally consulting around wikidata/wikibase and would be able to help writing fixes to those problems, we could study how to sponsor it. Please let me know.
Addshore I just scrolled up and it looks like it is a namespace issue Wikidata has namespace 0 (the main namespace) being wikidaa items,. On a default install of wikibase the item namespace is 120, and 0 is a wikitext namespace Either you'll have to configure your wikibase to be the same as Wikidata, or run something through the dump to change the namespace. I'm not sure there is anything built into the import process for XML for converting namespaces, I'll have a quick look, could be a good feature thoigh
Miquel Farre regarding the namespace issue, can I just add this config: https://www.mediawiki.org/wiki/Wikibase/Installation/Advanced_configuration#... chaning $baseNs = 0 ?
Addshore Yes, so if you look in the example settings you can find the rather verbose example of how to do this https://github.com/wikimedia/mediawiki-extensions-Wikibase/blob/master/repo/... defining $baseNs in local settings will not work, but defining those other things after you load the default settings will work changing this on an already existing wikibase might have unexpected consequences, and is not advised, but doing this on an empty one should be fine :)
Miquel Farre Adam, this is working, I can load a dump 🙂
On Tue, 19 Nov 2019 at 09:39, Miquel Àngel Farré <miquel.farre@gmail.com mailto:miquel.farre@gmail.com> wrote:
Hello, We are having issues launching a local copy of wikidata, when we use the 'importDump.php' tool, below the issues that we are facing. If somebody has an idea of how we could solve this, please let me know. We are also considering professional services to get fixes for this being released in case somebody is professionally consulting around wikibase. Thanks, Miquel Here the issues: if I try to load the full dump, the error I get is: root@4fc8cc9b76b3:/var/www/html/maintenance# php importDump.php --conf ../LocalSettings.php ../images/wikidatawiki-20191101-pages-articles-multistream.xml.bz2 Warning: XMLReader::read(): uploadsource://d0cd78c216b067ffdd60946c258db6a7:45: parser error : Extra content at the end of the document in /var/www/html/includes/import/WikiImporter.php on line 646 Warning: XMLReader::read(): </siteinfo> in /var/www/html/includes/import/WikiImporter.php on line 646 Warning: XMLReader::read(): ^ in /var/www/html/includes/import/WikiImporter.php on line 646 Done! You might want to run rebuildrecentchanges.php to regenerate RecentChanges, If I try to load a partial dump, the warnings that I get (which I think those mean nothing is loading) are: root@4fc8cc9b76b3:/var/www/html/maintenance# php importDump.php --conf ../LocalSettings.php ../images/wikidatawiki-20191020-pages-meta-current1.xml-p1p235321.bz2 Revision 1033865598 using content model wikibase-item cannot be stored on "Q15" on this wiki, since that model is not supported on that page. Revision 1034542603 using content model wikibase-item cannot be stored on "Q17" on this wiki, since that model is not supported on that page. Revision 1032554298 using content model wikibase-item cannot be stored on "Q18" on this wiki, since that model is not supported on that page. Revision 1032534215 using content model wikibase-item cannot be stored on "Q20" on this wiki, since that model is not supported on that page. Revision 1026713626 using content model wikibase-item cannot be stored on "Q21" on this wiki, since that model is not supported on that page. Revision 1023703278 using content model wikibase-item cannot be stored on "Q22" on this wiki, since that model is not supported on that page. Revision 1032815802 using content model wikibase-item cannot be stored on "Q25" on this wiki, since that model is not supported on that page. Revision 1032910600 using content model wikibase-item cannot be stored on "Q26" on this wiki, since that model is not supported on that page. _______________________________________________ Wikidata-tech mailing list Wikidata-tech@lists.wikimedia.org <mailto:Wikidata-tech@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/wikidata-tech
Wikidata-tech mailing list Wikidata-tech@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-tech
This should probably help:
https://addshore.com/2019/10/your-own-wikidata-query-service-with-no-limits-...
On Tue, Nov 19, 2019 at 8:27 AM Peter F. Patel-Schneider < pfpschneider@gmail.com> wrote:
Is this the recommended way to set up a local copy of Wikidata? (If not, what is the recommended way?)
peter
On 11/19/19 10:37 AM, Addshore wrote:
Hi all
We resolved this on the Wikibase telegram chat.
For anyone finding this email thread, here is a rough log of the chat....
Miquel Farre I just wrote to the wikidata-tech mail list asking for the question regarding the load of dumps https://t.me/c/1478172663/1845 if somebody here is professionally consulting around wikidata/wikibase
and
would be able to help writing fixes to those problems, we could study
how to
sponsor it. Please let me know.
Addshore I just scrolled up and it looks like it is a namespace issue Wikidata has namespace 0 (the main namespace) being wikidaa items,. On a default install of wikibase the item namespace is 120, and 0 is a wikitext namespace Either you'll have to configure your wikibase to be the same as
Wikidata, or
run something through the dump to change the namespace. I'm not sure there is anything built into the import process for XML for converting namespaces, I'll have a quick look, could be a good feature
thoigh
Miquel Farre regarding the namespace issue, can I just add this config:
https://www.mediawiki.org/wiki/Wikibase/Installation/Advanced_configuration#...
chaning $baseNs = 0 ?
Addshore Yes, so if you look in the example settings you can find the rather
verbose
example of how to do this
https://github.com/wikimedia/mediawiki-extensions-Wikibase/blob/master/repo/...
defining $baseNs in local settings will not work, but defining those
other
things after you load the default settings will work changing this on an already existing wikibase might have unexpected consequences, and is not advised, but doing this on an empty one should
be
fine :)
Miquel Farre Adam, this is working, I can load a dump 🙂
On Tue, 19 Nov 2019 at 09:39, Miquel Àngel Farré <miquel.farre@gmail.com mailto:miquel.farre@gmail.com> wrote:
Hello, We are having issues launching a local copy of wikidata, when we usethe
'importDump.php' tool, below the issues that we are facing. If somebody has an idea of how we could solve this, please let meknow.
We are also considering professional services to get fixes for this being released in case somebody is professionally consulting around wikibase. Thanks, Miquel Here the issues: if I try to load the full dump, the error I get is: root@4fc8cc9b76b3:/var/www/html/maintenance# php importDump.php--conf
../LocalSettings.php ../images/wikidatawiki-20191101-pages-articles-multistream.xml.bz2 Warning: XMLReader::read(): uploadsource://d0cd78c216b067ffdd60946c258db6a7:45: parser error :Extra
content at the end of the document in /var/www/html/includes/import/WikiImporter.php on line 646 Warning: XMLReader::read(): </siteinfo> in /var/www/html/includes/import/WikiImporter.php on line 646 Warning: XMLReader::read(): ^ in /var/www/html/includes/import/WikiImporter.php on line 646 Done! You might want to run rebuildrecentchanges.php to regenerateRecentChanges,
If I try to load a partial dump, the warnings that I get (which Ithink
those mean nothing is loading) are: root@4fc8cc9b76b3:/var/www/html/maintenance# php importDump.php--conf
../LocalSettings.php../images/wikidatawiki-20191020-pages-meta-current1.xml-p1p235321.bz2
Revision 1033865598 using content model wikibase-item cannot bestored
on "Q15" on this wiki, since that model is not supported on thatpage.
Revision 1034542603 using content model wikibase-item cannot bestored
on "Q17" on this wiki, since that model is not supported on thatpage.
Revision 1032554298 using content model wikibase-item cannot bestored
on "Q18" on this wiki, since that model is not supported on thatpage.
Revision 1032534215 using content model wikibase-item cannot bestored
on "Q20" on this wiki, since that model is not supported on thatpage.
Revision 1026713626 using content model wikibase-item cannot bestored
on "Q21" on this wiki, since that model is not supported on thatpage.
Revision 1023703278 using content model wikibase-item cannot bestored
on "Q22" on this wiki, since that model is not supported on thatpage.
Revision 1032815802 using content model wikibase-item cannot bestored
on "Q25" on this wiki, since that model is not supported on thatpage.
Revision 1032910600 using content model wikibase-item cannot bestored
on "Q26" on this wiki, since that model is not supported on thatpage.
_______________________________________________ Wikidata-tech mailing list Wikidata-tech@lists.wikimedia.org <mailto:Wikidata-tech@lists.wikimedia.org>
https://lists.wikimedia.org/mailman/listinfo/wikidata-tech
Wikidata-tech mailing list Wikidata-tech@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-tech
Wikidata-tech mailing list Wikidata-tech@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-tech
wikidata-tech@lists.wikimedia.org