Hi everyone,
The first weekend of May the Wikimedia Hackathon 2020 in Tirana was
supposed to happen. As an alternative, we're organizing a remote
hackathon. Please have a look at
https://www.mediawiki.org/wiki/Wikimedia_Hackathon_2020/Remote_Hackathon
and sign up if you're interested in participating.
All of the sessions, projects, and initiatives are run by the
participants themselves and not overseen by a core organizing team. All
ideas including technical projects and sessions, non-technical projects
and sessions, and social events/socal time are very welcome.
Please spread the word!
Maarten
Hi,
This message is relevant for people writing SPARQL queries and using the
Wikidata Query Service:
As part of the work of redesigning the WDQS updater[0] we identified that
blank nodes[1] are problematic[2] and we plan to deprecate their usage in
the wikibase RDF model[3]. To ease the deprecation process we are
introducing the new function wikibase:isSomeValue() that can be used in
place of isBlank() when it was used to filter SomeValue[4].
What does this mean for you: nothing will change for now, we are only
interested to know if you encounter any issues with the
wikibase:isSomeValue() function when used as a replacement of the isBlank()
function. More importantly, if you used the isBlank() function for other
purposes than identifying SomeValue (unknown values in the UI), please let
us know as soon as possible.
The current plan is as follow:
1. Introduce a new wikibase:isSomeValue() function
We are at this step. You can already use wikibase:isSomeValue() in the
Query Service. Here’s an example query (Humans whose gender we know we
don't know):
SELECT ?human WHERE {
?human wdt:P21 ?gender
FILTER wikibase:isSomeValue(?gender) .
}
You can also search the wikis[8] to find all the pages where the function
isBlank is referenced in a SPARQL query.
2. Generate stable labels for blank nodes in the wikibase RDF output
Instead of "autogenerated" blank node labels wikidata will now provide a
stable label for blank nodes. In other words the wikibase triples using
blank nodes such as:
s:Q2-6657d0b5-4aa4-b465-12ed-d1b8a04ef658 ps:P576 _:genid2 ;
will become
s:Q2-6657d0b5-4aa4-b465-12ed-d1b8a04ef658 ps:P576
_:1668ace9a6860f7b32569c45fe5a5c0d ;
This is not a breaking change.
3. [BREAKING CHANGE] Convert blank nodes to IRIs in the WDQS updater
At this point some WDQS servers will start returning IRIs such as
http://www.wikidata.org/somevalue/1668ace9a6860f7b32569c45fe5a5c0d (the
exact form of the IRI is still under discussion) instead of blank node
literals like t1514691780 auto-generated by blazegraph. Queries still using
isBlank() will stop functioning. Tools explicitly relying on the presence
of blank nodes (t1514691780) in the query results will also be affected.
We don’t have a defined date for this change yet, but we will follow the
Wikidata breaking change process (announcing the change 4 weeks in advance).
4. [BREAKING CHANGE] Change the RDF model and remove blank nodes completely
from the RDF dumps
Instead of doing the conversion and blank node removal in the WDQS updater
we will do it at RDF generation.
This is a breaking change of the somevalue section of the RDF model[5] and
the no value owl constraint for properties[6].
We don’t have a defined date for this change yet, but we will follow the
Wikidata breaking change process (announcing the change 4 weeks in advance).
If you encounter issues using wikibase:isSomeValue() or if you have
questions about the process, feel free to write a comment on the
Phabricator ticket[3] or the Contact the development team (query service
and search) wiki page[7].
Thanks!
--
David Causse
0: https://phabricator.wikimedia.org/T244590
1: https://en.wikipedia.org/wiki/Blank_node
2: https://phabricator.wikimedia.org/T244341#5889997
3: https://phabricator.wikimedia.org/T244341
4: https://www.mediawiki.org/wiki/Wikibase/DataModel#PropertySomeValueSnak
5:
https://www.mediawiki.org/wiki/Wikibase/Indexing/RDF_Dump_Format#Somevalue
6: https://www.mediawiki.org/wiki/Wikibase/Indexing/RDF_Dump_Format#Novalue
7:
https://www.wikidata.org/wiki/Wikidata:Contact_the_development_team/Query_S…
8:
https://www.wikidata.org/w/index.php?search=all%3Ainsource%3A%2FisBlank+%2A…
I recently asked <https://twitter.com/thadguidry/status/1247635376094707712>
a Freebase colleague @narphorium to graciously spend some time to write up
what was once a very cool Freebase app that many of us used in Freebase to
see overlap of Freebase Properties (like WD statements) between 1-5
Freebase Topics (like WD Items) that then ran a query to find other Topics
that matched that overlapping set.
I thought that sharing this knowledge would allow others to get inspired
and learn and possibly build a similar shape tool for Wikidata; if not
already existing.
Freebase Sets
https://github.com/narphorium/freebase-sets
Best of luck WD engineers!
Thad
https://www.linkedin.com/in/thadguidry/
Hello,
Is there any documentation to "wbeditentity", except from the few
examples on the API-document-page?
Any documentation about the format/grammar of the data?
And is there any API-documentation for lexicographical data at al?
The few bits on the source-code generated page aren't sufficient.
Cheers,
M
--
Michael F. Schönitzer
Prochintalstr. 2
80993 München
Mail: michael(a)schoenitzer.de
Jabber: schoenitzer(a)jabber.ccc.de
Tel: 089/37918949 - Mobil: 017657895702
Hello all,
This is an important announcement for all the tool builders and maintainers
who access Wikidata’s data by *querying Labs database replicas directly*.
In April 2019, the Wikidata development team started a heavy redesign of an
important table of the database, wb_terms. Over the years, this table has
become too big, causing various issues. That’s why we decided to split it
into several smaller tables. For more context, you can have a look at
previous announcements: April 2019
<https://lists.wikimedia.org/pipermail/wikidata/2019-April/012987.html>, May
2019 <https://lists.wikimedia.org/pipermail/wikidata/2019-May/013052.html>.
We encountered a lot of unexpected issues on the road which required us to
postpone the timeline, but the final steps of the migration are now
happening. We published a blog post
<https://phabricator.wikimedia.org/phame/post/view/195/coming_to_terms_with_…>
to explain in more detail what we did and why it was important to optimize
the database structure.
Here’s an overview of what happened, what are the next steps and how you
can update your code:
- The new structure is now in place on Labs. You can read more details
on this documentation page
<https://doc.wikimedia.org/mediawiki-extensions-Wikibase/master/php/md_docs_…>
- Wikidata is entirely reading from and writing to the new tables since
March 18th, which means the existing wb_terms table is *not updated
anymore*
- This means that any tools that are still reading from wb_terms and
updating Wikidata based on its content have to be updated as soon as
possible
- In order to fully deprecate the table and make it clear that it is
outdated, the wb_terms will be renamed to wb_terms_no_longer_updated on
Monday, *March 30th*. Tools still querying this table at that date will
certainly encounter errors.
- To get an overview on how to update your queries, you can have a look
at these examples <https://phabricator.wikimedia.org/T221767>
If you have questions or encounter any issues, please write on this
Phabricator ticket <https://phabricator.wikimedia.org/T208425> or write me
an email.
We apologize for the delay and the lack of recent updates regarding this
change.
Cheers,
--
Léa Lacroix
Project Manager Community Communication for Wikidata
Wikimedia Deutschland e.V.
Tempelhofer Ufer 23-24
10963 Berlin
www.wikimedia.de
Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.
Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für
Körperschaften I Berlin, Steuernummer 27/029/42207.
Hello all,
While cleaning (reviewing and rewriting) the code of Wikidata and Wikimedia
Commons backend in October 2019, The Wikidata team at WMDE together with
WMF worked on reducing the loading time of pages. We managed to reduce the
loading time of every Wikidata page by about 0.1-0.2 seconds. This is due
to a reduction of the modules (sets of code responsible for a certain
function) that need to be loaded every time a page is opened by someone.
Instead of 260 modules, which needed to be loaded before, only 85 modules
need to be loaded now when the page is called. By doing so, it is easier to
load Wikidata pages for people who only have a slow internet connection.
Link to picture on Commons:
https://commons.wikimedia.org/wiki/File:Reduced_loading_times_cut.png
Description: Size decrease of the initialization loader on Wikidata pages (see
on Grafana
<https://grafana.wikimedia.org/d/BvWJlaDWk/startup-manifest-size?orgId=1&fro…>
)
Reducing the amount of modules called when loading the page equals a
reduction of about 130 GB of network traffic for all users every day, or
47TB per year. The reduction of network traffic translates into a reduction
of electricity use, thus, this change is also good for the environment.
Additionally, the interdependencies between the modules were reduced from
4MB to 1MB, which improved the loading time per page as well.
Many thanks to everyone involved in this improvement! If you want to get
more details about the actions we performed, you can have a look at the
Phabricator board
<https://phabricator.wikimedia.org/project/board/4268/query/all/>.
If you are developing scripts or tools on top of the Wikidata UI, some
documentation will walk you through the architecture of RessourceLoader
<https://www.mediawiki.org/wiki/ResourceLoader/Architecture>, what’s page
load performance
<https://www.mediawiki.org/wiki/Wikimedia_Performance_Team/Page_load_perform…>
and how to create module bundles with ResourceLoader
<https://www.mediawiki.org/wiki/ResourceLoader/Developing_with_ResourceLoader>.
For further questions or feedback, feel free to contact us on this page
<https://www.wikidata.org/wiki/Wikidata:Contact_the_development_team>.
Cheers,
Max for the Wikidata team
--
Max Klemm
Working Student Community Communication for Technical Wishes
Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin
Phone: +49 (0)30 219 158 26-0https://wikimedia.de
Imagine a world in which every single human being can freely share in
the sum of all knowledge. Help us to achieve our
vision!https://spenden.wikimedia.de
Wikimedia Deutschland – Gesellschaft zur Förderung Freien Wissens e.
V. Eingetragen im Vereinsregister des Amtsgerichts
Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig
anerkannt durch das Finanzamt für Körperschaften I Berlin,
Steuernummer 27/029/42207.
FYI, this is related to Wikidata/Wikibase as well. Please check your tools
in case your code uses the API error codes.
---------- Forwarded message ---------
From: Brad Jorsch (Anomie) <bjorsch(a)wikimedia.org>
Date: Tue, 4 Feb 2020 at 19:30
Subject: [Wikitech-l] BREAKING CHANGES to Action API parameter validation
To: Wikimedia developers <wikitech-l(a)lists.wikimedia.org>
This notice is being sent to wikitech-l for the benefit of technical
subscribers who aren't subscribed to that list, due to concern that these
changes may affect many API clients. For notification of all breaking
changes to the Action API, please subscribe at
https://lists.wikimedia.org/mailman/listinfo/mediawiki-api-announce.
Error codes for parameter validation errors are changing. Among others,
"noX" becomes "missingparam" and "unknown_X" becomes "badvalue". See the
full announcement at
https://lists.wikimedia.org/pipermail/mediawiki-api-announce/2020-February/…
for details.
Various unusual values for integer-type parameters will no longer be
accepted, basically anything that isn't an optional ASCII sign ('+' or '-')
followed by ASCII digits. See
https://lists.wikimedia.org/pipermail/mediawiki-api-announce/2020-February/…
for details.
Both of these changes will most likely go out to Wikimedia wikis with
1.35.0-wmf.19. See https://www.mediawiki.org/wiki/MediaWiki_1.35/Roadmap
for a schedule.
--
Brad Jorsch (Anomie)
Senior Software Engineer
Wikimedia Foundation
_______________________________________________
Wikitech-l mailing list
Wikitech-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
--
Léa Lacroix
Project Manager Community Communication for Wikidata
Wikimedia Deutschland e.V.
Tempelhofer Ufer 23-24
10963 Berlin
www.wikimedia.de
Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.
Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für
Körperschaften I Berlin, Steuernummer 27/029/42207.
Hello all,
This email is important for you if you are running an instance of Wikibase.
As part of deprecating wb_terms table[1] for the whole Wikibase codebase,
the new term store is now wired to the Wikibase default setup and per the
default it's going to read from the new term store.
It means that next time you update your Wikibase (to master or 1.35 once
it’s out), you should run update.php [2] so it creates the new tables and
migrate your data to the new term store. This might take some time if you
have a large installation. If you’re not running the script, you might see
errors appear.
The default value until the next mediawiki major release (1.35) is going to
stay "read new, write both", meaning you will have both old and new data
side by side but it reads from the new one. By the next major release after
1.35, we are going to fully drop wb_terms table from Wikibase.
Please note that there’s an exception related to search: if you are not
using ElasticSearch, the search still uses the old term store (wb_terms
table). That’s why the default is to write to both term stores for now
until the search works for the new term store properly.
If you have any questions or issues, feel free to write me.
Cheers, Léa
[1]: https://phabricator.wikimedia.org/T208425
[2]:
https://github.com/wikimedia/mediawiki/blob/master/maintenance/update.php
--
Léa Lacroix
Project Manager Community Communication for Wikidata
Wikimedia Deutschland e.V.
Tempelhofer Ufer 23-24
10963 Berlin
www.wikimedia.de
Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.
Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für
Körperschaften I Berlin, Steuernummer 27/029/42207.
Hello,
We are having issues launching a local copy of wikidata, when we use the
'importDump.php' tool, below the issues that we are facing.
If somebody has an idea of how we could solve this, please let me know. We
are also considering professional services to get fixes for this being
released in case somebody is professionally consulting around wikibase.
Thanks,
Miquel
Here the issues:
if I try to load the full dump, the error I get is:
root@4fc8cc9b76b3:/var/www/html/maintenance# php importDump.php --conf
../LocalSettings.php
../images/wikidatawiki-20191101-pages-articles-multistream.xml.bz2
Warning: XMLReader::read():
uploadsource://d0cd78c216b067ffdd60946c258db6a7:45: parser error : Extra
content at the end of the document in
/var/www/html/includes/import/WikiImporter.php on line 646
Warning: XMLReader::read(): </siteinfo> in
/var/www/html/includes/import/WikiImporter.php on line 646
Warning: XMLReader::read(): ^ in
/var/www/html/includes/import/WikiImporter.php on line 646
Done!
You might want to run rebuildrecentchanges.php to regenerate RecentChanges,
If I try to load a partial dump, the warnings that I get (which I think
those mean nothing is loading) are:
root@4fc8cc9b76b3:/var/www/html/maintenance# php importDump.php --conf
../LocalSettings.php
../images/wikidatawiki-20191020-pages-meta-current1.xml-p1p235321.bz2
Revision 1033865598 using content model wikibase-item cannot be stored on
"Q15" on this wiki, since that model is not supported on that page.
Revision 1034542603 using content model wikibase-item cannot be stored on
"Q17" on this wiki, since that model is not supported on that page.
Revision 1032554298 using content model wikibase-item cannot be stored on
"Q18" on this wiki, since that model is not supported on that page.
Revision 1032534215 using content model wikibase-item cannot be stored on
"Q20" on this wiki, since that model is not supported on that page.
Revision 1026713626 using content model wikibase-item cannot be stored on
"Q21" on this wiki, since that model is not supported on that page.
Revision 1023703278 using content model wikibase-item cannot be stored on
"Q22" on this wiki, since that model is not supported on that page.
Revision 1032815802 using content model wikibase-item cannot be stored on
"Q25" on this wiki, since that model is not supported on that page.
Revision 1032910600 using content model wikibase-item cannot be stored on
"Q26" on this wiki, since that model is not supported on that page.
