Wikidata-tech

wikidata-tech@lists.wikimedia.org

2 participants
623 discussions

Best practices to poll the recent changes feed
by Antonin Delpeuch (lists) 06 May '20

06 May '20

Hi, I wonder if there is any guidance about how to poll the recent changes feed of a MediaWiki instance (in particular of a Wikibase one) to keep up with its stream of edits? In particular, how to do this responsibly (without hammering the server) and how to ensure that all changes are seen by the consumer? EditGroups (https://tools.wmflabs.org/editgroups/) currently uses the WMF Event Stream to do this, which works well but has the downside of not being available for non-WMF wikis, and the lack of server-side filtering support, so I have been looking into implementing recent changes polling in it, so it can be run on other wikis. So far it looks like my RC polling strategy misses some edits that the WMF Event Stream includes, so I need to improve this. RC polling is implemented in the WDQS updater here: https://github.com/wikimedia/wikidata-query-rdf/blob/master/tools/src/main/… Is this the best implementation to look at? And actually - is this really worth doing? Perhaps I should instead require that the target Wikibase runs the EventLogging extension (https://www.mediawiki.org/wiki/Extension:EventLogging) which exposes the edit stream in a Kafka instance, and then implement a Kafka topic consumer in EditGroups. It does add requirements on the Wikibase instance, but if RC polling is brittle, it would be wrong to promise that EditGroups can be run off a stock MediaWiki instance anyway. (Note that I still think EditGroups is not a long-term solution. We need a MediaWiki extension to replace it: https://phabricator.wikimedia.org/T203557. I am just looking into this to help our OpenRefine GSoC intern Lu Liu who will be working on Wikibase support in OpenRefine this summer.) Cheers, Antonin

1 0

Outdated copy of wb_terms to be dropped on April 29th
by Léa Lacroix 28 Apr '20

28 Apr '20

Hello all, This is a follow-up on previous announcements related to the migration of wb_terms table (March 23rd <https://www.wikidata.org/wiki/Wikidata:Project_chat/Archive/2020/03#Importa…>, April 6th <https://www.wikidata.org/wiki/Wikidata:Project_chat/Archive/2020/04#wb_term…>) that concerns tool developers querying the database replicas on Labs. As the last step of the migration, and as mentioned in the previous discussions, the table wb_terms_no_longer_updated will be dropped. We are doing this because the last steps of database migration have been achieved and announced a month ago, and we don’t want to keep a table containing outdated information accessible for too long. The table has already been deleted in production (without any impact on tools or users) and the replicas on Labs will be deleted on April 29th. (Ticket <https://phabricator.wikimedia.org/T248086>) If you’re maintaining tools querying the database with the replicas, you can adapt your code to the new table structure <https://doc.wikimedia.org/Wikibase/master/php/md_docs_storage_terms.html> (the former shared link doesn't work anymore). You can also have a look at suggestions to optimize your queries <https://www.wikidata.org/wiki/User:Amir_Sarabadani_(WMDE)/Database_normaliz…>. If you have any questions or issues, feel free to reach out to me. Thanks for your understanding, -- Léa Lacroix Project Manager Community Communication for Wikidata Wikimedia Deutschland e.V. Tempelhofer Ufer 23-24 10963 Berlin www.wikimedia.de Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.

1 0

Merge Wizard silent for Lexeme merging
by Thad Guidry 27 Apr '20

27 Apr '20

The only clue that I get that there is a problem is that the mediawiki-api-error field says "failed-modify" ? Here's my Request URL shown in the GET just before the POST : https://www.wikidata.org/w/api.php?action=wbgetentities&format=json&ids=L14… and here's the screenshot of the Response: [image: image.png] Thad https://www.linkedin.com/in/thadguidry/

2 2

New Community Communications Manager for Wikidata/Wikibase
by Mohammed Sadat Abdulai 22 Apr '20

22 Apr '20

Hi everyone, I hope you’re all having a great day! I’m super excited to announce that I’ll be joining the software department at Wikimedia Germany to help advance engagement between the software development team and the communities using and contributing to Wikidata/Wikibase. Together with Léa, Sam and Lydia, I will be liaising with the different user groups within the Wikibase community to provide information about software changes and promote a smooth and productive collaboration between stakeholders. You can share bug reports with us at < https://www.wikidata.org/wiki/Wikidata:Contact_the_development_team> Please leave a note on my talkpage < https://www.wikidata.org/wiki/User_talk:Mohammed_Sadat_(WMDE)> or write to me directly anytime you encounter issues with the Wikibase software so that I can bring them to the development team: * mohammed.sadat_ext(a)wikimedia.de * Telegram (@masssly) and IRC (mabdulai) Best Regards, -- Mohammed Sadat Abdulai *Community Communications Manager for Wikidata/Wikibase* Wikimedia Deutschland e.V. Tempelhofer Ufer 23-24 10963 Berlin www.wikimedia.de

4 3

Remote hackathon (May 9 - 11, 2020)
by Maarten Dammers 21 Apr '20

21 Apr '20

Hi everyone, The first weekend of May the Wikimedia Hackathon 2020 in Tirana was supposed to happen. As an alternative, we're organizing a remote hackathon. Please have a look at https://www.mediawiki.org/wiki/Wikimedia_Hackathon_2020/Remote_Hackathon and sign up if you're interested in participating. All of the sessions, projects, and initiatives are run by the participants themselves and not overseen by a core organizing team. All ideas including technical projects and sessions, non-technical projects and sessions, and social events/socal time are very welcome. Please spread the word! Maarten

3 2

Blank node deprecation in WDQS & Wikibase RDF model
by David Causse 17 Apr '20

17 Apr '20

Hi, This message is relevant for people writing SPARQL queries and using the Wikidata Query Service: As part of the work of redesigning the WDQS updater[0] we identified that blank nodes[1] are problematic[2] and we plan to deprecate their usage in the wikibase RDF model[3]. To ease the deprecation process we are introducing the new function wikibase:isSomeValue() that can be used in place of isBlank() when it was used to filter SomeValue[4]. What does this mean for you: nothing will change for now, we are only interested to know if you encounter any issues with the wikibase:isSomeValue() function when used as a replacement of the isBlank() function. More importantly, if you used the isBlank() function for other purposes than identifying SomeValue (unknown values in the UI), please let us know as soon as possible. The current plan is as follow: 1. Introduce a new wikibase:isSomeValue() function We are at this step. You can already use wikibase:isSomeValue() in the Query Service. Here’s an example query (Humans whose gender we know we don't know): SELECT ?human WHERE { ?human wdt:P21 ?gender FILTER wikibase:isSomeValue(?gender) . } You can also search the wikis[8] to find all the pages where the function isBlank is referenced in a SPARQL query. 2. Generate stable labels for blank nodes in the wikibase RDF output Instead of "autogenerated" blank node labels wikidata will now provide a stable label for blank nodes. In other words the wikibase triples using blank nodes such as: s:Q2-6657d0b5-4aa4-b465-12ed-d1b8a04ef658 ps:P576 _:genid2 ; will become s:Q2-6657d0b5-4aa4-b465-12ed-d1b8a04ef658 ps:P576 _:1668ace9a6860f7b32569c45fe5a5c0d ; This is not a breaking change. 3. [BREAKING CHANGE] Convert blank nodes to IRIs in the WDQS updater At this point some WDQS servers will start returning IRIs such as http://www.wikidata.org/somevalue/1668ace9a6860f7b32569c45fe5a5c0d (the exact form of the IRI is still under discussion) instead of blank node literals like t1514691780 auto-generated by blazegraph. Queries still using isBlank() will stop functioning. Tools explicitly relying on the presence of blank nodes (t1514691780) in the query results will also be affected. We don’t have a defined date for this change yet, but we will follow the Wikidata breaking change process (announcing the change 4 weeks in advance). 4. [BREAKING CHANGE] Change the RDF model and remove blank nodes completely from the RDF dumps Instead of doing the conversion and blank node removal in the WDQS updater we will do it at RDF generation. This is a breaking change of the somevalue section of the RDF model[5] and the no value owl constraint for properties[6]. We don’t have a defined date for this change yet, but we will follow the Wikidata breaking change process (announcing the change 4 weeks in advance). If you encounter issues using wikibase:isSomeValue() or if you have questions about the process, feel free to write a comment on the Phabricator ticket[3] or the Contact the development team (query service and search) wiki page[7]. Thanks! -- David Causse 0: https://phabricator.wikimedia.org/T244590 1: https://en.wikipedia.org/wiki/Blank_node 2: https://phabricator.wikimedia.org/T244341#5889997 3: https://phabricator.wikimedia.org/T244341 4: https://www.mediawiki.org/wiki/Wikibase/DataModel#PropertySomeValueSnak 5: https://www.mediawiki.org/wiki/Wikibase/Indexing/RDF_Dump_Format#Somevalue 6: https://www.mediawiki.org/wiki/Wikibase/Indexing/RDF_Dump_Format#Novalue 7: https://www.wikidata.org/wiki/Wikidata:Contact_the_development_team/Query_S… 8: https://www.wikidata.org/w/index.php?search=all%3Ainsource%3A%2FisBlank+%2A…

3 3

Freebase Sets
by Thad Guidry 08 Apr '20

08 Apr '20

I recently asked <https://twitter.com/thadguidry/status/1247635376094707712> a Freebase colleague @narphorium to graciously spend some time to write up what was once a very cool Freebase app that many of us used in Freebase to see overlap of Freebase Properties (like WD statements) between 1-5 Freebase Topics (like WD Items) that then ran a query to find other Topics that matched that overlapping set. I thought that sharing this knowledge would allow others to get inspired and learn and possibly build a similar shape tool for Wikidata; if not already existing. Freebase Sets https://github.com/narphorium/freebase-sets Best of luck WD engineers! Thad https://www.linkedin.com/in/thadguidry/

1 0

Wikidata API documentation
by Michael F. Schönitzer 01 Apr '20

01 Apr '20

Hello, Is there any documentation to "wbeditentity", except from the few examples on the API-document-page? Any documentation about the format/grammar of the data? And is there any API-documentation for lexicographical data at al? The few bits on the source-code generated page aren't sufficient. Cheers, M -- Michael F. Schönitzer Prochintalstr. 2 80993 München Mail: michael(a)schoenitzer.de Jabber: schoenitzer(a)jabber.ccc.de Tel: 089/37918949 - Mobil: 017657895702

4 3

Important for tool maintainers: last steps of wb_terms table migration
by Léa Lacroix 23 Mar '20

23 Mar '20

Hello all, This is an important announcement for all the tool builders and maintainers who access Wikidata’s data by *querying Labs database replicas directly*. In April 2019, the Wikidata development team started a heavy redesign of an important table of the database, wb_terms. Over the years, this table has become too big, causing various issues. That’s why we decided to split it into several smaller tables. For more context, you can have a look at previous announcements: April 2019 <https://lists.wikimedia.org/pipermail/wikidata/2019-April/012987.html>, May 2019 <https://lists.wikimedia.org/pipermail/wikidata/2019-May/013052.html>. We encountered a lot of unexpected issues on the road which required us to postpone the timeline, but the final steps of the migration are now happening. We published a blog post <https://phabricator.wikimedia.org/phame/post/view/195/coming_to_terms_with_…> to explain in more detail what we did and why it was important to optimize the database structure. Here’s an overview of what happened, what are the next steps and how you can update your code: - The new structure is now in place on Labs. You can read more details on this documentation page <https://doc.wikimedia.org/mediawiki-extensions-Wikibase/master/php/md_docs_…> - Wikidata is entirely reading from and writing to the new tables since March 18th, which means the existing wb_terms table is *not updated anymore* - This means that any tools that are still reading from wb_terms and updating Wikidata based on its content have to be updated as soon as possible - In order to fully deprecate the table and make it clear that it is outdated, the wb_terms will be renamed to wb_terms_no_longer_updated on Monday, *March 30th*. Tools still querying this table at that date will certainly encounter errors. - To get an overview on how to update your queries, you can have a look at these examples <https://phabricator.wikimedia.org/T221767> If you have questions or encounter any issues, please write on this Phabricator ticket <https://phabricator.wikimedia.org/T208425> or write me an email. We apologize for the delay and the lack of recent updates regarding this change. Cheers, -- Léa Lacroix Project Manager Community Communication for Wikidata Wikimedia Deutschland e.V. Tempelhofer Ufer 23-24 10963 Berlin www.wikimedia.de Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.

1 0

Reduced loading times for Wikidata/Wikimedia Commons
by Max Klemm 11 Feb '20

11 Feb '20

Hello all, While cleaning (reviewing and rewriting) the code of Wikidata and Wikimedia Commons backend in October 2019, The Wikidata team at WMDE together with WMF worked on reducing the loading time of pages. We managed to reduce the loading time of every Wikidata page by about 0.1-0.2 seconds. This is due to a reduction of the modules (sets of code responsible for a certain function) that need to be loaded every time a page is opened by someone. Instead of 260 modules, which needed to be loaded before, only 85 modules need to be loaded now when the page is called. By doing so, it is easier to load Wikidata pages for people who only have a slow internet connection. Link to picture on Commons: https://commons.wikimedia.org/wiki/File:Reduced_loading_times_cut.png Description: Size decrease of the initialization loader on Wikidata pages (see on Grafana <https://grafana.wikimedia.org/d/BvWJlaDWk/startup-manifest-size?orgId=1&fro…> ) Reducing the amount of modules called when loading the page equals a reduction of about 130 GB of network traffic for all users every day, or 47TB per year. The reduction of network traffic translates into a reduction of electricity use, thus, this change is also good for the environment. Additionally, the interdependencies between the modules were reduced from 4MB to 1MB, which improved the loading time per page as well. Many thanks to everyone involved in this improvement! If you want to get more details about the actions we performed, you can have a look at the Phabricator board <https://phabricator.wikimedia.org/project/board/4268/query/all/>. If you are developing scripts or tools on top of the Wikidata UI, some documentation will walk you through the architecture of RessourceLoader <https://www.mediawiki.org/wiki/ResourceLoader/Architecture>, what’s page load performance <https://www.mediawiki.org/wiki/Wikimedia_Performance_Team/Page_load_perform…> and how to create module bundles with ResourceLoader <https://www.mediawiki.org/wiki/ResourceLoader/Developing_with_ResourceLoader>. For further questions or feedback, feel free to contact us on this page <https://www.wikidata.org/wiki/Wikidata:Contact_the_development_team>. Cheers, Max for the Wikidata team -- Max Klemm Working Student Community Communication for Technical Wishes Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin Phone: +49 (0)30 219 158 26-0https://wikimedia.de Imagine a world in which every single human being can freely share in the sum of all knowledge. Help us to achieve our vision!https://spenden.wikimedia.de Wikimedia Deutschland – Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/029/42207.

1 0

← Newer
1
...
5
6
7
8
9
10
11
...
63
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

Wikidata-tech