thank you Guillaume, when do you expect a public update on the security incident [1]? Is any of our personal and private data (email, password etc) affected?

best, Marco [1] https://phabricator.wikimedia.org/T241410 On Fri, Feb 7, 2020 at 1:33 PM Guillaume Lederrey <glederrey(a)wikimedia.org> wrote: -- --- Marco Neumann KONA _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

On Fri, Feb 7, 2020 at 5:18 PM Guillaume Lederrey <glederrey(a)wikimedia.org> wrote: Corrections and apologies on what I said above. We are not actually ready to make this ticket public. The underlying issue is under control and does not require any user action to mitigate. Given the security aspect, I'm not going to do any further communication on this. Sorry to have been misleading on this. Enjoy your day! Guillaume -- Guillaume Lederrey Engineering Manager, Search Platform Wikimedia Foundation UTC+1 / CET _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

Better update granularity is probably good and may be a good priority. It is (still) unclear for me as a tool writer whether I can do anything. For instance it is not clear to me whether the parallel SPARQL queries that comes when a user visit a Scholia page is important for the load on WDQS (not likely) or it is miniscule (likely).

As far as I understand on http://ceur-ws.org/Vol-2073/article-03.pdf much of the query load comes via Magnus. I presume another big chunk is from the genewiki people. If robotic queries are sources of problems then tool writers/users can do something. But fixing issues would require the WMF to tell if it really is a problem and what the problems are.

best regards Finn On 07/02/2020 14:32, Guillaume Lederrey wrote: _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

Hoi, In my opinion, Wikidata is a flagship project of the Wikimedia Foundation. With the current underperformance vis a vis demand, it is not only a technical question what to do, it is also an organisational issue. As I have written in several blogposts, the current underperformance affects the projects using Wikidata as its infra structure. What is the official response to the underachievement of a key and strategic resource. thanks, GerardM http://ultimategerardm.blogspot.com/2020/02/dear-krmaher-wikipedia-is-not-f… On Fri, 7 Feb 2020 at 19:01, Pine W <wiki.pine(a)gmail.com> wrote: > > I don't know if this is helpful, as I'm not very familiar with > Wikidata's infrastructure, but I think that an idea that was discussed > in the Wikimedia Strategy 2030 process is charging real money to > organizations that consume large amounts of data from the Wikimedia > API. By extension, an idea to consider is charging real money to > consumers that want use Wikidata services in resource intensive ways. > That would have several potential benefits. Charging money for > resource intensive requests could make consumers be more value > conscious when deciding which queries to run, it would probably reduce > the workload on WMF's end, the reduced workload on WMF's end could > lead to faster performance, and the money could be used for the > maintenance and/or upgrade of Wikidata services. I think that offering > free services to consumers who are not making resource intensive > requests is good, but I am also fine with charging real money for > resource intensive requests by consumers. For anyone who wants to make > a resource intensive request and is unwilling or unable to pay > accordingly, an option is to put them into a "slow lane" where there > requests will be fulfilled but at a slower pace than paid resource > intensive requests will be fulfilled. > > Pine > ( https://meta.wikimedia.org/wiki/User:Pine )

Le ven. 7 févr. 2020 à 19:01, Pine W <wiki.pine(a)gmail.com> a écrit : I was told that charging on an API request basis is very difficult to get correctly in terms of software because measuring things is in general difficult. Take, for instance, the case of a failed query it should not be charged, should it? That is the reason why most cell phone plans are unlimited. _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

Hello Guillaume, Le ven. 7 févr. 2020 à 14:33, Guillaume Lederrey <glederrey(a)wikimedia.org> a écrit : terms of communication. I'll try my best to send a monthly update from now on. Keep me honest, remind me if I fail. It will be nice to have some feedback on my grant request at: https://meta.wikimedia.org/wiki/Grants:Project/Future-proof_WDQS Or one of the other threads on the very same mailing list. pressure on edits, by adding the WDQS update lag to the Wikdiata maxlag [6]. This is obviously less than ideal (at least as long as WDQS updates are lagging as often as they are), but does allow the service to recover from time to time. We probably need to iterate on this, provide better granularity, differentiate better between operations that have an impact on update lag and those which don't. > On the slightly better news side, we now have a much better understanding of the update process and of its shortcomings. The current process does a full diff between each updated entity and what we have in blazegraph. Even if a single triple needs to change, we still read tons of data from Blazegraph. While this approach is simple and robust, it is obviously not efficient. We need to rewrite the updater to take a more event streaming / reactive approach, and only work on the actual changes. When it will be done, it will be still a short term solution (something that our kafka queues don't offer). This is where we are focusing our energy at the moment, this looks like the best option to improve the situation in the medium term. This change will probably have some functional impacts [3]. Guaranteed ordering in a multi-party distributed setting has no easy solution, and apparently it is not provided by Kafka. For a non-technical person, you can read https://en.wikipedia.org/wiki/Two_Generals%27_Problem > Some longer term thoughts: > Keeping all of Wikidata in a single graph is most probably not going to work long term. :( triples and there is probably a good reason for that. Because Wikimedia is the only non-profit in the field? :( :( more insight). :( language-specific graphs, with only language-specific labels. :( :) possible in the long term. We need to think if / how we want to implement some form of authentication and quotas. With blacklists and whitelists, but this is huge anyway. strict for others. Again, we don't know how this will look like, but we're thinking about it. > What you can do to help: > Again, we're not sure. Of course, reducing the load (both in terms of edits on Wikidata and of reads on WDQS) will help. But not using those services makes them useless. What about making the lag part of the service. I mean, you could reload WDQS periodically, for instance daily, and drop the updater altogether. Who needs to see the updates live in WDQS as soon as edits are done in wikidata? property change to a large entity will require a comparatively insane amount of work to update it on the WDQS side). We'd like to have real data on the cost of various operations, but we only have guesses at this point. > If you've read this far, thanks a lot for your engagement! > Have fun! Will do. _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

Le lun. 10 févr. 2020 à 18:23, Marco Neumann <marco.neumann(a)gmail.com> a écrit : The semantic Web is already distributed. Everything into one graph, or if you prefer in one place, is the gist of the idea of a library or encyclopedia. ideas around efficient query distribution Maybe. But does not preclude the aggregation or sum of knowledge to happen. Like I wrote above, a distributed knowledge base is already the state of the things. I am not sure how to understand that part of the sentence. That is the gist of my proposal. Without the ability to run wikidata at a small scale, WMF will fail at knowledge equity. I proposed something along the lines of https://linkeddatafragments.org as known as "thin server, thick client" I had no feedback :( With that, I heartily agree. With the ability to downscale wikidata infrastructure, and make companies and institutions pay for the stream of changes to apply to their local instance, it will make things much easier. too literally. I disagree. actually, the SPARQL endpoint as an extension to the wikidata architecture around wikibase should be used more pro-actively to connect multiple RDF data providers for search. Read my proposal at https://meta.wikimedia.org/wiki/Grants:Project/Future-proof_WDQS The title is misleading, I intended to change it to Future-proof WikiData. WDQS or querying is an integral part of wikidata and must not be merely an addon. who enrich their remote queries with wikidata data. I do not understand. Yes, people enrich wikidata queries with their data. And? endpoint to where it is now. the future. Yes, I agree with that. Congratulations! I am very proud to be part of the Wikimedia community. The current WMF proposal that is called "sharding", see details at: https://en.wikipedia.org/wiki/Shard_(database_architecture) It is, not future proof. I have not done any analysis, but I bet that most of the 2TB of wikidata is English, so even if you shard by language, you will still end up with a gigantic graph. Also, most of the data is not specific to a natural language, so one can not possibly split the data by language. If WMF comes up with another sharding strategy, how will edits that span multiple regions will happen? How will it make entering the wikidata party easier? I dare to write in the open: it seems to me like we are witnessing "Earth is flat vs. Earth is not flat" kind of event. Thanks for the reply! amirouche.boubekki(a)gmail.com> wrote: in terms of communication. I'll try my best to send a monthly update from now on. Keep me honest, remind me if I fail. pressure on edits, by adding the WDQS update lag to the Wikdiata maxlag [6]. This is obviously less than ideal (at least as long as WDQS updates are lagging as often as they are), but does allow the service to recover from time to time. We probably need to iterate on this, provide better granularity, differentiate better between operations that have an impact on update lag and those which don't. understanding of the update process and of its shortcomings. The current process does a full diff between each updated entity and what we have in blazegraph. Even if a single triple needs to change, we still read tons of data from Blazegraph. While this approach is simple and robust, it is obviously not efficient. We need to rewrite the updater to take a more event streaming / reactive approach, and only work on the actual changes. (something that our kafka queues don't offer). This is where we are focusing our energy at the moment, this looks like the best option to improve the situation in the medium term. This change will probably have some functional impacts [3]. to work long term. triples and there is probably a good reason for that. more insight). language-specific graphs, with only language-specific labels. be possible in the long term. We need to think if / how we want to implement some form of authentication and quotas. strict for others. Again, we don't know how this will look like, but we're thinking about it. edits on Wikidata and of reads on WDQS) will help. But not using those services makes them useless. single property change to a large entity will require a comparatively insane amount of work to update it on the WDQS side). We'd like to have real data on the cost of various operations, but we only have guesses at this point. -- Amirouche ~ https://hyper.dev _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

Hello Guillaume, Le ven. 7 févr. 2020 à 14:33, Guillaume Lederrey <glederrey(a)wikimedia.org> a écrit : terms of communication. I'll try my best to send a monthly update from now on. Keep me honest, remind me if I fail. It will be nice to have some feedback on my grant request at: https://meta.wikimedia.org/wiki/Grants:Project/Future-proof_WDQS Or one of the other threads on the very same mailing list. pressure on edits, by adding the WDQS update lag to the Wikdiata maxlag [6]. This is obviously less than ideal (at least as long as WDQS updates are lagging as often as they are), but does allow the service to recover from time to time. We probably need to iterate on this, provide better granularity, differentiate better between operations that have an impact on update lag and those which don't. > On the slightly better news side, we now have a much better understanding of the update process and of its shortcomings. The current process does a full diff between each updated entity and what we have in blazegraph. Even if a single triple needs to change, we still read tons of data from Blazegraph. While this approach is simple and robust, it is obviously not efficient. We need to rewrite the updater to take a more event streaming / reactive approach, and only work on the actual changes. When it will be done, it will be still a short term solution (something that our kafka queues don't offer). This is where we are focusing our energy at the moment, this looks like the best option to improve the situation in the medium term. This change will probably have some functional impacts [3]. Guaranteed ordering in a multi-party distributed setting has no easy solution, and apparently it is not provided by Kafka. For a non-technical person, you can read https://en.wikipedia.org/wiki/Two_Generals%27_Problem > Some longer term thoughts: > Keeping all of Wikidata in a single graph is most probably not going to work long term. :( triples and there is probably a good reason for that. Because Wikimedia is the only non-profit in the field? :( :( more insight). :( language-specific graphs, with only language-specific labels. :( :) possible in the long term. We need to think if / how we want to implement some form of authentication and quotas. With blacklists and whitelists, but this is huge anyway. strict for others. Again, we don't know how this will look like, but we're thinking about it. > What you can do to help: > Again, we're not sure. Of course, reducing the load (both in terms of edits on Wikidata and of reads on WDQS) will help. But not using those services makes them useless. What about making the lag part of the service. I mean, you could reload WDQS periodically, for instance daily, and drop the updater altogether. Who needs to see the updates live in WDQS as soon as edits are done in wikidata? property change to a large entity will require a comparatively insane amount of work to update it on the WDQS side). We'd like to have real data on the cost of various operations, but we only have guesses at this point. > If you've read this far, thanks a lot for your engagement! > Have fun! Will do. _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

Hello Guillaume, Le ven. 7 févr. 2020 à 14:33, Guillaume Lederrey <glederrey(a)wikimedia.org> a écrit : It will be nice to have some feedback on my grant request at: https://meta.wikimedia.org/wiki/Grants:Project/Future-proof_WDQS Or one of the other threads on the very same mailing list.