On Fri, Jun 13, 2014 at 2:51 PM, Jeroen De Dauw jeroendedauw@gmail.com wrote:
Hey,
Are the claims a large part of the network traffic for items you are processing? Some client time might be saved by lazy loading the claim objects from _content. The claims data is even smaller when using raw revisions instead of the API JSON.
Is the size of the serialization something that is causing problems?
Not serious problems IMO. e.g. Q60 is 54K via the API, but that is <10K gziped
https://www.wikidata.org/w/api.php?action=wbgetentities&ids=Q60&lang...
Fetching only the claims only 'almost' halves the network traffic, but that results in the pywiki API cache not being as efficient if several labels or sitelinks are also fetched.
https://www.wikidata.org/w/api.php?action=wbgetentities&ids=Q60&lang...
If a bot is only working with wikidata and a single language wiki, this is the 'optimal' query, which is 5.7Kb gzipped.
https://www.wikidata.org/w/api.php?action=wbgetentities&ids=Q60&lang...
Prefetching many items also reduces the network activity as it lets gzip work harder.