<div dir="ltr">Plain text file would be best, one article title per row<br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Sep 9, 2014 at 10:43 AM, Navino Evans <span dir="ltr"><<a href="mailto:navino@histropedia.com" target="_blank">navino@histropedia.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir="ltr">Most definitely! That would be absolutely fantastic. </p>
<p dir="ltr">What format of list would be most useful for you to work with? </p><div class="HOEnZb"><div class="h5">
<div class="gmail_quote">On 9 Sep 2014 15:38, "John" <<a href="mailto:phoenixoverride@gmail.com" target="_blank">phoenixoverride@gmail.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Its not that big of a deal, once i set the system up. is it possible to have you post the list in a static location on your webserver? I could then just have the bot grab and use that list.<br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Sep 9, 2014 at 10:35 AM, Navino Evans <span dir="ltr"><<a href="mailto:navino@histropedia.com" target="_blank">navino@histropedia.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir="ltr">That's great to know, thank you. </p>
<p dir="ltr">We'll make sure we only use the API within that limit - basically just for individual calls when a user adds a new event to our database. </p>
<p dir="ltr">For the bulk processing, we would need to update the backlinks information as a monthly maintenance task, so I wouldn't want to trouble you with this each time.<br>
<br>
Would you rather we stick with data dump processing for the large scale stuff? <br><br><br><br></p><div><div>
<div class="gmail_quote">On 9 Sep 2014 15:05, "John" <<a href="mailto:phoenixoverride@gmail.com" target="_blank">phoenixoverride@gmail.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div>If you want a report on that many pages drop me a list of those titles and and I can write a report for you given that volume of affected pages. <br></div><br></div>I would say 1-2 seconds between quires should be reasonable for a moderate volume of quires. Any large scale request I will do server side and avoid hammering the web-servers for something that is better batched.<br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Sep 9, 2014 at 9:58 AM, Navino Evans <span dir="ltr"><<a href="mailto:navino@histropedia.com" target="_blank">navino@histropedia.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Once again, a huge thank you for taking the time to do this John - That's exactly what I was looking for! - the helpfulness of this community never ceases to amaze me :)<br></div><div><br></div><div>Hopefully I haven't initiated a journey down the rabbit hole into a fully fledged muliti-language counting machine ;)</div><div><br></div><div><div><br></div><div>Can I just ask what the limit of reasonable use would be for making API calls to this new tool? (e.g. number of calls per day)</div><div><br></div><div>It would be incredibly useful if we could use it to update the events in our database once a month (we are using it to rank historical events by 'importance'), but we are already have approximately 1.5 million events so am aware this may be way beyond what would be acceptable.</div></div></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Sep 9, 2014 at 2:56 PM, John <span dir="ltr"><<a href="mailto:phoenixoverride@gmail.com" target="_blank">phoenixoverride@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">That's doable, however it will require a little more time as I need to unearth some old code to handle multi-projects/languages<br></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Sep 9, 2014 at 9:51 AM, Jan Ainali <span dir="ltr"><<a href="mailto:jan.ainali@wikimedia.se" target="_blank">jan.ainali@wikimedia.se</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Awesome John!<div><br></div><div>Now I only wish that one could specify language code also ;)</div></div><div class="gmail_extra"><span><br clear="all"><div><div dir="ltr"><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px;background-color:rgb(255,255,255)"><i>Med vänliga hälsningar,<br>Jan Ainali</i></span></div><div><font size="1"><br></font></div><div><font size="1">Verksamhetschef, <a href="http://se.wikimedia.org/wiki/Huvudsida" target="_blank">Wikimedia Sverige</a> </font><span style="color:rgb(34,34,34);font-family:arial,sans-serif;background-color:rgb(255,255,255)"><font size="1"><br>0729 - 67 29 48</font></span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;background-color:rgb(255,255,255)"><font size="1"><br></font></span><img src="https://upload.wikimedia.org/wikipedia/commons/thumb/9/91/Wikimedia_Sverige_logo.svg/460px-Wikimedia_Sverige_logo.svg.png" style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px;background-color:rgb(255,255,255)" height="97" width="94">
</div><div><br></div><div><i><font size="1">Tänk dig en värld där varje människa har fri tillgång till mänsklighetens samlade kunskap. Det är det vi gör.</font></i></div><div><a href="http://blimedlem.wikimedia.se" target="_blank"><font size="1">Bli medlem.</font></a></div><div><br></div></div></div>
<br></span><div><div><div class="gmail_quote">2014-09-09 15:34 GMT+02:00 John <span dir="ltr"><<a href="mailto:phoenixoverride@gmail.com" target="_blank">phoenixoverride@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Per request, its no frills but what you what you asked for: <a href="http://tools.wmflabs.org/betacommand-dev/cgi-bin/backlinks" target="_blank">http://tools.wmflabs.org/betacommand-dev/cgi-bin/backlinks</a><br></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Sep 9, 2014 at 8:32 AM, Navino Evans <span dir="ltr"><<a href="mailto:navino@histropedia.com" target="_blank">navino@histropedia.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">That is fantastic news... I'm incredibly grateful for the help and advice.</div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Sep 9, 2014 at 1:27 PM, John <span dir="ltr"><<a href="mailto:phoenixoverride@gmail.com" target="_blank">phoenixoverride@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div>Given the overhead of the API and that he only needs a count getting that info should be fairly easy via a python cgi wrapper around an sql query. <br><br></div>The only thing that I cannot do is #3 since the software does not differentiate between links in templates and links not in templates. Its a requested feature for years now.<br><br></div>Give me a few hours and ill get you the tool you want. This should be less than 30 minutes work <br></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Sep 9, 2014 at 7:55 AM, Jan Ainali <span dir="ltr"><<a href="mailto:jan.ainali@wikimedia.se" target="_blank">jan.ainali@wikimedia.se</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Related tip: In the API you can get a list of backlinks (but you have to count them yourself) from the main namespace including all redirects by a query like this:<div><br></div><div><a href="https://en.wikipedia.org/w/api.php?action=query&list=backlinks&format=json&bltitle=Example&blnamespace=0&blfilterredir=all&bllimit=250&blredirect=" target="_blank">https://en.wikipedia.org/w/api.php?action=query&list=backlinks&format=json&bltitle=Example&blnamespace=0&blfilterredir=all&bllimit=250&blredirect=</a><br></div><div><br></div><div>More info at: <a href="https://www.mediawiki.org/wiki/API:Backlinks" target="_blank">https://www.mediawiki.org/wiki/API:Backlinks</a></div></div><div class="gmail_extra"><br clear="all"><div><div dir="ltr"><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px;background-color:rgb(255,255,255)"><i>Med vänliga hälsningar,<br>Jan Ainali</i></span></div><div><font size="1"><br></font></div><div><font size="1">Verksamhetschef, <a href="http://se.wikimedia.org/wiki/Huvudsida" target="_blank">Wikimedia Sverige</a> </font><span style="color:rgb(34,34,34);font-family:arial,sans-serif;background-color:rgb(255,255,255)"><font size="1"><br>0729 - 67 29 48</font></span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;background-color:rgb(255,255,255)"><font size="1"><br></font></span><img src="https://upload.wikimedia.org/wikipedia/commons/thumb/9/91/Wikimedia_Sverige_logo.svg/460px-Wikimedia_Sverige_logo.svg.png" style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px;background-color:rgb(255,255,255)" height="97" width="94">
</div><div><br></div><div><i><font size="1">Tänk dig en värld där varje människa har fri tillgång till mänsklighetens samlade kunskap. Det är det vi gör.</font></i></div><div><a href="http://blimedlem.wikimedia.se" target="_blank"><font size="1">Bli medlem.</font></a></div><div><br></div></div></div><div><div>
<br><div class="gmail_quote">2014-09-09 13:41 GMT+02:00 Navino Evans <span dir="ltr"><<a href="mailto:navino@histropedia.com" target="_blank">navino@histropedia.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Wow! That would be awesome :)<div><br></div><div><span style="font-family:arial,sans-serif;font-size:13px">The API we are looking for can be as simple as sending a GET request to a url (</span><a href="http://www.somewhere.com/api/count?t=wikipedia_title_goes_here" target="_blank">http://www.somewhere.com/api/count?t=wikipedia_title_goes_here</a><span style="font-family:arial,sans-serif;font-size:13px">), returning a number in "text/plain" format.</span><br></div><div><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div><div><span style="font-family:arial,sans-serif;font-size:13px">The actual count that we're interested is for English Wikipedia only, and would ideally include the following, all added up into a single number:</span></div><div><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div><div><span style="font-family:arial,sans-serif;font-size:13px">1) All links from articles in Main Namespace only (for our purpose it would be better to not include links from User pages, Talk pages etc if possible)</span></div><div><br></div><div>2) Including links from Redirect pages (e.g. counting a link from "Michel Jackson" redirect as part of the count from the article "Michael Jackson")</div><div><br></div><div>3) Excluding links that are within a template transcluded in an article (so we don't need to count the links inside Navboxes within an article for example)</div><div><br></div><div>4) For our purpose, it doesn't really matter whether transclusions of the actual page that is called are included in the count (we generally won't be using it for checking templates, timeline and list articles).</div><div><br></div><div>Just to give the full picture for this request - my use of this tool will be for a company (<a href="http://www.histropedia.com" target="_blank">www.histropedia.com</a>), so I wouldn't want to take up your time with this unless it's something you feel should be available for wider use. My plan was to get the developer working on our site to make this tool for the community if it didn't exist somewhere, but we would be reliant on datadumps so could not get live information (which would be incredibly useful for us, and I hope many others).</div><div><br></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div>On Mon, Sep 8, 2014 at 8:10 PM, John <span dir="ltr"><<a href="mailto:phoenixoverride@gmail.com" target="_blank">phoenixoverride@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div>What numbers/data do you want? I can whip up a replacement for it.<div><div><br><br>On Monday, September 8, 2014, Navino Evans <<a href="mailto:navino@histropedia.com" target="_blank">navino@histropedia.com</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi All,<div><br></div><div>Hi all, does anyone know if there is a tool currently available for counting backlinks to Wikipedia articles via an API? I have been using this tool <a href="http://dispenser.homenet.org/~dispenser/cgi-bin/backlinkscount.py" target="_blank">http://dispenser.homenet.org/~dispenser/cgi-bin/backlinkscount.py</a> - but it seems to have finally gone offline completely following some recent controversy with user:Dispenser.</div><div><br></div><div>Any advice much appreciated!</div><div><br></div><div>Regards, </div><div><br></div><div>Navino</div><div>
</div></div>
</blockquote>
</div></div><br></div></div>_______________________________________________<br>
Labs-l mailing list<br>
<a href="mailto:Labs-l@lists.wikimedia.org" target="_blank">Labs-l@lists.wikimedia.org</a><br>
<a href="https://lists.wikimedia.org/mailman/listinfo/labs-l" target="_blank">https://lists.wikimedia.org/mailman/listinfo/labs-l</a><br>
<br></blockquote></div><span><font color="#888888"><br><br clear="all"><div><br></div>-- <br><div dir="ltr"><div><font color="#000000" size="4">___________________________</font></div><div><font color="#000000" size="1"><br></font></div><div><font color="#000000" size="6">Histropedia</font></div><div><font color="#000000"><img src="http://www.histropedia.com/img/fblogo.png" height="96" width="96"></font></div><div><font color="#000000" size="4">The Timeline for all of History</font></div><div><font color="#000000"><a href="http://www.histropedia.com/" target="_blank">www.histropedia.com</a><br></font></div><div><font color="#000000"><br></font></div><div><span style="font-size:large"><font color="#000000">Follow us on:</font></span></div><div><font color="#000000"><a href="https://twitter.com/Histropedia" target="_blank">Twitter</a> <a href="https://www.facebook.com/Histropedia" target="_blank">Facebo</a><a href="https://www.facebook.com/Histropedia" target="_blank">ok</a> <a href="https://plus.google.com/u/0/b/104484373317792180682/104484373317792180682/posts" target="_blank">Google +</a> <a href="http://www.linkedin.com/company/histropedia-ltd" target="_blank">L</a><a href="http://www.linkedin.com/company/histropedia-ltd" target="_blank">inke</a><a href="http://www.linkedin.com/company/histropedia-ltd" target="_blank">dIn</a></font><br></div><div><br></div></div>
</font></span></div></div>
<br>_______________________________________________<br>
Labs-l mailing list<br>
<a href="mailto:Labs-l@lists.wikimedia.org" target="_blank">Labs-l@lists.wikimedia.org</a><br>
<a href="https://lists.wikimedia.org/mailman/listinfo/labs-l" target="_blank">https://lists.wikimedia.org/mailman/listinfo/labs-l</a><br>
<br></blockquote></div><br></div></div></div>
<br>_______________________________________________<br>
Labs-l mailing list<br>
<a href="mailto:Labs-l@lists.wikimedia.org" target="_blank">Labs-l@lists.wikimedia.org</a><br>
<a href="https://lists.wikimedia.org/mailman/listinfo/labs-l" target="_blank">https://lists.wikimedia.org/mailman/listinfo/labs-l</a><br>
<br></blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
Labs-l mailing list<br>
<a href="mailto:Labs-l@lists.wikimedia.org" target="_blank">Labs-l@lists.wikimedia.org</a><br>
<a href="https://lists.wikimedia.org/mailman/listinfo/labs-l" target="_blank">https://lists.wikimedia.org/mailman/listinfo/labs-l</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr"><div><font color="#000000" size="4">___________________________</font></div><div><font color="#000000" size="1"><br></font></div><div><font color="#000000" size="6">Histropedia</font></div><div><font color="#000000"><img src="http://www.histropedia.com/img/fblogo.png" height="96" width="96"></font></div><div><font color="#000000" size="4">The Timeline for all of History</font></div><div><font color="#000000"><a href="http://www.histropedia.com/" target="_blank">www.histropedia.com</a><br></font></div><div><font color="#000000"><br></font></div><div><span style="font-size:large"><font color="#000000">Follow us on:</font></span></div><div><font color="#000000"><a href="https://twitter.com/Histropedia" target="_blank">Twitter</a> <a href="https://www.facebook.com/Histropedia" target="_blank">Facebo</a><a href="https://www.facebook.com/Histropedia" target="_blank">ok</a> <a href="https://plus.google.com/u/0/b/104484373317792180682/104484373317792180682/posts" target="_blank">Google +</a> <a href="http://www.linkedin.com/company/histropedia-ltd" target="_blank">L</a><a href="http://www.linkedin.com/company/histropedia-ltd" target="_blank">inke</a><a href="http://www.linkedin.com/company/histropedia-ltd" target="_blank">dIn</a></font><br></div><div><br></div></div>
</div>
</div></div><br>_______________________________________________<br>
Labs-l mailing list<br>
<a href="mailto:Labs-l@lists.wikimedia.org" target="_blank">Labs-l@lists.wikimedia.org</a><br>
<a href="https://lists.wikimedia.org/mailman/listinfo/labs-l" target="_blank">https://lists.wikimedia.org/mailman/listinfo/labs-l</a><br>
<br></blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
Labs-l mailing list<br>
<a href="mailto:Labs-l@lists.wikimedia.org" target="_blank">Labs-l@lists.wikimedia.org</a><br>
<a href="https://lists.wikimedia.org/mailman/listinfo/labs-l" target="_blank">https://lists.wikimedia.org/mailman/listinfo/labs-l</a><br>
<br></blockquote></div><br></div></div></div>
<br>_______________________________________________<br>
Labs-l mailing list<br>
<a href="mailto:Labs-l@lists.wikimedia.org" target="_blank">Labs-l@lists.wikimedia.org</a><br>
<a href="https://lists.wikimedia.org/mailman/listinfo/labs-l" target="_blank">https://lists.wikimedia.org/mailman/listinfo/labs-l</a><br>
<br></blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
Labs-l mailing list<br>
<a href="mailto:Labs-l@lists.wikimedia.org" target="_blank">Labs-l@lists.wikimedia.org</a><br>
<a href="https://lists.wikimedia.org/mailman/listinfo/labs-l" target="_blank">https://lists.wikimedia.org/mailman/listinfo/labs-l</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr"><div><font color="#000000" size="4">___________________________</font></div><div><font color="#000000" size="1"><br></font></div><div><font color="#000000" size="6">Histropedia</font></div><div><font color="#000000"><img src="http://www.histropedia.com/img/fblogo.png" height="96" width="96"></font></div><div><font color="#000000" size="4">The Timeline for all of History</font></div><div><font color="#000000"><a href="http://www.histropedia.com/" target="_blank">www.histropedia.com</a><br></font></div><div><font color="#000000"><br></font></div><div><span style="font-size:large"><font color="#000000">Follow us on:</font></span></div><div><font color="#000000"><a href="https://twitter.com/Histropedia" target="_blank">Twitter</a> <a href="https://www.facebook.com/Histropedia" target="_blank">Facebo</a><a href="https://www.facebook.com/Histropedia" target="_blank">ok</a> <a href="https://plus.google.com/u/0/b/104484373317792180682/104484373317792180682/posts" target="_blank">Google +</a> <a href="http://www.linkedin.com/company/histropedia-ltd" target="_blank">L</a><a href="http://www.linkedin.com/company/histropedia-ltd" target="_blank">inke</a><a href="http://www.linkedin.com/company/histropedia-ltd" target="_blank">dIn</a></font><br></div><div><br></div></div>
</div>
</div></div><br>_______________________________________________<br>
Labs-l mailing list<br>
<a href="mailto:Labs-l@lists.wikimedia.org" target="_blank">Labs-l@lists.wikimedia.org</a><br>
<a href="https://lists.wikimedia.org/mailman/listinfo/labs-l" target="_blank">https://lists.wikimedia.org/mailman/listinfo/labs-l</a><br>
<br></blockquote></div><br></div>
<br>_______________________________________________<br>
Labs-l mailing list<br>
<a href="mailto:Labs-l@lists.wikimedia.org" target="_blank">Labs-l@lists.wikimedia.org</a><br>
<a href="https://lists.wikimedia.org/mailman/listinfo/labs-l" target="_blank">https://lists.wikimedia.org/mailman/listinfo/labs-l</a><br>
<br></blockquote></div>
</div></div><br>_______________________________________________<br>
Labs-l mailing list<br>
<a href="mailto:Labs-l@lists.wikimedia.org" target="_blank">Labs-l@lists.wikimedia.org</a><br>
<a href="https://lists.wikimedia.org/mailman/listinfo/labs-l" target="_blank">https://lists.wikimedia.org/mailman/listinfo/labs-l</a><br>
<br></blockquote></div><br></div>
<br>_______________________________________________<br>
Labs-l mailing list<br>
<a href="mailto:Labs-l@lists.wikimedia.org" target="_blank">Labs-l@lists.wikimedia.org</a><br>
<a href="https://lists.wikimedia.org/mailman/listinfo/labs-l" target="_blank">https://lists.wikimedia.org/mailman/listinfo/labs-l</a><br>
<br></blockquote></div>
</div></div><br>_______________________________________________<br>
Labs-l mailing list<br>
<a href="mailto:Labs-l@lists.wikimedia.org">Labs-l@lists.wikimedia.org</a><br>
<a href="https://lists.wikimedia.org/mailman/listinfo/labs-l" target="_blank">https://lists.wikimedia.org/mailman/listinfo/labs-l</a><br>
<br></blockquote></div><br></div>