<div dir="ltr">Ah, John, sorry. That's a known problem with the dumps process. It's been taking longer and longer and is harder and harder to manage because of the increased size. We weren't even able to update our reportcard lately because the process is taking so long it doesn't leave Erik Z. the time to run his analysis. I have started talking to people privately about revamping the dumps process. We need it in Analytics for some very important work that Aaron Halfaker is doing on diff analysis and folks like you need it for your work. From the start it's clear we need:<div><br></div><div>* incremental dumps</div><div>* fast access to them</div><div>* reliable bandwidth or a cluster to explore on</div><div><br></div><div>This is a million times easier said than done, but I'll keep making the case for it.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Feb 13, 2015 at 11:51 PM, John <span dir="ltr"><<a href="mailto:phoenixoverride@gmail.com" target="_blank">phoenixoverride@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div>I thought I included the link.... <a href="https://phabricator.wikimedia.org/T47646" target="_blank">https://phabricator.wikimedia.org/T47646</a> is for the two year old ticket. (that should make context a little clearer)<br><br></div>Dan the basic dumps from <a href="http://dumps.wikimedia.org" target="_blank">dumps.wikimedia.org</a> is all that I need, if you take a look at the path I provided the dumps for <br><br>20150112<br>20150204<br>20150205<br><br></div>are all missing.<br></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Feb 13, 2015 at 11:39 PM, Dan Andreescu <span dir="ltr"><<a href="mailto:dandreescu@wikimedia.org" target="_blank">dandreescu@wikimedia.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Sorry to hear, John. While I'm not ops, is there anything I can help with to get your immediate need filled? What would you do with the dump? Is labsdb a good alternative or do you already have scripts? Do you use <a href="http://dumps.wikimedia.org/" target="_blank">http://dumps.wikimedia.org/</a> ? Are the dumps you need not there? I know that site's experiencing some rate limiting but that's simply a budget issue.<div><br></div><div>I'm on the analytics team and one of my goals is to make datasets and raw data publicly available, so I appreciate your perspective and I'm sorry in advance if I can't help.</div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div>On Fri, Feb 13, 2015 at 11:23 PM, John <span dir="ltr"><<a href="mailto:phoenixoverride@gmail.com" target="_blank">phoenixoverride@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div dir="ltr"><div><div><div>I am looking at a ticket filed almost two years ago for labs to support the -latest format that the toolserver had, and guess what? Zero progress has been made. <br><br></div>This is getting sad, when labs was created it was supposed to be a replacement and improvement on the toolserver, yet a basic feature of running tools on database dumps has yet to be implemented,<br><br></div>So knowing that, I got a request to run a database scan today. I took a look at /public/dumps/public/enwiki to figure out the path to the most current dump. Guess what? we don't have it on labs. The most current dump for enwiki is from last year.... /public/dumps/public/enwiki/20141208/ <br><br></div>Something needs to happen, key, basic functionality of the toolserver is still missing, its not rocket science, yet ops has consistently failed to provide needed functionality in this area, filing tickets gets me nowhere, so the real question here is why is this still an issue and who do I need to call in order to get things resolved?<br></div>
<br></div></div>_______________________________________________<br>
Labs-l mailing list<br>
<a href="mailto:Labs-l@lists.wikimedia.org" target="_blank">Labs-l@lists.wikimedia.org</a><br>
<a href="https://lists.wikimedia.org/mailman/listinfo/labs-l" target="_blank">https://lists.wikimedia.org/mailman/listinfo/labs-l</a><br>
<br></blockquote></div><br></div>
<br>_______________________________________________<br>
Labs-l mailing list<br>
<a href="mailto:Labs-l@lists.wikimedia.org" target="_blank">Labs-l@lists.wikimedia.org</a><br>
<a href="https://lists.wikimedia.org/mailman/listinfo/labs-l" target="_blank">https://lists.wikimedia.org/mailman/listinfo/labs-l</a><br>
<br></blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
Labs-l mailing list<br>
<a href="mailto:Labs-l@lists.wikimedia.org">Labs-l@lists.wikimedia.org</a><br>
<a href="https://lists.wikimedia.org/mailman/listinfo/labs-l" target="_blank">https://lists.wikimedia.org/mailman/listinfo/labs-l</a><br>
<br></blockquote></div><br></div>