On Dec 22, 2014, at 10:57 AM, Nuria Ruiz &lt;nuria(a)wikimedia.org&gt; wrote: Adding mobile tech so they are aware, I am guessing we need to query for that data in a more efficient fashion. On Mon, Dec 22, 2014 at 4:10 AM, Sean Pringle &lt;springle(a)wikimedia.org <mailto:springle@wikimedia.org>> wrote: Had to kill queries, lest analytics-store grind to a halt and take even longer to recover. These ones: https://gerrit.wikimedia.org/r/#/c/178381/ <https://gerrit.wikimedia.org/r/#/c/178381/> On Sun, Dec 21, 2014 at 8:09 PM, Sean Pringle &lt;springle(a)wikimedia.org <mailto:springle@wikimedia.org>> wrote: This last few days analytics-store replication has started to lag by some hours. Currently s1 (enwiki) and s5 (dewiki, wikidatawiki) are most affected. Eventlogging is not lagging, due to the nicely batched writes it does now-a-days :-) There are many slow queries running from the research user on stat1003, referencing eventlogging tables like MobileWebDiffClickTracking*. I'm not sure who belongs to them, or if they're new, or if they're safe to kill, so this is mainly a heads-up email. Let us know if ops should kill stuff to let the box catch up again. BR Sean -- DBA @ WMF -- DBA @ WMF _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org <mailto:Analytics@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/analytics <https://lists.wikimedia.org/mailman/listinfo/analytics>

Thanks for the heads up -- the MobileWebDiffClickTracking schema has gotten way too big to query, it looks like. We'll figure out a way to break it up into more manageable chunks so it doesn't cause issues like this. On Dec 22, 2014, at 7:57 AM, Nuria Ruiz &lt;nuria(a)wikimedia.org&gt; wrote: Adding mobile tech so they are aware, I am guessing we need to query for that data in a more efficient fashion. On Mon, Dec 22, 2014 at 4:10 AM, Sean Pringle &lt;springle(a)wikimedia.org&gt; wrote:

https://phabricator.wikimedia.org/T76671 has been open for a while. We are just not sure how to deal with it. I'm a bit confused why MobileWebDiffClickTracking is so big. It should be tiny compared to the other tables. MobileWebClickTracking is the nasty one. For some reason I'm having issues sshing into stat1003 to explore further. On Mon, Dec 22, 2014 at 9:33 AM, Dan Garry &lt;dgarry(a)wikimedia.org&gt; wrote: Maryana, the break most writes it stat1003, safe should kill _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

On Dec 22, 2014, at 9:58 AM, Jon Robson &lt;jrobson(a)wikimedia.org&gt; wrote: It's fine. Andrew fixed issues for me. I just had to switch stat1003.wikimedia.org for stat1003.eqiad.wmnet :-) > On Mon, Dec 22, 2014 at 9:49 AM, Toby Negrin &lt;tnegrin(a)wikimedia.org&gt; wrote: > Jon -- we made some changes to stat1003 logins; I believe you need to use > the internal address now. I _thought_ we sent updated instructions to this > list but I can't find it. > > What specific issues are you having? > > -Toby > >> On Mon, Dec 22, 2014 at 9:37 AM, Jon Robson &lt;jrobson(a)wikimedia.org&gt; wrote: >> >> https://phabricator.wikimedia.org/T76671 has been open for a while. We >> are just not sure how to deal with it. >> >> I'm a bit confused why MobileWebDiffClickTracking is so big. It should >> be tiny compared to the other tables. MobileWebClickTracking is the >> nasty one. >> >> For some reason I'm having issues sshing into stat1003 to explore further. >> >>> On Mon, Dec 22, 2014 at 9:33 AM, Dan Garry &lt;dgarry(a)wikimedia.org&gt; wrote: >>> There is over six months of data in MobileWebClickTracking_5929948. >>> Maryana, >>> maybe the older data could be purged as well? That'd probably speed up >>> the >>> queries. >>> >>> Dan >>> >>> On 22 December 2014 at 09:29, Maryana Pinchuk &lt;mpinchuk(a)wikimedia.org&gt; >>> wrote: >>>> >>>> Thanks for the heads up -- the MobileWebDiffClickTracking schema has >>>> gotten way too big to query, it looks like. We'll figure out a way to >>>> break >>>> it up into more manageable chunks so it doesn't cause issues like this. >>>> >>>> >>>> >>>> On Dec 22, 2014, at 7:57 AM, Nuria Ruiz &lt;nuria(a)wikimedia.org&gt; wrote: >>>> >>>> Adding mobile tech so they are aware, I am guessing we need to query >>>> for >>>> that data in a more efficient fashion. >>>> >>>> >>>> >>>> On Mon, Dec 22, 2014 at 4:10 AM, Sean Pringle &lt;springle(a)wikimedia.org&gt; >>>> wrote: >>>>> >>>>> Had to kill queries, lest analytics-store grind to a halt and take >>>>> even >>>>> longer to recover. >>>>> >>>>> These ones: https://gerrit.wikimedia.org/r/#/c/178381/ >>>>> >>>>> On Sun, Dec 21, 2014 at 8:09 PM, Sean Pringle &lt;springle(a)wikimedia.org&gt; >>>>> wrote: >>>>>> >>>>>> This last few days analytics-store replication has started to lag by >>>>>> some hours. Currently s1 (enwiki) and s5 (dewiki, wikidatawiki) are >>>>>> most >>>>>> affected. Eventlogging is not lagging, due to the nicely batched >>>>>> writes it >>>>>> does now-a-days :-) >>>>>> >>>>>> There are many slow queries running from the research user on >>>>>> stat1003, >>>>>> referencing eventlogging tables like MobileWebDiffClickTracking*. >>>>>> >>>>>> I'm not sure who belongs to them, or if they're new, or if they're >>>>>> safe >>>>>> to kill, so this is mainly a heads-up email. Let us know if ops >>>>>> should kill >>>>>> stuff to let the box catch up again. >>>>>> >>>>>> BR >>>>>> Sean >>>>>> -- >>>>>> DBA @ WMF >>>>> >>>>> >>>>> >>>>> -- >>>>> DBA @ WMF >>>>> >>>>> _______________________________________________ >>>>> Analytics mailing list >>>>> Analytics(a)lists.wikimedia.org >>>>> https://lists.wikimedia.org/mailman/listinfo/analytics >>> >>> >>> >>> -- >>> Dan Garry >>> Associate Product Manager, Mobile Apps >>> Wikimedia Foundation >> >> _______________________________________________ >> Analytics mailing list >> Analytics(a)lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/analytics > >