This last few days analytics-store replication has started to lag by some hours. Currently s1 (enwiki) and s5 (dewiki, wikidatawiki) are most affected. Eventlogging is not lagging, due to the nicely batched writes it does now-a-days :-)
There are many slow queries running from the research user on stat1003, referencing eventlogging tables like MobileWebDiffClickTracking*.
I'm not sure who belongs to them, or if they're new, or if they're safe to kill, so this is mainly a heads-up email. Let us know if ops should kill stuff to let the box catch up again.
BR Sean
Had to kill queries, lest analytics-store grind to a halt and take even longer to recover.
These ones: https://gerrit.wikimedia.org/r/#/c/178381/
On Sun, Dec 21, 2014 at 8:09 PM, Sean Pringle springle@wikimedia.org wrote:
This last few days analytics-store replication has started to lag by some hours. Currently s1 (enwiki) and s5 (dewiki, wikidatawiki) are most affected. Eventlogging is not lagging, due to the nicely batched writes it does now-a-days :-)
There are many slow queries running from the research user on stat1003, referencing eventlogging tables like MobileWebDiffClickTracking*.
I'm not sure who belongs to them, or if they're new, or if they're safe to kill, so this is mainly a heads-up email. Let us know if ops should kill stuff to let the box catch up again.
BR Sean -- DBA @ WMF
Adding mobile tech so they are aware, I am guessing we need to query for that data in a more efficient fashion.
On Mon, Dec 22, 2014 at 4:10 AM, Sean Pringle springle@wikimedia.org wrote:
Had to kill queries, lest analytics-store grind to a halt and take even longer to recover.
These ones: https://gerrit.wikimedia.org/r/#/c/178381/
On Sun, Dec 21, 2014 at 8:09 PM, Sean Pringle springle@wikimedia.org wrote:
This last few days analytics-store replication has started to lag by some hours. Currently s1 (enwiki) and s5 (dewiki, wikidatawiki) are most affected. Eventlogging is not lagging, due to the nicely batched writes it does now-a-days :-)
There are many slow queries running from the research user on stat1003, referencing eventlogging tables like MobileWebDiffClickTracking*.
I'm not sure who belongs to them, or if they're new, or if they're safe to kill, so this is mainly a heads-up email. Let us know if ops should kill stuff to let the box catch up again.
BR Sean -- DBA @ WMF
-- DBA @ WMF
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
There is a follow up patch to the one mentioned by Sean: https://gerrit.wikimedia.org/r/#/c/181204/ https://gerrit.wikimedia.org/r/#/c/181204/
Since we split up the mobile click tracking schema into multiple schemas we had to join tables to get the data needed. However, I think we can look for older data in the historic table and only query new data that matches the new schemas.
On Dec 22, 2014, at 10:57 AM, Nuria Ruiz nuria@wikimedia.org wrote:
Adding mobile tech so they are aware, I am guessing we need to query for that data in a more efficient fashion.
On Mon, Dec 22, 2014 at 4:10 AM, Sean Pringle <springle@wikimedia.org mailto:springle@wikimedia.org> wrote: Had to kill queries, lest analytics-store grind to a halt and take even longer to recover.
These ones: https://gerrit.wikimedia.org/r/#/c/178381/ https://gerrit.wikimedia.org/r/#/c/178381/
On Sun, Dec 21, 2014 at 8:09 PM, Sean Pringle <springle@wikimedia.org mailto:springle@wikimedia.org> wrote: This last few days analytics-store replication has started to lag by some hours. Currently s1 (enwiki) and s5 (dewiki, wikidatawiki) are most affected. Eventlogging is not lagging, due to the nicely batched writes it does now-a-days :-)
There are many slow queries running from the research user on stat1003, referencing eventlogging tables like MobileWebDiffClickTracking*.
I'm not sure who belongs to them, or if they're new, or if they're safe to kill, so this is mainly a heads-up email. Let us know if ops should kill stuff to let the box catch up again.
BR Sean -- DBA @ WMF
-- DBA @ WMF
Analytics mailing list Analytics@lists.wikimedia.org mailto:Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics https://lists.wikimedia.org/mailman/listinfo/analytics
Thanks for the heads up -- the MobileWebDiffClickTracking schema has gotten way too big to query, it looks like. We'll figure out a way to break it up into more manageable chunks so it doesn't cause issues like this.
On Dec 22, 2014, at 7:57 AM, Nuria Ruiz nuria@wikimedia.org wrote:
Adding mobile tech so they are aware, I am guessing we need to query for that data in a more efficient fashion.
On Mon, Dec 22, 2014 at 4:10 AM, Sean Pringle springle@wikimedia.org wrote: Had to kill queries, lest analytics-store grind to a halt and take even longer to recover.
These ones: https://gerrit.wikimedia.org/r/#/c/178381/
On Sun, Dec 21, 2014 at 8:09 PM, Sean Pringle springle@wikimedia.org wrote: This last few days analytics-store replication has started to lag by some hours. Currently s1 (enwiki) and s5 (dewiki, wikidatawiki) are most affected. Eventlogging is not lagging, due to the nicely batched writes it does now-a-days :-)
There are many slow queries running from the research user on stat1003, referencing eventlogging tables like MobileWebDiffClickTracking*.
I'm not sure who belongs to them, or if they're new, or if they're safe to kill, so this is mainly a heads-up email. Let us know if ops should kill stuff to let the box catch up again.
BR Sean -- DBA @ WMF
-- DBA @ WMF
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
There is over six months of data in MobileWebClickTracking_5929948. Maryana, maybe the older data could be purged as well? That'd probably speed up the queries.
Dan
On 22 December 2014 at 09:29, Maryana Pinchuk mpinchuk@wikimedia.org wrote:
Thanks for the heads up -- the MobileWebDiffClickTracking schema has gotten way too big to query, it looks like. We'll figure out a way to break it up into more manageable chunks so it doesn't cause issues like this.
On Dec 22, 2014, at 7:57 AM, Nuria Ruiz nuria@wikimedia.org wrote:
Adding mobile tech so they are aware, I am guessing we need to query for that data in a more efficient fashion.
On Mon, Dec 22, 2014 at 4:10 AM, Sean Pringle springle@wikimedia.org wrote:
Had to kill queries, lest analytics-store grind to a halt and take even longer to recover.
These ones: https://gerrit.wikimedia.org/r/#/c/178381/
On Sun, Dec 21, 2014 at 8:09 PM, Sean Pringle springle@wikimedia.org wrote:
This last few days analytics-store replication has started to lag by some hours. Currently s1 (enwiki) and s5 (dewiki, wikidatawiki) are most affected. Eventlogging is not lagging, due to the nicely batched writes it does now-a-days :-)
There are many slow queries running from the research user on stat1003, referencing eventlogging tables like MobileWebDiffClickTracking*.
I'm not sure who belongs to them, or if they're new, or if they're safe to kill, so this is mainly a heads-up email. Let us know if ops should kill stuff to let the box catch up again.
BR Sean -- DBA @ WMF
-- DBA @ WMF
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
https://phabricator.wikimedia.org/T76671 has been open for a while. We are just not sure how to deal with it.
I'm a bit confused why MobileWebDiffClickTracking is so big. It should be tiny compared to the other tables. MobileWebClickTracking is the nasty one.
For some reason I'm having issues sshing into stat1003 to explore further.
On Mon, Dec 22, 2014 at 9:33 AM, Dan Garry dgarry@wikimedia.org wrote:
There is over six months of data in MobileWebClickTracking_5929948. Maryana, maybe the older data could be purged as well? That'd probably speed up the queries.
Dan
On 22 December 2014 at 09:29, Maryana Pinchuk mpinchuk@wikimedia.org wrote:
Thanks for the heads up -- the MobileWebDiffClickTracking schema has gotten way too big to query, it looks like. We'll figure out a way to break it up into more manageable chunks so it doesn't cause issues like this.
On Dec 22, 2014, at 7:57 AM, Nuria Ruiz nuria@wikimedia.org wrote:
Adding mobile tech so they are aware, I am guessing we need to query for that data in a more efficient fashion.
On Mon, Dec 22, 2014 at 4:10 AM, Sean Pringle springle@wikimedia.org wrote:
Had to kill queries, lest analytics-store grind to a halt and take even longer to recover.
These ones: https://gerrit.wikimedia.org/r/#/c/178381/
On Sun, Dec 21, 2014 at 8:09 PM, Sean Pringle springle@wikimedia.org wrote:
This last few days analytics-store replication has started to lag by some hours. Currently s1 (enwiki) and s5 (dewiki, wikidatawiki) are most affected. Eventlogging is not lagging, due to the nicely batched writes it does now-a-days :-)
There are many slow queries running from the research user on stat1003, referencing eventlogging tables like MobileWebDiffClickTracking*.
I'm not sure who belongs to them, or if they're new, or if they're safe to kill, so this is mainly a heads-up email. Let us know if ops should kill stuff to let the box catch up again.
BR Sean -- DBA @ WMF
-- DBA @ WMF
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
-- Dan Garry Associate Product Manager, Mobile Apps Wikimedia Foundation
Jon -- we made some changes to stat1003 logins; I believe you need to use the internal address now. I _thought_ we sent updated instructions to this list but I can't find it.
What specific issues are you having?
-Toby
On Mon, Dec 22, 2014 at 9:37 AM, Jon Robson jrobson@wikimedia.org wrote:
https://phabricator.wikimedia.org/T76671 has been open for a while. We are just not sure how to deal with it.
I'm a bit confused why MobileWebDiffClickTracking is so big. It should be tiny compared to the other tables. MobileWebClickTracking is the nasty one.
For some reason I'm having issues sshing into stat1003 to explore further.
On Mon, Dec 22, 2014 at 9:33 AM, Dan Garry dgarry@wikimedia.org wrote:
There is over six months of data in MobileWebClickTracking_5929948.
Maryana,
maybe the older data could be purged as well? That'd probably speed up
the
queries.
Dan
On 22 December 2014 at 09:29, Maryana Pinchuk mpinchuk@wikimedia.org wrote:
Thanks for the heads up -- the MobileWebDiffClickTracking schema has gotten way too big to query, it looks like. We'll figure out a way to
break
it up into more manageable chunks so it doesn't cause issues like this.
On Dec 22, 2014, at 7:57 AM, Nuria Ruiz nuria@wikimedia.org wrote:
Adding mobile tech so they are aware, I am guessing we need to query for that data in a more efficient fashion.
On Mon, Dec 22, 2014 at 4:10 AM, Sean Pringle springle@wikimedia.org wrote:
Had to kill queries, lest analytics-store grind to a halt and take even longer to recover.
These ones: https://gerrit.wikimedia.org/r/#/c/178381/
On Sun, Dec 21, 2014 at 8:09 PM, Sean Pringle springle@wikimedia.org wrote:
This last few days analytics-store replication has started to lag by some hours. Currently s1 (enwiki) and s5 (dewiki, wikidatawiki) are
most
affected. Eventlogging is not lagging, due to the nicely batched
writes it
does now-a-days :-)
There are many slow queries running from the research user on
stat1003,
referencing eventlogging tables like MobileWebDiffClickTracking*.
I'm not sure who belongs to them, or if they're new, or if they're
safe
to kill, so this is mainly a heads-up email. Let us know if ops
should kill
stuff to let the box catch up again.
BR Sean -- DBA @ WMF
-- DBA @ WMF
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
-- Dan Garry Associate Product Manager, Mobile Apps Wikimedia Foundation
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
It's fine. Andrew fixed issues for me. I just had to switch stat1003.wikimedia.org for stat1003.eqiad.wmnet :-)
On Mon, Dec 22, 2014 at 9:49 AM, Toby Negrin tnegrin@wikimedia.org wrote:
Jon -- we made some changes to stat1003 logins; I believe you need to use the internal address now. I _thought_ we sent updated instructions to this list but I can't find it.
What specific issues are you having?
-Toby
On Mon, Dec 22, 2014 at 9:37 AM, Jon Robson jrobson@wikimedia.org wrote:
https://phabricator.wikimedia.org/T76671 has been open for a while. We are just not sure how to deal with it.
I'm a bit confused why MobileWebDiffClickTracking is so big. It should be tiny compared to the other tables. MobileWebClickTracking is the nasty one.
For some reason I'm having issues sshing into stat1003 to explore further.
On Mon, Dec 22, 2014 at 9:33 AM, Dan Garry dgarry@wikimedia.org wrote:
There is over six months of data in MobileWebClickTracking_5929948. Maryana, maybe the older data could be purged as well? That'd probably speed up the queries.
Dan
On 22 December 2014 at 09:29, Maryana Pinchuk mpinchuk@wikimedia.org wrote:
Thanks for the heads up -- the MobileWebDiffClickTracking schema has gotten way too big to query, it looks like. We'll figure out a way to break it up into more manageable chunks so it doesn't cause issues like this.
On Dec 22, 2014, at 7:57 AM, Nuria Ruiz nuria@wikimedia.org wrote:
Adding mobile tech so they are aware, I am guessing we need to query for that data in a more efficient fashion.
On Mon, Dec 22, 2014 at 4:10 AM, Sean Pringle springle@wikimedia.org wrote:
Had to kill queries, lest analytics-store grind to a halt and take even longer to recover.
These ones: https://gerrit.wikimedia.org/r/#/c/178381/
On Sun, Dec 21, 2014 at 8:09 PM, Sean Pringle springle@wikimedia.org wrote:
This last few days analytics-store replication has started to lag by some hours. Currently s1 (enwiki) and s5 (dewiki, wikidatawiki) are most affected. Eventlogging is not lagging, due to the nicely batched writes it does now-a-days :-)
There are many slow queries running from the research user on stat1003, referencing eventlogging tables like MobileWebDiffClickTracking*.
I'm not sure who belongs to them, or if they're new, or if they're safe to kill, so this is mainly a heads-up email. Let us know if ops should kill stuff to let the box catch up again.
BR Sean -- DBA @ WMF
-- DBA @ WMF
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
-- Dan Garry Associate Product Manager, Mobile Apps Wikimedia Foundation
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Great - thanks all.
On Dec 22, 2014, at 9:58 AM, Jon Robson jrobson@wikimedia.org wrote:
It's fine. Andrew fixed issues for me. I just had to switch stat1003.wikimedia.org for stat1003.eqiad.wmnet :-)
On Mon, Dec 22, 2014 at 9:49 AM, Toby Negrin tnegrin@wikimedia.org wrote: Jon -- we made some changes to stat1003 logins; I believe you need to use the internal address now. I _thought_ we sent updated instructions to this list but I can't find it.
What specific issues are you having?
-Toby
On Mon, Dec 22, 2014 at 9:37 AM, Jon Robson jrobson@wikimedia.org wrote:
https://phabricator.wikimedia.org/T76671 has been open for a while. We are just not sure how to deal with it.
I'm a bit confused why MobileWebDiffClickTracking is so big. It should be tiny compared to the other tables. MobileWebClickTracking is the nasty one.
For some reason I'm having issues sshing into stat1003 to explore further.
On Mon, Dec 22, 2014 at 9:33 AM, Dan Garry dgarry@wikimedia.org wrote: There is over six months of data in MobileWebClickTracking_5929948. Maryana, maybe the older data could be purged as well? That'd probably speed up the queries.
Dan
On 22 December 2014 at 09:29, Maryana Pinchuk mpinchuk@wikimedia.org wrote:
Thanks for the heads up -- the MobileWebDiffClickTracking schema has gotten way too big to query, it looks like. We'll figure out a way to break it up into more manageable chunks so it doesn't cause issues like this.
On Dec 22, 2014, at 7:57 AM, Nuria Ruiz nuria@wikimedia.org wrote:
Adding mobile tech so they are aware, I am guessing we need to query for that data in a more efficient fashion.
On Mon, Dec 22, 2014 at 4:10 AM, Sean Pringle springle@wikimedia.org wrote:
Had to kill queries, lest analytics-store grind to a halt and take even longer to recover.
These ones: https://gerrit.wikimedia.org/r/#/c/178381/
On Sun, Dec 21, 2014 at 8:09 PM, Sean Pringle springle@wikimedia.org wrote: > > This last few days analytics-store replication has started to lag by > some hours. Currently s1 (enwiki) and s5 (dewiki, wikidatawiki) are > most > affected. Eventlogging is not lagging, due to the nicely batched > writes it > does now-a-days :-) > > There are many slow queries running from the research user on > stat1003, > referencing eventlogging tables like MobileWebDiffClickTracking*. > > I'm not sure who belongs to them, or if they're new, or if they're > safe > to kill, so this is mainly a heads-up email. Let us know if ops > should kill > stuff to let the box catch up again. > > BR > Sean > -- > DBA @ WMF
-- DBA @ WMF
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
-- Dan Garry Associate Product Manager, Mobile Apps Wikimedia Foundation
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics