Gerco:
On May 16th we lower the sampling rate of media viewer events as the event rate was ~170 events per second. It looks like as of a week and a half ago we are again at that rate.
Please see:
https://ganglia.wikimedia.org/latest/graph.php?r=month&z=xlarge&c=Mi...
This means that Media Viewer is generating about 15 million rows a day on EL database, a data flow that seems quite high for our capacity to analyze it.
Is this a mistake? Should sampling rates be lowered again?
So you know, right now media viewer is sampling more than twice as much the rest of the teams at the foundation together. If every team sampled at this ratio the system will go down. Now, at this time, event logging is not at risk of going down but the replication is affected.
Thanks,
Nuria
It's now been made configurable and the sampling rate has been lowered. While we're back to a similar rate as last time, we're actually measuring close to 3 times the activity we were back then, thanks to the sampling.
I'll see on what wikis/metrics we can lower the rates further while keeping meaningful data and I'll have a config change ready for tomorrow's swat window. I hadn't done it yet because I wanted to study the data to make less of a wild guess this time.
For today's launch to enwiki/dewiki I have already set the rates to be the same as the one currently applied to commons.
On Tue, Jun 3, 2014 at 2:17 PM, Nuria Ruiz nuria@wikimedia.org wrote:
Gerco:
On May 16th we lower the sampling rate of media viewer events as the event rate was ~170 events per second. It looks like as of a week and a half ago we are again at that rate.
Please see:
https://ganglia.wikimedia.org/latest/graph.php?r=month&z=xlarge&c=Mi...
This means that Media Viewer is generating about 15 million rows a day on EL database, a data flow that seems quite high for our capacity to analyze it.
Is this a mistake? Should sampling rates be lowered again?
So you know, right now media viewer is sampling more than twice as much the rest of the teams at the foundation together. If every team sampled at this ratio the system will go down. Now, at this time, event logging is not at risk of going down but the replication is affected.
Thanks,
Nuria
Related changesets, for reference:
https://gerrit.wikimedia.org/r/#/c/134064/ https://gerrit.wikimedia.org/r/#/c/134343/ https://gerrit.wikimedia.org/r/#/c/134804/ https://gerrit.wikimedia.org/r/#/c/134837/ https://gerrit.wikimedia.org/r/#/c/134065/ https://gerrit.wikimedia.org/r/#/c/136717/
On Tue, Jun 3, 2014 at 6:15 PM, Gilles Dubuc gilles@wikimedia.org wrote:
It's now been made configurable and the sampling rate has been lowered. While we're back to a similar rate as last time, we're actually measuring close to 3 times the activity we were back then, thanks to the sampling.
I'll see on what wikis/metrics we can lower the rates further while keeping meaningful data and I'll have a config change ready for tomorrow's swat window. I hadn't done it yet because I wanted to study the data to make less of a wild guess this time.
For today's launch to enwiki/dewiki I have already set the rates to be the same as the one currently applied to commons.
On Tue, Jun 3, 2014 at 2:17 PM, Nuria Ruiz nuria@wikimedia.org wrote:
Gerco:
On May 16th we lower the sampling rate of media viewer events as the event rate was ~170 events per second. It looks like as of a week and a half ago we are again at that rate.
Please see:
https://ganglia.wikimedia.org/latest/graph.php?r=month&z=xlarge&c=Mi...
This means that Media Viewer is generating about 15 million rows a day on EL database, a data flow that seems quite high for our capacity to analyze it.
Is this a mistake? Should sampling rates be lowered again?
So you know, right now media viewer is sampling more than twice as much the rest of the teams at the foundation together. If every team sampled at this ratio the system will go down. Now, at this time, event logging is not at risk of going down but the replication is affected.
Thanks,
Nuria
the replication is affected.
I take this back, as Christian mentioned there are other issues that might be affecting replication.
I'll see on what wikis/metrics we can lower the rates further while
keeping meaningful data and I'll have a config >change ready for tomorrow's swat window. I hadn't done it yet because I wanted to study the data to make less of a >wild guess this time. Sounds good, I think logging a lot to start with, studying the data and later lowering rates is what makes most sense. Thanks for the prompt response.
On Tue, Jun 3, 2014 at 6:19 PM, Gilles Dubuc gilles@wikimedia.org wrote:
Related changesets, for reference:
https://gerrit.wikimedia.org/r/#/c/134064/ https://gerrit.wikimedia.org/r/#/c/134343/ https://gerrit.wikimedia.org/r/#/c/134804/ https://gerrit.wikimedia.org/r/#/c/134837/ https://gerrit.wikimedia.org/r/#/c/134065/ https://gerrit.wikimedia.org/r/#/c/136717/
On Tue, Jun 3, 2014 at 6:15 PM, Gilles Dubuc gilles@wikimedia.org wrote:
It's now been made configurable and the sampling rate has been lowered. While we're back to a similar rate as last time, we're actually measuring close to 3 times the activity we were back then, thanks to the sampling.
I'll see on what wikis/metrics we can lower the rates further while keeping meaningful data and I'll have a config change ready for tomorrow's swat window. I hadn't done it yet because I wanted to study the data to make less of a wild guess this time.
For today's launch to enwiki/dewiki I have already set the rates to be the same as the one currently applied to commons.
On Tue, Jun 3, 2014 at 2:17 PM, Nuria Ruiz nuria@wikimedia.org wrote:
Gerco:
On May 16th we lower the sampling rate of media viewer events as the event rate was ~170 events per second. It looks like as of a week and a half ago we are again at that rate.
Please see:
https://ganglia.wikimedia.org/latest/graph.php?r=month&z=xlarge&c=Mi...
This means that Media Viewer is generating about 15 million rows a day on EL database, a data flow that seems quite high for our capacity to analyze it.
Is this a mistake? Should sampling rates be lowered again?
So you know, right now media viewer is sampling more than twice as much the rest of the teams at the foundation together. If every team sampled at this ratio the system will go down. Now, at this time, event logging is not at risk of going down but the replication is affected.
Thanks,
Nuria
On Tue, Jun 03, 2014 at 06:15:46PM +0200, Gilles Dubuc wrote:
For today's launch to enwiki/dewiki I have already set the rates to be the same as the one currently applied to commons.
Do we foresee 1/100 being enough of a cut for the launch to en and de? It seems like we'll be getting a massive uptake in the event count...
Nuria, thoughts?
Do we foresee 1/100 being enough of a cut for the launch to en and de?
According to http://reportcard.wmflabs.org/ dewiki gets about as much traffic as jawiki, which has the same sampling factors. enwiki is about 8 times more pageviews, so yes I guess we could make the sampling rate an order of magnitude smaller for enwiki on the high traffic metrics. I'll write the changeset for it, so that you can include it in the launch.
On Tue, Jun 3, 2014 at 9:45 PM, Mark Holmquist mtraceur@member.fsf.org wrote:
On Tue, Jun 03, 2014 at 06:15:46PM +0200, Gilles Dubuc wrote:
For today's launch to enwiki/dewiki I have already set the rates to be
the
same as the one currently applied to commons.
Do we foresee 1/100 being enough of a cut for the launch to en and de? It seems like we'll be getting a massive uptake in the event count...
Nuria, thoughts?
-- Mark Holmquist Software Engineer, Multimedia Wikimedia Foundation mtraceur@member.fsf.org https://wikimediafoundation.org/wiki/User:MHolmquist
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux)
iQIcBAEBAgAGBQJTjiXhAAoJEEPl+wghkjzxDr8P/Ro7YE+UE3ZA2NYI44pv6c9+ 65jIMJ5zhahw+PTdrB2D6F8WZuagE0+JraE/TJwtRarXyIi7lrHUVqpTxbsiAjW9 oPwyHoHYzs8W2BgTip8I9uPTK3M5MjeAWT8dWAJhG271aR5YgjG6Emz68wl0pZTd qaIVyTJFU9h/9IsD5rcQGQYXcvvBin2miUbPN5Iy0NVjJcMeci24ijEDkCNTYqdx 9udEswIvsBf48NhBCxakPNICBJp2EQIe29L8RSCn+x46LNjNMiL+Qf3dxZBPCIg7 ecx8qoU2jQbqveOVm2e2ygUYNXwDrugQmcKJbu6CZLhQiWbINu+5pforXP+P3mud 6dqYvPJsCMSoj+0x65szDgJXbM5pL7VRzd1Z9jZoWPB1I9lT552ZDyrcfM7Kf/gf WLNn9mOiujzvAuShIUfqpJuSRoEvqFF2B7NN/DsUhfrAE8wkYwr/9McLDnSKRClV L7RkU+oGVfdm+jRc2akT4aH7ngHIFpUN+KbQ8aUHrXAGGCz95mR9drsnyaBQl3vF sKuXxIF/ipjb9z71+w6nQ5XXofbFzCf/7p/NAmzY7KAmhEqQQC70RcVvl50VnluU 8pRzwsMSlrl1rUh19S/mrBPpcWut6IHi+RuYs/vcibKA5f9gkFpKfZE1ICvylPBu TZ4UC3ZcB4adCMg3XLAf =JjAR -----END PGP SIGNATURE-----
Multimedia mailing list Multimedia@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/multimedia
https://gerrit.wikimedia.org/r/137155
On Tue, Jun 3, 2014 at 10:15 PM, Gilles Dubuc gilles@wikimedia.org wrote:
Do we foresee 1/100 being enough of a cut for the launch to en and de?
According to http://reportcard.wmflabs.org/ dewiki gets about as much traffic as jawiki, which has the same sampling factors. enwiki is about 8 times more pageviews, so yes I guess we could make the sampling rate an order of magnitude smaller for enwiki on the high traffic metrics. I'll write the changeset for it, so that you can include it in the launch.
On Tue, Jun 3, 2014 at 9:45 PM, Mark Holmquist mtraceur@member.fsf.org wrote:
On Tue, Jun 03, 2014 at 06:15:46PM +0200, Gilles Dubuc wrote:
For today's launch to enwiki/dewiki I have already set the rates to be
the
same as the one currently applied to commons.
Do we foresee 1/100 being enough of a cut for the launch to en and de? It seems like we'll be getting a massive uptake in the event count...
Nuria, thoughts?
-- Mark Holmquist Software Engineer, Multimedia Wikimedia Foundation mtraceur@member.fsf.org https://wikimediafoundation.org/wiki/User:MHolmquist
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux)
iQIcBAEBAgAGBQJTjiXhAAoJEEPl+wghkjzxDr8P/Ro7YE+UE3ZA2NYI44pv6c9+ 65jIMJ5zhahw+PTdrB2D6F8WZuagE0+JraE/TJwtRarXyIi7lrHUVqpTxbsiAjW9 oPwyHoHYzs8W2BgTip8I9uPTK3M5MjeAWT8dWAJhG271aR5YgjG6Emz68wl0pZTd qaIVyTJFU9h/9IsD5rcQGQYXcvvBin2miUbPN5Iy0NVjJcMeci24ijEDkCNTYqdx 9udEswIvsBf48NhBCxakPNICBJp2EQIe29L8RSCn+x46LNjNMiL+Qf3dxZBPCIg7 ecx8qoU2jQbqveOVm2e2ygUYNXwDrugQmcKJbu6CZLhQiWbINu+5pforXP+P3mud 6dqYvPJsCMSoj+0x65szDgJXbM5pL7VRzd1Z9jZoWPB1I9lT552ZDyrcfM7Kf/gf WLNn9mOiujzvAuShIUfqpJuSRoEvqFF2B7NN/DsUhfrAE8wkYwr/9McLDnSKRClV L7RkU+oGVfdm+jRc2akT4aH7ngHIFpUN+KbQ8aUHrXAGGCz95mR9drsnyaBQl3vF sKuXxIF/ipjb9z71+w6nQ5XXofbFzCf/7p/NAmzY7KAmhEqQQC70RcVvl50VnluU 8pRzwsMSlrl1rUh19S/mrBPpcWut6IHi+RuYs/vcibKA5f9gkFpKfZE1ICvylPBu TZ4UC3ZcB4adCMg3XLAf =JjAR -----END PGP SIGNATURE-----
Multimedia mailing list Multimedia@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/multimedia
Thanks Gilles and Mark, much appreciated!
Fabrice
On Jun 3, 2014, at 1:20 PM, Gilles Dubuc gilles@wikimedia.org wrote:
https://gerrit.wikimedia.org/r/137155
On Tue, Jun 3, 2014 at 10:15 PM, Gilles Dubuc gilles@wikimedia.org wrote: Do we foresee 1/100 being enough of a cut for the launch to en and de?
According to http://reportcard.wmflabs.org/ dewiki gets about as much traffic as jawiki, which has the same sampling factors. enwiki is about 8 times more pageviews, so yes I guess we could make the sampling rate an order of magnitude smaller for enwiki on the high traffic metrics. I'll write the changeset for it, so that you can include it in the launch.
On Tue, Jun 3, 2014 at 9:45 PM, Mark Holmquist mtraceur@member.fsf.org wrote: On Tue, Jun 03, 2014 at 06:15:46PM +0200, Gilles Dubuc wrote:
For today's launch to enwiki/dewiki I have already set the rates to be the same as the one currently applied to commons.
Do we foresee 1/100 being enough of a cut for the launch to en and de? It seems like we'll be getting a massive uptake in the event count...
Nuria, thoughts?
-- Mark Holmquist Software Engineer, Multimedia Wikimedia Foundation mtraceur@member.fsf.org https://wikimediafoundation.org/wiki/User:MHolmquist
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux)
iQIcBAEBAgAGBQJTjiXhAAoJEEPl+wghkjzxDr8P/Ro7YE+UE3ZA2NYI44pv6c9+ 65jIMJ5zhahw+PTdrB2D6F8WZuagE0+JraE/TJwtRarXyIi7lrHUVqpTxbsiAjW9 oPwyHoHYzs8W2BgTip8I9uPTK3M5MjeAWT8dWAJhG271aR5YgjG6Emz68wl0pZTd qaIVyTJFU9h/9IsD5rcQGQYXcvvBin2miUbPN5Iy0NVjJcMeci24ijEDkCNTYqdx 9udEswIvsBf48NhBCxakPNICBJp2EQIe29L8RSCn+x46LNjNMiL+Qf3dxZBPCIg7 ecx8qoU2jQbqveOVm2e2ygUYNXwDrugQmcKJbu6CZLhQiWbINu+5pforXP+P3mud 6dqYvPJsCMSoj+0x65szDgJXbM5pL7VRzd1Z9jZoWPB1I9lT552ZDyrcfM7Kf/gf WLNn9mOiujzvAuShIUfqpJuSRoEvqFF2B7NN/DsUhfrAE8wkYwr/9McLDnSKRClV L7RkU+oGVfdm+jRc2akT4aH7ngHIFpUN+KbQ8aUHrXAGGCz95mR9drsnyaBQl3vF sKuXxIF/ipjb9z71+w6nQ5XXofbFzCf/7p/NAmzY7KAmhEqQQC70RcVvl50VnluU 8pRzwsMSlrl1rUh19S/mrBPpcWut6IHi+RuYs/vcibKA5f9gkFpKfZE1ICvylPBu TZ4UC3ZcB4adCMg3XLAf =JjAR -----END PGP SIGNATURE-----
Multimedia mailing list Multimedia@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/multimedia
Multimedia mailing list Multimedia@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/multimedia
_______________________________
Fabrice Florin Product Manager Wikimedia Foundation
https://ganglia.wikimedia.org/latest/graph.php?r=month&z=xlarge&c=Mi...
Looks like my finer-grained factor tweaking was reasonably effective, considering that we've just doubled Media Viewer traffic with the enwiki+dewiki launch. The EL usage still high in the grand scheme of things, but I'd like to have it running with the new values for a bit to see how much lower I can take it. I'm going on vacation for a week tonight, when I'm back I'll reduce the EL usage further in a new pass of studying the data.
On Tue, Jun 3, 2014 at 2:17 PM, Nuria Ruiz nuria@wikimedia.org wrote:
Gerco:
On May 16th we lower the sampling rate of media viewer events as the event rate was ~170 events per second. It looks like as of a week and a half ago we are again at that rate.
Please see:
https://ganglia.wikimedia.org/latest/graph.php?r=month&z=xlarge&c=Mi...
This means that Media Viewer is generating about 15 million rows a day on EL database, a data flow that seems quite high for our capacity to analyze it.
Is this a mistake? Should sampling rates be lowered again?
So you know, right now media viewer is sampling more than twice as much the rest of the teams at the foundation together. If every team sampled at this ratio the system will go down. Now, at this time, event logging is not at risk of going down but the replication is affected.
Thanks,
Nuria
The EL usage still high in the grand scheme of things, but I'd like to have it running with the new values for a bit to see how
much lower
I can take it. I'm going on vacation for a week tonight, when I'm back
I'll reduce the EL usage further in a new pass of >studying the data This makes total sense, thanks again.
On Fri, Jun 6, 2014 at 11:01 AM, Gilles Dubuc gilles@wikimedia.org wrote:
https://ganglia.wikimedia.org/latest/graph.php?r=month&z=xlarge&c=Mi...
Looks like my finer-grained factor tweaking was reasonably effective, considering that we've just doubled Media Viewer traffic with the enwiki+dewiki launch. The EL usage still high in the grand scheme of things, but I'd like to have it running with the new values for a bit to see how much lower I can take it. I'm going on vacation for a week tonight, when I'm back I'll reduce the EL usage further in a new pass of studying the data.
On Tue, Jun 3, 2014 at 2:17 PM, Nuria Ruiz nuria@wikimedia.org wrote:
Gerco:
On May 16th we lower the sampling rate of media viewer events as the event rate was ~170 events per second. It looks like as of a week and a half ago we are again at that rate.
Please see:
https://ganglia.wikimedia.org/latest/graph.php?r=month&z=xlarge&c=Mi...
This means that Media Viewer is generating about 15 million rows a day on EL database, a data flow that seems quite high for our capacity to analyze it.
Is this a mistake? Should sampling rates be lowered again?
So you know, right now media viewer is sampling more than twice as much the rest of the teams at the foundation together. If every team sampled at this ratio the system will go down. Now, at this time, event logging is not at risk of going down but the replication is affected.
Thanks,
Nuria
multimedia@lists.wikimedia.org