Hi folks,
As you've probably heard, last week we deployed ulsfo in production, reducing latency for Oceania, East/Southeast Asia & US/Canada pacific/west coast states. My estimation of the user base affected by this is 360 million users (as in, Internet users, not Wikipedia users).
I was wondering if you have an easy way to measure and plot the impact in page load time, perhaps using Navigation Timing data?
The operations team has spent a considerable amount of time and money to deploy ulsfo and I believe it'd be useful for us and the organization at large to be able to quantify this effort.
The exact dates of the rollout by country/region codes can be found in operations/dns' git history: https://git.wikimedia.org/summary/?r=operations/dns.git (the commits should be self-explanatory, but I'd be happy to clarify if needed)
Thanks! Faidon
Faidon Liambotis, 10/02/2014 15:37:
I was wondering if you have an easy way to measure and plot the impact in page load time, perhaps using Navigation Timing data?
+10^9. I was rather excited to see ulsfo back up and I tried to quickly figure some correlation out of https://gdash.wikimedia.org/dashboards/frontend/ , but I failed. I think what's needed here is mostly some brilliant idea on how to visualise / graph the data available there by region (and maybe also by project / kind of editor e.g. unregistered vs. registered vs. sysop) without making thousands graphs for each of the combinations. Or just one graph for the region moved to ulsfo to start with of course. :)
Nemo
I was wondering if you have an easy way to measure and plot the impact
in page load time, perhaps using Navigation Timing data?
+10^9. I was rather excited to see ulsfo back up and I tried to quickly figure some correlation out of https://gdash.wikimedia.org/ dashboards/frontend/ , but I failed. I think what's needed here is mostly some brilliant idea on how to visualise / graph the data available there by region (and maybe also by project / kind of editor e.g. unregistered vs. registered vs. sysop) without making thousands graphs for each of the combinations. Or just one graph for the region moved to ulsfo to start with of course. :)
Does the NavigationTiming data have the hostname of the varnish that served the request, and do we have data on which hostname served which geographic area at a particular time?
If so, we can generate a before and after average performance map by region served, to see if there are any differences.
The schema contains "originCountry" that should enable us to filter in this way.
I have two points:
*Browser bias: *I was under the impression that Nav timing was only available for modern browsers, so that might affect a bit of a bias into the situation.
*Browser cache: *Browser cache is going to reduce any gains we might see due to reduced latency or increased bandwidth. It would be nice if we could compare the speed changes of cached/non-cached page/asset loading. I don't think that the current schema will allow us to do this. Despite this, we should still be able to get a good sense for how cached/non-cached page loads were affected in general.
-Aaron
On Mon, Feb 10, 2014 at 9:43 AM, Dan Andreescu dandreescu@wikimedia.orgwrote:
I was wondering if you have an easy way to measure and plot the impact
in page load time, perhaps using Navigation Timing data?
+10^9. I was rather excited to see ulsfo back up and I tried to quickly figure some correlation out of https://gdash.wikimedia.org/ dashboards/frontend/ , but I failed. I think what's needed here is mostly some brilliant idea on how to visualise / graph the data available there by region (and maybe also by project / kind of editor e.g. unregistered vs. registered vs. sysop) without making thousands graphs for each of the combinations. Or just one graph for the region moved to ulsfo to start with of course. :)
Does the NavigationTiming data have the hostname of the varnish that served the request, and do we have data on which hostname served which geographic area at a particular time?
If so, we can generate a before and after average performance map by region served, to see if there are any differences.
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
I've created a card in trello to help us track this request. https://trello.com/c/dNODZnr0/108-measuring-ulsfo-s-impact-on-site-performan...
On Mon, Feb 10, 2014 at 10:02 AM, Aaron Halfaker ahalfaker@wikimedia.orgwrote:
The schema contains "originCountry" that should enable us to filter in this way.
I have two points:
*Browser bias: *I was under the impression that Nav timing was only available for modern browsers, so that might affect a bit of a bias into the situation.
*Browser cache: *Browser cache is going to reduce any gains we might see due to reduced latency or increased bandwidth. It would be nice if we could compare the speed changes of cached/non-cached page/asset loading. I don't think that the current schema will allow us to do this. Despite this, we should still be able to get a good sense for how cached/non-cached page loads were affected in general.
-Aaron
On Mon, Feb 10, 2014 at 9:43 AM, Dan Andreescu dandreescu@wikimedia.orgwrote:
I was wondering if you have an easy way to measure and plot the impact
in page load time, perhaps using Navigation Timing data?
+10^9. I was rather excited to see ulsfo back up and I tried to quickly figure some correlation out of https://gdash.wikimedia.org/ dashboards/frontend/ , but I failed. I think what's needed here is mostly some brilliant idea on how to visualise / graph the data available there by region (and maybe also by project / kind of editor e.g. unregistered vs. registered vs. sysop) without making thousands graphs for each of the combinations. Or just one graph for the region moved to ulsfo to start with of course. :)
Does the NavigationTiming data have the hostname of the varnish that served the request, and do we have data on which hostname served which geographic area at a particular time?
If so, we can generate a before and after average performance map by region served, to see if there are any differences.
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
I've created a card in trello to help us track this request. https://trello.com/c/dNODZnr0/108-measuring-ulsfo-s-impact-on-site-performan...
Thanks Aaron, I've subscribed and will help with any tricky visualization stuff
*Browser bias: *I was under the impression that Nav timing was only available for modern browsers, so that might affect a bit of a bias into
the situation.
It is well supported for IE >8 . The notable exception are iOS devices. http://caniuse.com/#search=navigation For details: http://www.html5rocks.com/en/tutorials/webperformance/basics/
*Browser cache: *Browser cache is going to reduce any gains we might see
due to reduced
latency or increased bandwidth. It would be nice if we could compare the speed changes of cached/non-cached page/asset loading.
Actually this is not an issue if you can assume cache hits/misses are the same before and after deploying ulsfo. What you care about in this case are the network times more than anything, as Faidon's changes do not affect the rendering of the page once all assets are downloaded. Now, if his changes also have the effect to clear cache, then we need to compare a data before his changes to data after his changes once caches have warmed up.
If there is an impact for network times should be "measurable" via transport times (they should be lower). If these smaller transport times have an effect on user experience (which is an entirely different matter) should be measurable via DOMContentReady (should be faster). Whether there is a real impact might get muddled as time passes and new features are deployed that affect number of assets on the page and rendering times. I would analyze desktop and mobile data separately.
Nuria
On Mon, Feb 10, 2014 at 5:50 PM, Aaron Halfaker ahalfaker@wikimedia.orgwrote:
I've created a card in trello to help us track this request. https://trello.com/c/dNODZnr0/108-measuring-ulsfo-s-impact-on-site-performan...
On Mon, Feb 10, 2014 at 10:02 AM, Aaron Halfaker ahalfaker@wikimedia.orgwrote:
The schema contains "originCountry" that should enable us to filter in this way.
I have two points:
*Browser bias: *I was under the impression that Nav timing was only available for modern browsers, so that might affect a bit of a bias into the situation.
*Browser cache: *Browser cache is going to reduce any gains we might see due to reduced latency or increased bandwidth. It would be nice if we could compare the speed changes of cached/non-cached page/asset loading. I don't think that the current schema will allow us to do this. Despite this, we should still be able to get a good sense for how cached/non-cached page loads were affected in general.
-Aaron
On Mon, Feb 10, 2014 at 9:43 AM, Dan Andreescu dandreescu@wikimedia.orgwrote:
I was wondering if you have an easy way to measure and plot the impact
in page load time, perhaps using Navigation Timing data?
+10^9. I was rather excited to see ulsfo back up and I tried to quickly figure some correlation out of https://gdash.wikimedia.org/ dashboards/frontend/ , but I failed. I think what's needed here is mostly some brilliant idea on how to visualise / graph the data available there by region (and maybe also by project / kind of editor e.g. unregistered vs. registered vs. sysop) without making thousands graphs for each of the combinations. Or just one graph for the region moved to ulsfo to start with of course. :)
Does the NavigationTiming data have the hostname of the varnish that served the request, and do we have data on which hostname served which geographic area at a particular time?
If so, we can generate a before and after average performance map by region served, to see if there are any differences.
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Federico Leva (Nemo), 10/02/2014 15:57:
Faidon Liambotis, 10/02/2014 15:37:
I was wondering if you have an easy way to measure and plot the impact in page load time, perhaps using Navigation Timing data?
+10^9. I was rather excited to see ulsfo back up and I tried to quickly figure some correlation out of https://gdash.wikimedia.org/dashboards/frontend/ , but I failed.
For those who missed it, https://www.mediawiki.org/wiki/Analytics/Reports/ULSFOImpact gave a lot of insight.
I think what's needed here is mostly some brilliant idea on how to visualise / graph the data available there by region (and maybe also by project / kind of editor e.g. unregistered vs. registered vs. sysop) without making thousands graphs for each of the combinations. Or just one graph for the region moved to ulsfo to start with of course. :)
We still need this though. Currently I'm in need of per-wiki and/or diachronic/over-time plots for it.wiki: https://meta.wikimedia.org/wiki/Research:The_sudden_decline_of_Italian_Wikipedia
If nobody has ideas I guess I'll just go for the 800 lb gorilla approach and submit a patch to have performance graphs for the top 10 wikis as we currently do for edits (cf. https://bugzilla.wikimedia.org/56039 )? In the meanwhile: https://gerrit.wikimedia.org/r/#/c/136631/
Nemo
Federico Leva (Nemo), 01/06/2014 14:11:
Currently I'm in need of per-wiki and/or diachronic/over-time plots for it.wiki: https://meta.wikimedia.org/wiki/Research:The_sudden_decline_of_Italian_Wikipedia
If nobody has ideas I guess I'll just go for the 800 lb gorilla approach and submit a patch to have performance graphs for the top 10 wikis as we currently do for edits (cf. https://bugzilla.wikimedia.org/56039 )? In the meanwhile: https://gerrit.wikimedia.org/r/#/c/136631/
Those two are closed but I can't just steal the editswiki approach because it's based on the series' values; absent better ideas I'll need to hardcode the wikis' names.
Not for that goal, but mostDeviant can be of use too: https://graphite.readthedocs.org/en/latest/functions.html#graphite.render.functions.mostDeviant For instance, a graph of sites/kinds of assets whose payload varied the most across last week. http://ur1.ca/hi4oa Can something like this be of use anywhere?
Nemo
I thiiink (don't quote me on this) the NT schema contains wiki data. I'll see if I can stitch together an aggregate dataset for you, if you think it'd help. Per-day granular enough?
On 11 June 2014 06:54, Federico Leva (Nemo) nemowiki@gmail.com wrote:
Federico Leva (Nemo), 01/06/2014 14:11:
Currently I'm in need of per-wiki and/or
diachronic/over-time plots for it.wiki: https://meta.wikimedia.org/wiki/Research:The_sudden_ decline_of_Italian_Wikipedia
If nobody has ideas I guess I'll just go for the 800 lb gorilla approach and submit a patch to have performance graphs for the top 10 wikis as we currently do for edits (cf. https://bugzilla.wikimedia.org/56039 )? In the meanwhile: https://gerrit.wikimedia.org/r/#/c/136631/
Those two are closed but I can't just steal the editswiki approach because it's based on the series' values; absent better ideas I'll need to hardcode the wikis' names.
Not for that goal, but mostDeviant can be of use too: < https://graphite.readthedocs.org/en/latest/functions.html# graphite.render.functions.mostDeviant> For instance, a graph of sites/kinds of assets whose payload varied the most across last week. http://ur1.ca/hi4oa Can something like this be of use anywhere?
Nemo
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Bah - all covered above. I should not answer emails early in the morning. But: if you still want that data, I am happy to see if I can grab it ;).
On 11 June 2014 08:04, Oliver Keyes okeyes@wikimedia.org wrote:
I thiiink (don't quote me on this) the NT schema contains wiki data. I'll see if I can stitch together an aggregate dataset for you, if you think it'd help. Per-day granular enough?
On 11 June 2014 06:54, Federico Leva (Nemo) nemowiki@gmail.com wrote:
Federico Leva (Nemo), 01/06/2014 14:11:
Currently I'm in need of per-wiki and/or
diachronic/over-time plots for it.wiki: https://meta.wikimedia.org/wiki/Research:The_sudden_ decline_of_Italian_Wikipedia
If nobody has ideas I guess I'll just go for the 800 lb gorilla approach and submit a patch to have performance graphs for the top 10 wikis as we currently do for edits (cf. https://bugzilla.wikimedia.org/56039 )? In the meanwhile: https://gerrit.wikimedia.org/r/#/c/136631/
Those two are closed but I can't just steal the editswiki approach because it's based on the series' values; absent better ideas I'll need to hardcode the wikis' names.
Not for that goal, but mostDeviant can be of use too: < https://graphite.readthedocs.org/en/latest/functions.html# graphite.render.functions.mostDeviant> For instance, a graph of sites/kinds of assets whose payload varied the most across last week. http://ur1.ca/hi4oa Can something like this be of use anywhere?
Nemo
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
-- Oliver Keyes Research Analyst Wikimedia Foundation
Faidon,
Ori correct me if I am wrong but analytics team has no instrumentation code to measure performance. We store data that other teams produce.
In the case of navigation timing there are events storing the data on EventLogging database for example:
https://meta.wikimedia.org/wiki/Schema:NavigationTiming
This data is been collected by the performance team and analyzed by Aaron prior for other purposes, I think you should consult with Ori to see if it would be suitable to measure what you are interested on, seems like if you have a big enough dataset it would work.
Thanks,
Nuria
On Mon, Feb 10, 2014 at 3:37 PM, Faidon Liambotis faidon@wikimedia.orgwrote:
Hi folks,
As you've probably heard, last week we deployed ulsfo in production, reducing latency for Oceania, East/Southeast Asia & US/Canada pacific/west coast states. My estimation of the user base affected by this is 360 million users (as in, Internet users, not Wikipedia users).
I was wondering if you have an easy way to measure and plot the impact in page load time, perhaps using Navigation Timing data?
The operations team has spent a considerable amount of time and money to deploy ulsfo and I believe it'd be useful for us and the organization at large to be able to quantify this effort.
The exact dates of the rollout by country/region codes can be found in operations/dns' git history: https://git.wikimedia.org/ summary/?r=operations/dns.git (the commits should be self-explanatory, but I'd be happy to clarify if needed)
Thanks! Faidon
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics