I like videos because it's so easy for non technical people to understand the value of performance. For me it has been a really good way to show and explain performance changes and what it looks like for the user.
Picking a representative connection is hard and the good thing is I think we don't need to. Connections in the real world vary so much: You could have 3G but a really bad connection to the nearest station because there's a lot of other phones connected etc. We have tests running with real connections 2G/3G from Bangalore and LTE in San Francisco, you can check it out how it looks here: https://grafana.wikimedia.org/dashboard/db/mobile-webpagetest?panelId=30&... I really like that graph because it shows how much variances you can have on the same connection type from the same location. Check out how it looks like in Bangalore.
When we throttle the connection for WebPageTest (or other tools) we don't get that type of variations, we just use hard coded values for a specific connection. And that's good, because it makes it possible for us to measure improvements so that when we do fixes, we can easily see if it makes it better or worse. Picking a slow connection like 2G is good because then it is easier to spot what impact we get from our changes.
I like automated testing with real devices, we have automatic tests up and running with a Iphone 6, Ipad Mini and Motorola G (using the ones provided in WebPageTest) and it looks like this: https://grafana.wikimedia.org/dashboard/db/mobile-webpagetest?panelId=28&...
You can click on the each row in the table to only show one graph, that makes it easier to see what numbers we have. Click on Dulles_iPhone6_iPhone_6_iOS_9.render and you can see that the start rendering time (when something is first showed on the screen) is pretty stable for the Iphone 6 but if you choose Dulles_iPhone6_iPhone_6_iOS_9.SpeedIndex (SpeedIndex is a way of calculate when content within the viewport is visible for the end user) vary a lot. The thing is that we only have limited access to automated tests with real phones (we don't have anything setup for ourselves). So we use it now to just get a feeling of what it looks like on real devices.
Most IMPORTANT: When we measure performance it is important that we always do it exactly the same way, so we try to limit the changes to as few as possible (=our code change). Using WebPageTest I think it's ok to use Mobile 2G Fast, emulating mobile using Chrome. When we run the automated tests in Jenkins we test each URL 5 times, it seems to be pretty stable. Our tests will not give us the same numbers as the users gets, but it will give us constant numbers so that we can see that the changes we do impact the site. When we push these changes to production we can hopefully pick them up with our RUM metrics.
One last thing: We will not pickup things as CPU usage faking real devices but to do that, I think we need to look at another tool than WebPageTest to do that correctly. Do we have one already?
On Mon, Jan 18, 2016 at 1:20 PM, Federico Leva (Nemo) nemowiki@gmail.com wrote:
I'm allergic to videos as replacement of data, but does your video substantially differ from what tests give us, e.g. http://www.webpagetest.org/video/view.php?id=160118_YK_HKT.1.0 ?
It's easy to test multiple speeds (some tests still pending): http://www.webpagetest.org/video/compare.php?tests=160118_AZ_JH3,160118_VB_J...
Some of the tests show significant CPU usage after the "everything and a kitchen sink" ResourceLoader phase in which some stuff for CentralNotice, Gather etc. etc. is loaded.
Mainly, the issue is to pick representative connections. There is some official data, some of which based on actual tests by probes, although it's limited:
https://ec.europa.eu/digital-agenda/en/news/quality-broadband-services-eu (no mobile)
https://ec.europa.eu/digital-agenda/en/news/use-commercial-mobile-networks-a... (needs some parsing!)
Nemo
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l