https://grafana.wikimedia.org/dashboard/db/mobile-webpagetest?panelId=28&...
I'm guessing they can be milliseconds?
yes sorry I forgot to change that, now it is easier to understand.
On Wed, Jan 20, 2016 at 5:09 PM, Joaquin Oltra Hernandez < jhernandez@wikimedia.org> wrote:
Thanks for the responses Federico & Peter.
Federico, I fully agree that a video is not a replacement for data, but they serve different purposes. Peter expanded on it better than I what I could do, but basically data driven tests serve a great deal for development purposes to assessing impact of changes since they give us a controlled environment to see the impact of changes trying to improve performance.
The reality is a lot messier and hard to measure, and I find it greatly useful to see real experiences on real devices, where the network conditions vary a lot, and there are multiple applications/services competing for CPU and bandwidth on the device, making the experience completely different to what it is on the numbers that the performance tests provide.
Surely it is anecdotal data, but it is an anecdotal experience that a lot of people probably experience, so I end up caring about it.
Thanks Peter for the links and the thoughtful response. Great points and advice. I didn't know about that dashboard, it is great!
What are the units on the Y axis here for example?
https://grafana.wikimedia.org/dashboard/db/mobile-webpagetest?panelId=28&... I'm guessing they can be milliseconds?
One last thing: We will not pickup things as CPU usage faking real devices
but to do that, I think we need to look at another tool than WebPageTest
to
do that correctly. Do we have one already?
Not that I know of, we were thinking about doing some searching to see what we could find. It would be a great addition to have.
Joaquin
On Wed, Jan 20, 2016 at 10:28 AM, Peter Hedenskog peter@wikimedia.org wrote:
I like videos because it's so easy for non technical people to understand the value of performance. For me it has been a really good way to show
and
explain performance changes and what it looks like for the user.
Picking a representative connection is hard and the good thing is I think we don't need to. Connections in the real world vary so much: You could have 3G but a really bad connection to the nearest station because
there's
a lot of other phones connected etc. We have tests running with real connections 2G/3G from Bangalore and LTE in San Francisco, you can check
it
out how it looks here:
https://grafana.wikimedia.org/dashboard/db/mobile-webpagetest?panelId=30&...
I really like that graph because it shows how much variances you can have on the same connection type from the same location. Check out how it
looks
like in Bangalore.
When we throttle the connection for WebPageTest (or other tools) we don't get that type of variations, we just use hard coded values for a specific connection. And that's good, because it makes it possible for us to
measure
improvements so that when we do fixes, we can easily see if it makes it better or worse. Picking a slow connection like 2G is good because then
it
is easier to spot what impact we get from our changes.
I like automated testing with real devices, we have automatic tests up
and
running with a Iphone 6, Ipad Mini and Motorola G (using the ones
provided
in WebPageTest) and it looks like this:
https://grafana.wikimedia.org/dashboard/db/mobile-webpagetest?panelId=28&...
You can click on the each row in the table to only show one graph, that makes it easier to see what numbers we have. Click on Dulles_iPhone6_iPhone_6_iOS_9.render and you can see that the start rendering time (when something is first showed on the screen) is pretty stable for the Iphone 6 but if you choose Dulles_iPhone6_iPhone_6_iOS_9.SpeedIndex (SpeedIndex is a way of calculate when content within the viewport is visible for the end user) vary a lot. The thing is that we only have limited access to automated tests with real phones (we don't have anything setup for ourselves). So
we
use it now to just get a feeling of what it looks like on real devices.
Most IMPORTANT: When we measure performance it is important that we
always
do it exactly the same way, so we try to limit the changes to as few as possible (=our code change). Using WebPageTest I think it's ok to use Mobile 2G Fast, emulating mobile using Chrome. When we run the automated tests in Jenkins we test each URL 5 times, it seems to be pretty stable. Our tests will not give us the same numbers as the users gets, but it
will
give us constant numbers so that we can see that the changes we do impact the site. When we push these changes to production we can hopefully pick them up with our RUM metrics.
One last thing: We will not pickup things as CPU usage faking real
devices
but to do that, I think we need to look at another tool than WebPageTest
to
do that correctly. Do we have one already?
On Mon, Jan 18, 2016 at 1:20 PM, Federico Leva (Nemo) <
nemowiki@gmail.com>
wrote:
I'm allergic to videos as replacement of data, but does your video substantially differ from what tests give us, e.g. http://www.webpagetest.org/video/view.php?id=160118_YK_HKT.1.0 ?
It's easy to test multiple speeds (some tests still pending):
http://www.webpagetest.org/video/compare.php?tests=160118_AZ_JH3,160118_VB_J...
Some of the tests show significant CPU usage after the "everything and
a
kitchen sink" ResourceLoader phase in which some stuff for
CentralNotice,
Gather etc. etc. is loaded.
Mainly, the issue is to pick representative connections. There is some official data, some of which based on actual tests by probes, although
it's
limited:
https://ec.europa.eu/digital-agenda/en/news/quality-broadband-services-eu
(no mobile)
https://ec.europa.eu/digital-agenda/en/news/use-commercial-mobile-networks-a...
(needs some parsing!)
Nemo
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l