Wait... I'm not convinced by this. I'm not convinced you measured the right things. How exactly did you measure this? Did you include the HTTP response time?
I struggle to believe that non-lazy loaded pages could ever be faster and that the improvement was on a half second.
Can you post more details on the tests you ran? Really you should be looking at numerous things, namely: 1) Time from user request (refresh page) to being able to read the content (DOM content loaded) - e.g. time the HTTP request takes when just using the api and just using HTML 2) Time the JavaScript loads and the page becomes interactive (with a lazy loaded page this will always be 0s and on a non-lazy loaded page this will always be more as the entire HTML, JS and CSS has to be loaded)
cc'ing mobile-l
On Mon, Jul 28, 2014 at 11:39 AM, Maryana Pinchuk mpinchuk@wikimedia.org wrote:
Thanks, Kaldari – I've changed the description in this card[1] to reflect your recommendation. Unless anyone objects, we'll remove lazy-loading entirely in the next sprint.
On Mon, Jul 28, 2014 at 11:00 AM, Ryan Kaldari rkaldari@wikimedia.org wrote:
I did a spike on the effects of lazy loading pages on save. I tried a variety of different articles and did at least 10 tests in each mode.
On stub size articles, lazy loading resulted in a half-second improvement in page loading on average. On larger articles, there was greater variation in lazy-loading time and the averages were virtually identical (non-lazy-loaded time was actually 0.15 seconds faster on average, but this was not statistically significant).
So basically, lazy loading results in a small improvement for small articles and no improvement for large articles. Given the extra maintenance required (we have to keep maintaining virtually all of the lazy loading code for this specific use even if we don't use it elsewhere), and the frequent bugs that arise, I would still favor removing this feature.
Ryan Kaldari
-- Maryana Pinchuk Product Manager, Wikimedia Foundation wikimediafoundation.org
Yuvi on irc also points out that if you were logged in during these tests the cache works differently to anonymous, not such an issue right now while anonymous editing is not supported but might be in future.
I'm also curious how this effects follow up edits. With lazy loaded pages the edit button is actionable straight away. When loading via HTTP request, we are introducing a delay of X seconds before the user can edit again (the user will need to pull down the new HTML and pull down again the new JavaScript for the editor interface). This number X is the one that concerns me most. I wonder if that alone might be worth an A/B test. Are editors more likely to do follow up edits after an edit during which the page reloads via JavaScript instead of HTML?
Ps. I ran some similar tests measuring DOMContentLoaded and the JavaScript for toggling being enabled which showed big discrepancies (albeit much less HTML was being served up in this case) [1]. May or may not be helpful to demonstrate the impact this sort of JavaScript loading can have.
[1] https://docs.google.com/spreadsheet/ccc?key=0ArbzKvV50qF6dEFzR0pCbUxfVy1ybGV...
On Mon, Jul 28, 2014 at 12:02 PM, Jon Robson jrobson@wikimedia.org wrote:
Wait... I'm not convinced by this. I'm not convinced you measured the right things. How exactly did you measure this? Did you include the HTTP response time?
I struggle to believe that non-lazy loaded pages could ever be faster and that the improvement was on a half second.
Can you post more details on the tests you ran? Really you should be looking at numerous things, namely:
- Time from user request (refresh page) to being able to read the
content (DOM content loaded) - e.g. time the HTTP request takes when just using the api and just using HTML 2) Time the JavaScript loads and the page becomes interactive (with a lazy loaded page this will always be 0s and on a non-lazy loaded page this will always be more as the entire HTML, JS and CSS has to be loaded)
cc'ing mobile-l
On Mon, Jul 28, 2014 at 11:39 AM, Maryana Pinchuk mpinchuk@wikimedia.org wrote:
Thanks, Kaldari – I've changed the description in this card[1] to reflect your recommendation. Unless anyone objects, we'll remove lazy-loading entirely in the next sprint.
On Mon, Jul 28, 2014 at 11:00 AM, Ryan Kaldari rkaldari@wikimedia.org wrote:
I did a spike on the effects of lazy loading pages on save. I tried a variety of different articles and did at least 10 tests in each mode.
On stub size articles, lazy loading resulted in a half-second improvement in page loading on average. On larger articles, there was greater variation in lazy-loading time and the averages were virtually identical (non-lazy-loaded time was actually 0.15 seconds faster on average, but this was not statistically significant).
So basically, lazy loading results in a small improvement for small articles and no improvement for large articles. Given the extra maintenance required (we have to keep maintaining virtually all of the lazy loading code for this specific use even if we don't use it elsewhere), and the frequent bugs that arise, I would still favor removing this feature.
Ryan Kaldari
-- Maryana Pinchuk Product Manager, Wikimedia Foundation wikimediafoundation.org
Why would we measure when the page becomes interactive (rather than viewable)? I can't imagine many people interacting with a page within the first second of it being viewable.
The reason lazy-loading wasn't faster on the larger articles is that the loading speed was less reliable. Non-lazy-load times varied by up to 1 second. Lazy-load times varied by over 2 seconds. So even though the fastest load times were from lazy-loading, so were the slowest. For small articles, the load times were very consistent for both load types.
Ryan Kaldari
On Mon, Jul 28, 2014 at 12:02 PM, Jon Robson jrobson@wikimedia.org wrote:
Wait... I'm not convinced by this. I'm not convinced you measured the right things. How exactly did you measure this? Did you include the HTTP response time?
I struggle to believe that non-lazy loaded pages could ever be faster and that the improvement was on a half second.
Can you post more details on the tests you ran? Really you should be looking at numerous things, namely:
- Time from user request (refresh page) to being able to read the
content (DOM content loaded) - e.g. time the HTTP request takes when just using the api and just using HTML 2) Time the JavaScript loads and the page becomes interactive (with a lazy loaded page this will always be 0s and on a non-lazy loaded page this will always be more as the entire HTML, JS and CSS has to be loaded)
cc'ing mobile-l
On Mon, Jul 28, 2014 at 11:39 AM, Maryana Pinchuk mpinchuk@wikimedia.org wrote:
Thanks, Kaldari – I've changed the description in this card[1] to reflect your recommendation. Unless anyone objects, we'll remove lazy-loading entirely in the next sprint.
On Mon, Jul 28, 2014 at 11:00 AM, Ryan Kaldari rkaldari@wikimedia.org wrote:
I did a spike on the effects of lazy loading pages on save. I tried a variety of different articles and did at least 10 tests in each mode.
On stub size articles, lazy loading resulted in a half-second
improvement
in page loading on average. On larger articles, there was greater
variation
in lazy-loading time and the averages were virtually identical (non-lazy-loaded time was actually 0.15 seconds faster on average, but
this
was not statistically significant).
So basically, lazy loading results in a small improvement for small articles and no improvement for large articles. Given the extra
maintenance
required (we have to keep maintaining virtually all of the lazy loading
code
for this specific use even if we don't use it elsewhere), and the
frequent
bugs that arise, I would still favor removing this feature.
Ryan Kaldari
-- Maryana Pinchuk Product Manager, Wikimedia Foundation wikimediafoundation.org