Hi,
Cold cache concerns aside (see the other email I just sent), I now have preliminary performance figures comparing Media Viewer to the status quo (File: page) in production. I'm still working with QA on getting our test merged and running automatically on cloudbees. The results I have were performed on a 2MB DSL connection in France, with nothing else running on the network.
Since we don't collect browser resolutions in our own stats (that I could find) I picked "average" and "large" desktop browser window sizes based on 3rd-party "internet-wide" statistics I could find online. These statistics differentiated desktop and mobile, so mobile's lower resolutions didn't affect the average. The "small" size was picked specifically to hit a bucket size identical to the File: page's image.
On an small browser window size (900x700), Media Viewer displays the image in a little less than 125% of the time it takes to display the image on the File: page
On an average browser window size (1366x768), Media Viewer displays the image in a little less than 125% of the time it takes to display the image on the File: page
On a large browser window size (1920x1080) I found the same result, but I suspect it might not be true, because my desktop wasn't big enough to accommodate that size in selenium's Firefox. I think we'll need to wait for cloudbees to run the tests headless for the real figure.
For the sake of argument, let's look at the difference in file size between the file page and the "average" browser size:
http://en.wikipedia.beta.wmflabs.org/wiki/File:Sunrise_over_fishing_boats_in...
- on the File: page, the image displayed is 800x600 (480,000 pixels) and weighs 52.7kb - on Media Viewer, at the "average" browser resolution, the image displayed is 1024x768 (786,432 pixels) and weighs 79.6kb
Therefore, the Media Viewer image has 63.8% more pixels on the screen and weighs 51% more.
But, this is only a consolation prize, because the small browser window size (900x700) which serves the same 800x600 image as the File: page still takes 20-25% longer, which seems to indicate that Media Viewer's overhead isn't due to the image size. Basically, if we made the File: page serve a 1024x768 image, it would certainly still beat Media Viewer.
We've already done multiple rounds of optimizing the performance of Media Viewer, but I'll investigate further to see if there's anything obvious that makes Media Viewer slower than opening the File: page.
In my opinion between 20 and 25% of overhead isn't a show-stopper for launch, considering that Media Viewer serves a bigger image in most situations, but it's worth giving optimizing one last try to see if we've missed anything. However, it could very well be that we can't do better and that the overhead is due to doing adding the image dynamically with JS instead of it being present in the pageload DOM of a relatively lightweight page.
Looking at the timing waterfall, I think that the answer is simply the thumbnail info API call, for which Gergo has a WIP solution: https://gerrit.wikimedia.org/r/#/c/124796/
Basically at the moment Media Viewer (unlike the File: page) has to get the results of a PHP API call before it knows the URL of the image to load. My hunch, seeing that on my machine the API calls can take up to a full second, is that the 25% overhead will go away entirely when that improvement gets merged.
On Wed, Apr 9, 2014 at 1:28 PM, Gilles Dubuc gilles@wikimedia.org wrote:
Hi,
Cold cache concerns aside (see the other email I just sent), I now have preliminary performance figures comparing Media Viewer to the status quo (File: page) in production. I'm still working with QA on getting our test merged and running automatically on cloudbees. The results I have were performed on a 2MB DSL connection in France, with nothing else running on the network.
Since we don't collect browser resolutions in our own stats (that I could find) I picked "average" and "large" desktop browser window sizes based on 3rd-party "internet-wide" statistics I could find online. These statistics differentiated desktop and mobile, so mobile's lower resolutions didn't affect the average. The "small" size was picked specifically to hit a bucket size identical to the File: page's image.
On an small browser window size (900x700), Media Viewer displays the image in a little less than 125% of the time it takes to display the image on the File: page
On an average browser window size (1366x768), Media Viewer displays the image in a little less than 125% of the time it takes to display the image on the File: page
On a large browser window size (1920x1080) I found the same result, but I suspect it might not be true, because my desktop wasn't big enough to accommodate that size in selenium's Firefox. I think we'll need to wait for cloudbees to run the tests headless for the real figure.
For the sake of argument, let's look at the difference in file size between the file page and the "average" browser size:
http://en.wikipedia.beta.wmflabs.org/wiki/File:Sunrise_over_fishing_boats_in...
- on the File: page, the image displayed is 800x600 (480,000 pixels) and
weighs 52.7kb
- on Media Viewer, at the "average" browser resolution, the image
displayed is 1024x768 (786,432 pixels) and weighs 79.6kb
Therefore, the Media Viewer image has 63.8% more pixels on the screen and weighs 51% more.
But, this is only a consolation prize, because the small browser window size (900x700) which serves the same 800x600 image as the File: page still takes 20-25% longer, which seems to indicate that Media Viewer's overhead isn't due to the image size. Basically, if we made the File: page serve a 1024x768 image, it would certainly still beat Media Viewer.
We've already done multiple rounds of optimizing the performance of Media Viewer, but I'll investigate further to see if there's anything obvious that makes Media Viewer slower than opening the File: page.
In my opinion between 20 and 25% of overhead isn't a show-stopper for launch, considering that Media Viewer serves a bigger image in most situations, but it's worth giving optimizing one last try to see if we've missed anything. However, it could very well be that we can't do better and that the overhead is due to doing adding the image dynamically with JS instead of it being present in the pageload DOM of a relatively lightweight page.
Another secondary thing we can improve (and which we've mentioned in passing during previous meetings) is to cache the results of our API requests across users, presumably at the squid level. I've emailed Ops to see what can be done in that area.
If we achieve this, it would allow us to considerably reduce the average amount of time it takes for the metadata to show up in Media Viewer. This hasn't been the focus of my performance comparison test, which is focusing solely on when the sharp image appears.
On Wed, Apr 9, 2014 at 1:51 PM, Gilles Dubuc gilles@wikimedia.org wrote:
Looking at the timing waterfall, I think that the answer is simply the thumbnail info API call, for which Gergo has a WIP solution: https://gerrit.wikimedia.org/r/#/c/124796/
Basically at the moment Media Viewer (unlike the File: page) has to get the results of a PHP API call before it knows the URL of the image to load. My hunch, seeing that on my machine the API calls can take up to a full second, is that the 25% overhead will go away entirely when that improvement gets merged.
On Wed, Apr 9, 2014 at 1:28 PM, Gilles Dubuc gilles@wikimedia.org wrote:
Hi,
Cold cache concerns aside (see the other email I just sent), I now have preliminary performance figures comparing Media Viewer to the status quo (File: page) in production. I'm still working with QA on getting our test merged and running automatically on cloudbees. The results I have were performed on a 2MB DSL connection in France, with nothing else running on the network.
Since we don't collect browser resolutions in our own stats (that I could find) I picked "average" and "large" desktop browser window sizes based on 3rd-party "internet-wide" statistics I could find online. These statistics differentiated desktop and mobile, so mobile's lower resolutions didn't affect the average. The "small" size was picked specifically to hit a bucket size identical to the File: page's image.
On an small browser window size (900x700), Media Viewer displays the image in a little less than 125% of the time it takes to display the image on the File: page
On an average browser window size (1366x768), Media Viewer displays the image in a little less than 125% of the time it takes to display the image on the File: page
On a large browser window size (1920x1080) I found the same result, but I suspect it might not be true, because my desktop wasn't big enough to accommodate that size in selenium's Firefox. I think we'll need to wait for cloudbees to run the tests headless for the real figure.
For the sake of argument, let's look at the difference in file size between the file page and the "average" browser size:
http://en.wikipedia.beta.wmflabs.org/wiki/File:Sunrise_over_fishing_boats_in...
- on the File: page, the image displayed is 800x600 (480,000 pixels) and
weighs 52.7kb
- on Media Viewer, at the "average" browser resolution, the image
displayed is 1024x768 (786,432 pixels) and weighs 79.6kb
Therefore, the Media Viewer image has 63.8% more pixels on the screen and weighs 51% more.
But, this is only a consolation prize, because the small browser window size (900x700) which serves the same 800x600 image as the File: page still takes 20-25% longer, which seems to indicate that Media Viewer's overhead isn't due to the image size. Basically, if we made the File: page serve a 1024x768 image, it would certainly still beat Media Viewer.
We've already done multiple rounds of optimizing the performance of Media Viewer, but I'll investigate further to see if there's anything obvious that makes Media Viewer slower than opening the File: page.
In my opinion between 20 and 25% of overhead isn't a show-stopper for launch, considering that Media Viewer serves a bigger image in most situations, but it's worth giving optimizing one last try to see if we've missed anything. However, it could very well be that we can't do better and that the overhead is due to doing adding the image dynamically with JS instead of it being present in the pageload DOM of a relatively lightweight page.
On Apr 9, 2014 12:16 PM, "Gilles Dubuc" gilles@wikimedia.org wrote:
Another secondary thing we can improve (and which we've mentioned in
passing during previous meetings) is to cache the results of our API requests across users, presumably at the squid level. I've emailed Ops to see what can be done in that area.
Api has support for doing this already. See smaxage url parameter and getCacheMode method.
--bawolff
Thanks for the pointers, Brian. I imagine this affects the HTTP caching headers and can't be too aggressive in terms of duration, otherwise if someone updates the metadata for a file, it won't be reflected in media viewer until the max age is reached, right?
I was hoping for a silver bullet that would allow us to not only cache API responses, but also invalidate that cache when someone updates a file's information, but maybe it's not possible with our architecture. The main thing I want to check with Ops is whether squid is affected by the HTTP caching headers for API calls, to make sure that the caching benefit is effective across users, not just for someone who would re-request the same image across pageloads.
On Wed, Apr 9, 2014 at 5:27 PM, Brian Wolff bawolff@gmail.com wrote:
On Apr 9, 2014 12:16 PM, "Gilles Dubuc" gilles@wikimedia.org wrote:
Another secondary thing we can improve (and which we've mentioned in
passing during previous meetings) is to cache the results of our API requests across users, presumably at the squid level. I've emailed Ops to see what can be done in that area.
Api has support for doing this already. See smaxage url parameter and getCacheMode method.
--bawolff
Multimedia mailing list Multimedia@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/multimedia
If you know the exact url of the api request, you can purge it on specific events (e.g. edits to a file page). But you need to know the precise url. The class that sends the actual purge is named SquidUpdate I think (writing on a phone. Didnt check name. Its something like that.)
The varnish cache (we dont use squid anymore, but the various methods still have squid in their names) would work across all users if the api module declares the correct caching mode.
--bawolff On Apr 9, 2014 2:12 PM, "Gilles Dubuc" gilles@wikimedia.org wrote:
Thanks for the pointers, Brian. I imagine this affects the HTTP caching
headers and can't be too aggressive in terms of duration, otherwise if someone updates the metadata for a file, it won't be reflected in media viewer until the max age is reached, right?
I was hoping for a silver bullet that would allow us to not only cache
API responses, but also invalidate that cache when someone updates a file's information, but maybe it's not possible with our architecture. The main thing I want to check with Ops is whether squid is affected by the HTTP caching headers for API calls, to make sure that the caching benefit is effective across users, not just for someone who would re-request the same image across pageloads.
On Wed, Apr 9, 2014 at 5:27 PM, Brian Wolff bawolff@gmail.com wrote:
On Apr 9, 2014 12:16 PM, "Gilles Dubuc" gilles@wikimedia.org wrote:
Another secondary thing we can improve (and which we've mentioned in
passing during previous meetings) is to cache the results of our API requests across users, presumably at the squid level. I've emailed Ops to see what can be done in that area.
Api has support for doing this already. See smaxage url parameter and
getCacheMode method.
--bawolff
Multimedia mailing list Multimedia@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/multimedia
Multimedia mailing list Multimedia@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/multimedia
On Wed, Apr 9, 2014 at 6:29 PM, Brian Wolff bawolff@gmail.com wrote:
If you know the exact url of the api request, you can purge it on specific events (e.g. edits to a file page). But you need to know the precise url.
That's just sad... I was assuming Varnish supports some sort of tag-based invalidation, but apparently you have to pay for that [1].
It might be easier then to add a cache-breaking parameter to the API request URLs (something like the last file / file page modification date), and store that in the HTML. At least there is a sane invalidation mechanism for HTML pages. This would mean HTML pages needed to be invalidated when a new file is uploaded or the file page is edited, which sounds hard (cross-wiki invalidations, ugh) but not impossible. I assume something like that already exists for file uploads since the thumbnail URL might potentially change as a result.
Per the IRC discussion with Gilles, we should start caching filerepoinfo, userinfo, imageusage and globalusage right now, that's an easy win. Filerepoinfo and userinfo (gender) don't change much, imageusage and globalusage can, but meh, they are not very useful anyway, we only show the first three entries. Coming up with some sort of caching mechanism for imageinfo would be great, but sounds like an epic task.
[1] https://www.varnish-software.com/blog/advanced-cache-invalidation-strategies
Hi Gilles,
Thanks so much for doing this important study, which gives us a better idea on the overall image load performance of Media Viewer compared to the current file pages.
I agree with you that a 25% increase in load time between these two solutions on a warm cache is not a show-stopper for this release -- considering that we are displaying larger images and providing a more immersive experience that can make up for this slight delay.
But I am more concerned with the 200% increase in load time on a cold cache, particularly if it happens often. I encourage us to look at practical solutions to address this issue sooner rather than later — whether it is your proposal to get rid of mmv.bootstrap entirely, caching the results of our API requests across users, or other solutions.
These issues will be particularly important in world regions where low bandwidth / slow connections are more prevalent, so we will want to monitor performance closely for those sites. If we can identify early on projects that might be impacted the most by this issue, we can make those communities aware of the trade-off between richer experience and image load time.
In any case, many thanks for these useful test results, and I look forward to learning more together in coming days :)
Fabrice
On Apr 9, 2014, at 4:51 AM, Gilles Dubuc gilles@wikimedia.org wrote:
Looking at the timing waterfall, I think that the answer is simply the thumbnail info API call, for which Gergo has a WIP solution: https://gerrit.wikimedia.org/r/#/c/124796/
Basically at the moment Media Viewer (unlike the File: page) has to get the results of a PHP API call before it knows the URL of the image to load. My hunch, seeing that on my machine the API calls can take up to a full second, is that the 25% overhead will go away entirely when that improvement gets merged.
On Wed, Apr 9, 2014 at 1:28 PM, Gilles Dubuc gilles@wikimedia.org wrote: Hi,
Cold cache concerns aside (see the other email I just sent), I now have preliminary performance figures comparing Media Viewer to the status quo (File: page) in production. I'm still working with QA on getting our test merged and running automatically on cloudbees. The results I have were performed on a 2MB DSL connection in France, with nothing else running on the network.
Since we don't collect browser resolutions in our own stats (that I could find) I picked "average" and "large" desktop browser window sizes based on 3rd-party "internet-wide" statistics I could find online. These statistics differentiated desktop and mobile, so mobile's lower resolutions didn't affect the average. The "small" size was picked specifically to hit a bucket size identical to the File: page's image.
On an small browser window size (900x700), Media Viewer displays the image in a little less than 125% of the time it takes to display the image on the File: page
On an average browser window size (1366x768), Media Viewer displays the image in a little less than 125% of the time it takes to display the image on the File: page
On a large browser window size (1920x1080) I found the same result, but I suspect it might not be true, because my desktop wasn't big enough to accommodate that size in selenium's Firefox. I think we'll need to wait for cloudbees to run the tests headless for the real figure.
For the sake of argument, let's look at the difference in file size between the file page and the "average" browser size:
http://en.wikipedia.beta.wmflabs.org/wiki/File:Sunrise_over_fishing_boats_in...
- on the File: page, the image displayed is 800x600 (480,000 pixels) and weighs 52.7kb
- on Media Viewer, at the "average" browser resolution, the image displayed is 1024x768 (786,432 pixels) and weighs 79.6kb
Therefore, the Media Viewer image has 63.8% more pixels on the screen and weighs 51% more.
But, this is only a consolation prize, because the small browser window size (900x700) which serves the same 800x600 image as the File: page still takes 20-25% longer, which seems to indicate that Media Viewer's overhead isn't due to the image size. Basically, if we made the File: page serve a 1024x768 image, it would certainly still beat Media Viewer.
We've already done multiple rounds of optimizing the performance of Media Viewer, but I'll investigate further to see if there's anything obvious that makes Media Viewer slower than opening the File: page.
In my opinion between 20 and 25% of overhead isn't a show-stopper for launch, considering that Media Viewer serves a bigger image in most situations, but it's worth giving optimizing one last try to see if we've missed anything. However, it could very well be that we can't do better and that the overhead is due to doing adding the image dynamically with JS instead of it being present in the pageload DOM of a relatively lightweight page.
Multimedia mailing list Multimedia@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/multimedia
_______________________________
On Apr 9, 2014, at 3:03 AM, Gilles Dubuc gilles@wikimedia.org wrote:
Hi,
Since mediawiki.org finally moved to '21, I was able to run the performance tests against production again. We now have a figure for how bad the cold cache experience can be. On an average-sized browser window, the cold cache image load in media viewer takes around 200% of the time it takes for the image to appear on the File: page with a warm cache*. And it takes around 230% of the time for a large browser window. We're talking seconds of difference, that's a long time to wait.
When we first built mmv.bootstrap (the on-demand loading of JS), we didn't have the click catcher (mmv.head). Now that we do, I think we can afford to sacrifice some user bandwidth and just load Media Viewer's JS with the async attribute (which means it wouldn't block page rendering). And mmv.head would take care of catching and replaying clicks that happened before Media Viewer's JS was loaded.
Doing that means we could get rid of mmv.bootstrap entirely, increase the cold cache performance (chances are, the thumbnails on a given page will take longer to load than Media Viewer's JS and CSS) and simplify our code.
Any objections to doing this?
*the reason why we're not comparing figures with the File: page on a cold cache is that it's an unfair comparison: all of mediawiki's JS and CSS would be cold in that case, whereas for Media Viewer it's only Media Viewer's JS and CSS that is cold. _______________________________________________ Multimedia mailing list Multimedia@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/multimedia
On Wed, Apr 9, 2014 at 4:28 AM, Gilles Dubuc gilles@wikimedia.org wrote:
Cold cache concerns aside (see the other email I just sent), I now have preliminary performance figures comparing Media Viewer to the status quo (File: page) in production. I'm still working with QA on getting our test merged and running automatically on cloudbees. The results I have were performed on a 2MB DSL connection in France, with nothing else running on the network.
Awesome! It is great to have hard stats about performance. (The request performance maps were great too.)
Is it possible to run this tests at given geographical locations, with Cloudbees or some similar service? It would be really good to know what the overhead is for the wikis which will participate in the first round of the pilot.
Is it possible to run this tests at given geographical locations, with Cloudbees or some similar service?
I'm still waiting on QA to accept the changeset in order to get it on cloudbees.
On Wed, Apr 9, 2014 at 11:10 PM, Gergo Tisza gtisza@wikimedia.org wrote:
On Wed, Apr 9, 2014 at 4:28 AM, Gilles Dubuc gilles@wikimedia.org wrote:
Cold cache concerns aside (see the other email I just sent), I now have preliminary performance figures comparing Media Viewer to the status quo (File: page) in production. I'm still working with QA on getting our test merged and running automatically on cloudbees. The results I have were performed on a 2MB DSL connection in France, with nothing else running on the network.
Awesome! It is great to have hard stats about performance. (The request performance maps were great too.)
Is it possible to run this tests at given geographical locations, with Cloudbees or some similar service? It would be really good to know what the overhead is for the wikis which will participate in the first round of the pilot.
Multimedia mailing list Multimedia@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/multimedia
multimedia@lists.wikimedia.org