Hello, Startup module, is the Javascript code [1] that is being served at almost every request to Wikimedia as part of <head> (non-blocking though) to load other RL modules. It does important things for example it checks if the requested modules already exist with the given hash in the local storage cache.
Because of the given reasons, the startup module needs and includes list of all registered modules with their version hash in the code. As the result if you register a module, even if you don't load it, it adds a rather small overhead (around 19 bytes after minification and compression) to every request to any wiki the code is enabled. So if you register a new module in core or one of the extensions that is deployed in all wikis (CX, WikibaseClient, etc.), it's going to be added to every page request, meaning 30 GBs more traffic every day to our users. Even if you don't use it anywhere.
If you're adding a feature, 30GB/day is acceptable but sometimes developers use Resource loader for dependency management or class loading (yours truly used to do that) and introduce 20 modules instead of one and each one causing an extra 600 GB/day. The big overhead for our users is bad for three reasons: 1- RL has to handle the dependency management, making every page view slightly slower and worse UX for our users 2- The extra network is bad with places without access to broadband connection, where Wikipedia is the most likely place that people learn and grow [2] 3- The scale we are talking is so massive (petabytes a year) that It has environmental impact.
Let me give you an example, a couple of weeks ago we dropped 137 modules from WikibaseClient. After deployment, it dropped 4TB/day from our network (= 1.5 PB/year) and synthetic data shows, in an average case we drop 40 ms from every response time [3].
We now have a dashboard to track size of size of RL registry [4] plus a weekly metrics for changes [5][6]
If you're looking for ways to help. I wrote a tool [7] to do some graph analysis and it gives list of extensions that has modules that can be merged. The extensions that according to the analysis (that can have false positives) can get better are TimedMediaHandler, PageTriage, Graph, RevisionSlider, CodeMirror, Citoid, TemplateData, TwoColConflict, Collection, CentralNotice, AdvancedSearch, 3D, MobileFrontend and many more including some bits and pieces in core. I put the graphs of modules that can be merged at [8] and I invite you to have fun with those modules. Modules can be merged using package modules [9]
Most of the is work done by the performance team [10] and volunteers and developers in lots of teams. I joined the party later as volunteer/WMDE staff and I'm sharing mostly the results and writing long emails. Big kudos to Krinkle, James Forrester, Santhosh Thottingal, Jon Robson, Alaa Sarhaan, Alexandros Kosiaris, and so many others that helped and I forgot.
[1] An example from English Wikipedia: https://en.wikipedia.org/w/load.php?lang=en&modules=startup&only=scr... [2] https://arxiv.org/abs/1812.00474 [3] https://phabricator.wikimedia.org/T203696#5387672 [4] https://grafana.wikimedia.org/d/BvWJlaDWk/startup-module-size?orgId=1 [5] https://gist.github.com/Krinkle/f76229f512fead79fb4868824b5bee07 [6] https://docs.google.com/document/d/1SESOADAH9phJTeLo4lqipAjYUMaLpGsQTAUqdgyZ... [7] https://phabricator.wikimedia.org/T232728 [8] https://phabricator.wikimedia.org/T233048 [9] https://www.mediawiki.org/wiki/ResourceLoader/Package_modules [10] https://phabricator.wikimedia.org/T202154
Best
TimedMediaHandler is particularly bad at this right now because it's still in the middle of a switch from an old set of frontend modules to a new one, so we have to register both. :) I'll make sure we chop the old ones out soon...
-- brion
On Mon, Sep 16, 2019 at 2:11 PM Amir Sarabadani ladsgroup@gmail.com wrote:
Hello, Startup module, is the Javascript code [1] that is being served at almost every request to Wikimedia as part of <head> (non-blocking though) to load other RL modules. It does important things for example it checks if the requested modules already exist with the given hash in the local storage cache.
Because of the given reasons, the startup module needs and includes list of all registered modules with their version hash in the code. As the result if you register a module, even if you don't load it, it adds a rather small overhead (around 19 bytes after minification and compression) to every request to any wiki the code is enabled. So if you register a new module in core or one of the extensions that is deployed in all wikis (CX, WikibaseClient, etc.), it's going to be added to every page request, meaning 30 GBs more traffic every day to our users. Even if you don't use it anywhere.
If you're adding a feature, 30GB/day is acceptable but sometimes developers use Resource loader for dependency management or class loading (yours truly used to do that) and introduce 20 modules instead of one and each one causing an extra 600 GB/day. The big overhead for our users is bad for three reasons: 1- RL has to handle the dependency management, making every page view slightly slower and worse UX for our users 2- The extra network is bad with places without access to broadband connection, where Wikipedia is the most likely place that people learn and grow [2] 3- The scale we are talking is so massive (petabytes a year) that It has environmental impact.
Let me give you an example, a couple of weeks ago we dropped 137 modules from WikibaseClient. After deployment, it dropped 4TB/day from our network (= 1.5 PB/year) and synthetic data shows, in an average case we drop 40 ms from every response time [3].
We now have a dashboard to track size of size of RL registry [4] plus a weekly metrics for changes [5][6]
If you're looking for ways to help. I wrote a tool [7] to do some graph analysis and it gives list of extensions that has modules that can be merged. The extensions that according to the analysis (that can have false positives) can get better are TimedMediaHandler, PageTriage, Graph, RevisionSlider, CodeMirror, Citoid, TemplateData, TwoColConflict, Collection, CentralNotice, AdvancedSearch, 3D, MobileFrontend and many more including some bits and pieces in core. I put the graphs of modules that can be merged at [8] and I invite you to have fun with those modules. Modules can be merged using package modules [9]
Most of the is work done by the performance team [10] and volunteers and developers in lots of teams. I joined the party later as volunteer/WMDE staff and I'm sharing mostly the results and writing long emails. Big kudos to Krinkle, James Forrester, Santhosh Thottingal, Jon Robson, Alaa Sarhaan, Alexandros Kosiaris, and so many others that helped and I forgot.
[1] An example from English Wikipedia:
https://en.wikipedia.org/w/load.php?lang=en&modules=startup&only=scr... [2] https://arxiv.org/abs/1812.00474 [3] https://phabricator.wikimedia.org/T203696#5387672 [4] https://grafana.wikimedia.org/d/BvWJlaDWk/startup-module-size?orgId=1 [5] https://gist.github.com/Krinkle/f76229f512fead79fb4868824b5bee07 [6]
https://docs.google.com/document/d/1SESOADAH9phJTeLo4lqipAjYUMaLpGsQTAUqdgyZ... [7] https://phabricator.wikimedia.org/T232728 [8] https://phabricator.wikimedia.org/T233048 [9] https://www.mediawiki.org/wiki/ResourceLoader/Package_modules [10] https://phabricator.wikimedia.org/T202154
Best
Amir (he/him) _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Amazing work!
See also https://phabricator.wikimedia.org/phame/post/view/175/wikipedia_s_javascript... https://phabricator.wikimedia.org/phame/post/view/175/wikipedia_s_javascript_initialisation_on_a_budget/
Kosta
On Sep 16, 2019, at 11:10 PM, Amir Sarabadani ladsgroup@gmail.com wrote:
Hello, Startup module, is the Javascript code [1] that is being served at almost every request to Wikimedia as part of <head> (non-blocking though) to load other RL modules. It does important things for example it checks if the requested modules already exist with the given hash in the local storage cache.
Because of the given reasons, the startup module needs and includes list of all registered modules with their version hash in the code. As the result if you register a module, even if you don't load it, it adds a rather small overhead (around 19 bytes after minification and compression) to every request to any wiki the code is enabled. So if you register a new module in core or one of the extensions that is deployed in all wikis (CX, WikibaseClient, etc.), it's going to be added to every page request, meaning 30 GBs more traffic every day to our users. Even if you don't use it anywhere.
If you're adding a feature, 30GB/day is acceptable but sometimes developers use Resource loader for dependency management or class loading (yours truly used to do that) and introduce 20 modules instead of one and each one causing an extra 600 GB/day. The big overhead for our users is bad for three reasons: 1- RL has to handle the dependency management, making every page view slightly slower and worse UX for our users 2- The extra network is bad with places without access to broadband connection, where Wikipedia is the most likely place that people learn and grow [2] 3- The scale we are talking is so massive (petabytes a year) that It has environmental impact.
Let me give you an example, a couple of weeks ago we dropped 137 modules from WikibaseClient. After deployment, it dropped 4TB/day from our network (= 1.5 PB/year) and synthetic data shows, in an average case we drop 40 ms from every response time [3].
We now have a dashboard to track size of size of RL registry [4] plus a weekly metrics for changes [5][6]
If you're looking for ways to help. I wrote a tool [7] to do some graph analysis and it gives list of extensions that has modules that can be merged. The extensions that according to the analysis (that can have false positives) can get better are TimedMediaHandler, PageTriage, Graph, RevisionSlider, CodeMirror, Citoid, TemplateData, TwoColConflict, Collection, CentralNotice, AdvancedSearch, 3D, MobileFrontend and many more including some bits and pieces in core. I put the graphs of modules that can be merged at [8] and I invite you to have fun with those modules. Modules can be merged using package modules [9]
Most of the is work done by the performance team [10] and volunteers and developers in lots of teams. I joined the party later as volunteer/WMDE staff and I'm sharing mostly the results and writing long emails. Big kudos to Krinkle, James Forrester, Santhosh Thottingal, Jon Robson, Alaa Sarhaan, Alexandros Kosiaris, and so many others that helped and I forgot.
[1] An example from English Wikipedia: https://en.wikipedia.org/w/load.php?lang=en&modules=startup&only=scr... [2] https://arxiv.org/abs/1812.00474 [3] https://phabricator.wikimedia.org/T203696#5387672 [4] https://grafana.wikimedia.org/d/BvWJlaDWk/startup-module-size?orgId=1 [5] https://gist.github.com/Krinkle/f76229f512fead79fb4868824b5bee07 [6] https://docs.google.com/document/d/1SESOADAH9phJTeLo4lqipAjYUMaLpGsQTAUqdgyZ... [7] https://phabricator.wikimedia.org/T232728 [8] https://phabricator.wikimedia.org/T233048 [9] https://www.mediawiki.org/wiki/ResourceLoader/Package_modules [10] https://phabricator.wikimedia.org/T202154
Best
Amir (he/him) _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
My friend @wetterfrosch points out on twitter that reducing our daily bandwidth usage by 4.3TB reduces CO2 emission by 30 tons PER DAY. That's roughly equivalent to flying 20 people from Berlin to San Francisco and back. Every day.
Thank you all!
Sources: https://twitter.com/wetterfrosch/status/1174675018795638785 https://alistapart.com/article/sustainable-web-design/ https://en.wikipedia.org/wiki/Environmental_impact_of_transport https://www.carbonfootprint.com/calculator.aspx
Am 16.09.19 um 23:10 schrieb Amir Sarabadani:
Hello, Startup module, is the Javascript code [1] that is being served at almost every request to Wikimedia as part of <head> (non-blocking though) to load other RL modules. It does important things for example it checks if the requested modules already exist with the given hash in the local storage cache.
Because of the given reasons, the startup module needs and includes list of all registered modules with their version hash in the code. As the result if you register a module, even if you don't load it, it adds a rather small overhead (around 19 bytes after minification and compression) to every request to any wiki the code is enabled. So if you register a new module in core or one of the extensions that is deployed in all wikis (CX, WikibaseClient, etc.), it's going to be added to every page request, meaning 30 GBs more traffic every day to our users. Even if you don't use it anywhere.
If you're adding a feature, 30GB/day is acceptable but sometimes developers use Resource loader for dependency management or class loading (yours truly used to do that) and introduce 20 modules instead of one and each one causing an extra 600 GB/day. The big overhead for our users is bad for three reasons: 1- RL has to handle the dependency management, making every page view slightly slower and worse UX for our users 2- The extra network is bad with places without access to broadband connection, where Wikipedia is the most likely place that people learn and grow [2] 3- The scale we are talking is so massive (petabytes a year) that It has environmental impact.
Let me give you an example, a couple of weeks ago we dropped 137 modules from WikibaseClient. After deployment, it dropped 4TB/day from our network (= 1.5 PB/year) and synthetic data shows, in an average case we drop 40 ms from every response time [3].
We now have a dashboard to track size of size of RL registry [4] plus a weekly metrics for changes [5][6]
If you're looking for ways to help. I wrote a tool [7] to do some graph analysis and it gives list of extensions that has modules that can be merged. The extensions that according to the analysis (that can have false positives) can get better are TimedMediaHandler, PageTriage, Graph, RevisionSlider, CodeMirror, Citoid, TemplateData, TwoColConflict, Collection, CentralNotice, AdvancedSearch, 3D, MobileFrontend and many more including some bits and pieces in core. I put the graphs of modules that can be merged at [8] and I invite you to have fun with those modules. Modules can be merged using package modules [9]
Most of the is work done by the performance team [10] and volunteers and developers in lots of teams. I joined the party later as volunteer/WMDE staff and I'm sharing mostly the results and writing long emails. Big kudos to Krinkle, James Forrester, Santhosh Thottingal, Jon Robson, Alaa Sarhaan, Alexandros Kosiaris, and so many others that helped and I forgot.
[1] An example from English Wikipedia: https://en.wikipedia.org/w/load.php?lang=en&modules=startup&only=scr... [2] https://arxiv.org/abs/1812.00474 [3] https://phabricator.wikimedia.org/T203696#5387672 [4] https://grafana.wikimedia.org/d/BvWJlaDWk/startup-module-size?orgId=1 [5] https://gist.github.com/Krinkle/f76229f512fead79fb4868824b5bee07 [6] https://docs.google.com/document/d/1SESOADAH9phJTeLo4lqipAjYUMaLpGsQTAUqdgyZ... [7] https://phabricator.wikimedia.org/T232728 [8] https://phabricator.wikimedia.org/T233048 [9] https://www.mediawiki.org/wiki/ResourceLoader/Package_modules [10] https://phabricator.wikimedia.org/T202154
Best
On Thu, Sep 19, 2019 at 05:09:59PM +0200, Daniel Kinzler wrote:
My friend @wetterfrosch points out on twitter that reducing our daily bandwidth usage by 4.3TB reduces CO2 emission by 30 tons PER DAY. That's roughly equivalent to flying 20 people from Berlin to San Francisco and back. Every day.
More events/conferences then, got it! :)
Congrats and thanks Performance and others!
Greg
wikitech-l@lists.wikimedia.org