Hi! As many of you know, a central global repository for Lua modules and templates has been a frequent request since the early days of the movement.
This year, I programmed a JavaScript tool called Synchronizer https://www.mediawiki.org/wiki/Synchronizer (inspired on a previous tool called DiBabel by User:Yurik) The tool allows to synchronize (that is, automatically copy) Lua modules across Wikimedia wikis, and provides other features to help developers update and maintain global modules.
I also re-wrote the documentation at https://www.mediawiki.org/wiki/Multilingual_Templates_and_Modules to account for the new tool. It basically describes how to develop a Lua module that can be copied unchanged to any wiki, by abstracting things like user-readable strings and config.
Admittedly, this is a "poor man's version" of a proper solution to the problem, but one I find invaluable while developing and maintaining modules. Hopefully some of you may find it useful too!
Hi! I poked around it a bit; it looks super neat! I had a few design-related questions while digging into this:
- "Update all outdated" looks like a scary button. In which scenarios someone might want to edit all modules at once? - Is there a way to automatically populate the master version for a module without letting users hunt for it? - Why don't some modules have an analyze/update button next to them? Certain terminologies used on the tool page are confusing, for example, what unrelated means?
As the target is Lua developers, which maybe aren't that many in Wikiverse, I wonder how to recruit them to use this tool and share more feedback. If you haven't done so already, you could consider dropping a note in two of these Telegram channels (Wikimedia Hackathon https://t.me/wmhack, Small wiki toolkits https://t.me/+Z_b1MR8O0wAzZmVh) and consider sharing more about the tool in a synchronous format here: < https://meta.wikimedia.org/wiki/Grants:Knowledge_Sharing/Connect/Resources#O...
.
Cheers,
Srishti
On Wed, Jul 26, 2023 at 6:09 AM Felipe Schenone schenonef@gmail.com wrote:
Hi! As many of you know, a central global repository for Lua modules and templates has been a frequent request since the early days of the movement.
This year, I programmed a JavaScript tool called Synchronizer https://www.mediawiki.org/wiki/Synchronizer (inspired on a previous tool called DiBabel by User:Yurik) The tool allows to synchronize (that is, automatically copy) Lua modules across Wikimedia wikis, and provides other features to help developers update and maintain global modules.
I also re-wrote the documentation at https://www.mediawiki.org/wiki/Multilingual_Templates_and_Modules to account for the new tool. It basically describes how to develop a Lua module that can be copied unchanged to any wiki, by abstracting things like user-readable strings and config.
Admittedly, this is a "poor man's version" of a proper solution to the problem, but one I find invaluable while developing and maintaining modules. Hopefully some of you may find it useful too! _______________________________________________ Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
Hi! Thanks for taking the time to poke around and write back!
1. "Update all outdated" is designed for mass updates once a module has been effectively "globalized". In other words, once a developer has analyzed the situation of a particular module, went through the process of designing the master version of a module so that it works in all wikis (following the guide https://www.mediawiki.org/wiki/Multilingual_Templates_and_Modules), and carefully updated one-by-one all the copies to the master version, then, whenever a new version of the master module goes live, all the other copies can be updated in a single click (but after a confirmation message). At least, that's what I did and currently do with the modules I developed and maintain (Module:Excerpt and Module:Transcluder).
2. I've had a long conversation https://www.mediawiki.org/wiki/Talk:Synchronizer with User:Uzume in which we discussed this topic (and many others). Probably the best solution will be to set a "sitelink badge" in Wikidata to set the master version of each module. However, most modules will start out without such a badge, so a fallback system and corresponding UI will be needed, which does take a few hours to write. I'll do so eventually, but until some people actually start using the tool, I see little point in continuing to throw hours into it. It's quite sophisticated and friendly already.
3. If you hover over the terms and buttons, a tooltip will appear with an explanation. Some modules don't have an Update button because they are already updated to the latest version. The Analyze button appears only next to Forked modules because it displays information only relevant to such cases. Let me know if you can think of a better way to convey any of this.
Finally, thanks for letting me know about those sharing channels, I'll drop a note! Kind regards,
On Fri, Jul 28, 2023 at 8:43 PM Srishti Sethi ssethi@wikimedia.org wrote:
Hi! I poked around it a bit; it looks super neat! I had a few design-related questions while digging into this:
- "Update all outdated" looks like a scary button. In which scenarios
someone might want to edit all modules at once?
- Is there a way to automatically populate the master version for a
module without letting users hunt for it?
- Why don't some modules have an analyze/update button next to them?
Certain terminologies used on the tool page are confusing, for example, what unrelated means?
As the target is Lua developers, which maybe aren't that many in Wikiverse, I wonder how to recruit them to use this tool and share more feedback. If you haven't done so already, you could consider dropping a note in two of these Telegram channels (Wikimedia Hackathon https://t.me/wmhack, Small wiki toolkits https://t.me/+Z_b1MR8O0wAzZmVh) and consider sharing more about the tool in a synchronous format here: < https://meta.wikimedia.org/wiki/Grants:Knowledge_Sharing/Connect/Resources#O...
.
Cheers,
Srishti
On Wed, Jul 26, 2023 at 6:09 AM Felipe Schenone schenonef@gmail.com wrote:
Hi! As many of you know, a central global repository for Lua modules and templates has been a frequent request since the early days of the movement.
This year, I programmed a JavaScript tool called Synchronizer https://www.mediawiki.org/wiki/Synchronizer (inspired on a previous tool called DiBabel by User:Yurik) The tool allows to synchronize (that is, automatically copy) Lua modules across Wikimedia wikis, and provides other features to help developers update and maintain global modules.
I also re-wrote the documentation at https://www.mediawiki.org/wiki/Multilingual_Templates_and_Modules to account for the new tool. It basically describes how to develop a Lua module that can be copied unchanged to any wiki, by abstracting things like user-readable strings and config.
Admittedly, this is a "poor man's version" of a proper solution to the problem, but one I find invaluable while developing and maintaining modules. Hopefully some of you may find it useful too! _______________________________________________ Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
I think this is one of the most awesome things I've seen in a long time. Has my full support! Thank you for building this and sharing it with everyone!
I've been asked whether we should be concerned about this gadget in terms of site load or site performance. I'll share my analysis here also for Felipe's benefit and for transparency to others.
*== Performance ==* ** Firstly, I'll look at potential database and web server load. What is the exposure of this feature? How many queries does it make for a typical interaction? Are those queries expensive? Are they well-cached within MediaWiki? Are they cachable at the HTTP level (e.g CDN/Varnish cache)?
Secondly, I'll look at how it makes edits. Does it always attribute edits to the actor? What security risk might there be? Does it act on stale data? Does it prevent conflicts under race conditions? Does it handle rate limits?
*== Server load ==*
The gadget looks well-written and well-guarded. It was easy to figure out what it does and how it does it. For a typical interaction, the gadget initially makes ~100 requests to the Action API, which one be seen in the browser DevTools under the Network tab. It uses the *MediaWiki* *Action API* to fetch page metadata and raw page content.
There's a couple of things Synchronizer does that I like:
* The user interface requires specific input in the form of a Wikidata entity and a wiki ID. This means upon casually stumbling on the feature, it won't do anything until and unless the person has understood what needs to be entered and enters that. A good example is shown through placeholder text, but you have to actually enter something before continuing. * Once submitted, before anything else, Synchronizer checks whether the client is logged-in and thus likely to be able to later make edits through this gadget. If you're not logged-in, it won't make that burst of 100 API requests. Even though a logged-out user could successfully make those API requests to render the initial overview, this avoids significant load that would only leave the user stranded later on. This removes the vast majority of risk right here. In the event of viral linking or misunderstanding of what the feature is for, it would have little to no impact. * When gathering data, it uses *Wikidata* to discover which wikis are known to have a version of what the same template. This greatly limits the fan-out to "only" 10-100 wikis as opposed to ~1000 wikis. It also makes sense from a correctness standpoint as templates don't have the same name on all wikis, and those that do share a name may not be compatible or otherwise conceptually the same. Wikidata is awesome for this.
Synchronizer uses the Action API to fetch page metadata and raw page content. This kind of request is fairly cheap and benefits from various forms of caching within MediaWiki. These API modules don't currently offer CDN caching, but, I don't think that's warranted today given they're fast enough. If this feature were accessible to logged-out users and if it made these queries directly when merely visiting a URL, then we'd need to think about HTTP caching to ensure that any spikes in traffic can be absorbed by our edge cache data centres.
There is one improvement I can suggest, which is to limit the number of concurrent requests. It works fine today but it does technically violate *MediaWiki API Etiquette* https://www.mediawiki.org/wiki/API:Etiquette, and may get caught in API throttling. To mitigate this, you could tweak the for-loop in "getMasterData". This currently starts each `updateStatus` call in parallel to each other. If updateStatus returned a Promise, then this loop could instead chain together the calls in a linear sequence. For example, `sequence = Promise.resolve(); for (…) { sequence = sequence.finally( updateStatus.bind( null, module ) ); }`.
*== Editing ==*
For editing, Synchronizer uses all the built-in affordances correctly. E.g. mw.Api and mw.ForeignApi https://doc.wikimedia.org/mediawiki-core/master/js/#!/api/mw.Api as exposed by the *mediawiki.api ResourceLoader module*. This gives the gadget a stable surface, greatly reduces complexity of the implementation, and automatically does all the right things.
I really like that the gadget goes the extra mile of cleverly figuring out why a local template that differs from the central one. For example, does the local copy match one of the previous versions of the central template? Or was it locally forked in a new direction? It then also allows you to preview a diff before syncing any given wiki. This is really powerful and empowers people to understand their actions before doing it.
To show what could happen if someone didn't use the provided mw.Api JS utility and naively made requests directly to the API endpoint: * edit() takes care of preventing unresolved *edit conflicts*. It uses the Edit API's `basetimestamp` parameter, so that if someone else edited the template between when Synchronizer fetches metadata and when the actor saves their edit, it will not overwrite that edit but instead fail safely. * edit() takes care to ensure the user *remains logged-in* so that if cross-wiki logins expire or become invalid for any reason, the edit won't be saved as an anon but fail safely. * edit() takes care to *not (re-)create a page* if the page in question was deleted in the mean time. It uses the Edit API's `createonly` and `nocreate` parameters. https://www.mediawiki.org/wiki/API:Edit
-- Timo Tijhof, Principal Engineer, Wikimedia Foundation.
On Wed, 26 Jul 2023, at 14:08, Felipe Schenone wrote:
Hi! As many of you know, a central global repository for Lua modules and templates has been a frequent request since the early days of the movement.
This year, I programmed a JavaScript tool called Synchronizer https://www.mediawiki.org/wiki/Synchronizer (inspired on a previous tool called DiBabel by User:Yurik) The tool allows to synchronize (that is, automatically copy) Lua modules across Wikimedia wikis, and provides other features to help developers update and maintain global modules.
I also re-wrote the documentation at https://www.mediawiki.org/wiki/Multilingual_Templates_and_Modules to account for the new tool. It basically describes how to develop a Lua module that can be copied unchanged to any wiki, by abstracting things like user-readable strings and config.
Admittedly, this is a "poor man's version" of a proper solution to the problem, but one I find invaluable while developing and maintaining modules. Hopefully some of you may find it useful too! _______________________________________________ Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
Hi! Thanks for the amazing review and encouragement! I just implemented the "promise sequencing" you suggested. Very neat pattern! I also did some improvements to the error handling, including an earlier detection of logged-out users. Hopefully your support will encourage more Lua developers to start using the tool. :-) Unfortunately we can't track usage until https://phabricator.wikimedia.org/T343104 is fixed, but I trust it will be soon. Kind regards,
On Tue, Aug 29, 2023 at 4:03 PM Krinkle krinkle@fastmail.com wrote:
I think this is one of the most awesome things I've seen in a long time. Has my full support! Thank you for building this and sharing it with everyone!
I've been asked whether we should be concerned about this gadget in terms of site load or site performance. I'll share my analysis here also for Felipe's benefit and for transparency to others.
*== Performance ==*
Firstly, I'll look at potential database and web server load. What is the exposure of this feature? How many queries does it make for a typical interaction? Are those queries expensive? Are they well-cached within MediaWiki? Are they cachable at the HTTP level (e.g CDN/Varnish cache)?
Secondly, I'll look at how it makes edits. Does it always attribute edits to the actor? What security risk might there be? Does it act on stale data? Does it prevent conflicts under race conditions? Does it handle rate limits?
*== Server load ==*
The gadget looks well-written and well-guarded. It was easy to figure out what it does and how it does it. For a typical interaction, the gadget initially makes ~100 requests to the Action API, which one be seen in the browser DevTools under the Network tab. It uses the *MediaWiki* *Action API* to fetch page metadata and raw page content.
There's a couple of things Synchronizer does that I like:
- The user interface requires specific input in the form of a Wikidata
entity and a wiki ID. This means upon casually stumbling on the feature, it won't do anything until and unless the person has understood what needs to be entered and enters that. A good example is shown through placeholder text, but you have to actually enter something before continuing.
- Once submitted, before anything else, Synchronizer checks whether the
client is logged-in and thus likely to be able to later make edits through this gadget. If you're not logged-in, it won't make that burst of 100 API requests. Even though a logged-out user could successfully make those API requests to render the initial overview, this avoids significant load that would only leave the user stranded later on. This removes the vast majority of risk right here. In the event of viral linking or misunderstanding of what the feature is for, it would have little to no impact.
- When gathering data, it uses *Wikidata* to discover which wikis are
known to have a version of what the same template. This greatly limits the fan-out to "only" 10-100 wikis as opposed to ~1000 wikis. It also makes sense from a correctness standpoint as templates don't have the same name on all wikis, and those that do share a name may not be compatible or otherwise conceptually the same. Wikidata is awesome for this.
Synchronizer uses the Action API to fetch page metadata and raw page content. This kind of request is fairly cheap and benefits from various forms of caching within MediaWiki. These API modules don't currently offer CDN caching, but, I don't think that's warranted today given they're fast enough. If this feature were accessible to logged-out users and if it made these queries directly when merely visiting a URL, then we'd need to think about HTTP caching to ensure that any spikes in traffic can be absorbed by our edge cache data centres.
There is one improvement I can suggest, which is to limit the number of concurrent requests. It works fine today but it does technically violate *MediaWiki API Etiquette* https://www.mediawiki.org/wiki/API:Etiquette, and may get caught in API throttling. To mitigate this, you could tweak the for-loop in "getMasterData". This currently starts each `updateStatus` call in parallel to each other. If updateStatus returned a Promise, then this loop could instead chain together the calls in a linear sequence. For example, `sequence = Promise.resolve(); for (…) { sequence = sequence.finally( updateStatus.bind( null, module ) ); }`.
*== Editing ==*
For editing, Synchronizer uses all the built-in affordances correctly. E.g. mw.Api and mw.ForeignApi https://doc.wikimedia.org/mediawiki-core/master/js/#!/api/mw.Api as exposed by the *mediawiki.api ResourceLoader module*. This gives the gadget a stable surface, greatly reduces complexity of the implementation, and automatically does all the right things.
I really like that the gadget goes the extra mile of cleverly figuring out why a local template that differs from the central one. For example, does the local copy match one of the previous versions of the central template? Or was it locally forked in a new direction? It then also allows you to preview a diff before syncing any given wiki. This is really powerful and empowers people to understand their actions before doing it.
To show what could happen if someone didn't use the provided mw.Api JS utility and naively made requests directly to the API endpoint:
- edit() takes care of preventing unresolved *edit conflicts*. It uses
the Edit API's `basetimestamp` parameter, so that if someone else edited the template between when Synchronizer fetches metadata and when the actor saves their edit, it will not overwrite that edit but instead fail safely.
- edit() takes care to ensure the user *remains logged-in* so that if
cross-wiki logins expire or become invalid for any reason, the edit won't be saved as an anon but fail safely.
- edit() takes care to *not (re-)create a page* if the page in question
was deleted in the mean time. It uses the Edit API's `createonly` and `nocreate` parameters. https://www.mediawiki.org/wiki/API:Edit
-- Timo Tijhof, Principal Engineer, Wikimedia Foundation.
On Wed, 26 Jul 2023, at 14:08, Felipe Schenone wrote:
Hi! As many of you know, a central global repository for Lua modules and templates has been a frequent request since the early days of the movement.
This year, I programmed a JavaScript tool called Synchronizer https://www.mediawiki.org/wiki/Synchronizer (inspired on a previous tool called DiBabel by User:Yurik) The tool allows to synchronize (that is, automatically copy) Lua modules across Wikimedia wikis, and provides other features to help developers update and maintain global modules.
I also re-wrote the documentation at https://www.mediawiki.org/wiki/Multilingual_Templates_and_Modules to account for the new tool. It basically describes how to develop a Lua module that can be copied unchanged to any wiki, by abstracting things like user-readable strings and config.
Admittedly, this is a "poor man's version" of a proper solution to the problem, but one I find invaluable while developing and maintaining modules. Hopefully some of you may find it useful too! _______________________________________________ Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
wikitech-l@lists.wikimedia.org