Hey,
I just wanted to check in about the status of enabling JavaScript package management usage in MediaWiki. I am basically talking about an equivalent for JS to what we have with Composer for PHP.
Real-world example: The "data-values/value-view" package[0] is defining "jquery.event.special.eachchange.js": ValueView/lib/jquery.event/jquery.event.special.eachchange.js
Now, recently I needed the same functionality in one of my extensions, so I just copied it over. [1]
I know that this is the worst way one could do this, but as far as I can see we don't have that much of a choice right now. Here are the alternative options I can see:
Moving "jquery.event.special.eachchange.js" out of the "data-values/value-view" package into its own "WMDE/jquery-eachchange" package...
1. ... and using it in my extension via composer. + pro: two or more extensions or other packages requiring this package will still result in having only one MW-wide installation.. - con: requires MW specific code which is actually not related to the MW-independent package to feed the resource loader. - con: using Composer to manage pure JavaScript packages! Uuuh, ugly!
2. ... and having a build step in other packages using the package, pulling the "WMDE/jquery-eachchange" somewhere into the file structure of the packages/extensions using it. + pro: don't need to abuse composer, we can use "npm", "Bower" or any other arbitrary JS package manager here. - con: got to tell resource loader somehow... (didn't think so much about that yet) - con: if more than one extensions or other packages require this package we still end up with the same code twice or more often in one MW installation.
3. Combining 1 and 2: Start with 2, using a JS package manager. Then going to 1, creating a composer package and pulling the "WMDE/jquery-eachchange" package in via some build script. + pro: The two pros from 1 + 2 + pro: ^^ - con: still got to tell resource loader somehow... - con: Overhead; We now create two packages where the Composer one is just a bridge to the MW-world, still polluting packagist.org. Still kind of ugly and more effort for publishing a package and therefore potentially scaring programmers away from doing so since they've got better things to do than doing work that could be automated.
I have not seen Approach 2 and 3 yet. Though I could imagine that the VisualEditor team has used something like that. Approach 1 is the way the "data-values/value-view" package itself is being handled. And that package should actually be a MW independent pure JS package but right now it contains MW specific code and uses composer for distribution! There is still another option but that had to be properly implemented:
4. Choose one native JS package manager for now and go with it (add support for others later perhaps). Integrate it properly with MW (resource loader to begin with), document how to use it and finally distribute JS code coming from the MW world but useful for other projects in a way where it can actually be used in a non-MW context.
This has already been bugging me when working on Wikidata. Now I'd like to reuse some of the code I have written there without spending hours and hours with option 3 because there should be support for option 4 rather sooner or later. So I am wondering; Does anyone have any thoughts, any alternatives perhaps or is there any roadmap on anything like the option 4 that I have shown?
Cheers, Daniel
[0]: https://packagist.org/packages/data-values/value-view [1]: https://github.com/DanweDE/mediawiki-ext-UserBitcoinAddresses/blob/master/re...
On Wed, Jul 22, 2015 at 12:24 PM, Daniel Werner daniel.a.r.werner@gmail.com wrote:
Hey,
I just wanted to check in about the status of enabling JavaScript package management usage in MediaWiki. I am basically talking about an equivalent for JS to what we have with Composer for PHP.
Real-world example: The "data-values/value-view" package[0] is defining "jquery.event.special.eachchange.js": ValueView/lib/jquery.event/jquery.event.special.eachchange.js
Now, recently I needed the same functionality in one of my extensions, so I just copied it over. [1]
I know that this is the worst way one could do this, but as far as I can see we don't have that much of a choice right now. Here are the alternative options I can see:
Moving "jquery.event.special.eachchange.js" out of the "data-values/value-view" package into its own "WMDE/jquery-eachchange" package...
- ... and using it in my extension via composer.
- pro: two or more extensions or other packages requiring this package
will still result in having only one MW-wide installation..
- con: requires MW specific code which is actually not related to the
MW-independent package to feed the resource loader.
- con: using Composer to manage pure JavaScript packages! Uuuh, ugly!
- ... and having a build step in other packages using the package, pulling
the "WMDE/jquery-eachchange" somewhere into the file structure of the packages/extensions using it.
- pro: don't need to abuse composer, we can use "npm", "Bower" or any
other arbitrary JS package manager here.
- con: got to tell resource loader somehow... (didn't think so much about
that yet)
- con: if more than one extensions or other packages require this package
we still end up with the same code twice or more often in one MW installation.
- Combining 1 and 2: Start with 2, using a JS package manager. Then going
to 1, creating a composer package and pulling the "WMDE/jquery-eachchange" package in via some build script.
- pro: The two pros from 1 + 2
- pro: ^^
- con: still got to tell resource loader somehow...
- con: Overhead; We now create two packages where the Composer one is
just a bridge to the MW-world, still polluting packagist.org. Still kind of ugly and more effort for publishing a package and therefore potentially scaring programmers away from doing so since they've got better things to do than doing work that could be automated.
I have not seen Approach 2 and 3 yet. Though I could imagine that the VisualEditor team has used something like that. Approach 1 is the way the "data-values/value-view" package itself is being handled. And that package should actually be a MW independent pure JS package but right now it contains MW specific code and uses composer for distribution! There is still another option but that had to be properly implemented:
- Choose one native JS package manager for now and go with it (add support
for others later perhaps). Integrate it properly with MW (resource loader to begin with), document how to use it and finally distribute JS code coming from the MW world but useful for other projects in a way where it can actually be used in a non-MW context.
This has already been bugging me when working on Wikidata. Now I'd like to reuse some of the code I have written there without spending hours and hours with option 3 because there should be support for option 4 rather sooner or later. So I am wondering; Does anyone have any thoughts, any alternatives perhaps or is there any roadmap on anything like the option 4 that I have shown?
Cheers, Daniel
Option 4 was discussed last October as part of the Librarization project [0]. At the time the front end standards group wasn't ready to pick a winner in the javascript packaging landscape. They did want to revisit that in 3-6 months however so maybe it is time to find someone to look into the various options and their pros and cons again?
[0]: https://phabricator.wikimedia.org/T807
We should choose something, any of them is better than nothing, and manually copying files.
My vote goes to npm, now that it tries (in v3) its best to flatten the node_modules folder. See the changelog ( https://github.com/npm/npm/releases/tag/v3.0.0)
*Flat, flat, flat!*
Your dependencies will now be installed *maximally flat*. Insofar as is
possible, all of your dependencies, and their dependencies, and THEIR dependencies will be installed in your project's node_modules folder with no nesting. You'll only see modules nested underneath one another when two (or more) modules have conflicting dependencies.
And it is the biggest JS package manager.
On Thu, Jul 23, 2015 at 1:21 AM, Bryan Davis bd808@wikimedia.org wrote:
On Wed, Jul 22, 2015 at 12:24 PM, Daniel Werner daniel.a.r.werner@gmail.com wrote:
Hey,
I just wanted to check in about the status of enabling JavaScript package management usage in MediaWiki. I am basically talking about an equivalent for JS to what we have with Composer for PHP.
Real-world example: The "data-values/value-view" package[0] is defining "jquery.event.special.eachchange.js": ValueView/lib/jquery.event/jquery.event.special.eachchange.js
Now, recently I needed the same functionality in one of my extensions,
so
I just copied it over. [1]
I know that this is the worst way one could do this, but as far as I can see we don't have that much of a choice right now. Here are the
alternative
options I can see:
Moving "jquery.event.special.eachchange.js" out of the "data-values/value-view" package into its own "WMDE/jquery-eachchange" package...
- ... and using it in my extension via composer.
- pro: two or more extensions or other packages requiring this package
will still result in having only one MW-wide installation..
- con: requires MW specific code which is actually not related to the
MW-independent package to feed the resource loader.
- con: using Composer to manage pure JavaScript packages! Uuuh, ugly!
- ... and having a build step in other packages using the package,
pulling
the "WMDE/jquery-eachchange" somewhere into the file structure of the packages/extensions using it.
- pro: don't need to abuse composer, we can use "npm", "Bower" or any
other arbitrary JS package manager here.
- con: got to tell resource loader somehow... (didn't think so much
about
that yet)
- con: if more than one extensions or other packages require this
package
we still end up with the same code twice or more often in one MW installation.
- Combining 1 and 2: Start with 2, using a JS package manager. Then
going
to 1, creating a composer package and pulling the
"WMDE/jquery-eachchange"
package in via some build script.
- pro: The two pros from 1 + 2
- pro: ^^
- con: still got to tell resource loader somehow...
- con: Overhead; We now create two packages where the Composer one is
just a bridge to the MW-world, still polluting packagist.org. Still
kind of
ugly and more effort for publishing a package and therefore potentially scaring programmers away from doing so since they've got better things to do than doing work that could be automated.
I have not seen Approach 2 and 3 yet. Though I could imagine that the VisualEditor team has used something like that. Approach 1 is the way the "data-values/value-view" package itself is
being
handled. And that package should actually be a MW independent pure JS package but right now it contains MW specific code and uses composer for distribution! There is still another option but that had to be properly implemented:
- Choose one native JS package manager for now and go with it (add
support
for others later perhaps). Integrate it properly with MW (resource loader to begin with), document how to use it and finally distribute JS code coming from the MW world but useful for other projects in a way where it can actually be used in a non-MW context.
This has already been bugging me when working on Wikidata. Now I'd like
to
reuse some of the code I have written there without spending hours and hours with option 3 because there should be support for option 4 rather sooner or later. So I am wondering; Does anyone have any thoughts, any alternatives
perhaps
or is there any roadmap on anything like the option 4 that I have shown?
Cheers, Daniel
[1]:
https://github.com/DanweDE/mediawiki-ext-UserBitcoinAddresses/blob/master/re...
Option 4 was discussed last October as part of the Librarization project [0]. At the time the front end standards group wasn't ready to pick a winner in the javascript packaging landscape. They did want to revisit that in 3-6 months however so maybe it is time to find someone to look into the various options and their pros and cons again?
[0]: https://phabricator.wikimedia.org/T807
Bryan Davis Wikimedia Foundation bd808@wikimedia.org [[m:User:BDavis_(WMF)]] Sr Software Engineer Boise, ID USA irc: bd808 v:415.839.6885 x6855
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
It's been a year now, over 3-6 months. Volker can this be given a priority in the next frontend standards meeting. I think the lack of any standard is extremely damaging to the project.
On Wed, Jul 22, 2015 at 4:21 PM, Bryan Davis bd808@wikimedia.org wrote:
On Wed, Jul 22, 2015 at 12:24 PM, Daniel Werner daniel.a.r.werner@gmail.com wrote:
Hey,
I just wanted to check in about the status of enabling JavaScript package management usage in MediaWiki. I am basically talking about an equivalent for JS to what we have with Composer for PHP.
Real-world example: The "data-values/value-view" package[0] is defining "jquery.event.special.eachchange.js": ValueView/lib/jquery.event/jquery.event.special.eachchange.js
Now, recently I needed the same functionality in one of my extensions, so I just copied it over. [1]
I know that this is the worst way one could do this, but as far as I can see we don't have that much of a choice right now. Here are the alternative options I can see:
Moving "jquery.event.special.eachchange.js" out of the "data-values/value-view" package into its own "WMDE/jquery-eachchange" package...
- ... and using it in my extension via composer.
- pro: two or more extensions or other packages requiring this package
will still result in having only one MW-wide installation..
- con: requires MW specific code which is actually not related to the
MW-independent package to feed the resource loader.
- con: using Composer to manage pure JavaScript packages! Uuuh, ugly!
- ... and having a build step in other packages using the package, pulling
the "WMDE/jquery-eachchange" somewhere into the file structure of the packages/extensions using it.
- pro: don't need to abuse composer, we can use "npm", "Bower" or any
other arbitrary JS package manager here.
- con: got to tell resource loader somehow... (didn't think so much about
that yet)
- con: if more than one extensions or other packages require this package
we still end up with the same code twice or more often in one MW installation.
- Combining 1 and 2: Start with 2, using a JS package manager. Then going
to 1, creating a composer package and pulling the "WMDE/jquery-eachchange" package in via some build script.
- pro: The two pros from 1 + 2
- pro: ^^
- con: still got to tell resource loader somehow...
- con: Overhead; We now create two packages where the Composer one is
just a bridge to the MW-world, still polluting packagist.org. Still kind of ugly and more effort for publishing a package and therefore potentially scaring programmers away from doing so since they've got better things to do than doing work that could be automated.
I have not seen Approach 2 and 3 yet. Though I could imagine that the VisualEditor team has used something like that. Approach 1 is the way the "data-values/value-view" package itself is being handled. And that package should actually be a MW independent pure JS package but right now it contains MW specific code and uses composer for distribution! There is still another option but that had to be properly implemented:
- Choose one native JS package manager for now and go with it (add support
for others later perhaps). Integrate it properly with MW (resource loader to begin with), document how to use it and finally distribute JS code coming from the MW world but useful for other projects in a way where it can actually be used in a non-MW context.
This has already been bugging me when working on Wikidata. Now I'd like to reuse some of the code I have written there without spending hours and hours with option 3 because there should be support for option 4 rather sooner or later. So I am wondering; Does anyone have any thoughts, any alternatives perhaps or is there any roadmap on anything like the option 4 that I have shown?
Cheers, Daniel
Option 4 was discussed last October as part of the Librarization project [0]. At the time the front end standards group wasn't ready to pick a winner in the javascript packaging landscape. They did want to revisit that in 3-6 months however so maybe it is time to find someone to look into the various options and their pros and cons again?
[0]: https://phabricator.wikimedia.org/T807
Bryan Davis Wikimedia Foundation bd808@wikimedia.org [[m:User:BDavis_(WMF)]] Sr Software Engineer Boise, ID USA irc: bd808 v:415.839.6885 x6855
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
The flat approach to NPM is a game changer for us, and a Bower killer.
Timo? Had a lot of insight at the time, I'd like to see him be invoked in this decision. Any thoughts Timo?
- Trevor
On Thursday, November 5, 2015, Jon Robson jrobson@wikimedia.org wrote:
It's been a year now, over 3-6 months. Volker can this be given a priority in the next frontend standards meeting. I think the lack of any standard is extremely damaging to the project.
On Wed, Jul 22, 2015 at 4:21 PM, Bryan Davis <bd808@wikimedia.org javascript:;> wrote:
On Wed, Jul 22, 2015 at 12:24 PM, Daniel Werner <daniel.a.r.werner@gmail.com javascript:;> wrote:
Hey,
I just wanted to check in about the status of enabling JavaScript
package
management usage in MediaWiki. I am basically talking about an
equivalent
for JS to what we have with Composer for PHP.
Real-world example: The "data-values/value-view" package[0] is defining "jquery.event.special.eachchange.js": ValueView/lib/jquery.event/jquery.event.special.eachchange.js
Now, recently I needed the same functionality in one of my
extensions, so
I just copied it over. [1]
I know that this is the worst way one could do this, but as far as I can see we don't have that much of a choice right now. Here are the
alternative
options I can see:
Moving "jquery.event.special.eachchange.js" out of the "data-values/value-view" package into its own "WMDE/jquery-eachchange" package...
- ... and using it in my extension via composer.
- pro: two or more extensions or other packages requiring this package
will still result in having only one MW-wide installation..
- con: requires MW specific code which is actually not related to the
MW-independent package to feed the resource loader.
- con: using Composer to manage pure JavaScript packages! Uuuh, ugly!
- ... and having a build step in other packages using the package,
pulling
the "WMDE/jquery-eachchange" somewhere into the file structure of the packages/extensions using it.
- pro: don't need to abuse composer, we can use "npm", "Bower" or any
other arbitrary JS package manager here.
- con: got to tell resource loader somehow... (didn't think so much
about
that yet)
- con: if more than one extensions or other packages require this
package
we still end up with the same code twice or more often in one MW installation.
- Combining 1 and 2: Start with 2, using a JS package manager. Then
going
to 1, creating a composer package and pulling the
"WMDE/jquery-eachchange"
package in via some build script.
- pro: The two pros from 1 + 2
- pro: ^^
- con: still got to tell resource loader somehow...
- con: Overhead; We now create two packages where the Composer one is
just a bridge to the MW-world, still polluting packagist.org. Still
kind of
ugly and more effort for publishing a package and therefore potentially scaring programmers away from doing so since they've got better things
to
do than doing work that could be automated.
I have not seen Approach 2 and 3 yet. Though I could imagine that the VisualEditor team has used something like that. Approach 1 is the way the "data-values/value-view" package itself is
being
handled. And that package should actually be a MW independent pure JS package but right now it contains MW specific code and uses composer for distribution! There is still another option but that had to be properly implemented:
- Choose one native JS package manager for now and go with it (add
support
for others later perhaps). Integrate it properly with MW (resource
loader
to begin with), document how to use it and finally distribute JS code coming from the MW world but useful for other projects in a way where it can actually be used in a non-MW context.
This has already been bugging me when working on Wikidata. Now I'd like
to
reuse some of the code I have written there without spending hours and hours with option 3 because there should be support for option 4 rather sooner or later. So I am wondering; Does anyone have any thoughts, any alternatives
perhaps
or is there any roadmap on anything like the option 4 that I have shown?
Cheers, Daniel
[1]:
https://github.com/DanweDE/mediawiki-ext-UserBitcoinAddresses/blob/master/re...
Option 4 was discussed last October as part of the Librarization project [0]. At the time the front end standards group wasn't ready to pick a winner in the javascript packaging landscape. They did want to revisit that in 3-6 months however so maybe it is time to find someone to look into the various options and their pros and cons again?
[0]: https://phabricator.wikimedia.org/T807
Bryan Davis Wikimedia Foundation <bd808@wikimedia.org
[[m:User:BDavis_(WMF)]] Sr Software Engineer Boise, ID USA irc: bd808 v:415.839.6885 x6855
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org javascript:; https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org javascript:; https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Note that as part of https://phabricator.wikimedia.org/T114457 I'm experimenting with distributing mediawiki-core itself as an npm-installable package. It ought to be possible to package up extensions, including Parsoid and VE, using npm as well. --scott
On Thu, Nov 5, 2015 at 11:32 AM, Trevor Parscal tparscal@wikimedia.org wrote:
The flat approach to NPM is a game changer for us, and a Bower killer.
Timo? Had a lot of insight at the time, I'd like to see him be invoked in this decision. Any thoughts Timo?
- Trevor
On Thursday, November 5, 2015, Jon Robson jrobson@wikimedia.org wrote:
It's been a year now, over 3-6 months. Volker can this be given a priority in the next frontend standards meeting. I think the lack of any standard is extremely damaging to the project.
On Wed, Jul 22, 2015 at 4:21 PM, Bryan Davis <bd808@wikimedia.org javascript:;> wrote:
On Wed, Jul 22, 2015 at 12:24 PM, Daniel Werner <daniel.a.r.werner@gmail.com javascript:;> wrote:
Hey,
I just wanted to check in about the status of enabling JavaScript
package
management usage in MediaWiki. I am basically talking about an
equivalent
for JS to what we have with Composer for PHP.
Real-world example: The "data-values/value-view" package[0] is defining "jquery.event.special.eachchange.js": ValueView/lib/jquery.event/jquery.event.special.eachchange.js
Now, recently I needed the same functionality in one of my
extensions, so
I just copied it over. [1]
I know that this is the worst way one could do this, but as far as I
can
see we don't have that much of a choice right now. Here are the
alternative
options I can see:
Moving "jquery.event.special.eachchange.js" out of the "data-values/value-view" package into its own "WMDE/jquery-eachchange" package...
- ... and using it in my extension via composer.
- pro: two or more extensions or other packages requiring this
package
will still result in having only one MW-wide installation..
- con: requires MW specific code which is actually not related to
the
MW-independent package to feed the resource loader.
- con: using Composer to manage pure JavaScript packages! Uuuh,
ugly!
- ... and having a build step in other packages using the package,
pulling
the "WMDE/jquery-eachchange" somewhere into the file structure of the packages/extensions using it.
- pro: don't need to abuse composer, we can use "npm", "Bower" or
any
other arbitrary JS package manager here.
- con: got to tell resource loader somehow... (didn't think so much
about
that yet)
- con: if more than one extensions or other packages require this
package
we still end up with the same code twice or more often in one MW installation.
- Combining 1 and 2: Start with 2, using a JS package manager. Then
going
to 1, creating a composer package and pulling the
"WMDE/jquery-eachchange"
package in via some build script.
- pro: The two pros from 1 + 2
- pro: ^^
- con: still got to tell resource loader somehow...
- con: Overhead; We now create two packages where the Composer one
is
just a bridge to the MW-world, still polluting packagist.org. Still
kind of
ugly and more effort for publishing a package and therefore
potentially
scaring programmers away from doing so since they've got better things
to
do than doing work that could be automated.
I have not seen Approach 2 and 3 yet. Though I could imagine that the VisualEditor team has used something like that. Approach 1 is the way the "data-values/value-view" package itself is
being
handled. And that package should actually be a MW independent pure JS package but right now it contains MW specific code and uses composer
for
distribution! There is still another option but that had to be properly implemented:
- Choose one native JS package manager for now and go with it (add
support
for others later perhaps). Integrate it properly with MW (resource
loader
to begin with), document how to use it and finally distribute JS code coming from the MW world but useful for other projects in a way where
it
can actually be used in a non-MW context.
This has already been bugging me when working on Wikidata. Now I'd
like
to
reuse some of the code I have written there without spending hours and hours with option 3 because there should be support for option 4
rather
sooner or later. So I am wondering; Does anyone have any thoughts, any alternatives
perhaps
or is there any roadmap on anything like the option 4 that I have
shown?
Cheers, Daniel
[1]:
https://github.com/DanweDE/mediawiki-ext-UserBitcoinAddresses/blob/master/re...
Option 4 was discussed last October as part of the Librarization project [0]. At the time the front end standards group wasn't ready to pick a winner in the javascript packaging landscape. They did want to revisit that in 3-6 months however so maybe it is time to find someone to look into the various options and their pros and cons again?
[0]: https://phabricator.wikimedia.org/T807
Bryan Davis Wikimedia Foundation <bd808@wikimedia.org
[[m:User:BDavis_(WMF)]] Sr Software Engineer Boise, ID USA irc: bd808 v:415.839.6885 x6855
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org javascript:; https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org javascript:; https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
As someone who now works in js/node primarily I might be able to add a little outside insight.
IMHO bower was probably killed far before npm@3 came out.
- npm has always been more advanced - bower's main isn't as reliable as it is in npm. Even if modules aren't forgetting to set it as much as they used to I still see hard to deal with things like modules that include .sass sources in it. - bower's distribution model is actually very problematic for upstream developers. bower clones git repositories directly expecting to find built/compiled js and has no local build hooks. Few major js libraries work in classic js/css sources anymore, instead having some required build step. Bower makes a complete mess of this. Requiring the build result to be committed with each change, an automated build bot, a second git repo for build results only, or the upstream dev to just not care about bower. - Thanks to the rise of browserify and the like more and more client-only libraries have moved to npm despite people traditionally thinking of it as being for nodejs modules. Most client only libraries now exist in npm. And if you wave the name browserify around you can get just about any js library to start distributing to npm. - This has actually gone so far that once when I added a contribution to a library I noticed that they were actually forgetting to keep their bower.json up to date.
npm@3 is also probably not as important as it's being made out here.
npm@3 still doesn't guarantee a tree will be 100% flat. Most of npm@3's changes fix small quirks in front-end usage and stability issues with npm.
The 'major' change of 'flatness' is really that when installing `a` that depends on `b`, `b` will be hoisted up to the same level as `a`, if possible, instead of always being installed under `a`. npm@2 did have some annoying quirks during development/updating that could leave a duplicate module until you recreated your node_modules tree. And there was the side case of installing two modules that both depended on something you did not depend on. But that could be dealt with by either adding it as a dep yourself or running dedupe.
----
The bad news is that while more and more libraries are moving to npm the underlying reason for many being on npm is for the many users using tools like browserify and webpack. So the assumption of many libraries is that when you use npm for client-side libraries is you are using it in a node-style/CommonJS way where require is available other npm dependencies can be used through it and you're not including things in the traditional way of libraries being run one after another on the page in the global scope like in ResourceLoader.
It's probably going to get harder and harder to reconcile deps shared between extensions and/or use library code itself without having node installed on servers. For that matter half the advancements in the css space probably won't be making their way to php either.
Though I do have some hints to things that might open up some possibilities.
Firstly browserify can build standalone versions of libraries. It's possible you could make tweaks to that pattern and have a development build step that would convert npm packages into modules built to run in a new type of ResourceLoader that would probably be a hybrid of the current RL and npm. However you would need to take care to exclude things or else you'll end up with a script that will duplicate the source of a bunch of other modules. You also have to be wary of handling the cases where npm intentionally duplicates a library.
This actually may be even more difficult than it sounds due to npm packages that require modules inside a package rather than just the main module. You might need to go even deeper into browserify, substituting one of the last steps so it outputs a different format with objects that have metadata on the individual modules intended for use in a special loader instead of as a plain script tag.
Current RL on its own just won't work for this. Globals don't work well in the module pattern and so the ability to get a result from require() and allow npm modules to require() others are necessary. Circular requires also need consideration.
Whether you want a RL api that has a bridge to the npm ecosystem or want to go all-in to the modern js landscape and build a more CommonJS/node/npm-like RL api that loads things according to MW's use cases will be the important bike-shedding.
~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://danielfriesen.name/]
On 2015-11-05 8:32 AM, Trevor Parscal wrote:
The flat approach to NPM is a game changer for us, and a Bower killer.
Timo? Had a lot of insight at the time, I'd like to see him be invoked in this decision. Any thoughts Timo?
- Trevor
On Thursday, November 5, 2015, Jon Robson jrobson@wikimedia.org wrote:
It's been a year now, over 3-6 months. Volker can this be given a priority in the next frontend standards meeting. I think the lack of any standard is extremely damaging to the project.
On Wed, Jul 22, 2015 at 4:21 PM, Bryan Davis <bd808@wikimedia.org javascript:;> wrote:
On Wed, Jul 22, 2015 at 12:24 PM, Daniel Werner <daniel.a.r.werner@gmail.com javascript:;> wrote:
Hey,
I just wanted to check in about the status of enabling JavaScript
package
management usage in MediaWiki. I am basically talking about an
equivalent
for JS to what we have with Composer for PHP.
Real-world example: The "data-values/value-view" package[0] is defining "jquery.event.special.eachchange.js": ValueView/lib/jquery.event/jquery.event.special.eachchange.js
Now, recently I needed the same functionality in one of my
extensions, so
I just copied it over. [1]
I know that this is the worst way one could do this, but as far as I can see we don't have that much of a choice right now. Here are the
alternative
options I can see:
Moving "jquery.event.special.eachchange.js" out of the "data-values/value-view" package into its own "WMDE/jquery-eachchange" package...
- ... and using it in my extension via composer.
- pro: two or more extensions or other packages requiring this package
will still result in having only one MW-wide installation..
- con: requires MW specific code which is actually not related to the
MW-independent package to feed the resource loader.
- con: using Composer to manage pure JavaScript packages! Uuuh, ugly!
- ... and having a build step in other packages using the package,
pulling
the "WMDE/jquery-eachchange" somewhere into the file structure of the packages/extensions using it.
- pro: don't need to abuse composer, we can use "npm", "Bower" or any
other arbitrary JS package manager here.
- con: got to tell resource loader somehow... (didn't think so much
about
that yet)
- con: if more than one extensions or other packages require this
package
we still end up with the same code twice or more often in one MW installation.
- Combining 1 and 2: Start with 2, using a JS package manager. Then
going
to 1, creating a composer package and pulling the
"WMDE/jquery-eachchange"
package in via some build script.
- pro: The two pros from 1 + 2
- pro: ^^
- con: still got to tell resource loader somehow...
- con: Overhead; We now create two packages where the Composer one is
just a bridge to the MW-world, still polluting packagist.org. Still
kind of
ugly and more effort for publishing a package and therefore potentially scaring programmers away from doing so since they've got better things
to
do than doing work that could be automated.
I have not seen Approach 2 and 3 yet. Though I could imagine that the VisualEditor team has used something like that. Approach 1 is the way the "data-values/value-view" package itself is
being
handled. And that package should actually be a MW independent pure JS package but right now it contains MW specific code and uses composer for distribution! There is still another option but that had to be properly implemented:
- Choose one native JS package manager for now and go with it (add
support
for others later perhaps). Integrate it properly with MW (resource
loader
to begin with), document how to use it and finally distribute JS code coming from the MW world but useful for other projects in a way where it can actually be used in a non-MW context.
This has already been bugging me when working on Wikidata. Now I'd like
to
reuse some of the code I have written there without spending hours and hours with option 3 because there should be support for option 4 rather sooner or later. So I am wondering; Does anyone have any thoughts, any alternatives
perhaps
or is there any roadmap on anything like the option 4 that I have shown?
Cheers, Daniel
[1]:
https://github.com/DanweDE/mediawiki-ext-UserBitcoinAddresses/blob/master/re...
Option 4 was discussed last October as part of the Librarization project [0]. At the time the front end standards group wasn't ready to pick a winner in the javascript packaging landscape. They did want to revisit that in 3-6 months however so maybe it is time to find someone to look into the various options and their pros and cons again?
[0]: https://phabricator.wikimedia.org/T807
Bryan Davis Wikimedia Foundation <bd808@wikimedia.org
[[m:User:BDavis_(WMF)]] Sr Software Engineer Boise, ID USA irc: bd808 v:415.839.6885 x6855
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org javascript:; https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org javascript:; https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Two other interesting pieces:
1. http://requirejs.org/ is still the goal-standard for async browser-type loading, AFAIK, and there are good packages (`npm install requirejs`) that allow interoperability with the "npm style".
2. The recently-completed ES6 spec contains a module format. You can already use it via compatibility thunks from many existing module systems. It may well see increased use, especially on the web as browsers implement the spec (which is happening quickly). There is resistance in the node community to adopting ES6 modules, but it may be that we are at an inflection point and ES6 will eventually win out. --scott
On Thu, Nov 5, 2015 at 3:24 PM, Daniel Friesen daniel@nadir-seen-fire.com wrote:
As someone who now works in js/node primarily I might be able to add a little outside insight.
IMHO bower was probably killed far before npm@3 came out.
- npm has always been more advanced
- bower's main isn't as reliable as it is in npm. Even if modules aren't
forgetting to set it as much as they used to I still see hard to deal with things like modules that include .sass sources in it.
- bower's distribution model is actually very problematic for upstream
developers. bower clones git repositories directly expecting to find built/compiled js and has no local build hooks. Few major js libraries work in classic js/css sources anymore, instead having some required build step. Bower makes a complete mess of this. Requiring the build result to be committed with each change, an automated build bot, a second git repo for build results only, or the upstream dev to just not care about bower.
- Thanks to the rise of browserify and the like more and more
client-only libraries have moved to npm despite people traditionally thinking of it as being for nodejs modules. Most client only libraries now exist in npm. And if you wave the name browserify around you can get just about any js library to start distributing to npm.
- This has actually gone so far that once when I added a contribution
to a library I noticed that they were actually forgetting to keep their bower.json up to date.
npm@3 is also probably not as important as it's being made out here.
npm@3 still doesn't guarantee a tree will be 100% flat. Most of npm@3's changes fix small quirks in front-end usage and stability issues with npm.
The 'major' change of 'flatness' is really that when installing `a` that depends on `b`, `b` will be hoisted up to the same level as `a`, if possible, instead of always being installed under `a`. npm@2 did have some annoying quirks during development/updating that could leave a duplicate module until you recreated your node_modules tree. And there was the side case of installing two modules that both depended on something you did not depend on. But that could be dealt with by either adding it as a dep yourself or running dedupe.
The bad news is that while more and more libraries are moving to npm the underlying reason for many being on npm is for the many users using tools like browserify and webpack. So the assumption of many libraries is that when you use npm for client-side libraries is you are using it in a node-style/CommonJS way where require is available other npm dependencies can be used through it and you're not including things in the traditional way of libraries being run one after another on the page in the global scope like in ResourceLoader.
It's probably going to get harder and harder to reconcile deps shared between extensions and/or use library code itself without having node installed on servers. For that matter half the advancements in the css space probably won't be making their way to php either.
Though I do have some hints to things that might open up some possibilities.
Firstly browserify can build standalone versions of libraries. It's possible you could make tweaks to that pattern and have a development build step that would convert npm packages into modules built to run in a new type of ResourceLoader that would probably be a hybrid of the current RL and npm. However you would need to take care to exclude things or else you'll end up with a script that will duplicate the source of a bunch of other modules. You also have to be wary of handling the cases where npm intentionally duplicates a library.
This actually may be even more difficult than it sounds due to npm packages that require modules inside a package rather than just the main module. You might need to go even deeper into browserify, substituting one of the last steps so it outputs a different format with objects that have metadata on the individual modules intended for use in a special loader instead of as a plain script tag.
Current RL on its own just won't work for this. Globals don't work well in the module pattern and so the ability to get a result from require() and allow npm modules to require() others are necessary. Circular requires also need consideration.
Whether you want a RL api that has a bridge to the npm ecosystem or want to go all-in to the modern js landscape and build a more CommonJS/node/npm-like RL api that loads things according to MW's use cases will be the important bike-shedding.
~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://danielfriesen.name/]
On 2015-11-05 8:32 AM, Trevor Parscal wrote:
The flat approach to NPM is a game changer for us, and a Bower killer.
Timo? Had a lot of insight at the time, I'd like to see him be invoked in this decision. Any thoughts Timo?
- Trevor
On Thursday, November 5, 2015, Jon Robson jrobson@wikimedia.org wrote:
It's been a year now, over 3-6 months. Volker can this be given a priority in the next frontend standards meeting. I think the lack of any standard is extremely damaging to the project.
On Wed, Jul 22, 2015 at 4:21 PM, Bryan Davis <bd808@wikimedia.org javascript:;> wrote:
On Wed, Jul 22, 2015 at 12:24 PM, Daniel Werner <daniel.a.r.werner@gmail.com javascript:;> wrote:
Hey,
I just wanted to check in about the status of enabling JavaScript
package
management usage in MediaWiki. I am basically talking about an
equivalent
for JS to what we have with Composer for PHP.
Real-world example: The "data-values/value-view" package[0] is defining "jquery.event.special.eachchange.js": ValueView/lib/jquery.event/jquery.event.special.eachchange.js
Now, recently I needed the same functionality in one of my
extensions, so
I just copied it over. [1]
I know that this is the worst way one could do this, but as far as I
can
see we don't have that much of a choice right now. Here are the
alternative
options I can see:
Moving "jquery.event.special.eachchange.js" out of the "data-values/value-view" package into its own "WMDE/jquery-eachchange" package...
- ... and using it in my extension via composer.
- pro: two or more extensions or other packages requiring this
package
will still result in having only one MW-wide installation..
- con: requires MW specific code which is actually not related to
the
MW-independent package to feed the resource loader.
- con: using Composer to manage pure JavaScript packages! Uuuh,
ugly!
- ... and having a build step in other packages using the package,
pulling
the "WMDE/jquery-eachchange" somewhere into the file structure of the packages/extensions using it.
- pro: don't need to abuse composer, we can use "npm", "Bower" or
any
other arbitrary JS package manager here.
- con: got to tell resource loader somehow... (didn't think so much
about
that yet)
- con: if more than one extensions or other packages require this
package
we still end up with the same code twice or more often in one MW installation.
- Combining 1 and 2: Start with 2, using a JS package manager. Then
going
to 1, creating a composer package and pulling the
"WMDE/jquery-eachchange"
package in via some build script.
- pro: The two pros from 1 + 2
- pro: ^^
- con: still got to tell resource loader somehow...
- con: Overhead; We now create two packages where the Composer one
is
just a bridge to the MW-world, still polluting packagist.org. Still
kind of
ugly and more effort for publishing a package and therefore
potentially
scaring programmers away from doing so since they've got better things
to
do than doing work that could be automated.
I have not seen Approach 2 and 3 yet. Though I could imagine that the VisualEditor team has used something like that. Approach 1 is the way the "data-values/value-view" package itself is
being
handled. And that package should actually be a MW independent pure JS package but right now it contains MW specific code and uses composer
for
distribution! There is still another option but that had to be properly implemented:
- Choose one native JS package manager for now and go with it (add
support
for others later perhaps). Integrate it properly with MW (resource
loader
to begin with), document how to use it and finally distribute JS code coming from the MW world but useful for other projects in a way where
it
can actually be used in a non-MW context.
This has already been bugging me when working on Wikidata. Now I'd
like
to
reuse some of the code I have written there without spending hours and hours with option 3 because there should be support for option 4
rather
sooner or later. So I am wondering; Does anyone have any thoughts, any alternatives
perhaps
or is there any roadmap on anything like the option 4 that I have
shown?
Cheers, Daniel
[1]:
https://github.com/DanweDE/mediawiki-ext-UserBitcoinAddresses/blob/master/re...
Option 4 was discussed last October as part of the Librarization project [0]. At the time the front end standards group wasn't ready to pick a winner in the javascript packaging landscape. They did want to revisit that in 3-6 months however so maybe it is time to find someone to look into the various options and their pros and cons again?
[0]: https://phabricator.wikimedia.org/T807
Bryan Davis Wikimedia Foundation <bd808@wikimedia.org
[[m:User:BDavis_(WMF)]] Sr Software Engineer Boise, ID USA irc: bd808 v:415.839.6885 x6855
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org javascript:; https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org javascript:; https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On 2015-11-05 1:30 PM, C. Scott Ananian wrote:
Two other interesting pieces:
- http://requirejs.org/ is still the goal-standard for async browser-type
loading, AFAIK, and there are good packages (`npm install requirejs`) that allow interoperability with the "npm style".
requirejs is still built for the same single application type as the other non-async loaders. You may not be able to get even requirejs to work with MediaWiki's needed pattern of different packages required by different extensions all integrated together without the use of node on the server that needs to combine them together.
- The recently-completed ES6 spec contains a module format. You can
already use it via compatibility thunks from many existing module systems. It may well see increased use, especially on the web as browsers implement the spec (which is happening quickly). There is resistance in the node community to adopting ES6 modules, but it may be that we are at an inflection point and ES6 will eventually win out.
ES6 modules have a different pattern for how exports are treated, especially in regards to the 'default' export. To handle this case babel inserts the following when you are transpiling ES6/ES2015 module syntax: Object.defineProperty(exports, '__esModule', { value: true });
Unless you explicitly enable loose mode any library you develop as an es module and anything that uses it will instantly break in browsers like IE8.
Use of ES6 import/export is gaining more and more adoption. But this might be one reason some people are holding back on it.
es modules also have different behavior in regards to cycles that I know esperanto handles but I'm not sure about the other transpilers.
~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://danielfriesen.name/]
--scott
On Thu, Nov 5, 2015 at 3:24 PM, Daniel Friesen daniel@nadir-seen-fire.com wrote:
As someone who now works in js/node primarily I might be able to add a little outside insight.
IMHO bower was probably killed far before npm@3 came out.
- npm has always been more advanced
- bower's main isn't as reliable as it is in npm. Even if modules aren't
forgetting to set it as much as they used to I still see hard to deal with things like modules that include .sass sources in it.
- bower's distribution model is actually very problematic for upstream
developers. bower clones git repositories directly expecting to find built/compiled js and has no local build hooks. Few major js libraries work in classic js/css sources anymore, instead having some required build step. Bower makes a complete mess of this. Requiring the build result to be committed with each change, an automated build bot, a second git repo for build results only, or the upstream dev to just not care about bower.
- Thanks to the rise of browserify and the like more and more
client-only libraries have moved to npm despite people traditionally thinking of it as being for nodejs modules. Most client only libraries now exist in npm. And if you wave the name browserify around you can get just about any js library to start distributing to npm.
- This has actually gone so far that once when I added a contribution
to a library I noticed that they were actually forgetting to keep their bower.json up to date.
npm@3 is also probably not as important as it's being made out here.
npm@3 still doesn't guarantee a tree will be 100% flat. Most of npm@3's changes fix small quirks in front-end usage and stability issues with npm.
The 'major' change of 'flatness' is really that when installing `a` that depends on `b`, `b` will be hoisted up to the same level as `a`, if possible, instead of always being installed under `a`. npm@2 did have some annoying quirks during development/updating that could leave a duplicate module until you recreated your node_modules tree. And there was the side case of installing two modules that both depended on something you did not depend on. But that could be dealt with by either adding it as a dep yourself or running dedupe.
The bad news is that while more and more libraries are moving to npm the underlying reason for many being on npm is for the many users using tools like browserify and webpack. So the assumption of many libraries is that when you use npm for client-side libraries is you are using it in a node-style/CommonJS way where require is available other npm dependencies can be used through it and you're not including things in the traditional way of libraries being run one after another on the page in the global scope like in ResourceLoader.
It's probably going to get harder and harder to reconcile deps shared between extensions and/or use library code itself without having node installed on servers. For that matter half the advancements in the css space probably won't be making their way to php either.
Though I do have some hints to things that might open up some possibilities.
Firstly browserify can build standalone versions of libraries. It's possible you could make tweaks to that pattern and have a development build step that would convert npm packages into modules built to run in a new type of ResourceLoader that would probably be a hybrid of the current RL and npm. However you would need to take care to exclude things or else you'll end up with a script that will duplicate the source of a bunch of other modules. You also have to be wary of handling the cases where npm intentionally duplicates a library.
This actually may be even more difficult than it sounds due to npm packages that require modules inside a package rather than just the main module. You might need to go even deeper into browserify, substituting one of the last steps so it outputs a different format with objects that have metadata on the individual modules intended for use in a special loader instead of as a plain script tag.
Current RL on its own just won't work for this. Globals don't work well in the module pattern and so the ability to get a result from require() and allow npm modules to require() others are necessary. Circular requires also need consideration.
Whether you want a RL api that has a bridge to the npm ecosystem or want to go all-in to the modern js landscape and build a more CommonJS/node/npm-like RL api that loads things according to MW's use cases will be the important bike-shedding.
~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://danielfriesen.name/]
On 2015-11-05 8:32 AM, Trevor Parscal wrote:
The flat approach to NPM is a game changer for us, and a Bower killer.
Timo? Had a lot of insight at the time, I'd like to see him be invoked in this decision. Any thoughts Timo?
- Trevor
On Thursday, November 5, 2015, Jon Robson jrobson@wikimedia.org wrote:
It's been a year now, over 3-6 months. Volker can this be given a priority in the next frontend standards meeting. I think the lack of any standard is extremely damaging to the project.
On Wed, Jul 22, 2015 at 4:21 PM, Bryan Davis <bd808@wikimedia.org javascript:;> wrote:
On Wed, Jul 22, 2015 at 12:24 PM, Daniel Werner <daniel.a.r.werner@gmail.com javascript:;> wrote:
Hey,
I just wanted to check in about the status of enabling JavaScript
package
management usage in MediaWiki. I am basically talking about an
equivalent
for JS to what we have with Composer for PHP.
Real-world example: The "data-values/value-view" package[0] is defining "jquery.event.special.eachchange.js": ValueView/lib/jquery.event/jquery.event.special.eachchange.js
Now, recently I needed the same functionality in one of my
extensions, so
I just copied it over. [1]
I know that this is the worst way one could do this, but as far as I
can
see we don't have that much of a choice right now. Here are the
alternative
options I can see:
Moving "jquery.event.special.eachchange.js" out of the "data-values/value-view" package into its own "WMDE/jquery-eachchange" package...
- ... and using it in my extension via composer.
- pro: two or more extensions or other packages requiring this
package
will still result in having only one MW-wide installation..
- con: requires MW specific code which is actually not related to
the
MW-independent package to feed the resource loader.
- con: using Composer to manage pure JavaScript packages! Uuuh,
ugly!
- ... and having a build step in other packages using the package,
pulling
the "WMDE/jquery-eachchange" somewhere into the file structure of the packages/extensions using it.
- pro: don't need to abuse composer, we can use "npm", "Bower" or
any
other arbitrary JS package manager here.
- con: got to tell resource loader somehow... (didn't think so much
about
that yet)
- con: if more than one extensions or other packages require this
package
we still end up with the same code twice or more often in one MW installation.
- Combining 1 and 2: Start with 2, using a JS package manager. Then
going
to 1, creating a composer package and pulling the
"WMDE/jquery-eachchange"
package in via some build script.
- pro: The two pros from 1 + 2
- pro: ^^
- con: still got to tell resource loader somehow...
- con: Overhead; We now create two packages where the Composer one
is
just a bridge to the MW-world, still polluting packagist.org. Still
kind of
ugly and more effort for publishing a package and therefore
potentially
scaring programmers away from doing so since they've got better things
to
do than doing work that could be automated.
I have not seen Approach 2 and 3 yet. Though I could imagine that the VisualEditor team has used something like that. Approach 1 is the way the "data-values/value-view" package itself is
being
handled. And that package should actually be a MW independent pure JS package but right now it contains MW specific code and uses composer
for
distribution! There is still another option but that had to be properly implemented:
- Choose one native JS package manager for now and go with it (add
support
for others later perhaps). Integrate it properly with MW (resource
loader
to begin with), document how to use it and finally distribute JS code coming from the MW world but useful for other projects in a way where
it
can actually be used in a non-MW context.
This has already been bugging me when working on Wikidata. Now I'd
like
to
reuse some of the code I have written there without spending hours and hours with option 3 because there should be support for option 4
rather
sooner or later. So I am wondering; Does anyone have any thoughts, any alternatives
perhaps
or is there any roadmap on anything like the option 4 that I have
shown?
Cheers, Daniel
[1]:
https://github.com/DanweDE/mediawiki-ext-UserBitcoinAddresses/blob/master/re...
Option 4 was discussed last October as part of the Librarization project [0]. At the time the front end standards group wasn't ready to pick a winner in the javascript packaging landscape. They did want to revisit that in 3-6 months however so maybe it is time to find someone to look into the various options and their pros and cons again?
[0]: https://phabricator.wikimedia.org/T807
Bryan Davis Wikimedia Foundation <bd808@wikimedia.org
[[m:User:BDavis_(WMF)]] Sr Software Engineer Boise, ID USA irc: bd808 v:415.839.6885 x6855
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org javascript:; https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org javascript:; https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
(such top posting! much discussion!)
It would be great to see some of the salient points of this discussion captured on the current phab ticket [0] on the topic.
[0]: https://phabricator.wikimedia.org/T107561
Bryan
On Fri, Nov 6, 2015 at 1:03 AM, Daniel Friesen daniel@nadir-seen-fire.com wrote:
On 2015-11-05 1:30 PM, C. Scott Ananian wrote:
Two other interesting pieces:
- http://requirejs.org/ is still the goal-standard for async browser-type
loading, AFAIK, and there are good packages (`npm install requirejs`) that allow interoperability with the "npm style".
requirejs is still built for the same single application type as the other non-async loaders. You may not be able to get even requirejs to work with MediaWiki's needed pattern of different packages required by different extensions all integrated together without the use of node on the server that needs to combine them together.
- The recently-completed ES6 spec contains a module format. You can
already use it via compatibility thunks from many existing module systems. It may well see increased use, especially on the web as browsers implement the spec (which is happening quickly). There is resistance in the node community to adopting ES6 modules, but it may be that we are at an inflection point and ES6 will eventually win out.
ES6 modules have a different pattern for how exports are treated, especially in regards to the 'default' export. To handle this case babel inserts the following when you are transpiling ES6/ES2015 module syntax: Object.defineProperty(exports, '__esModule', { value: true });
Unless you explicitly enable loose mode any library you develop as an es module and anything that uses it will instantly break in browsers like IE8.
Use of ES6 import/export is gaining more and more adoption. But this might be one reason some people are holding back on it.
es modules also have different behavior in regards to cycles that I know esperanto handles but I'm not sure about the other transpilers.
~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://danielfriesen.name/]
--scott
On Thu, Nov 5, 2015 at 3:24 PM, Daniel Friesen daniel@nadir-seen-fire.com wrote:
As someone who now works in js/node primarily I might be able to add a little outside insight.
IMHO bower was probably killed far before npm@3 came out.
- npm has always been more advanced
- bower's main isn't as reliable as it is in npm. Even if modules aren't
forgetting to set it as much as they used to I still see hard to deal with things like modules that include .sass sources in it.
- bower's distribution model is actually very problematic for upstream
developers. bower clones git repositories directly expecting to find built/compiled js and has no local build hooks. Few major js libraries work in classic js/css sources anymore, instead having some required build step. Bower makes a complete mess of this. Requiring the build result to be committed with each change, an automated build bot, a second git repo for build results only, or the upstream dev to just not care about bower.
- Thanks to the rise of browserify and the like more and more
client-only libraries have moved to npm despite people traditionally thinking of it as being for nodejs modules. Most client only libraries now exist in npm. And if you wave the name browserify around you can get just about any js library to start distributing to npm.
- This has actually gone so far that once when I added a contribution
to a library I noticed that they were actually forgetting to keep their bower.json up to date.
npm@3 is also probably not as important as it's being made out here.
npm@3 still doesn't guarantee a tree will be 100% flat. Most of npm@3's changes fix small quirks in front-end usage and stability issues with npm.
The 'major' change of 'flatness' is really that when installing `a` that depends on `b`, `b` will be hoisted up to the same level as `a`, if possible, instead of always being installed under `a`. npm@2 did have some annoying quirks during development/updating that could leave a duplicate module until you recreated your node_modules tree. And there was the side case of installing two modules that both depended on something you did not depend on. But that could be dealt with by either adding it as a dep yourself or running dedupe.
The bad news is that while more and more libraries are moving to npm the underlying reason for many being on npm is for the many users using tools like browserify and webpack. So the assumption of many libraries is that when you use npm for client-side libraries is you are using it in a node-style/CommonJS way where require is available other npm dependencies can be used through it and you're not including things in the traditional way of libraries being run one after another on the page in the global scope like in ResourceLoader.
It's probably going to get harder and harder to reconcile deps shared between extensions and/or use library code itself without having node installed on servers. For that matter half the advancements in the css space probably won't be making their way to php either.
Though I do have some hints to things that might open up some possibilities.
Firstly browserify can build standalone versions of libraries. It's possible you could make tweaks to that pattern and have a development build step that would convert npm packages into modules built to run in a new type of ResourceLoader that would probably be a hybrid of the current RL and npm. However you would need to take care to exclude things or else you'll end up with a script that will duplicate the source of a bunch of other modules. You also have to be wary of handling the cases where npm intentionally duplicates a library.
This actually may be even more difficult than it sounds due to npm packages that require modules inside a package rather than just the main module. You might need to go even deeper into browserify, substituting one of the last steps so it outputs a different format with objects that have metadata on the individual modules intended for use in a special loader instead of as a plain script tag.
Current RL on its own just won't work for this. Globals don't work well in the module pattern and so the ability to get a result from require() and allow npm modules to require() others are necessary. Circular requires also need consideration.
Whether you want a RL api that has a bridge to the npm ecosystem or want to go all-in to the modern js landscape and build a more CommonJS/node/npm-like RL api that loads things according to MW's use cases will be the important bike-shedding.
~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://danielfriesen.name/]
On 2015-11-05 8:32 AM, Trevor Parscal wrote:
The flat approach to NPM is a game changer for us, and a Bower killer.
Timo? Had a lot of insight at the time, I'd like to see him be invoked in this decision. Any thoughts Timo?
- Trevor
On Thursday, November 5, 2015, Jon Robson jrobson@wikimedia.org wrote:
It's been a year now, over 3-6 months. Volker can this be given a priority in the next frontend standards meeting. I think the lack of any standard is extremely damaging to the project.
On Wed, Jul 22, 2015 at 4:21 PM, Bryan Davis <bd808@wikimedia.org javascript:;> wrote:
On Wed, Jul 22, 2015 at 12:24 PM, Daniel Werner <daniel.a.r.werner@gmail.com javascript:;> wrote: > Hey, > > I just wanted to check in about the status of enabling JavaScript
package
> management usage in MediaWiki. I am basically talking about an
equivalent
> for JS to what we have with Composer for PHP. > > Real-world example: > The "data-values/value-view" package[0] is defining > "jquery.event.special.eachchange.js": > ValueView/lib/jquery.event/jquery.event.special.eachchange.js > > Now, recently I needed the same functionality in one of my
extensions, so
> I just copied it over. [1] > > I know that this is the worst way one could do this, but as far as I
can
> see we don't have that much of a choice right now. Here are the
alternative
> options I can see: > > Moving "jquery.event.special.eachchange.js" out of the > "data-values/value-view" package into its own "WMDE/jquery-eachchange" > package... > > 1. ... and using it in my extension via composer. > + pro: two or more extensions or other packages requiring this
package
> will still result in having only one MW-wide installation.. > - con: requires MW specific code which is actually not related to
the
> MW-independent package to feed the resource loader. > - con: using Composer to manage pure JavaScript packages! Uuuh,
ugly!
> 2. ... and having a build step in other packages using the package,
pulling
> the "WMDE/jquery-eachchange" somewhere into the file structure of the > packages/extensions using it. > + pro: don't need to abuse composer, we can use "npm", "Bower" or
any
> other arbitrary JS package manager here. > - con: got to tell resource loader somehow... (didn't think so much
about
> that yet) > - con: if more than one extensions or other packages require this
package
> we still end up with the same code twice or more often in one MW > installation. > > 3. Combining 1 and 2: Start with 2, using a JS package manager. Then
going
> to 1, creating a composer package and pulling the
"WMDE/jquery-eachchange"
> package in via some build script. > + pro: The two pros from 1 + 2 > + pro: ^^ > - con: still got to tell resource loader somehow... > - con: Overhead; We now create two packages where the Composer one
is
> just a bridge to the MW-world, still polluting packagist.org. Still
kind of
> ugly and more effort for publishing a package and therefore
potentially
> scaring programmers away from doing so since they've got better things
to
> do than doing work that could be automated. > > I have not seen Approach 2 and 3 yet. Though I could imagine that the > VisualEditor team has used something like that. > Approach 1 is the way the "data-values/value-view" package itself is
being
> handled. And that package should actually be a MW independent pure JS > package but right now it contains MW specific code and uses composer
for
> distribution! > There is still another option but that had to be properly implemented: > > 4. Choose one native JS package manager for now and go with it (add
support
> for others later perhaps). Integrate it properly with MW (resource
loader
> to begin with), document how to use it and finally distribute JS code > coming from the MW world but useful for other projects in a way where
it
> can actually be used in a non-MW context. > > This has already been bugging me when working on Wikidata. Now I'd
like
to
> reuse some of the code I have written there without spending hours and > hours with option 3 because there should be support for option 4
rather
> sooner or later. > So I am wondering; Does anyone have any thoughts, any alternatives
perhaps
> or is there any roadmap on anything like the option 4 that I have
shown?
> Cheers, > Daniel > > [0]: https://packagist.org/packages/data-values/value-view > [1]: >
https://github.com/DanweDE/mediawiki-ext-UserBitcoinAddresses/blob/master/re...
Option 4 was discussed last October as part of the Librarization project [0]. At the time the front end standards group wasn't ready to pick a winner in the javascript packaging landscape. They did want to revisit that in 3-6 months however so maybe it is time to find someone to look into the various options and their pros and cons again?
wikitech-l@lists.wikimedia.org