Hello,
One of the blockers for the effort to package HHVM for Debian is the licensing of the JSON extension. This is the same problem that has afflicted PHP (see https://bugs.php.net/bug.php?id=63520: "JSON extension includes a problematic license statement") and that we have discussed in the context of bug 47413 (< https://bugzilla.wikimedia.org/show_bug.cgi?id=47431%3E: "JSON extension dependency has a non-free component".
There is a drop-in replacement which is not encumbered with licensing issues: Remi Collet's pecl-json-c, available at < https://github.com/remicollet/pecl-json-c%3E. Paul Tarjan replied on the packaging thread (see < https://www.mail-archive.com/debian-bugs-dist@lists.debian.org/msg1208263.ht...), saying that the HHVM team would be interested in swapping out the JSON library and replacing it with pecl-json-c, if someone were to port it. Since both extensions provide the same API, I imagine "porting" in this context means tweaking the include paths and the build configuration.
Are any of you interested in taking this on?
I took a look at this over the weekend. Based on the existing code in hhvm only the json_decode pathway needs to be replaced. encoding is implemented independently and shares a common pathway with a variety of other serializations. I've gone ahead and done the initial work of porting over the linked pecl extension's json_decode into hhvm's codebase[1]. There are two issues that i'm not sure how to solve:
1) the json_encode pathway doesn't need to be replaced, but it does make a single method call[2] to a function from the previous library:
void utf16_to_utf8(HPHP::StringBuffer &buf, unsigned short utf16);
I don't really know enough in this realm to go about re-implementing this. It takes the unsigned short and append its utf-8 representation to the buffer. hhvm does have an intl_convert_utf16_to_utf8 method, but it works on full strings so passing it two bytes at a time would be undesirable.
2) hhvm has a special mode FB_LOOSE which allows invalid json. json-c has a strict mode which i've only enabled when FB_LOOSE is not set, but this does not pass their unit tests of previously accepted invalid json. I have not changed the method signatures between the json extension and the parser, rather implemented the existing method signatures with the json-c parser, so with a bit more work json-c could just be a compile time flag, but not sure i like that solution either.
Suggestions?
Erik B.
[1] https://github.com/ebernhardson/hiphop-php/commit/2619eb96c29f17e84ed0781f57... [2] https://github.com/facebook/hhvm/blob/master/hphp/runtime/base/variable-seri...
On Wed, Apr 16, 2014 at 3:42 PM, Ori Livneh ori@wikimedia.org wrote:
Hello,
One of the blockers for the effort to package HHVM for Debian is the licensing of the JSON extension. This is the same problem that has afflicted PHP (see https://bugs.php.net/bug.php?id=63520: "JSON extension includes a problematic license statement") and that we have discussed in the context of bug 47413 (< https://bugzilla.wikimedia.org/show_bug.cgi?id=47431%3E: "JSON extension dependency has a non-free component".
There is a drop-in replacement which is not encumbered with licensing issues: Remi Collet's pecl-json-c, available at < https://github.com/remicollet/pecl-json-c%3E. Paul Tarjan replied on the packaging thread (see < https://www.mail-archive.com/debian-bugs-dist@lists.debian.org/msg1208263.ht...), saying that the HHVM team would be interested in swapping out the JSON library and replacing it with pecl-json-c, if someone were to port it. Since both extensions provide the same API, I imagine "porting" in this context means tweaking the include paths and the build configuration.
Are any of you interested in taking this on?
+Faidon!
On Sun, Apr 27, 2014 at 1:10 PM, Erik Bernhardson < ebernhardson@wikimedia.org> wrote:
I took a look at this over the weekend. Based on the existing code in hhvm only the json_decode pathway needs to be replaced. encoding is implemented independently and shares a common pathway with a variety of other serializations. I've gone ahead and done the initial work of porting over the linked pecl extension's json_decode into hhvm's codebase[1]. There are two issues that i'm not sure how to solve:
- the json_encode pathway doesn't need to be replaced, but it does make a
single method call[2] to a function from the previous library:
void utf16_to_utf8(HPHP::StringBuffer &buf, unsigned short utf16);I don't really know enough in this realm to go about re-implementing this. It takes the unsigned short and append its utf-8 representation to the buffer. hhvm does have an intl_convert_utf16_to_utf8 method, but it works on full strings so passing it two bytes at a time would be undesirable.
- hhvm has a special mode FB_LOOSE which allows invalid json. json-c has
a strict mode which i've only enabled when FB_LOOSE is not set, but this does not pass their unit tests of previously accepted invalid json. I have not changed the method signatures between the json extension and the parser, rather implemented the existing method signatures with the json-c parser, so with a bit more work json-c could just be a compile time flag, but not sure i like that solution either.
Suggestions?
Erik B.
[1] https://github.com/ebernhardson/hiphop-php/commit/2619eb96c29f17e84ed0781f57... [2] https://github.com/facebook/hhvm/blob/master/hphp/runtime/base/variable-seri...
On Wed, Apr 16, 2014 at 3:42 PM, Ori Livneh ori@wikimedia.org wrote:
Hello,
One of the blockers for the effort to package HHVM for Debian is the licensing of the JSON extension. This is the same problem that has afflicted PHP (see https://bugs.php.net/bug.php?id=63520: "JSON extension includes a problematic license statement") and that we have discussed in the context of bug 47413 (< https://bugzilla.wikimedia.org/show_bug.cgi?id=47431%3E: "JSON extension dependency has a non-free component".
There is a drop-in replacement which is not encumbered with licensing issues: Remi Collet's pecl-json-c, available at < https://github.com/remicollet/pecl-json-c%3E. Paul Tarjan replied on the packaging thread (see < https://www.mail-archive.com/debian-bugs-dist@lists.debian.org/msg1208263.ht...), saying that the HHVM team would be interested in swapping out the JSON library and replacing it with pecl-json-c, if someone were to port it. Since both extensions provide the same API, I imagine "porting" in this context means tweaking the include paths and the build configuration.
Are any of you interested in taking this on?
Erik,
As I told Ori before -but probably after his email asking for help :(-, my understanding from posts on public mail threads by Luis, WMF Legal doesn't consider the evil JSON license as undistributable, but "just" FSF/OSI non-free. In fact, the PHP that we run in production right now has the exact same code in it, so this is not a new HHVM issue by any chance. Therefore, this problem is *not* a blocker for our HHVM deployment.
That being said, this is really awesome work. Many thanks for doing this. I've discussed this matter with Facebook engineers in the past and my understanding is that they would like to rip this code off from their codebase as well; their lawyers are probably okay with it, but it harms their reputation as an open-source project, as for example it would be a blocker for Debian/Ubuntu inclusion.
I would suggest to take this preliminary work of yours and post it as a pull-request on HHVM's GitHub (but marking it as draft). I'm sure they'd appreciate it and we might avoid some duplication of effort here. They also would be able to give their opinion to you or even finish it off. I can also imagine one of the possible outcomes being "FB_LOOSE is a Facebook internal thing anyway, so we'll merge this and keep a patch to revert to the old code in our internal fork of HHVM until a better solution is found".
Thanks again! Faidon
On Mon, Apr 28, 2014 at 10:34:58AM -0700, Ori Livneh wrote:
+Faidon!
On Sun, Apr 27, 2014 at 1:10 PM, Erik Bernhardson < ebernhardson@wikimedia.org> wrote:
I took a look at this over the weekend. Based on the existing code in hhvm only the json_decode pathway needs to be replaced. encoding is implemented independently and shares a common pathway with a variety of other serializations. I've gone ahead and done the initial work of porting over the linked pecl extension's json_decode into hhvm's codebase[1]. There are two issues that i'm not sure how to solve:
- the json_encode pathway doesn't need to be replaced, but it does make a
single method call[2] to a function from the previous library:
void utf16_to_utf8(HPHP::StringBuffer &buf, unsigned short utf16);I don't really know enough in this realm to go about re-implementing this. It takes the unsigned short and append its utf-8 representation to the buffer. hhvm does have an intl_convert_utf16_to_utf8 method, but it works on full strings so passing it two bytes at a time would be undesirable.
- hhvm has a special mode FB_LOOSE which allows invalid json. json-c has
a strict mode which i've only enabled when FB_LOOSE is not set, but this does not pass their unit tests of previously accepted invalid json. I have not changed the method signatures between the json extension and the parser, rather implemented the existing method signatures with the json-c parser, so with a bit more work json-c could just be a compile time flag, but not sure i like that solution either.
Suggestions?
Erik B.
[1] https://github.com/ebernhardson/hiphop-php/commit/2619eb96c29f17e84ed0781f57... [2] https://github.com/facebook/hhvm/blob/master/hphp/runtime/base/variable-seri...
On Wed, Apr 16, 2014 at 3:42 PM, Ori Livneh ori@wikimedia.org wrote:
Hello,
One of the blockers for the effort to package HHVM for Debian is the licensing of the JSON extension. This is the same problem that has afflicted PHP (see https://bugs.php.net/bug.php?id=63520: "JSON extension includes a problematic license statement") and that we have discussed in the context of bug 47413 (< https://bugzilla.wikimedia.org/show_bug.cgi?id=47431%3E: "JSON extension dependency has a non-free component".
There is a drop-in replacement which is not encumbered with licensing issues: Remi Collet's pecl-json-c, available at < https://github.com/remicollet/pecl-json-c%3E. Paul Tarjan replied on the packaging thread (see < https://www.mail-archive.com/debian-bugs-dist@lists.debian.org/msg1208263.ht...), saying that the HHVM team would be interested in swapping out the JSON library and replacing it with pecl-json-c, if someone were to port it. Since both extensions provide the same API, I imagine "porting" in this context means tweaking the include paths and the build configuration.
Are any of you interested in taking this on?
mediawiki-core@lists.wikimedia.org