Here's my requirement: - a wiki page is one JSON document - when editing, the user edits the JSON directly - when viewing, I have a viewer that turns the JSON into wikitext, and that wikitext gets rendered as wikitext and turned into HTML by MediaWiki
I have several options, including:
1) hook for a tag like <json>, and write an extension that parses the content between the tags and turns it into wikitext (not ideal, as I don't use any of the existing support for JSON stuff, and also I could have several such tags per page, which does not fit with my requirements)
2) I found the JsonConfig extension by yurik. This allows me to do almost all of the things above - but it returns HTML directly, not wikitext. It doesn't seem trivial to be able to return wikitext instead of HTML, but hopefully I am wrong? Also, this ties in nicely with the Code Editor.
3) there is actually a JsonContentHandler in core. But looking through it it seems that this suffers from the same limitations - I can return HTML, but not wikitext.
3 seems to have the advantage to be more actively worked on that 2 (which is not based on 3, probably because it is older than 3). So future goodies like a Json Schema validator will probably go to 3, but not to 2, so I should probably go to 3.
Writing this down, one solution could be to create the wikitext, and then call the wikitext parser manually and have it create HTML?
I have already developed the extension in 1, and then fully rewritten it in 2. Before I go and rewrite it again in 3, I wanted to ask whether I am doing it right, or if should do it completely differently, and also if there are examples of stuff developed in 3, i.e. of extensions or features using the JsonContent class.
Example: I have a JSON document { "username": "Denny" } which gets turned into wikitext ''Hello, [[User:Denny|Denny]]!'' which then gets turned into the right HTML and displayed to the user, e.g. <i>Hello, <a href="...">Denny</a>!</i>
Cheers, Denny
Why, exactly, do you want a wikitext intermediary between your JSON and your HTML? The value of wikitext is that it’s a syntax that is easier to edit than HTML. But if it’s not the native format of your data, nor can browsers render it directly, what’s the point of having it?
The CollaborationKit extension [0] has two content models, CollaborationHubContent and CollaborationListContent, and they have two different strategies for parsing. CollaborationHubContent takes validated JSON and puts out raw HTML; this is the most straightforward to work with. CollaborationListContent instead puts out wikitext that is fed into the parser, since it is expected that lists can be transcluded onto other pages, and this means using wikitext (as transcluded HTML is not an option). However, this creates a lot of limitations, including parser restrictions that make sense in the context of arbitrary wikitext parsing but not when the markup is provided directly by an extension. The long term plan is for CollaborationListContent to put out HTML, since it’s more straightforward than using a wikitext intermediary that the user does not see anyway.
[0] https://www.mediawiki.org/wiki/Extension:CollaborationKit
On April 8, 2017 at 11:23:38 PM, Denny Vrandečić (vrandecic@gmail.com) wrote:
Here's my requirement: - a wiki page is one JSON document - when editing, the user edits the JSON directly - when viewing, I have a viewer that turns the JSON into wikitext, and that wikitext gets rendered as wikitext and turned into HTML by MediaWiki
I have several options, including:
1) hook for a tag like <json>, and write an extension that parses the content between the tags and turns it into wikitext (not ideal, as I don't use any of the existing support for JSON stuff, and also I could have several such tags per page, which does not fit with my requirements)
2) I found the JsonConfig extension by yurik. This allows me to do almost all of the things above - but it returns HTML directly, not wikitext. It doesn't seem trivial to be able to return wikitext instead of HTML, but hopefully I am wrong? Also, this ties in nicely with the Code Editor.
3) there is actually a JsonContentHandler in core. But looking through it it seems that this suffers from the same limitations - I can return HTML, but not wikitext.
3 seems to have the advantage to be more actively worked on that 2 (which is not based on 3, probably because it is older than 3). So future goodies like a Json Schema validator will probably go to 3, but not to 2, so I should probably go to 3.
Writing this down, one solution could be to create the wikitext, and then call the wikitext parser manually and have it create HTML?
I have already developed the extension in 1, and then fully rewritten it in 2. Before I go and rewrite it again in 3, I wanted to ask whether I am doing it right, or if should do it completely differently, and also if there are examples of stuff developed in 3, i.e. of extensions or features using the JsonContent class.
Example: I have a JSON document { "username": "Denny" } which gets turned into wikitext ''Hello, [[User:Denny|Denny]]!'' which then gets turned into the right HTML and displayed to the user, e.g. <i>Hello, <a href="...">Denny</a>!</i>
Cheers, Denny _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Sounds like what they want is what collaborationlistcontent/handler does, specifically, at least for now.
On 09/04/17 06:30, James Hare wrote:
Why, exactly, do you want a wikitext intermediary between your JSON and your HTML? The value of wikitext is that it’s a syntax that is easier to edit than HTML. But if it’s not the native format of your data, nor can browsers render it directly, what’s the point of having it?
The CollaborationKit extension [0] has two content models, CollaborationHubContent and CollaborationListContent, and they have two different strategies for parsing. CollaborationHubContent takes validated JSON and puts out raw HTML; this is the most straightforward to work with. CollaborationListContent instead puts out wikitext that is fed into the parser, since it is expected that lists can be transcluded onto other pages, and this means using wikitext (as transcluded HTML is not an option). However, this creates a lot of limitations, including parser restrictions that make sense in the context of arbitrary wikitext parsing but not when the markup is provided directly by an extension. The long term plan is for CollaborationListContent to put out HTML, since it’s more straightforward than using a wikitext intermediary that the user does not see anyway.
[0] https://www.mediawiki.org/wiki/Extension:CollaborationKit
On April 8, 2017 at 11:23:38 PM, Denny Vrandečić (vrandecic@gmail.com) wrote:
Here's my requirement:
- a wiki page is one JSON document
- when editing, the user edits the JSON directly
- when viewing, I have a viewer that turns the JSON into wikitext, and that
wikitext gets rendered as wikitext and turned into HTML by MediaWiki
I have several options, including:
- hook for a tag like <json>, and write an extension that parses the
content between the tags and turns it into wikitext (not ideal, as I don't use any of the existing support for JSON stuff, and also I could have several such tags per page, which does not fit with my requirements)
- I found the JsonConfig extension by yurik. This allows me to do almost
all of the things above - but it returns HTML directly, not wikitext. It doesn't seem trivial to be able to return wikitext instead of HTML, but hopefully I am wrong? Also, this ties in nicely with the Code Editor.
- there is actually a JsonContentHandler in core. But looking through it
it seems that this suffers from the same limitations - I can return HTML, but not wikitext.
3 seems to have the advantage to be more actively worked on that 2 (which is not based on 3, probably because it is older than 3). So future goodies like a Json Schema validator will probably go to 3, but not to 2, so I should probably go to 3.
Writing this down, one solution could be to create the wikitext, and then call the wikitext parser manually and have it create HTML?
I have already developed the extension in 1, and then fully rewritten it in 2. Before I go and rewrite it again in 3, I wanted to ask whether I am doing it right, or if should do it completely differently, and also if there are examples of stuff developed in 3, i.e. of extensions or features using the JsonContent class.
Example: I have a JSON document { "username": "Denny" } which gets turned into wikitext ''Hello, [[User:Denny|Denny]]!'' which then gets turned into the right HTML and displayed to the user, e.g. <i>Hello, <a href="...">Denny</a>!</i>
Cheers, Denny _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
An other maybe relevant example : ProofreadPage has a content model for proofreading pages with multiple areas of Wikitext and use some for transclusion and others for rendering:
* https://github.com/wikimedia/mediawiki-extensions-ProofreadPage/blob/master/... * https://github.com/wikimedia/mediawiki-extensions-ProofreadPage/blob/master/...
Thomas
Le 9 avr. 2017 à 18:44, Isarra Yos zhorishna@gmail.com a écrit :
Sounds like what they want is what collaborationlistcontent/handler does, specifically, at least for now.
On 09/04/17 06:30, James Hare wrote:
Why, exactly, do you want a wikitext intermediary between your JSON and your HTML? The value of wikitext is that it’s a syntax that is easier to edit than HTML. But if it’s not the native format of your data, nor can browsers render it directly, what’s the point of having it?
The CollaborationKit extension [0] has two content models, CollaborationHubContent and CollaborationListContent, and they have two different strategies for parsing. CollaborationHubContent takes validated JSON and puts out raw HTML; this is the most straightforward to work with. CollaborationListContent instead puts out wikitext that is fed into the parser, since it is expected that lists can be transcluded onto other pages, and this means using wikitext (as transcluded HTML is not an option). However, this creates a lot of limitations, including parser restrictions that make sense in the context of arbitrary wikitext parsing but not when the markup is provided directly by an extension. The long term plan is for CollaborationListContent to put out HTML, since it’s more straightforward than using a wikitext intermediary that the user does not see anyway.
[0] https://www.mediawiki.org/wiki/Extension:CollaborationKit
On April 8, 2017 at 11:23:38 PM, Denny Vrandečić (vrandecic@gmail.com) wrote:
Here's my requirement:
- a wiki page is one JSON document
- when editing, the user edits the JSON directly
- when viewing, I have a viewer that turns the JSON into wikitext, and that
wikitext gets rendered as wikitext and turned into HTML by MediaWiki
I have several options, including:
- hook for a tag like <json>, and write an extension that parses the
content between the tags and turns it into wikitext (not ideal, as I don't use any of the existing support for JSON stuff, and also I could have several such tags per page, which does not fit with my requirements)
- I found the JsonConfig extension by yurik. This allows me to do almost
all of the things above - but it returns HTML directly, not wikitext. It doesn't seem trivial to be able to return wikitext instead of HTML, but hopefully I am wrong? Also, this ties in nicely with the Code Editor.
- there is actually a JsonContentHandler in core. But looking through it
it seems that this suffers from the same limitations - I can return HTML, but not wikitext.
3 seems to have the advantage to be more actively worked on that 2 (which is not based on 3, probably because it is older than 3). So future goodies like a Json Schema validator will probably go to 3, but not to 2, so I should probably go to 3.
Writing this down, one solution could be to create the wikitext, and then call the wikitext parser manually and have it create HTML?
I have already developed the extension in 1, and then fully rewritten it in 2. Before I go and rewrite it again in 3, I wanted to ask whether I am doing it right, or if should do it completely differently, and also if there are examples of stuff developed in 3, i.e. of extensions or features using the JsonContent class.
Example: I have a JSON document { "username": "Denny" } which gets turned into wikitext ''Hello, [[User:Denny|Denny]]!'' which then gets turned into the right HTML and displayed to the user, e.g. <i>Hello, <a href="...">Denny</a>!</i>
Cheers, Denny _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Thanks Tpt, that sounds relevant! I will take a look into the code. More code examples are really useful to figure this all out :)
On Sun, Apr 9, 2017 at 9:53 AM Thomas PT thomaspt@hotmail.fr wrote:
An other maybe relevant example : ProofreadPage has a content model for proofreading pages with multiple areas of Wikitext and use some for transclusion and others for rendering:
https://github.com/wikimedia/mediawiki-extensions-ProofreadPage/blob/master/...
https://github.com/wikimedia/mediawiki-extensions-ProofreadPage/blob/master/...
Thomas
Le 9 avr. 2017 à 18:44, Isarra Yos zhorishna@gmail.com a écrit :
Sounds like what they want is what collaborationlistcontent/handler
does, specifically, at least for now.
On 09/04/17 06:30, James Hare wrote:
Why, exactly, do you want a wikitext intermediary between your JSON and your HTML? The value of wikitext is that it’s a syntax that is easier to edit than HTML. But if it’s not the native format of your data, nor can browsers render it directly, what’s the point of having it?
The CollaborationKit extension [0] has two content models, CollaborationHubContent and CollaborationListContent, and they have two different strategies for parsing. CollaborationHubContent takes
validated
JSON and puts out raw HTML; this is the most straightforward to work
with.
CollaborationListContent instead puts out wikitext that is fed into the parser, since it is expected that lists can be transcluded onto other pages, and this means using wikitext (as transcluded HTML is not an option). However, this creates a lot of limitations, including parser restrictions that make sense in the context of arbitrary wikitext
parsing
but not when the markup is provided directly by an extension. The long
term
plan is for CollaborationListContent to put out HTML, since it’s more straightforward than using a wikitext intermediary that the user does
not
see anyway.
[0] https://www.mediawiki.org/wiki/Extension:CollaborationKit
On April 8, 2017 at 11:23:38 PM, Denny Vrandečić (vrandecic@gmail.com) wrote:
Here's my requirement:
- a wiki page is one JSON document
- when editing, the user edits the JSON directly
- when viewing, I have a viewer that turns the JSON into wikitext, and
that
wikitext gets rendered as wikitext and turned into HTML by MediaWiki
I have several options, including:
- hook for a tag like <json>, and write an extension that parses the
content between the tags and turns it into wikitext (not ideal, as I
don't
use any of the existing support for JSON stuff, and also I could have several such tags per page, which does not fit with my requirements)
- I found the JsonConfig extension by yurik. This allows me to do
almost
all of the things above - but it returns HTML directly, not wikitext. It doesn't seem trivial to be able to return wikitext instead of HTML, but hopefully I am wrong? Also, this ties in nicely with the Code Editor.
- there is actually a JsonContentHandler in core. But looking through
it
it seems that this suffers from the same limitations - I can return
HTML,
but not wikitext.
3 seems to have the advantage to be more actively worked on that 2
(which
is not based on 3, probably because it is older than 3). So future
goodies
like a Json Schema validator will probably go to 3, but not to 2, so I should probably go to 3.
Writing this down, one solution could be to create the wikitext, and
then
call the wikitext parser manually and have it create HTML?
I have already developed the extension in 1, and then fully rewritten
it in
- Before I go and rewrite it again in 3, I wanted to ask whether I am
doing it right, or if should do it completely differently, and also if there are examples of stuff developed in 3, i.e. of extensions or
features
using the JsonContent class.
Example: I have a JSON document { "username": "Denny" } which gets turned into wikitext ''Hello, [[User:Denny|Denny]]!'' which then gets turned into the right HTML and displayed to the user,
e.g.
<i>Hello, <a href="...">Denny</a>!</i>
Cheers, Denny _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Sat, Apr 8, 2017 at 11:30 PM James Hare jamesmhare@gmail.com wrote:
Why, exactly, do you want a wikitext intermediary between your JSON and your HTML? The value of wikitext is that it’s a syntax that is easier to edit than HTML. But if it’s not the native format of your data, nor can browsers render it directly, what’s the point of having it?
Ah, good question indeed. The reason is that users would be actually putting fragments of wikitext into the JSON structure, and then the JSON structure gets assembled into wikitext. Not only would I prefer to have the users work with fragments of wikitext than fragments of HTML, but some things are almost impossible with HTML - e.g. making internal links red or blue depending on the existence of the article, etc.
The CollaborationKit extension [0] has two content models, CollaborationHubContent and CollaborationListContent, and they have two different strategies for parsing. CollaborationHubContent takes validated JSON and puts out raw HTML; this is the most straightforward to work with. CollaborationListContent instead puts out wikitext that is fed into the parser, since it is expected that lists can be transcluded onto other pages, and this means using wikitext (as transcluded HTML is not an option). However, this creates a lot of limitations, including parser restrictions that make sense in the context of arbitrary wikitext parsing but not when the markup is provided directly by an extension. The long term plan is for CollaborationListContent to put out HTML, since it’s more straightforward than using a wikitext intermediary that the user does not see anyway.
Thanks, that is super helpful to know! And thanks for the rationale in particular.
[0] https://www.mediawiki.org/wiki/Extension:CollaborationKit
On April 8, 2017 at 11:23:38 PM, Denny Vrandečić (vrandecic@gmail.com) wrote:
Here's my requirement:
- a wiki page is one JSON document
- when editing, the user edits the JSON directly
- when viewing, I have a viewer that turns the JSON into wikitext, and that
wikitext gets rendered as wikitext and turned into HTML by MediaWiki
I have several options, including:
- hook for a tag like <json>, and write an extension that parses the
content between the tags and turns it into wikitext (not ideal, as I don't use any of the existing support for JSON stuff, and also I could have several such tags per page, which does not fit with my requirements)
- I found the JsonConfig extension by yurik. This allows me to do almost
all of the things above - but it returns HTML directly, not wikitext. It doesn't seem trivial to be able to return wikitext instead of HTML, but hopefully I am wrong? Also, this ties in nicely with the Code Editor.
- there is actually a JsonContentHandler in core. But looking through it
it seems that this suffers from the same limitations - I can return HTML, but not wikitext.
3 seems to have the advantage to be more actively worked on that 2 (which is not based on 3, probably because it is older than 3). So future goodies like a Json Schema validator will probably go to 3, but not to 2, so I should probably go to 3.
Writing this down, one solution could be to create the wikitext, and then call the wikitext parser manually and have it create HTML?
I have already developed the extension in 1, and then fully rewritten it in 2. Before I go and rewrite it again in 3, I wanted to ask whether I am doing it right, or if should do it completely differently, and also if there are examples of stuff developed in 3, i.e. of extensions or features using the JsonContent class.
Example: I have a JSON document { "username": "Denny" } which gets turned into wikitext ''Hello, [[User:Denny|Denny]]!'' which then gets turned into the right HTML and displayed to the user, e.g. <i>Hello, <a href="...">Denny</a>!</i>
Cheers, Denny
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Sun, Apr 9, 2017 at 11:38 AM, Daniel Kinzler <daniel.kinzler@wikimedia.de
wrote:
Generating wikitext from some other thing is what Scribunto does.
Not really. What Scribunto does is let you run a program to generate wikitext.
If you wanted to write code that took some JSON and turned it into wikitext without going through all the trouble of writing an extension and getting it deployed, you might write that code in Lua as a Scribunto module.
On Mon, Apr 10, 2017 at 12:17 AM, Denny Vrandečić vrandecic@gmail.com wrote:
On Sat, Apr 8, 2017 at 11:30 PM James Hare jamesmhare@gmail.com wrote:
Why, exactly, do you want a wikitext intermediary between your JSON and your HTML? The value of wikitext is that it’s a syntax that is easier to edit than HTML. But if it’s not the native format of your data, nor can browsers render it directly, what’s the point of having it?
Ah, good question indeed. The reason is that users would be actually putting fragments of wikitext into the JSON structure, and then the JSON structure gets assembled into wikitext. Not only would I prefer to have the users work with fragments of wikitext than fragments of HTML, but some things are almost impossible with HTML - e.g. making internal links red or blue depending on the existence of the article, etc.
What you probably want to do then is to extend JsonContent and JsonContentHandler. In the fillParserOutput() method, you'd convert the JSON to wikitext and then pass that wikitext to the Parser; for the latter step you could look at how WikitextContent does it.
You might also look at implementing Content::getWikitextForTransclusion() to let people transclude the resulting wikitext.
Thanks Brad, that's perfect! I'll proceed as suggested.
Thanks list, that was really helpful and saved me plenty of trial and errors!
On Mon, Apr 10, 2017 at 7:07 AM Brad Jorsch (Anomie) bjorsch@wikimedia.org wrote:
On Sun, Apr 9, 2017 at 11:38 AM, Daniel Kinzler < daniel.kinzler@wikimedia.de
wrote:
Generating wikitext from some other thing is what Scribunto does.
Not really. What Scribunto does is let you run a program to generate wikitext.
If you wanted to write code that took some JSON and turned it into wikitext without going through all the trouble of writing an extension and getting it deployed, you might write that code in Lua as a Scribunto module.
On Mon, Apr 10, 2017 at 12:17 AM, Denny Vrandečić vrandecic@gmail.com wrote:
On Sat, Apr 8, 2017 at 11:30 PM James Hare jamesmhare@gmail.com wrote:
Why, exactly, do you want a wikitext intermediary between your JSON and your HTML? The value of wikitext is that it’s a syntax that is easier
to
edit than HTML. But if it’s not the native format of your data, nor can browsers render it directly, what’s the point of having it?
Ah, good question indeed. The reason is that users would be actually putting fragments of wikitext into the JSON structure, and then the JSON structure gets assembled into wikitext. Not only would I prefer to have
the
users work with fragments of wikitext than fragments of HTML, but some things are almost impossible with HTML - e.g. making internal links red
or
blue depending on the existence of the article, etc.
What you probably want to do then is to extend JsonContent and JsonContentHandler. In the fillParserOutput() method, you'd convert the JSON to wikitext and then pass that wikitext to the Parser; for the latter step you could look at how WikitextContent does it.
You might also look at implementing Content::getWikitextForTransclusion() to let people transclude the resulting wikitext.
-- Brad Jorsch (Anomie) Senior Software Engineer Wikimedia Foundation _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Am 10.04.2017 um 06:17 schrieb Denny Vrandečić:
Ah, good question indeed. The reason is that users would be actually putting fragments of wikitext into the JSON structure, and then the JSON structure gets assembled into wikitext. Not only would I prefer to have the users work with fragments of wikitext than fragments of HTML, but some things are almost impossible with HTML - e.g. making internal links red or blue depending on the existence of the article, etc.
Why JSON? JSON is an absolute pain to edit by hand. HTML/XML, as annoying as it is, is still much better for that than JSON is. And Wikitext is designed to mix well with HTML.
Ah, just an internal data structure. In the end, the UI will be form-based anyway. But in these forms, the user will be able to enter some wikitext fragments. Yucky, I know.
I prefer JSON over XML only because it is the Zeitgeist, and I expect the tool support for JSON to grow whereas I don't see similar activity around XML in the MediaWiki development community.
Or, put differently, the same reason Wikibase is using JSON.
On Mon, Apr 10, 2017 at 11:06 AM Daniel Kinzler daniel.kinzler@wikimedia.de wrote:
Am 10.04.2017 um 06:17 schrieb Denny Vrandečić:
Ah, good question indeed. The reason is that users would be actually putting fragments of wikitext into the JSON structure, and then the JSON structure gets assembled into wikitext. Not only would I prefer to have
the
users work with fragments of wikitext than fragments of HTML, but some things are almost impossible with HTML - e.g. making internal links red
or
blue depending on the existence of the article, etc.
Why JSON? JSON is an absolute pain to edit by hand. HTML/XML, as annoying as it is, is still much better for that than JSON is. And Wikitext is designed to mix well with HTML.
-- Daniel Kinzler Principal Platform Engineer
Wikimedia Deutschland Gesellschaft zur Förderung Freien Wissens e.V.
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On 2017-04-10 06:17, Denny Vrandečić wrote:
On Sat, Apr 8, 2017 at 11:30 PM James Hare jamesmhare@gmail.com wrote:
Why, exactly, do you want a wikitext intermediary between your JSON and your HTML? The value of wikitext is that it’s a syntax that is easier to edit than HTML. But if it’s not the native format of your data, nor can browsers render it directly, what’s the point of having it?
Ah, good question indeed. The reason is that users would be actually putting fragments of wikitext into the JSON structure, and then the JSON structure gets assembled into wikitext. Not only would I prefer to have the users work with fragments of wikitext than fragments of HTML, but some things are almost impossible with HTML - e.g. making internal links red or blue depending on the existence of the article, etc.
It would be more reliable to parse each fragment of wikitext separately, and then build the HTML out of them. If you try to make a big chunk of wikitext with user-supplied fragments, you'll soon run into two problems:
* Users will accidentally put '<nowiki>' or something into one of the fields and completely mess up the rendering. * Users will intentionally put '[[Foo|' in one field and ']]' into another and then be mad at you when you change the structure in a minor way and their link no longer works.
Good points. All too familiar from trying to parse and understand the ways the templating system has been used :)
Unfortunately, I am unsure whether I can restrict the usage this tightly, and whether I'd rather let the users shoot themselves in the foot or restrict them from doing so... age old question!
On Mon, Apr 10, 2017 at 2:42 PM Bartosz Dziewoński matma.rex@gmail.com wrote:
On 2017-04-10 06:17, Denny Vrandečić wrote:
On Sat, Apr 8, 2017 at 11:30 PM James Hare jamesmhare@gmail.com wrote:
Why, exactly, do you want a wikitext intermediary between your JSON and your HTML? The value of wikitext is that it’s a syntax that is easier to edit than HTML. But if it’s not the native format of your data, nor can browsers render it directly, what’s the point of having it?
Ah, good question indeed. The reason is that users would be actually putting fragments of wikitext into the JSON structure, and then the JSON structure gets assembled into wikitext. Not only would I prefer to have
the
users work with fragments of wikitext than fragments of HTML, but some things are almost impossible with HTML - e.g. making internal links red
or
blue depending on the existence of the article, etc.
It would be more reliable to parse each fragment of wikitext separately, and then build the HTML out of them. If you try to make a big chunk of wikitext with user-supplied fragments, you'll soon run into two problems:
- Users will accidentally put '<nowiki>' or something into one of the
fields and completely mess up the rendering.
- Users will intentionally put '[[Foo|' in one field and ']]' into
another and then be mad at you when you change the structure in a minor way and their link no longer works.
-- Bartosz Dziewoński
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Am 11.04.2017 um 02:35 schrieb Denny Vrandečić:
Unfortunately, I am unsure whether I can restrict the usage this tightly, and whether I'd rather let the users shoot themselves in the foot or restrict them from doing so... age old question!
That indeed is one of the ultimate questions, from UI design to law making!
You probably want to subclass JsonContentHandler and add wikitext transform and whatever else you need. For schemas, have a look at JsonSchemaContentHandler in EventLogging.
That seems like a performance hog in the making in my opinion to not mention, mostly unneeded as far as I can tell.
Sent from my iPhone
On Apr 9, 2017, at 6:11 AM, Gergo Tisza gtisza@wikimedia.org wrote:
You probably want to subclass JsonContentHandler and add wikitext transform and whatever else you need. For schemas, have a look at JsonSchemaContentHandler in EventLogging. _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Sun, Apr 9, 2017 at 4:11 AM Gergo Tisza gtisza@wikimedia.org wrote:
You probably want to subclass JsonContentHandler and add wikitext transform
What does it mean to add wikitext transform?
and whatever else you need. For schemas, have a look at JsonSchemaContentHandler in EventLogging.
Thanks, that looks very helpful! I am surprised this is not part of Core, or a stand-alone component, but it is in EventLogging. Thanks!
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Am 09.04.2017 um 08:23 schrieb Denny Vrandečić:
Here's my requirement:
- a wiki page is one JSON document
- when editing, the user edits the JSON directly
- when viewing, I have a viewer that turns the JSON into wikitext, and that
wikitext gets rendered as wikitext and turned into HTML by MediaWiki
Quick thoughts off the top of my head:
Generating wikitext from some other thing is what Scribunto does. Instead of using the Lua handler, you would make a handler for Json, with whetever rules you like for generating wikitext.
I have ne4ver looked at customizing Scribunto, but this seems to right place to plug this in.
The alternative is to try to have some kind of "cascading" ContentHandler, that generates a WikiTextContent object first, and then turns that into HTML.
On Sun, Apr 9, 2017 at 8:38 AM Daniel Kinzler daniel.kinzler@wikimedia.de wrote:
Am 09.04.2017 um 08:23 schrieb Denny Vrandečić:
Here's my requirement:
- a wiki page is one JSON document
- when editing, the user edits the JSON directly
- when viewing, I have a viewer that turns the JSON into wikitext, and
that
wikitext gets rendered as wikitext and turned into HTML by MediaWiki
Quick thoughts off the top of my head:
Generating wikitext from some other thing is what Scribunto does. Instead of using the Lua handler, you would make a handler for Json, with whetever rules you like for generating wikitext.
Thanks, I will take a look into how Scribunto does that.
I have ne4ver looked at customizing Scribunto, but this seems to right place to plug this in.
The alternative is to try to have some kind of "cascading" ContentHandler, that generates a WikiTextContent object first, and then turns that into HTML.
Not sure what you mean here - just create wikitext and then run it through the parser?
-- Daniel Kinzler Senior Software Developer
Wikimedia Deutschland Gesellschaft zur Förderung Freien Wissens e.V.
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
More related functionality: Parsoid supports JSON as a contenttype, and emits a special type of table (the same as the HTML table generated by the HTML handler for JSON in mediawiki). You can edit this in VE, although w/o any special support, and Parsoid will serialize it back to JSON.
This could be turned into a very pleasant type-aware editor for JSON in VE pretty easily. --scott
On Sun, Apr 9, 2017 at 2:23 AM, Denny Vrandečić vrandecic@gmail.com wrote:
Here's my requirement:
- a wiki page is one JSON document
- when editing, the user edits the JSON directly
- when viewing, I have a viewer that turns the JSON into wikitext, and that
wikitext gets rendered as wikitext and turned into HTML by MediaWiki
I have several options, including:
- hook for a tag like <json>, and write an extension that parses the
content between the tags and turns it into wikitext (not ideal, as I don't use any of the existing support for JSON stuff, and also I could have several such tags per page, which does not fit with my requirements)
- I found the JsonConfig extension by yurik. This allows me to do almost
all of the things above - but it returns HTML directly, not wikitext. It doesn't seem trivial to be able to return wikitext instead of HTML, but hopefully I am wrong? Also, this ties in nicely with the Code Editor.
- there is actually a JsonContentHandler in core. But looking through it
it seems that this suffers from the same limitations - I can return HTML, but not wikitext.
3 seems to have the advantage to be more actively worked on that 2 (which is not based on 3, probably because it is older than 3). So future goodies like a Json Schema validator will probably go to 3, but not to 2, so I should probably go to 3.
Writing this down, one solution could be to create the wikitext, and then call the wikitext parser manually and have it create HTML?
I have already developed the extension in 1, and then fully rewritten it in 2. Before I go and rewrite it again in 3, I wanted to ask whether I am doing it right, or if should do it completely differently, and also if there are examples of stuff developed in 3, i.e. of extensions or features using the JsonContent class.
Example: I have a JSON document { "username": "Denny" } which gets turned into wikitext ''Hello, [[User:Denny|Denny]]!'' which then gets turned into the right HTML and displayed to the user, e.g. <i>Hello, <a href="...">Denny</a>!</i>
Cheers, Denny _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On April 10, 2017 at 12:15:34 PM, C. Scott Ananian (cananian@wikimedia.org) wrote:
More related functionality: Parsoid supports JSON as a contenttype, and emits a special type of table (the same as the HTML table generated by the HTML handler for JSON in mediawiki). You can edit this in VE, although w/o any special support, and Parsoid will serialize it back to JSON.
This could be turned into a very pleasant type-aware editor for JSON in VE pretty easily. --scott
This is actually very interesting.
So, take a CollaborationHubContent page: https://wpx.wmflabs.org/w/index.php/WPX:WikiProject_Dogs
At some point we would like to create a VisualEditor-type editing interface. Note that behind the scenes, that page is JSON. So in this “VisualEditor” you would edit fields and such and it would appear you are editing the page as though it were a regular document, but really you are just changing values in a JSON blob.
Since Parsoid supports JSON as a content type, would it thus be trivial to create an extension for VisualEditor to support editing this content type and others like it?
Thanks, James Hare
I *believe* so. Both Parsoid and Visual Editor have extension interfaces. However, they haven't really been used by folks outside WMF yet. I'm eager to have that change.
We're thinking about proposing a "Workshop" session at wikimania for "How to write extensions for Parsoid and Visual Editor". But the area's only half-baked, it might be more appropriate to do it as a hackathon session instead. Thoughts? --scott
On Mon, Apr 10, 2017 at 3:20 PM, James Hare jamesmhare@gmail.com wrote:
On April 10, 2017 at 12:15:34 PM, C. Scott Ananian (cananian@wikimedia.org) wrote:
More related functionality: Parsoid supports JSON as a contenttype, and emits a special type of table (the same as the HTML table generated by the HTML handler for JSON in mediawiki). You can edit this in VE, although w/o any special support, and Parsoid will serialize it back to JSON.
This could be turned into a very pleasant type-aware editor for JSON in VE pretty easily. --scott
This is actually very interesting.
So, take a CollaborationHubContent page: https://wpx.wmflabs.org/ w/index.php/WPX:WikiProject_Dogs
At some point we would like to create a VisualEditor-type editing interface. Note that behind the scenes, that page is JSON. So in this “VisualEditor” you would edit fields and such and it would appear you are editing the page as though it were a regular document, but really you are just changing values in a JSON blob.
Since Parsoid supports JSON as a content type, would it thus be trivial to create an extension for VisualEditor to support editing this content type and others like it?
Thanks, James Hare
This upcoming hackathon might be too soon, considering we have more urgent priorities that need to be addressed first (like having it not be broken on the Beta Cluster). Beyond that, it’s something that can be considered, but how many people are actually interested in this?
On April 10, 2017 at 12:26:26 PM, C. Scott Ananian (cananian@wikimedia.org) wrote:
I *believe* so. Both Parsoid and Visual Editor have extension interfaces. However, they haven't really been used by folks outside WMF yet. I'm eager to have that change.
We're thinking about proposing a "Workshop" session at wikimania for "How to write extensions for Parsoid and Visual Editor". But the area's only half-baked, it might be more appropriate to do it as a hackathon session instead. Thoughts? --scott
On Mon, Apr 10, 2017 at 3:20 PM, James Hare jamesmhare@gmail.com wrote:
On April 10, 2017 at 12:15:34 PM, C. Scott Ananian (cananian@wikimedia.org) wrote:
More related functionality: Parsoid supports JSON as a contenttype, and emits a special type of table (the same as the HTML table generated by the HTML handler for JSON in mediawiki). You can edit this in VE, although w/o any special support, and Parsoid will serialize it back to JSON.
This could be turned into a very pleasant type-aware editor for JSON in VE pretty easily. --scott
This is actually very interesting.
So, take a CollaborationHubContent page: https://wpx.wmflabs.org/ w/index.php/WPX:WikiProject_Dogs
At some point we would like to create a VisualEditor-type editing interface. Note that behind the scenes, that page is JSON. So in this “VisualEditor” you would edit fields and such and it would appear you are editing the page as though it were a regular document, but really you are just changing values in a JSON blob.
Since Parsoid supports JSON as a content type, would it thus be trivial to create an extension for VisualEditor to support editing this content type and others like it?
Thanks, James Hare
-- (http://cscott.net)
I was thinking of the pre-WIkimania hackathon.
There are a *lot* of extensions installed on WMF wikis. Ideally we'd start getting extension maintainers to take over the Parsoid/VE support for their extensions. --scott
On Mon, Apr 10, 2017 at 3:37 PM, James Hare jamesmhare@gmail.com wrote:
This upcoming hackathon might be too soon, considering we have more urgent priorities that need to be addressed first (like having it not be broken on the Beta Cluster). Beyond that, it’s something that can be considered, but how many people are actually interested in this?
On April 10, 2017 at 12:26:26 PM, C. Scott Ananian (cananian@wikimedia.org) wrote:
I *believe* so. Both Parsoid and Visual Editor have extension interfaces. However, they haven't really been used by folks outside WMF yet. I'm eager to have that change.
We're thinking about proposing a "Workshop" session at wikimania for "How to write extensions for Parsoid and Visual Editor". But the area's only half-baked, it might be more appropriate to do it as a hackathon session instead. Thoughts? --scott
On Mon, Apr 10, 2017 at 3:20 PM, James Hare jamesmhare@gmail.com wrote:
On April 10, 2017 at 12:15:34 PM, C. Scott Ananian ( cananian@wikimedia.org) wrote:
More related functionality: Parsoid supports JSON as a contenttype, and emits a special type of table (the same as the HTML table generated by the HTML handler for JSON in mediawiki). You can edit this in VE, although w/o any special support, and Parsoid will serialize it back to JSON.
This could be turned into a very pleasant type-aware editor for JSON in VE pretty easily. --scott
This is actually very interesting.
So, take a CollaborationHubContent page: https://wpx.wmflabs.org/ w/index.php/WPX:WikiProject_Dogs
At some point we would like to create a VisualEditor-type editing interface. Note that behind the scenes, that page is JSON. So in this “VisualEditor” you would edit fields and such and it would appear you are editing the page as though it were a regular document, but really you are just changing values in a JSON blob.
Since Parsoid supports JSON as a content type, would it thus be trivial to create an extension for VisualEditor to support editing this content type and others like it?
Thanks, James Hare
-- (http://cscott.net)
Uh, that sounds nifty. Thanks for that!
On Mon, Apr 10, 2017 at 12:15 PM C. Scott Ananian cananian@wikimedia.org wrote:
More related functionality: Parsoid supports JSON as a contenttype, and emits a special type of table (the same as the HTML table generated by the HTML handler for JSON in mediawiki). You can edit this in VE, although w/o any special support, and Parsoid will serialize it back to JSON.
This could be turned into a very pleasant type-aware editor for JSON in VE pretty easily. --scott
On Sun, Apr 9, 2017 at 2:23 AM, Denny Vrandečić vrandecic@gmail.com wrote:
Here's my requirement:
- a wiki page is one JSON document
- when editing, the user edits the JSON directly
- when viewing, I have a viewer that turns the JSON into wikitext, and
that
wikitext gets rendered as wikitext and turned into HTML by MediaWiki
I have several options, including:
- hook for a tag like <json>, and write an extension that parses the
content between the tags and turns it into wikitext (not ideal, as I
don't
use any of the existing support for JSON stuff, and also I could have several such tags per page, which does not fit with my requirements)
- I found the JsonConfig extension by yurik. This allows me to do almost
all of the things above - but it returns HTML directly, not wikitext. It doesn't seem trivial to be able to return wikitext instead of HTML, but hopefully I am wrong? Also, this ties in nicely with the Code Editor.
- there is actually a JsonContentHandler in core. But looking through it
it seems that this suffers from the same limitations - I can return HTML, but not wikitext.
3 seems to have the advantage to be more actively worked on that 2 (which is not based on 3, probably because it is older than 3). So future
goodies
like a Json Schema validator will probably go to 3, but not to 2, so I should probably go to 3.
Writing this down, one solution could be to create the wikitext, and then call the wikitext parser manually and have it create HTML?
I have already developed the extension in 1, and then fully rewritten it
in
- Before I go and rewrite it again in 3, I wanted to ask whether I am
doing it right, or if should do it completely differently, and also if there are examples of stuff developed in 3, i.e. of extensions or
features
using the JsonContent class.
Example: I have a JSON document { "username": "Denny" } which gets turned into wikitext ''Hello, [[User:Denny|Denny]]!'' which then gets turned into the right HTML and displayed to the user,
e.g.
<i>Hello, <a href="...">Denny</a>!</i>
Cheers, Denny _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
-- (http://cscott.net) _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
...and inspired, I just submitted https://wikimania2017.wikimedia.org/wiki/Submissions/Writing_Visual_Editor_a... --scott
On Mon, Apr 10, 2017 at 3:15 PM, C. Scott Ananian cananian@wikimedia.org wrote:
More related functionality: Parsoid supports JSON as a contenttype, and emits a special type of table (the same as the HTML table generated by the HTML handler for JSON in mediawiki). You can edit this in VE, although w/o any special support, and Parsoid will serialize it back to JSON.
This could be turned into a very pleasant type-aware editor for JSON in VE pretty easily. --scott
On Sun, Apr 9, 2017 at 2:23 AM, Denny Vrandečić vrandecic@gmail.com wrote:
Here's my requirement:
- a wiki page is one JSON document
- when editing, the user edits the JSON directly
- when viewing, I have a viewer that turns the JSON into wikitext, and
that wikitext gets rendered as wikitext and turned into HTML by MediaWiki
I have several options, including:
- hook for a tag like <json>, and write an extension that parses the
content between the tags and turns it into wikitext (not ideal, as I don't use any of the existing support for JSON stuff, and also I could have several such tags per page, which does not fit with my requirements)
- I found the JsonConfig extension by yurik. This allows me to do almost
all of the things above - but it returns HTML directly, not wikitext. It doesn't seem trivial to be able to return wikitext instead of HTML, but hopefully I am wrong? Also, this ties in nicely with the Code Editor.
- there is actually a JsonContentHandler in core. But looking through it
it seems that this suffers from the same limitations - I can return HTML, but not wikitext.
3 seems to have the advantage to be more actively worked on that 2 (which is not based on 3, probably because it is older than 3). So future goodies like a Json Schema validator will probably go to 3, but not to 2, so I should probably go to 3.
Writing this down, one solution could be to create the wikitext, and then call the wikitext parser manually and have it create HTML?
I have already developed the extension in 1, and then fully rewritten it in 2. Before I go and rewrite it again in 3, I wanted to ask whether I am doing it right, or if should do it completely differently, and also if there are examples of stuff developed in 3, i.e. of extensions or features using the JsonContent class.
Example: I have a JSON document { "username": "Denny" } which gets turned into wikitext ''Hello, [[User:Denny|Denny]]!'' which then gets turned into the right HTML and displayed to the user, e.g. <i>Hello, <a href="...">Denny</a>!</i>
Cheers, Denny _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
-- (http://cscott.net)
wikitech-l@lists.wikimedia.org