(Continuing the crunching. huh? But this message is only 4 pages long.)
Forked into the new thread.
I'm afraid latest maillist messages were considered TLDR by most of
the subscribers so I will put this in the beginning: is there any
point in continuing our discussion on the subject? Platonides is a
constructive company but he seems to be the only one participating.
Is the community truly interested in reworking the markup?
I have some knowledge and code assets that I will be happy to
contribute; I will gladly take part in discussions or help improve the
situation in some other way. But if Wikimedia team has different views
onto the markup evolution it's fruitless to spend so much time
chatting before the closed doors.
My reply follows.
On 08.02.2012 2:27, Platonides wrote:
> Nobody proposed to change the template in that way? :)
You mean that nobody has actually studied markup usability?
> If you start creating inline, block and mixed template modes, I
> suspect the syntax will end up being chaotic (I'm thinking in
> concrete cases in MW syntax).
True, that's why I propose only two modes: block and inline, both with
clear distinctions and features.
> That assumes that there's a non-ambiguous way to express that in
> natural language (plus that it is easily parseable by a machine).
Yes, added a few simple rules an unambiguous language can be created.
I'm sure most of those business e-mails and official documents can be
processed by the machine without much effort. And we're talking about
even more formalized language here - text markup.
> So, how do you split {{About Bijection, injection and surjection}} ?
If that is supposed to be a long caption (4 words and a comma) then just
use quotes - like in natural handwriting. {{About "Bijection, injection
and surjection"}}
> The point of using an additional character not used in normal
> language is precisely for working the metalanguage.
I disagree, it only means that this subject has not yet been researched
enough.
>>> Also, there are colons as parameters. How would you write as the
>>> parameter the article [[Gypsy: A Musical Fable]] or [[Batman:
>>> Year One]] ? By banning ':' in titles?
>> Have I said something about colons and links? Links are fine with
>> colons or any other symbols.
> You mentioned colons for template arguments I'm acting as the
> devil's advocate asking you how to provide those titles as parameters
> to a template.
Uh, I have mistyped "comma" instead of "colon". Let me correct this:
1. {{About Something}}
2. {{About Something, of kind}}
3. {{About "Something, something and something", of kind}}
4. {{About "Something, something and something", "of kind, kind and kind"}}
As you can see, no character is banned from the title while in current
pipe-centric approach I don't thing it's possible to have pipes there
without a headache.
>> But if we're touching this pipes in links are not that intuitive
>> either. Pipes are actually not present on many keyboard layouts but
>> even apart from that it's more natural to use an equality sign. Or
>> double, for the purpose of text markup.
> It's consistent with the use of pipes in templates (which do use
> equal in that way to name parameters). Although link syntax was
> probably earlier.
Right, and pipes should not appear in templates either. It's too special
symbol.
> So is [[Batman Forever]] your syntax for [[Batman Forever|Batman
> Forever]] or [[Batman|Forever]] ? So much cases are bad, KISS.
I do not see your point. The processing is straightforward:
1. Link contains == - it separates address from title.
2. Link contains no == but contains a space - the first space separates
address from title.
3. There is neither == nor ' ' - link is titleless. This means that:
* local links get titles from page name, not page address (this is
important and differs from current MediaWiki implementation in a better way)
* remote links can also get their title from <title> after fetching
first 4 KiB of that page or something
Use cases:
* "[[http://google/search?q=%61%62%63 Google it]]" - for external links
== delimiter won't be used at all
* "See this [[page]]" - current wikitext is the same
* "See [[page that page]]" vs. current [[page|that page]]. Looks more
clean and easier to type (space is present on all keyboards and is quite
large in size). This covers not less than half local links.
* "See [[Some page==this page]]" vs. current [[Some page|this page]].
This case has less drastic differences than previous 3 but a pipe is
still both special to English layouts and less noticeable to human eye
than double equality sign.
Does "KISS" mean that every use case should be created with uniform but
because of this equally inconvenient syntax? I agree that more complex
cases should have correspondingly more complex syntax but this scaling
must be adequate. By placing pipe everywhere not only cross-language
usability is reduced but the fact that it's redundant in some cases (#1
and #3 items above) is ignored.
>> 4. Finally, in very rare cases when both space and equality symbol
>> is necessary a special markup-wise (!) escape symbol can be used.
> As an example: [[2 + 2 = 5]]
Your example contains no double equality symbol and is treated as
space-separated title: [[2| + 2 = 5]] in current wikitext.
> Would you remove === headings?
No, headings are consistent because the first heading starts with double
equality sign.
>> Currently wikitext uses terrible "<nowiki>stuff</nowiki>" but it
>> doesn't always work and HTMLTidy comes in handy with its< and
>> >. And some places (such as link titles) cannot be escaped
>> altogether.
> Really? I think you can.
Give some examples and we will examine their adequateness.
> Your proposal for forcing to edit the urls is very bad. You can't
> just paste, you need to go changing every = on it (which is a
> frequent character) to ~==.
No, no, no, you have got a completely wrong idea. You don't have to
escape SINGLE = because it is not special. You only need to escape
double ==. How much double == have you seen in the links? I have seen
them being used on my local bookstore site but it's surely an exception.
> Pipes are banned from titles.
Great, let's make machine's life easier.
> I'm not sure this is a good analogy. Copy-pasting chunks of code look
> like copying phrases from other articles to make your own. That
> should be original. OTOH, reusing the existing LaTeX template is much
> more appropiate than writing your own from scratch trying to copy the
> style of the provided one.
For such things templates must be created that will reduce the number of
entities identical to all of their use cases to minimum. In MediaWiki
this is done using {{templates and=parameters}} and this is good. If you
were talking about copy-pasting these templates, their parameters and
empty values - this is fine. But if it was about copy-pasting the same
code with all rendering tricks ( , {{iejrhgy}} and other cryptic
things) - this is bad.
> Even if I write a program from scratch, I should make it consistent
> with other tools. That means an appropiate arguments would be sort
> -r --ignore-case --sort=month ./myfile instead of sort<- !case (sort
> as month) \\\\./myfile\\\\
Standardizing is fine unless it starts looking unnatural. The following
example might be argued but I can't think of another one quickly:
tar -czf file.tar.gz .
While this uses standard CLI syntax is in true *nix ideology this is
what (among other things) separate POSIX from Windows. For instance, I
could write:
tar file.tar.gz .
...and the program will detect -czf arguments on its own based on
-f is simply implied because there are 2 unnamed arguments (without
leading -X)
-c target file doesn't exist
-z target file has extension .gz
It's the same with templates or other markup: while {{About page=Earth
kind=planet}} or something similar is fine, {{About Earth, planet}} or
some other form is more appropriate in this particular use case.
> You are giving many attributions to the machine. Personally, I would
> spit out an error, just in they were eg. in different units.
Yes, this is one of the ways and I would opt for it if we want to have a
strict syntax.
> But you are making up your syntax, then requiring the system to adapt
> for you.
Can you elaborate more on this point?
>>> The goal of wikitext is to make html editing easy.
>> HTML editing? I thought wikitext was about text editing. Why not
>> edit HTML using HTML?
> Because it's considered cumbersome. (Actually, it's presentational
> editing, but as the presentation is obtained by using HTML as an
> intermediate language...)
Indeed, HTML is cumbersome, that's why wikitext and all other text
markups have been invented. But they don't have to copy HTML syntax -
just the opposite.
> And you have complicated the originally clean syntax of 1, 2, 3
Clean syntax for whom? For Englishmen? And are hashes actually clean? If
so, why don't we use them in our e-mail messages?
> Would html links become italic? (that was a problem of wikicreole, it
> was defined as 'italic unless in links')
Not at all because we are talking about context-specific grammar.
Addresses in links can hold no formatting and thus all but context
ending tokens (]], space and ==) are ignored there.
And yes, context-specific grammar is more than regular expressions can
handle. Regexps are good but this doesn't mean anything incompatible
with sed is beyond "too complex".
As already mentioned, I am using my own markup processor written in
PHP on my projects and it implements all markup already described
including the [[http://italic]] (context-specific grammar) case. And
its parsing loop is under 350 lines of code.
> Well, I have to say it seems well though, it "doesn't look bad".
Thank you. I have given it a lot of thinking and practice but I'm sure
there still are things to improve. I would be ecstatic if my
experience can help the world's largest free knowledge community.
Thanks again for your mail, Platonides.
Signed,
P. Tkachenko
Hi wikitext-l!
I've read http://www.mediawiki.org/wiki/Future/Parser_plan recently,
and the plans seemed strange and scary to me.
In several places, there is the following stuff said:
...rich text editor which will let most editors and commenters
contribute text without encountering source markup...
...further reducing the need of advanced editors to work with markup
directly...
...by integrating well with a rich text editor, we reduce the
dependence of editors and commenters on dealing with low-level
wikitext...
..."oh there's that funky apostrophe thing in this old-style page".
Most editors will never need to encounter it...
Such plans seem very scary to me, as I think the PLAIN-TEXT is one of
the MOST IMPORTANT features of Wiki software! And you basically say you
want to move away from it and turn MediaWiki to another Word, having all
problems of "WYSIWYdnG" (...Is What Wou don't Get) editors. I don't
think I need to explain the advantages of plain-text markup to core
developers of MediaWiki... :)
I've patched the parser code slightly (in mediawiki4intranet) and I
understand it's not perfect, so I support any effort for creating a new
parser, but if it involves moving away from markup in the future
fully...
So, my question is - is this all true, or did I misunderstand the
plans?
Hi Good People.
I'd like to thank everyone for helping me with labs, code reviews and other difficulties.
Recent Search Related Activity:
1. Branched the project to svn:https://svn.wikimedia.org/svnroot/mediawiki/trunk/lucene-search-3
2. Upgraded the code from is Lucene 2.4.0 to 2.9.1 last December and I've been reviewing and committing to svn.
3. I've migrated the project from Ant To Maven.
4. We have placed the Maven based code is in Continuous Integration on Jenkins with JUnit PMD & Coverage report in place.
5. With the help of some excellent volunteers, I've set up a lab to test the build using simple English Wikipedia.
6. One major setback is that there is no proper testing or deployment possible for update. For this reason I've not closed any of the bugs I've worked on. (Access to the production machines is considered too sensitive now that there are labs. At this time labs do not have capacity for setting up a At this Labs do not have There is no labs Setting up a lab which replicates the production has been unsuccessful,. Once the scripts are sanitized, and production search will be put into puppet it may be possible. However as the labs environment is a far cry from the production in terms of both content, and updating.
7. I've done some rough analysis and design for the next version of search which will feature computational linguistics support for the many languages being used in wikipedias. Search analytics (optimizing ranking) and innovative content analytics for Ranking but including objective metrics on neutral point of view (via. sentiment analysis), notability (via. semantic algorithms), checking of external links (anti-link spam).
8. We are trying to relicense the search code so that the Lucene community in the apache projects will become more involved. It may be necessary also to relicense MWdumper since the two projects are related.
In the pipeline:
1. Testing and Integration into Lucene a ANTLR grammar which parses wiki syntax Tables. Once successful it will be also integrated
into SOLR and become the prototype of more difficult wiki syntax analysis tasks.
2. I've started on some of the NLP tasks:
a. a Transducer of Devanagari scripts to IPA. (in HFST)
b. a Transducer of English to IPA with the common goal to index named entities based on their sound in a language agnostic fashion.
(also in HFST)
c. Extraction of phonetics data from English Wiktionary.
d. conversion of CMU pronunciation dictionary to IPA.
e. Extraction of bi-lingual lexicons from Wiktionary and conversion to Apertium Formats.
f. Unsupervised learning of morphologies using minimum description length.
g. Sentence boundary detection (SVM and MaxEnt Models).
h. Topological text alignment algorithm.
3. A Maven based POM for building and packaging SOLR + our extension for distributed use.
4. A repository for NLP artifacts built from WikiContent.
Oren Bochman
MediaWiki Search Lead.
(branched from the long thread)
Dear Jay and list,
if there are 97-99% compatible alternate parsers that is in fact
something I should know about!
I tried to figure out which are those 4-5 you mentioned but from
http://www.mediawiki.org/wiki/Alternative_parsers it's not so easy.
FlexBisonParse? Magnus's?
So I thought I ask everyone: which are the top alternate parsers? (in
compatibility with wikitext syntax)
Thanks
Mihály
On 6 February 2012 22:32, Jay Ashworth <jra(a)baylink.com> wrote:
> ----- Original Message -----
>> From: "Mihály Héder" <hedermisi(a)gmail.com>
>
>> By following this list I hope I gathered how they plan to tackle this
>> really hard problem:
>> -a functional decomposition of what the current parser does to a
>> separate tokenizer, an AST(aka WOM or now just DOM) builder and a
>> serializer. Also, AST building might be further decomposed to the
>> builder part and an error handling according to html specs.
>> -in architecture terms, all this will be a separate component, unlike
>> the old php parser which is really hard to take out from the rest of
>> the code.
>> In this setup there is hope that the tokenizing task can be specified
>> with a set of rules, thus effectively creating a wikitext tokenizing
>> standard (already a great leap forward!)
>> Then the really custom stuff (because wikitext still lacks a formal
>> grammar) can be encapsulated in AST building.
>
> As I noted in a reply I wrote on this thread a few minutes ago (but it
> was kinda buried): there are between 4 and 7 projects with varying
> stages of seriosity that are already in work, some of them having posted
> to this list one or more times.
>
> At least a couple of them had as a serious goal producing a formalized,
> architecturally cleaner parser that could be dropped into Mediawiki.
>
> The framing of your reply suggests that you needed to know that and
> didn't.
>
> Cheers,
> -- jra
> --
> Jay R. Ashworth Baylink jra(a)baylink.com
> Designer The Things I Think RFC 2100
> Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII
> St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274
>
> _______________________________________________
> Wikitext-l mailing list
> Wikitext-l(a)lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitext-l
It's now possible to request a parsed wiki page via the MediaWiki API.
You have to have node (and all the dependencies) installed. See the
README file in the modules/parse directory. It simply shells out to
node.js. A more clever version might make use of a node.js daemon, but
it was just easier to reuse the STDIN/STDERR/STDOUT model this way.
It's implemented as a 'property' of a title, i.e.:
http://wiki.ivy.local/w/api.php?action=query&prop=parsetree&titles=Main%20P…
If I've read the docs right, this should work well with caching.
It doesn't work with format=xml, for some reason.
https://bugzilla.wikimedia.org/show_bug.cgi?id=34058
--
Neil Kandalgaonkar ) <neilk(a)wikimedia.org>
The current code of the Visual Editor reimplements many basic editing
functions, which are available in the browsers: text selection,
alignment, cursor movement and appearance and even cursor blinking.
This is a major problem for internationalization. These functions work
very differently in different languages. Most browsers already support
most of the needed functionality in textarea and in content-editable.
Overriding them in the Visual Editor means that it will be unusable in
any language except English.
To name just a few basic examples:
* Cursor shape: In bidirectional environments the cursor has an arrow
pointing in the writing direction. In the current VE it's just a
stick. See https://en.wikipedia.org/wiki/Text_cursor#Bi-directional_text
* Cursor position: VE always positions the cursor as it the text is
left-to-right. The same goes for text selection.
* Word boundary: Ctrl-left/right arrow moves the cursor word by word.
Browsers support all Unicode scripts for word boundary identification,
but in the current VE it only works with ASCII.
* Character segmentation and cursor movement: A letter with one or
more diacritics is one character segment according to Unicode (UAX
#29), even if internally it's composed of several Unicode characters.
For example, <שָּׁ> is composed of four characters, but only one arrow
key press is supposed to move the cursor over it. That's what the
browsers already do for years, but in the Visual Editor, four key
presses are needed.
This is just a small sample of the potential issues.
Bugs are already reported for some of these issues, but the basic
question is - why reimplement it all in the first place? Why not use
what the browsers offer as much as possible?
--
Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי
http://aharoni.wordpress.com
“We're living in pieces,
I want to live in peace.” – T. Moore
Some of you wikitext workers might find this interesting as well ...
Erik
---------- Forwarded message ----------
From: Erik Moeller <erik(a)wikimedia.org>
Date: Fri, Dec 30, 2011 at 5:31 PM
Subject: In-browser offline wiki viewer
To: Using Wikimedia projects and MediaWiki offline
<offline-l(a)lists.wikimedia.org>
Text only for now, but very cool:
http://offline-wiki.googlecode.com/git/app.html
In modern browsers, this gives you access to the most popular WP
articles offline without any additional software.
(via NeilK)
--
Erik Möller
VP of Engineering and Product Development, Wikimedia Foundation
Support Free Knowledge: http://wikimediafoundation.org/wiki/Donate
--
Erik Möller
VP of Engineering and Product Development, Wikimedia Foundation
Support Free Knowledge: http://wikimediafoundation.org/wiki/Donate