We can use Visual Editor to edit templates, and in the process better
separate code, content, and presentation. I'm going to call these "Visual
Templates".
Here are the key ideas
<https://wikimania2015.wikimedia.org/w/index.php?title=File%3ATemplates_are_…>
:
1. A minimal "logic-less" template language, like the Spacebars
<https://github.com/meteor/meteor/blob/devel/packages/spacebars/README.md>
variant
of Handlebars <http://handlebarsjs.com/>/mustache
<https://mustache.github.io/> templates. There are only three basic
template constructs (variable interpolation, block tags, and partials),
which can all be visualized in the editor. There are no hidden directives.
2. All logic is written in a real programming language, via Scribunto
<https://www.mediawiki.org/wiki/Extension:Scribunto> (or something very
much like it). In the examples below I'll be using a version of Scribunto
that supports <https://github.com/preillyme/v8js> JavaScript 2015 as
well as Lua; forgive my eccentricities.
3. All data is escaped and hygienic by default. String data is
automatically escaped. The Scribunto code manipulates and returns
DocumentFragments
<https://developer.mozilla.org/en-US/docs/Web/API/DocumentFragment>, not
wikitext strings.
4. Similarly, arguments are provided as JSON types (strings/numerical
data), DocumentFragments, or wikidata queries, not wikitext.
5. Composition of content is structural, not via string concatenation.
You can't "accidentally" create a list item if your template result happens
to start with a : or *.
So: every Visual Template is composed of two parts: a Scribunto module
(edited with a code editor), and a "layout" (edited with Visual Editor).
Here's what the layout of a particular template might look like:
https://phabricator.wikimedia.org/F2651938
Forgive the terrible yellow; a designer could make this look much nicer.
Note that there are two "block helper" invocations, `each` and `caption`.
The `each` helper is a built-in, and it is given `row` from the template
data context object. The `caption` helper is defined by the Scribunto
module. Each row of the table gets "variable" invocations to fill in the
table cells.
There are only three new functions added to the VE toolbar in template
mode, for inserting "variable", "block helper", and "partial" elements.
Context-sensitive help will be given to choose from available variable,
block helper, and partial names.
The invocation of this template would, for this example, take structured
data of the following form, perhaps from a wikidata query:
| event | gold | silver | bronze |
| Shotput | USA | GER | MEX |
| Discus | MEX | USA | GER |
| Javelin | GER | MEX | USA |
The Scribunto module transforms this raw data by computing medal counts for
each country. Here's what it might look like, using JavaScript 2015 (ES6)
<https://esdiscuss.org/topic/javascript-2015> module syntax,
```
// The main module entry point is a transformation of a
// "data context".
// In this example the data context comes from a wikidata query.
export default function(events) {
// `events` is an array: each item has `event`, `gold`,
// `silver`, and `bronze` props (the latter three holding
// country names)
let totals = new Map(),
countries = new Set(),
allColors = ['gold','silver','bronze'];
function mget(m, key, defaultval) {
if (!m.has(key)) { m.set(key, defaultval); }
return m.get(key);
}
function inc(country, color) {
let old = mget(mget(totals, country, new Map()), color, 0);
totals.get(country).set(color, old + 1);
countries.add(country);
}
// count up how many medals of each color a given country has.
events.forEach( (e) => {
for (let color of allColors) { inc(e[color], color); }
}):
// Now sort the countries by medal count
let rows = Array.from(countries).sort( (a,b) => {
let aa = totals.get(a), bb = totals.get(b);
for (let color of allColors) {
let c = aa[color] - bb[color];
if (c!==0) { return c; }
}
return c;
}).map( (c) => totals.get(c) );
// our resulting data context object has just one
// property, named 'rows', which is an array.
return { row: rows };
}
// Additional block helper. Helpers must return a
// `DocumentFragment`, though for brevity we allow `Node` or
// `Array<Node>` as well. In fact, if you returned a string
// we'd make a text node for you, but that would make this
// example even more trivial.
export function caption() {
return document.createTextNode('caption');
}
```
The default entry point transforms the template arguments to set up the
template data context. //*It does no text manipulation.*// There is a
second `caption` entry point, which defines the `caption` block helper.
The exact API used in block helpers definitions might be tweaked: one
obvious improvement would be to allow jquery methods for DOM construction,
instead of using the DOM API directly. The key point is that the result is
conceptually a chunk of structural DOM, not a wikitext string.
***Visual Template invocation in Visual Editor***
The primary authoring mechanism for Visual Templates is expected to be
Visual Editor. The UX is expected to be roughly the same as exists now for
templates: a mechanism like TemplateData exposes the expected types of
arguments and descriptive text, so that Visual Editor can expose
appropriate editing widgets. Initially the expected types are: "markup"
(edited with VE recursively), JSON (edited with a JSON editor, as with
TemplateData), and "wikidata" (a SPARQL wikidata query
<https://www.mediawiki.org/wiki/Wikidata_query_service/User_Manual>).
***Visual Template Invocation in wikitext***
This is a strawman for the wikitext serialization of a template invocation:
```
{{#visual
template_name
param1=<markup>'''bold'''</markup>
param2=<json>{"foo":"bar"}</json>
param3=<wikidata>
SELECT ?spaceProbeLabel ?date ?picture
WHERE {
?spaceProbe wdt:P31 wd:Q26529 ;
wdt:P18 ?picture ;
wdt:P619 ?date .
SERVICE wikibase:label {
bd:serviceParam wikibase:language "fr,en" .
}
}
ORDER BY ?date
</wikidata>
}}
```
The parameter name `entity` is reserved; the data context object for the
template always includes the wikidata entity object (equivalent to `
mw.wikibase.getEntityObject()`) as an implicit parameter named `entity`.
This makes it easy to generate infoboxes (for example), since you can just
insert `{{#formatproperty entity.P17}}` in your template (equivalent
to `entity.formatPropertyValues(
'P17' ).value`). Hopefully there will be a friendly shortcut for `
entity.<something>`, which will include mapping wikidata identifier `
<something>` to a label in your language.
Since arguments are delimited by html-style tags, the usual & and <
escapes are all that are required to include arbitrary data inside a
parameter, and whitespace can be used to separate arguments. The parameter
names can be omitted for positional arguments.
//*Perhaps we can allow for an abbreivated format for certain arguments:
`"..."` -> literal string, `\d+` -> number, `{...}` -> JSON value. Then
unit conversion (for example) can still be simply `{{#visual convertToMiles
5 km}}`*//
****Syntax details can be tweaked: your input wanted!****
Phabricator ticket: https://phabricator.wikimedia.org/T114454
--scott
--
(http://cscott.net)
Hello,
in an effort to standardize all post parameters in the REST v1 API to use
snake_case names, we have deprecated the bodyOnly flag in the wikitext to
HTML transform end point [1]. Instead, clients should use body_only, as
mentioned in the documentation.
We plan to remove support for the old form by the end of November.
Thanks,
Gabriel
[1]:
https://en.wikipedia.org/api/rest_v1/?doc#!/Transforms/post_transform_wikit…
We currently have several projects which can not be tested with our current
Jenkins test infrastructure, and as a consequence are hosting their primary
code repositories on github:
* RESTBase <https://github.com/wikimedia/restbase> -- missing Cassandra
support
* iOS Mobile app
<https://lists.wikimedia.org/pipermail/wikitech-l/2015-September/083005.html>
--
missing OS X platform
Other projects can only run a small portion of their test suite via Jenkins:
* mw-ocg-latexer
<https://github.com/wikimedia/mediawiki-extensions-Collection-OfflineContent…>
-- requires LaTeX from PPA, image utilities (`jpegtran`).
An alternative to allow these apps to be hosted on Wikimedia infrastructure
(gerrit, eventually phabricator) is to allow travis integration with
jenkins as an optional service.
npm-travis <https://github.com/cscott/npm-travis> (
https://github.com/cscott/npm-travis) is a tool which will trigger Travis
builds from NPM by pushing to a throwaway branch, which is then cleaned up
after the tests complete. It integrates well with the Gerrit access
control mechanism: the "Travis Bot" user can be granted push access only,
and only to branches prefixed with `npm-travis/`, so it cannot be used to
push changes to the master or deployment branches.
This isn't a replacement for our jenkins test infrastructure, but it allows
us to accommodate oddball repositories without taxing our infrastructure
team or resorting to offsite repository hosting.
There are WIP patches for integrating `npm-travis` with our jenkins
infrastructure (https://gerrit.wikimedia.org/r/173045,
https://gerrit.wikimedia.org/r/173046) but they seem to be blocked on
policy disagreements. This RFC aims to resolve the policy issues.
This RFC in Phabricator: https://phabricator.wikimedia.org/T114421
--scott
--
(http://cscott.net)
*(Not to be confused with hygenic template arguments
<https://phabricator.wikimedia.org/T114432>.)*
As described in my Wikimania 2015 talk
<https://wikimania2015.wikimedia.org/wiki/Submissions/Templates_are_dead!_Lo…!>
(starting at slide 27
<https://wikimania2015.wikimedia.org/w/index.php?title=File:Templates_are_de…>),
there are a number of reasons to mark certain templates as "hygienic".
Among them: to allow high-performance incremental update of page contents
after templates are modified, and to allow safe editing of template uses
using HTML-based tools such as Visual Editor or jsapi
<https://doc.wikimedia.org/Parsoid/master/#!/guide/jsapi>.
This means that the output of the template is a complete DocumentFragment
<https://developer.mozilla.org/en-US/docs/Web/API/DocumentFragment>: every
open tag is closed and there are no nodes which the HTML adoption agency
algorithm
<http://dev.w3.org/html5/spec-LC/tree-construction.html#adoptionAgency> will
reorder.
This can be enforced in practice by running tidy (or something like it:
T89331 <https://phabricator.wikimedia.org/T89331>) on the template output
before it is inserted into the document. We are likely to be doing
something like this already for T114072
<https://phabricator.wikimedia.org/T114072>.
The purpose of this RFC is to settle the bikeshed-color questions about how
templates will be marked hygienic/unhygienic. There are two main scenarios:
***Opt-in Hygiene***
In this scenario, unmarked templates are assumed "dirty", and will render
correctly (as before), just more slowly. Some of the unmarked templates
(the "actually dirty" ones) may be uneditable in VisualEditor.
In the opt-in scenario, it is expected that profiling the "costliest"/"most
frequently used/changed" templates on wikimedia projects will be done, and
opt-in tags will be added first to those templates where the greatest
potential performance gain may be achieved. (Opting in will enable
incremental update of the parser cache after template modifications.)
Proposals:
1. An explicit tag surrounding "hygienic" contents; this tag will
exactly encapsulate the content which will be run through tidy. Proposed
names:
1. `<tidy>...</tidy>`
2. `<domscope>...</domscope>`
3. `<domparse>...</domparse>`
2. A magic word, present in templates known to be hygienic. Proposed
names:
1. `__SAFE__`
2. `__HYGIENIC__`
3. Some `TemplateData
<https://www.mediawiki.org/wiki/Extension:TemplateData>` property?
4. Others?
***Opt-out Hygiene***
In this scenario, all templates are assumed "hygienic", and an explicit
marker of some kind is required to opt-out of the usual tidy of the
contents. Templates which opt-out are not editable in Visual Editor, but
all other templates are (because the contents will have been automatically
cleaned to close any stray open tags).
In the opt-out scenario, it is expected that Parsoid (or other tools) will
be run against the wikimedia projects to automatically add opt-out tags
where possible; perhaps based on logging during the parse process.
Proposals:
1. Magic word, present in unhygienic templates
1. `__UNSAFE__`
2. Others?
Phabricator task for this RFC: https://phabricator.wikimedia.org/T114445
--scott
--
(http://cscott.net)
Howdy (my apologies for cross-posting),
I'm pleased to announce that the Community Tech team has a new Product
Manager – Danny Horn, who will be moving from the Collaboration team to
join this new initiative. The Community Tech team is focused on building
improved curation and moderation tools for experienced Wikimedia
editors.[1] Danny will be working with the team to develop the upcoming
Community Wishlist Survey, which will help the team define and prioritize
future projects.[2] We're excited to have someone with Danny's product
management experience as well as extensive wiki community experience
working with this team.
Danny’s role on the Collaboration team will be filled by a new Product
Manager which the Foundation is hiring for now. There’s an open job posting
on the WMF job board.[3]
Looking forward to working with Danny to build some awesome tools for the
community!
1. https://www.mediawiki.org/wiki/Community_Tech_team
2.
https://www.mediawiki.org/wiki/Community_Tech_team/Community_Wishlist_Survey
3. https://boards.greenhouse.io/wikimedia/jobs/104264?t=k7xsjm
Ryan Kaldari
Engineering Manager, Community Tech
Wikimedia Foundation
Hi (and sorry for cross-posting)
I’d first like to say that I’m excited about Danny’s new role and the
positive impact I know he will have on the relatively new Community Tech
team, and that he will be missed in the Editing group. This change now
leaves an open position, which we are in the process of hiring a new
Product Manager to fill. There’s an open position posted.[1]
The Collaboration team, meanwhile, will continue their work to provide Flow
as an opt-in Beta feature, allowing contributors to use Flow on their user
talk page. This feature is currently available on MediaWiki.org, and will
be enabled on other wikis upon request. Especially as more users enable
Flow, the team will continue supporting users of the product by promptly
triaging and resolving bugs.
This quarter, which began yesterday, The Collaboration team will be
focusing on the development of cross-wiki notifications and other Echo
improvements. They’re also continuing to advance efforts to research and
prototype solutions for advanced editor workflows. Further feature
development on Flow discussion tools will be based on an assessment
following the 'workflows' research work, and on editor feedback at the
wikis already using Flow.[2]
Please ask any questions you may have on-wiki at
https://www.mediawiki.org/wiki/Talk:Flow to reduce fragmentation.
- Trevor
[1] https://boards.greenhouse.io/wikimedia/jobs/104264?t=k7xsjm
[2] https://www.mediawiki.org/wiki/Flow/Rollout
As described in my Wikimania 2015 talk
<https://wikimania2015.wikimedia.org/wiki/Submissions/Templates_are_dead!_Lo…!>
(starting at slide 31
<https://wikimania2015.wikimedia.org/w/index.php?title=File:Templates_are_de…>),
the existing template argument syntax offers a number of traps for the
unwary (for example, with the characters `=`, `|`, etc). As a result, it
is difficult to easily move large blocks of text into templates.
As a result, we often have constructs such as:
```
{{tablestart|class="shiny"}}
| Hello || world
{{tableend}}
```
which pose a number of issues:
1. There is no mechanism to ensure `{{tablestart}}` and `{{tableend}}`
are properly matched.
2. Both `{{tablestart}}` and `{{tableend}}` emit unbalanced HTML, which
complicates work on efficiently updating the parser cache after template
changes.
3. Due to the tag matching issues, this whole block is uneditable by
Visual Editor.
***Hygienic arguments*** provide a new form of template invocation which
avoids these issues.
The above example would be written as:
```
{{>Table|class="shiny"}}
| Hello || world
{{< Table}}
```
and the contents between the template start `{{>...}}` and template end `
{{<...}}` are passed as (WLOG) the first positional argument of the
template. Named arguments (like `class` in this example) can be passed
using the existing argument syntax as before. The new `Template:Table` can
now emit properly balanced HTML, with both `<table>` and `</table>`
generated by the same template (instead of by two separate templates).
Visual Editor can now edit this block as a single template invocation,
invoking itself recursively to edit the template arguments as it does now.
Further, there are no special characters or odd escape rules in the
argument when expressed this way. The only "special" construct inside the
argument is `{{<...}}` and any of those characters can be replaced by an
html entity to allow that character sequence to be included literally if
needed. That is, we don't need special `{{!}}`, `{{=}}`, etc escapes.
Another example, from Brion Vibber's talk on citations
<https://wikimania2015.wikimedia.org/wiki/Submissions/Future_of_structured_d…!>
:
```
{{>cite|id=“32412”}}
First person plural pronouns in Isthmus-Mecayapan Nahuat:
:''nejamēn'' ({{IPA|[nehameːn]}}) "We, but not you" (= me & them)
:''tejamēn'' ({{IPA|[tehameːn]}}) "We along with you" (= me & you & them)
{{<cite}}
```
Note that it was easy to surround the entire text covered by the citation
into the `{{cite}}` template, since I didn't need to worry about the fact
that the text included the special character `=`.
***Additional arguments*** (optional extension)
>From experience with mustache templates <https://mustache.github.io/> which
have a similar construct
<https://mustache.github.io/mustache.5.html#Sections> (well, actually with
the spacebars
<https://github.com/meteor/meteor/blob/devel/packages/spacebars/README.md>
equivalent,
which shares a concern that template contents be well-balanced), it seems
that a large number of use cases can be satisfied with only one or two
"block arguments".
But let's say we want to allow more generality. We could support:
```
{{>Foo|namedArg=bar|anotherNamedArg=bat}}
This first block goes in argument "1".
{{else}}
Now this goes in argument "2".
{{else}}
Now this goes in argument "3". It's a little more natural if there are
only one or two arguments,
but we could keep going if we wanted.
{{<Foo}}
```
Additional proposals for this syntax welcome, I'm just trying to avoid too
many crazy new constructs.
***Keeping things terse***
We might allow `{{<}}` as a shortcut to end the block argument, instead of
requiring the author to repeat the name of the template. Some folks will
like the brevity, others may prefer the more explicit error messages
provided when you spell out the name of the template you intend to close.
**Strict start-of-line constraints**
For ease of parsing (and reading) we can enforce the constraint that the `
{{>...}}` and `{{<...}}` tags must be alone on their line, and that the
newline following the `{{>...}}` and `{{<...}}` tags is not copied into the
output.
We could enforce start-of-line context on the result to avoid the
T14974/T2529 hacks and make behavior consistent. (More thought welcome
here.)
***Migrating template use***
We might allow configuring details of the parameter name(s) used for the
block argument(s) via `TemplateData`, to more easily use the new syntax
with existing templates.
Phabricator task for this RFC: https://phabricator.wikimedia.org/T114432
--scott
--
(http://cscott.net)
Hi Community Metrics team,
this is your automatic monthly Phabricator statistics mail.
Number of accounts created in (2015-09): 363
Number of active users (any activity) in (2015-09): 863
Number of task authors in (2015-09): 477
Number of users who have closed tasks in (2015-09): 268
Number of projects which had at least one task moved from one column
to another on their workboard in (2015-09): 209
Number of tasks created in (2015-09): 3351
Number of tasks closed in (2015-09): 2981
Number of open and stalled tasks in total: 25811
Median age in days of open tasks by priority:
Unbreak now: 11
Needs Triage: 125
High: 164
Normal: 301
Low: 660
Lowest: 528
(How long tasks have been open, not how long they have had that priority)
TODO: Numbers which refer to closed tasks might not be correct, as described in T1003.
Yours sincerely,
Fab Rick Aytor
(via community_metrics.sh on iridium at Thu Oct 1 00:00:08 UTC 2015)