Wikitech-l October 2015

wikitech-l@lists.wikimedia.org

122 participants
120 discussions

Semantic MediaWiki Conference Fall 2015: Program Announcement
by kghbln 03 Oct '15

03 Oct '15

Dear users, developers and all people interested in semantic wikis, We are very happy to announce the program of the 12th Semantic MediaWiki Conference in Barcelona! People from business, academia and non-profit organizations will give a variety of very interesting talks and presentations about applying semantic wikis as well as about newest developments in the field. We would also like to remind you of the upcoming end to the early bird registration registration period on October 5, 2015. Please register via Tito [0] if you have not done so already. Important facts reminder: =========================================== * Dates: October 28th to October 30th 2015 (Wednesday to Friday) * Location: Fabra i Coats, Art Factory. Carrer Sant Adrià 20 (Sant Andreu), Barcelona, Catalonia, Spain. * Conference page: https://semantic-mediawiki.org/wiki/SMWCon_Fall_2015 * Participants: Everybody interested in semantic wikis, especially in Semantic MediaWiki, e.g. users, developers, consultants, business representatives and researchers. News on the program: =========================================== * The tutorial program has been announced and made available [1] * The conference program has now also been announced and made available [2]. The keynote will be held jointly by Denny Vrandečić and Markus Krötzsch. * A social program is also being prepared [3]. We still welcome contributions from you: =========================================== * We encourage contributions for workshops and poster sessions about semantic wikis; for a list of topics, see [4]. Please add your proposals here: [5] * Presentations will generally be video and audio recorded and made available for others after the conference. Organization: =========================================== * Amical Wikimedia [6] and Open Semantic Data Association e. V. [7] have become the official organisers of SMWCon Fall 2015 * Thanks to Institut de Cultura - Ajuntament de Barcelona [8] for providing free access to the conference location and its infrastructure as well as ArchiXL/XL&Knowledge [9] for being our silver sponsor. If you have questions you may contact Lia Veja and Karsten Hoffmeyer (Program Chairs), Alina Mierluș (General Chair) or Toni Hermoso (Local Chair) per e-mail (Cc). We will be happy to see you in Barcelona! Lia Veja, Karsten Hoffmeyer (Program Board) [0] <https://ti.to/wikisofia/smwcon2015-fall> [1] <https://semantic-mediawiki.org/wiki/SMWCon_Fall_2015/Tutorial_Day> [2] <https://semantic-mediawiki.org/wiki/SMWCon_Fall_2015/Conference_Days> [3] <https://semantic-mediawiki.org/wiki/SMWCon_Fall_2015#Social_program> [4] <https://semantic-mediawiki.org/wiki/SMWCon_Fall_2015/Announcement> [5] <https://semantic-mediawiki.org/wiki/SMWCon_Fall_2015#Program_proposals> [6] <https://www.wikimedia.cat/> [7] <https://opensemanticdata.org/> [8] <http://lameva.barcelona.cat/barcelonacultura/en/> [9] <http://www.xl-knowledge.nl/index.php/Hoofdpagina/en>

1 0

[RFC] Visual Templates
by C. Scott Ananian 02 Oct '15

02 Oct '15

We can use Visual Editor to edit templates, and in the process better separate code, content, and presentation. I'm going to call these "Visual Templates". Here are the key ideas <https://wikimania2015.wikimedia.org/w/index.php?title=File%3ATemplates_are_…> : 1. A minimal "logic-less" template language, like the Spacebars <https://github.com/meteor/meteor/blob/devel/packages/spacebars/README.md> variant of Handlebars <http://handlebarsjs.com/>/mustache <https://mustache.github.io/> templates. There are only three basic template constructs (variable interpolation, block tags, and partials), which can all be visualized in the editor. There are no hidden directives. 2. All logic is written in a real programming language, via Scribunto <https://www.mediawiki.org/wiki/Extension:Scribunto> (or something very much like it). In the examples below I'll be using a version of Scribunto that supports <https://github.com/preillyme/v8js> JavaScript 2015 as well as Lua; forgive my eccentricities. 3. All data is escaped and hygienic by default. String data is automatically escaped. The Scribunto code manipulates and returns DocumentFragments <https://developer.mozilla.org/en-US/docs/Web/API/DocumentFragment>, not wikitext strings. 4. Similarly, arguments are provided as JSON types (strings/numerical data), DocumentFragments, or wikidata queries, not wikitext. 5. Composition of content is structural, not via string concatenation. You can't "accidentally" create a list item if your template result happens to start with a : or *. So: every Visual Template is composed of two parts: a Scribunto module (edited with a code editor), and a "layout" (edited with Visual Editor). Here's what the layout of a particular template might look like: https://phabricator.wikimedia.org/F2651938 Forgive the terrible yellow; a designer could make this look much nicer. Note that there are two "block helper" invocations, `each` and `caption`. The `each` helper is a built-in, and it is given `row` from the template data context object. The `caption` helper is defined by the Scribunto module. Each row of the table gets "variable" invocations to fill in the table cells. There are only three new functions added to the VE toolbar in template mode, for inserting "variable", "block helper", and "partial" elements. Context-sensitive help will be given to choose from available variable, block helper, and partial names. The invocation of this template would, for this example, take structured data of the following form, perhaps from a wikidata query: | event | gold | silver | bronze | | Shotput | USA | GER | MEX | | Discus | MEX | USA | GER | | Javelin | GER | MEX | USA | The Scribunto module transforms this raw data by computing medal counts for each country. Here's what it might look like, using JavaScript 2015 (ES6) <https://esdiscuss.org/topic/javascript-2015> module syntax, ``` // The main module entry point is a transformation of a // "data context". // In this example the data context comes from a wikidata query. export default function(events) { // `events` is an array: each item has `event`, `gold`, // `silver`, and `bronze` props (the latter three holding // country names) let totals = new Map(), countries = new Set(), allColors = ['gold','silver','bronze']; function mget(m, key, defaultval) { if (!m.has(key)) { m.set(key, defaultval); } return m.get(key); } function inc(country, color) { let old = mget(mget(totals, country, new Map()), color, 0); totals.get(country).set(color, old + 1); countries.add(country); } // count up how many medals of each color a given country has. events.forEach( (e) => { for (let color of allColors) { inc(e[color], color); } }): // Now sort the countries by medal count let rows = Array.from(countries).sort( (a,b) => { let aa = totals.get(a), bb = totals.get(b); for (let color of allColors) { let c = aa[color] - bb[color]; if (c!==0) { return c; } } return c; }).map( (c) => totals.get(c) ); // our resulting data context object has just one // property, named 'rows', which is an array. return { row: rows }; } // Additional block helper. Helpers must return a // `DocumentFragment`, though for brevity we allow `Node` or // `Array<Node>` as well. In fact, if you returned a string // we'd make a text node for you, but that would make this // example even more trivial. export function caption() { return document.createTextNode('caption'); } ``` The default entry point transforms the template arguments to set up the template data context. //*It does no text manipulation.*// There is a second `caption` entry point, which defines the `caption` block helper. The exact API used in block helpers definitions might be tweaked: one obvious improvement would be to allow jquery methods for DOM construction, instead of using the DOM API directly. The key point is that the result is conceptually a chunk of structural DOM, not a wikitext string. ***Visual Template invocation in Visual Editor*** The primary authoring mechanism for Visual Templates is expected to be Visual Editor. The UX is expected to be roughly the same as exists now for templates: a mechanism like TemplateData exposes the expected types of arguments and descriptive text, so that Visual Editor can expose appropriate editing widgets. Initially the expected types are: "markup" (edited with VE recursively), JSON (edited with a JSON editor, as with TemplateData), and "wikidata" (a SPARQL wikidata query <https://www.mediawiki.org/wiki/Wikidata_query_service/User_Manual>). ***Visual Template Invocation in wikitext*** This is a strawman for the wikitext serialization of a template invocation: ``` {{#visual template_name param1=<markup>'''bold'''</markup> param2=<json>{"foo":"bar"}</json> param3=<wikidata> SELECT ?spaceProbeLabel ?date ?picture WHERE { ?spaceProbe wdt:P31 wd:Q26529 ; wdt:P18 ?picture ; wdt:P619 ?date . SERVICE wikibase:label { bd:serviceParam wikibase:language "fr,en" . } } ORDER BY ?date </wikidata> }} ``` The parameter name `entity` is reserved; the data context object for the template always includes the wikidata entity object (equivalent to ` mw.wikibase.getEntityObject()`) as an implicit parameter named `entity`. This makes it easy to generate infoboxes (for example), since you can just insert `{{#formatproperty entity.P17}}` in your template (equivalent to `entity.formatPropertyValues( 'P17' ).value`). Hopefully there will be a friendly shortcut for ` entity.<something>`, which will include mapping wikidata identifier ` <something>` to a label in your language. Since arguments are delimited by html-style tags, the usual & and < escapes are all that are required to include arbitrary data inside a parameter, and whitespace can be used to separate arguments. The parameter names can be omitted for positional arguments. //*Perhaps we can allow for an abbreivated format for certain arguments: `"..."` -> literal string, `\d+` -> number, `{...}` -> JSON value. Then unit conversion (for example) can still be simply `{{#visual convertToMiles 5 km}}`*// ****Syntax details can be tweaked: your input wanted!**** Phabricator ticket: https://phabricator.wikimedia.org/T114454 --scott -- (http://cscott.net)

1 0

REST v1 API: Replacing bodyOnly flag in wikitext to HTML transform end point with body_only
by Gabriel Wicke 02 Oct '15

02 Oct '15

Hello, in an effort to standardize all post parameters in the REST v1 API to use snake_case names, we have deprecated the bodyOnly flag in the wikitext to HTML transform end point [1]. Instead, clients should use body_only, as mentioned in the documentation. We plan to remove support for the old form by the end of November. Thanks, Gabriel [1]: https://en.wikipedia.org/api/rest_v1/?doc#!/Transforms/post_transform_wikit…

1 0

RFC: Optional Travis integration for Jenkins
by C. Scott Ananian 02 Oct '15

02 Oct '15

We currently have several projects which can not be tested with our current Jenkins test infrastructure, and as a consequence are hosting their primary code repositories on github: * RESTBase <https://github.com/wikimedia/restbase> -- missing Cassandra support * iOS Mobile app <https://lists.wikimedia.org/pipermail/wikitech-l/2015-September/083005.html> -- missing OS X platform Other projects can only run a small portion of their test suite via Jenkins: * mw-ocg-latexer <https://github.com/wikimedia/mediawiki-extensions-Collection-OfflineContent…> -- requires LaTeX from PPA, image utilities (`jpegtran`). An alternative to allow these apps to be hosted on Wikimedia infrastructure (gerrit, eventually phabricator) is to allow travis integration with jenkins as an optional service. npm-travis <https://github.com/cscott/npm-travis> ( https://github.com/cscott/npm-travis) is a tool which will trigger Travis builds from NPM by pushing to a throwaway branch, which is then cleaned up after the tests complete. It integrates well with the Gerrit access control mechanism: the "Travis Bot" user can be granted push access only, and only to branches prefixed with `npm-travis/`, so it cannot be used to push changes to the master or deployment branches. This isn't a replacement for our jenkins test infrastructure, but it allows us to accommodate oddball repositories without taxing our infrastructure team or resorting to offsite repository hosting. There are WIP patches for integrating `npm-travis` with our jenkins infrastructure (https://gerrit.wikimedia.org/r/173045, https://gerrit.wikimedia.org/r/173046) but they seem to be blocked on policy disagreements. This RFC aims to resolve the policy issues. This RFC in Phabricator: https://phabricator.wikimedia.org/T114421 --scott -- (http://cscott.net)

5 9

[RFC] Hygienic templates
by C. Scott Ananian 02 Oct '15

02 Oct '15

*(Not to be confused with hygenic template arguments <https://phabricator.wikimedia.org/T114432>.)* As described in my Wikimania 2015 talk <https://wikimania2015.wikimedia.org/wiki/Submissions/Templates_are_dead!_Lo…!> (starting at slide 27 <https://wikimania2015.wikimedia.org/w/index.php?title=File:Templates_are_de…>), there are a number of reasons to mark certain templates as "hygienic". Among them: to allow high-performance incremental update of page contents after templates are modified, and to allow safe editing of template uses using HTML-based tools such as Visual Editor or jsapi <https://doc.wikimedia.org/Parsoid/master/#!/guide/jsapi>. This means that the output of the template is a complete DocumentFragment <https://developer.mozilla.org/en-US/docs/Web/API/DocumentFragment>: every open tag is closed and there are no nodes which the HTML adoption agency algorithm <http://dev.w3.org/html5/spec-LC/tree-construction.html#adoptionAgency> will reorder. This can be enforced in practice by running tidy (or something like it: T89331 <https://phabricator.wikimedia.org/T89331>) on the template output before it is inserted into the document. We are likely to be doing something like this already for T114072 <https://phabricator.wikimedia.org/T114072>. The purpose of this RFC is to settle the bikeshed-color questions about how templates will be marked hygienic/unhygienic. There are two main scenarios: ***Opt-in Hygiene*** In this scenario, unmarked templates are assumed "dirty", and will render correctly (as before), just more slowly. Some of the unmarked templates (the "actually dirty" ones) may be uneditable in VisualEditor. In the opt-in scenario, it is expected that profiling the "costliest"/"most frequently used/changed" templates on wikimedia projects will be done, and opt-in tags will be added first to those templates where the greatest potential performance gain may be achieved. (Opting in will enable incremental update of the parser cache after template modifications.) Proposals: 1. An explicit tag surrounding "hygienic" contents; this tag will exactly encapsulate the content which will be run through tidy. Proposed names: 1. `<tidy>...</tidy>` 2. `<domscope>...</domscope>` 3. `<domparse>...</domparse>` 2. A magic word, present in templates known to be hygienic. Proposed names: 1. `__SAFE__` 2. `__HYGIENIC__` 3. Some `TemplateData <https://www.mediawiki.org/wiki/Extension:TemplateData>` property? 4. Others? ***Opt-out Hygiene*** In this scenario, all templates are assumed "hygienic", and an explicit marker of some kind is required to opt-out of the usual tidy of the contents. Templates which opt-out are not editable in Visual Editor, but all other templates are (because the contents will have been automatically cleaned to close any stray open tags). In the opt-out scenario, it is expected that Parsoid (or other tools) will be run against the wikimedia projects to automatically add opt-out tags where possible; perhaps based on logging during the parse process. Proposals: 1. Magic word, present in unhygienic templates 1. `__UNSAFE__` 2. Others? Phabricator task for this RFC: https://phabricator.wikimedia.org/T114445 --scott -- (http://cscott.net)

3 5

Announcing Danny Horn's move to Community Tech
by Ryan Kaldari 01 Oct '15

01 Oct '15

Howdy (my apologies for cross-posting), I'm pleased to announce that the Community Tech team has a new Product Manager – Danny Horn, who will be moving from the Collaboration team to join this new initiative. The Community Tech team is focused on building improved curation and moderation tools for experienced Wikimedia editors.[1] Danny will be working with the team to develop the upcoming Community Wishlist Survey, which will help the team define and prioritize future projects.[2] We're excited to have someone with Danny's product management experience as well as extensive wiki community experience working with this team. Danny’s role on the Collaboration team will be filled by a new Product Manager which the Foundation is hiring for now. There’s an open job posting on the WMF job board.[3] Looking forward to working with Danny to build some awesome tools for the community! 1. https://www.mediawiki.org/wiki/Community_Tech_team 2. https://www.mediawiki.org/wiki/Community_Tech_team/Community_Wishlist_Survey 3. https://boards.greenhouse.io/wikimedia/jobs/104264?t=k7xsjm Ryan Kaldari Engineering Manager, Community Tech Wikimedia Foundation

2 1

Collaboration Team Update
by Trevor Parscal 01 Oct '15

01 Oct '15

Hi (and sorry for cross-posting) I’d first like to say that I’m excited about Danny’s new role and the positive impact I know he will have on the relatively new Community Tech team, and that he will be missed in the Editing group. This change now leaves an open position, which we are in the process of hiring a new Product Manager to fill. There’s an open position posted.[1] The Collaboration team, meanwhile, will continue their work to provide Flow as an opt-in Beta feature, allowing contributors to use Flow on their user talk page. This feature is currently available on MediaWiki.org, and will be enabled on other wikis upon request. Especially as more users enable Flow, the team will continue supporting users of the product by promptly triaging and resolving bugs. This quarter, which began yesterday, The Collaboration team will be focusing on the development of cross-wiki notifications and other Echo improvements. They’re also continuing to advance efforts to research and prototype solutions for advanced editor workflows. Further feature development on Flow discussion tools will be based on an assessment following the 'workflows' research work, and on editor feedback at the wikis already using Flow.[2] Please ask any questions you may have on-wiki at https://www.mediawiki.org/wiki/Talk:Flow to reduce fragmentation. - Trevor [1] https://boards.greenhouse.io/wikimedia/jobs/104264?t=k7xsjm [2] https://www.mediawiki.org/wiki/Flow/Rollout

1 0

[RFC] Hygienic template arguments
by C. Scott Ananian 01 Oct '15

01 Oct '15

As described in my Wikimania 2015 talk <https://wikimania2015.wikimedia.org/wiki/Submissions/Templates_are_dead!_Lo…!> (starting at slide 31 <https://wikimania2015.wikimedia.org/w/index.php?title=File:Templates_are_de…>), the existing template argument syntax offers a number of traps for the unwary (for example, with the characters `=`, `|`, etc). As a result, it is difficult to easily move large blocks of text into templates. As a result, we often have constructs such as: ``` {{tablestart|class="shiny"}} | Hello || world {{tableend}} ``` which pose a number of issues: 1. There is no mechanism to ensure `{{tablestart}}` and `{{tableend}}` are properly matched. 2. Both `{{tablestart}}` and `{{tableend}}` emit unbalanced HTML, which complicates work on efficiently updating the parser cache after template changes. 3. Due to the tag matching issues, this whole block is uneditable by Visual Editor. ***Hygienic arguments*** provide a new form of template invocation which avoids these issues. The above example would be written as: ``` {{>Table|class="shiny"}} | Hello || world {{< Table}} ``` and the contents between the template start `{{>...}}` and template end ` {{<...}}` are passed as (WLOG) the first positional argument of the template. Named arguments (like `class` in this example) can be passed using the existing argument syntax as before. The new `Template:Table` can now emit properly balanced HTML, with both `<table>` and `</table>` generated by the same template (instead of by two separate templates). Visual Editor can now edit this block as a single template invocation, invoking itself recursively to edit the template arguments as it does now. Further, there are no special characters or odd escape rules in the argument when expressed this way. The only "special" construct inside the argument is `{{<...}}` and any of those characters can be replaced by an html entity to allow that character sequence to be included literally if needed. That is, we don't need special `{{!}}`, `{{=}}`, etc escapes. Another example, from Brion Vibber's talk on citations <https://wikimania2015.wikimedia.org/wiki/Submissions/Future_of_structured_d…!> : ``` {{>cite|id=“32412”}} First person plural pronouns in Isthmus-Mecayapan Nahuat: :''nejamēn'' ({{IPA|[nehameːn]}}) "We, but not you" (= me & them) :''tejamēn'' ({{IPA|[tehameːn]}}) "We along with you" (= me & you & them) {{<cite}} ``` Note that it was easy to surround the entire text covered by the citation into the `{{cite}}` template, since I didn't need to worry about the fact that the text included the special character `=`. ***Additional arguments*** (optional extension) >From experience with mustache templates <https://mustache.github.io/> which have a similar construct <https://mustache.github.io/mustache.5.html#Sections> (well, actually with the spacebars <https://github.com/meteor/meteor/blob/devel/packages/spacebars/README.md> equivalent, which shares a concern that template contents be well-balanced), it seems that a large number of use cases can be satisfied with only one or two "block arguments". But let's say we want to allow more generality. We could support: ``` {{>Foo|namedArg=bar|anotherNamedArg=bat}} This first block goes in argument "1". {{else}} Now this goes in argument "2". {{else}} Now this goes in argument "3". It's a little more natural if there are only one or two arguments, but we could keep going if we wanted. {{<Foo}} ``` Additional proposals for this syntax welcome, I'm just trying to avoid too many crazy new constructs. ***Keeping things terse*** We might allow `{{<}}` as a shortcut to end the block argument, instead of requiring the author to repeat the name of the template. Some folks will like the brevity, others may prefer the more explicit error messages provided when you spell out the name of the template you intend to close. **Strict start-of-line constraints** For ease of parsing (and reading) we can enforce the constraint that the ` {{>...}}` and `{{<...}}` tags must be alone on their line, and that the newline following the `{{>...}}` and `{{<...}}` tags is not copied into the output. We could enforce start-of-line context on the result to avoid the T14974/T2529 hacks and make behavior consistent. (More thought welcome here.) ***Migrating template use*** We might allow configuring details of the parameter name(s) used for the block argument(s) via `TemplateData`, to more easily use the new syntax with existing templates. Phabricator task for this RFC: https://phabricator.wikimedia.org/T114432 --scott -- (http://cscott.net)

1 0

MediaWiki 1.26 release blockers
by Andre Klapper 01 Oct '15

01 Oct '15

MediaWiki 1.26 will be released later this year. There are 62 open tasks in Phabricator tagged with "MW-1.26-release". Some tasks miss an assignee; some tasks welcome patch review. Dropping some links here to raise awareness and ask for help: 12 without any assignee and without a "Patch-For-Review": https://phabricator.wikimedia.org/maniphest/query/MoC9YKZ4gsR./#R 48 with "Patch-For-Review": https://phabricator.wikimedia.org/maniphest/query/w6gwbg0JIWRl/#R 23 with Assignees set (Aaron, Addshore, Bawolff, bd808, Catrope, Cenarium, Dchan, Gilles, Jdlrobson, Hexmode, Legoktm, Matmarex, Seb35, Sn1per, Tgr, Željko): https://phabricator.wikimedia.org/maniphest/query/KWoEsNbaAjFX/#R andre -- Andre Klapper | Wikimedia Bugwrangler http://blogs.gnome.org/aklapper/

7 7

Phabricator monthly statistics - 2015-09
by communitymetrics＠wikimedia.org 30 Sep '15

30 Sep '15

Hi Community Metrics team, this is your automatic monthly Phabricator statistics mail. Number of accounts created in (2015-09): 363 Number of active users (any activity) in (2015-09): 863 Number of task authors in (2015-09): 477 Number of users who have closed tasks in (2015-09): 268 Number of projects which had at least one task moved from one column to another on their workboard in (2015-09): 209 Number of tasks created in (2015-09): 3351 Number of tasks closed in (2015-09): 2981 Number of open and stalled tasks in total: 25811 Median age in days of open tasks by priority: Unbreak now: 11 Needs Triage: 125 High: 164 Normal: 301 Low: 660 Lowest: 528 (How long tasks have been open, not how long they have had that priority) TODO: Numbers which refer to closed tasks might not be correct, as described in T1003. Yours sincerely, Fab Rick Aytor (via community_metrics.sh on iridium at Thu Oct 1 00:00:08 UTC 2015)

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Wikitech-l October 2015