Wikitext-l

wikitext-l@lists.wikimedia.org

239 discussions

Google Code-in: Get your tasks for young contributors prepared!
by Andre Klapper 16 Oct '17

16 Oct '17

Google Code-in is an annual contest for 13-17 year old students. It will take place from Nov28 to Jan17 and is not only about coding tasks. While we wait whether Wikimedia will get accepted: * You have small, self-contained bugs you'd like to see fixed? * Your documentation needs specific improvements? * Your user interface has small design issues? * Your Outreachy/Summer of Code project welcomes small tweaks? * You'd enjoy helping someone port your template to Lua? * Your gadget code uses some deprecated API calls? * You have tasks in mind that welcome some research? Also note that "Beginner tasks" (e.g. "Set up Vagrant" etc) and "generic" tasks are very welcome (e.g. "Choose & fix 2 PHP7 issues from the list in https://phabricator.wikimedia.org/T120336 "). Because we will need hundreds of tasks. :) And we also have more than 400 unassigned open 'easy' tasks listed: https://phabricator.wikimedia.org/maniphest/query/HCyOonSbFn.z/#R Would you be willing to mentor some of those in your area? Please take a moment to find / update [Phabricator etc.] tasks in your project(s) which would take an experienced contributor 2-3 hours. Check https://www.mediawiki.org/wiki/Google_Code-in/Mentors and please ask if you have any questions! For some achievements from last round, see https://blog.wikimedia.org/2017/02/03/google-code-in/ Thanks!, andre -- Andre Klapper | Wikimedia Bugwrangler http://blogs.gnome.org/aklapper/

1 0

Styling definition lists
by Thomas Douillard 22 Aug '17

22 Aug '17

Definition lists are underused. People usually prefers to format themselves to fit their preferences than to use the "; :„ syntax, or custom templates (theorem, examples, …) It’s not without good reasons : it’s kird of unaesy to do multilne definitions, they can’t be easily styled, the imbrication beetween dls and ul and on is tricky once you want to do something elaborated. Yet using dls could allow us to better annotate semantically our articles. Any idea on how to make their usage easier and to style them ? We can style tables, lists are not as easy to style.

3 6

Replacement of Tidy on WMF wikis, and how you can help
by Erica Litrenta 07 Jul '17

07 Jul '17

Hi all, apologies for the cross-posting! How to read this post? ---------------------- * For those without time to read lengthy technical emails, read the TL;DR section. * For those who don't care about all the details but want to help with this project, you can read sections 1 and 2 about Tidy, and then skip to section 7. * For those who like all their details, read the post in its entirety, and follow the links. Please ask follow up questions on wiki *on the FAQ’s talk page* [0]. If you find a bug, please report it *on Phabricator or on the page mentioned above*. TL;DR ----- The Parsing team wants to replace Tidy with a RemexHTML-based solution on the Wikimedia cluster by June 2018. This will require editors to fix pages and templates to address wikitext patterns that behave differently with RemexHTML. Please see 'What editors will need to do' section on the Tidy replacement FAQ [1]. 1. What is Tidy? ---------------- Tidy [2] is a library currently used by MediaWiki to fix some HTML errors found in wiki pages. Badly formed markup is common on wiki pages when editors use HTML tags in templates and on the page itself. (Ex: unclosed HTML tags, such as a <small> without a </small>, are common). In some cases, MediaWiki can generate erroneous HTML by itself. If we didn't fix these before sending it to browsers, some would display things in a broken way to readers. But Tidy also does other "cleanup" on its own that is not required for correctness. Ex: it removes empty elements and adds whitespace between HTML tags, which can sometimes change rendering. 2. Why replace it? ------------------ Since Tidy is based on HTML4 semantics and the Web has moved to HTML5, it also makes some incorrect changes to HTML to 'fix' things that used to not work; for example, Tidy will unexpectedly move a bullet list out of a table caption even though that's allowed. HTML4 Tidy is no longer maintained or packaged. There have also been a number of bug reports filed against Tidy [3]. Since Parsoid is based on HTML5 semantics, there are differences in rendering between Parsoid's rendering of a page and current read view that is based on Tidy. 3. Project status ----------------- Given all these considerations, the Parsing team started work to replace Tidy [4] around mid-2015. Tim Starling started this work and after a survey of existing options, decided to write a wrapper over a Java-based HTML5 parser. At the time we started the project, we thought we could probably have Tidy replaced by mid-2016. Alas! 4. What is replacing Tidy? -------------------------- Tidy will be replaced by a RemexHTML-based solution that uses the RemexHTML[5] library along with some Tidy-compatibility shims to ensure better parity with the current rendering. RemexHTML is a PHP library that Tim wrote with C.Scott’s input that implements the HTML5 parsing spec. 5. Testing and followup ----------------------- We knew that some pages will be affected and need fixing due to this change. In order to more precisely identify what that would be, we wanted to do some thorough testing. So, we built some new tools [6][7] and overhauled and upgraded other test infrastructure [8][9] to let us evaluate the impacts of replacing Tidy (among other such things in the future) which can be a subject of a post all on its own. You can find the details of our testing on the wiki [1][10], but we found that a large number of pages had rendering differences. We analyzed the results and categorized the source of differences. Based on that, to ease the process of replacement, we added a bunch of compatibility shims to mimic what Tidy does. I am skipping the details in this post. Even after that, newer testing showed that this nevertheless still leaves us with a few patterns that need fixing that we cannot / don't want to work around automatically. 6. Tools to assist editors: Linter & ParserMigration ---------------------------------------------------- In October 2016, at the parsing team offsite, Kunal ([[User:Legoktm (WMF)]]) dusted off the stalled wikitext linting project [11] and (with the help from a bunch of people on the Parsoid, db/security/code review areas) built the Linter extension that surfaces wikitext errors that Parsoid knows about to let editors fix them. Earlier this year, we decided to use Linter in service of Tidy replacement. Based on our earlier testing results, we have added a set of high-priority linter categories that identifies specific wikitext markup patterns on wiki pages that need to be fixed [12]. Separately, Tim built the ParserMigration extension to let editors evaluate their fixes to pages [13]. You can enable this in your editing preferences or replace '&action=edit' in your url bar with '&action=parsermigration-edit'. 7. What editors have to do -------------------------- The part that you have all been waiting for! Please see 'What editors will need to do' section on the Tidy replacement FAQ [1]. We have added simplified instructions, so that even community members who do not consider themselves "techies" can still learn about ways to fix pages. We'll keep that section up to date based on feedback and questions. But since it is a wiki, please also edit and tweak as required to make the text useful for yourselves! This is a first call for fixes and it is about the problems defined as "high priority". We'll issue other calls in the future for any other necessary Tidy fixups. Caveats: * As noted on that page, the linter categories don't cover all the possible sources of rendering differences. For example, there is still T157418 [14] left to address. For those who have an opinion about this, please chime in on that task. We are still evaluating the best solution for this without adding more cruft to wikitext behavior or kicking the cleanup can down the road. * As the issues in the identified linter categories are fixed, we might be better able to isolate other issues that need addressing. 8. So, when will Tidy actually be replaced? ------------------------------------------- We really would like to get Tidy removed from the cluster latest by June 2018 (or sooner if possible), and your assistance and prompt attention to these markup issues would be very helpful. We will do this in a phased manner on different wikis rather than all at once on all wikis. We really want to do this as smoothly as possible without disrupting the work of editors or affecting the rendering of the large corpus of pages on the various wikis. As you might have gathered from the text above, we have built and leveraged a wide variety of tools to assist with this. 9. Monitoring progress ---------------------- In order to monitor progress, we plan to do a weekly (or some such periodic frequency) test run that compares the rendering of pages with Tidy and with RemexHTML on a large sample of pages (in the 50K range) from a large subset of Wikimedia wikis (~50 or so). This will give us a pulse of how fixups are going, and when we might be able to flip the switch on different wikis. Best, Elitre (WMF) on behalf of the Parsing Team (https://www.mediawiki.org/wiki/Parsing ) References ---------- 0. https://www.mediawiki.org/wiki/Talk:Parsing/Replacing_Tidy/FAQ 1. https://www.mediawiki.org/wiki/Parsing/Replacing_Tidy/FAQ#Wh at_will_editors_need_to_do.3F 2. https://en.wikipedia.org/wiki/HTML_Tidy 3. https://phabricator.wikimedia.org/tag/tidy/ 4. https://phabricator.wikimedia.org/T89331 5. https://github.com/wikimedia/mediawiki-libs-RemexHtml 6. https://phabricator.wikimedia.org/T120345 7. https://github.com/wikimedia/integration-uprightdiff 8. https://github.com/wikimedia/integration-visualdiff 9. https://github.com/wikimedia/mediawiki-services-parsoid-testreduce 10. https://www.mediawiki.org/wiki/Parsing/Replacing_Tidy 11. https://phabricator.wikimedia.org/T48705 12. https://www.mediawiki.org/wiki/Help:Extension:Linter# Goal:_Replacing_Tidy 13. https://www.mediawiki.org/wiki/Help:Extension:Linter# Verifying_fixes_for_these_lint_categories 14. https://phabricator.wikimedia.org/T157418

1 0

Parsoid: node 0.1x deprecated now; node 0.1x support will end March 31st, 2017
by Subramanya Sastry 07 Apr '17

07 Apr '17

The Parsing team at the Wikimedia Foundation that develops the Parsoid service is deprecating support for node 0.1x. Parsoid is the service that powers VisualEditor, Content Translation, and Flow. If you don't run a MediaWiki install that uses VisualEditor, then this announcement does not affect you. Node 0.10 has reached end of life on October 31st, 2016 [1] and node 0.12 is scheduled to reach end of life December 31st, 2016 [1]. Yesterday, we released a 0.6.1 debian package [2] and a 0.6.1 npm version of Parsoid [3]. This will be the last release that will have node 0.1x support. We'll continue to provide any necessary critical bug fixes and security fixes for the 0.6.1 release till March 31st 2017 and will be completely dropping support for all node versions before node v4.x starting April 2017. If you are running a Parsoid service on your wiki and are still using node 0.1x, please upgrade your node version by April 2017. The Wikimedia cluster runs node v4.6 right now and will soon be upgraded to node v6.x [4]. Parsoid has been tested with node 0.1x, node v4.x and node v6.x and works with all these versions. However, we are dropping support for node 0.1x right away from the master branch of Parsoid. Going forward, the Parsoid codebase will adopt ES6 features available in node v4.x and higher which aren't supported in node 0.1x and will constitute a breaking change. Subramanya Sastry (Subbu), Technical Lead and Manager, Parsing Team, Wikimedia Foundation. [1] Node.js Long Term Support schedule @ https://github.com/nodejs/LTS [2] https://www.mediawiki.org/wiki/Parsoid/Releases [3] https://www.npmjs.com/package/parsoid [4] https://phabricator.wikimedia.org/T149331

1 1

Developer Wishlist Survey: propose your ideas
by Srishti Sethi 20 Jan '17

20 Jan '17

That's right! At the Wikimedia Developer Summit, we decided to organize a Developer Wishlist Survey, and here we go: https://www.mediawiki.org/wiki/Developer_Wishlist The Wikimedia technical community seeks input from developers for developers, to create a high-profile list of desired improvements. The scope of the survey includes the MediaWiki platform (core software, APIs, developer environment, enablers for extensions, gadgets, templates, bots, dumps), the Wikimedia server infrastructure, the contribution process, and documentation. The best part: we want to have the results published by Wednesday, February 15. Yes, in a month, to have a higher chance to influence the Wikimedia Foundation annual plan FY 2017-18. There's no time to lose. *Propose your ideas before the end of January, * either by pushing existing tasks in Phabricator or by creating new ones. You can find instructions on the wiki page <https://www.mediawiki.org/wiki/Developer_Wishlist>. Questions and feedback are welcome especially on the related Talk page. The voting phase is expected to start on February 6 (tentative). Watch this space (or even better, the wiki page). Cheers, Srishti Sethi Developer Advocate, Technical Collaboration team Wikimedia Foundation https://www.mediawiki.org/wiki/User:SSethi_(WMF)

1 0

Parsoid rejects a valid SSL cert?
by Daniel Barrett 07 Dec '16

07 Dec '16

I'm seeking help with a MediaWiki 1.27.1 site that uses VisualEditor and parsoid. Everything worked perfectly until I switched from a non-secure site to using SSL (with a valid, commercial cert). Now I get an error 500 each time I try to launch VisualEditor. The problem goes away if I set: parsoidConfig.strictSSL = false; The cert is purchased from Comodo (PositiveSSL), not self-signed, and it yields a green padlock in Chrome. I believe I'm using parsoid 0.6.1all on Ubuntu 16.04LTS. I also see Comodo root certs in /etc/ssl/certs. The errors in the parsoid log are: {"name":"../src/lib/index.js","hostname":"example","pid":23005,"level":40,"logType":"warning/api/unable_to_verify_leaf_signature","wiki":"example.com","title":"Home","oldId":null,"reqId":null,"userAgent":"VisualEditor-MediaWiki/1.27.1","msg":"Failed API request, {\"error\":{\"code\":\"UNABLE_TO_VERIFY_LEAF_SIGNATURE\"},\"retries-remaining\":1}","longMsg":"Failed API request,\n{\"error\":{\"code\":\"UNABLE_TO_VERIFY_LEAF_SIGNATURE\"},\"retries-remaining\":1}","levelPath":"warn/api/unable_to_verify_leaf_signature","time":"2016-12-07T01:00:28.297Z","v":0} {"name":"../src/lib/index.js","hostname":"example","pid":23005,"level":40,"logType":"warning/api/unable_to_verify_leaf_signature","wiki":"example.com","title":"Home","oldId":null,"reqId":null,"userAgent":"VisualEditor-MediaWiki/1.27.1","msg":"Failed API request, {\"error\":{\"code\":\"UNABLE_TO_VERIFY_LEAF_SIGNATURE\"},\"retries-remaining\":0}","longMsg":"Failed API request,\n{\"error\":{\"code\":\"UNABLE_TO_VERIFY_LEAF_SIGNATURE\"},\"retries-remaining\":0}","levelPath":"warn/api/unable_to_verify_leaf_signature","time":"2016-12-07T01:00:28.333Z","v":0} {"name":"../src/lib/index.js","hostname":"example","pid":23005,"level":60,"logType":"fatal/request","wiki":"example.com","title":"Home","oldId":null,"reqId":null,"userAgent":"VisualEditor-MediaWiki/1.27.1","msg":"Template Fetch failure for \"Home\": Error: unable to verify the first certificate","stack":"Error: Template Fetch failure for \"Home\": Error: unable to verify the first certificate\n at TemplateRequest.ApiRequest._requestCB (/usr/lib/parsoid/src/lib/mw/ApiRequest.js:395:15)\n at self.callback (/usr/lib/parsoid/node_modules/request/request.js:187:22)\n at emitOne (events.js:77:13)\n at Request.emit (events.js:169:7)\n at Request.onRequestError (/usr/lib/parsoid/node_modules/request/request.js:813:8)\n at emitOne (events.js:77:13)\n at ClientRequest.emit (events.js:169:7)\n at TLSSocket.socketErrorListener (_http_client.js:258:9)\n at emitOne (events.js:77:13)\n at TLSSocket.emit (events.js:169:7)\n at emitErrorNT (net.js:1256:8)\n at nextTickCallbackWith2Args (node.js:441:9)\n at process._tickCallback (node.js:355:17)","longMsg":"Template Fetch failure for \"Home\": Error: unable to verify the first certificate","levelPath":"fatal/request","time":"2016-12-07T01:00:28.340Z","v":0} Any help appreciated! Dan

1 1

Parsoid Security Update and Release
by Subramanya Sastry 01 Nov '16

01 Nov '16

The parsing team has fixed a security bug in Parsoid [1]. * Users could send invalid prefixes, formats, or domains and run javascript code on the error page that Parsoid displayed. * This fix has been applied to the Wikimedia cluster [2] and also merged into Parsoid master [1]. * We have also released a 0.5.3 deb version with this patch applied. [3] * We have also released a 0.5.3 npm version of Parsoid. [4] * Parsoid is a stateless service and doesn't retain any state between requests. In private wikis, VisualEditor can be configured to forward the user cookie to Parsoid to pass along to the MediaWiki API to parse a page, but this exploit is not exposed through VE. In addition, Parsoid doesn't receive any user credentials on public wikis. * However, if a wiki's Parsoid service is publicly accessible on the internet *and* is accessible through the wiki's domain, then, this exploit can be used to leak user cookies for that wiki. For all wikis that use Parsoid in this fashion, we recommend they patch their Parsoid installation immediately. * On the Wikimedia cluster, Parsoid is proxied behind RESTBase and is not public accessible and as such, this exploit wasn't available for an exploit to steal user sessions. Thanks to the reporter of this exploit, Darian Patrick from the Security Team, Arlo Breault from the Parsing Team, Daniel Zahn and others from Ops for their assistance handling this bug and preparing this release. [1] https://gerrit.wikimedia.org/r/#/c/319115 [2] https://www.mediawiki.org/wiki/Parsoid/Deployments#Monday.2C_October_31.2C_… [3] https://releases.wikimedia.org/debian/pool/main/p/parsoid/ [4] https://www.npmjs.com/package/parsoid Subramanya Sastry, Technical Lead and Manager, Parsing Team, Wikimedia Foundation.

1 0

The Wikimedia Developer Summit wants YOU!
by Srishti Sethi 17 Oct '16

17 Oct '16

Hello! The Wikimedia Developer Summit <https://www.mediawiki.org/wiki/Wikimedia_Developer_Summit> is the annual meeting to push the evolution of MediaWiki and other technologies supporting the Wikimedia movement. The next edition will be held in San Francisco on January 9-11, 2017. We welcome all Wikimedia technical contributors, third party developers, and users of MediaWiki and the Wikimedia APIs. We specifically want to increase the participation of volunteer developers and other contributors dealing with extensions, apps, tools, bots, gadgets, and templates. Important deadlines: - Monday, October 24: This is the last day to request travel sponsorship. Applying takes less than five minutes. - Monday, October 31: This is the last day to propose an activity. Bring the topics you care about! Subscribe to weekly updates: https://www.mediawiki.org/wiki/Topic: Td5wfd70vptn8eu4 Please feel free to forward this email to anyone who might be interested in attending! Thanks, Srishti -- Srishti Sethi ssethi(a)wikimedia.org

1 0

Deprecating rest.wikimedia.org in favor of /api/rest_v1/
by Gabriel Wicke 14 Apr '16

14 Apr '16

We have decided to officially retire the rest.wikimedia.org domain in favor of /api/rest_v1/ at each individual project domain. For example, https://rest.wikimedia.org/en.wikipedia.org/v1/?doc becomes https://en.wikipedia.org/api/rest_v1/?doc Most clients already use the new path, and benefit from better performance from geo-distributed caching, no additional DNS lookups, and sharing of TLS / HTTP2 connections. We intend to shut down the rest.wikimedia.org entry point around March, so please adjust your clients to use /api/rest_v1/ soon. Thank you for your cooperation, Gabriel -- Gabriel Wicke Principal Engineer, Wikimedia Foundation

1 1

Trouble with Parsoid
by Шотт Ольга 01 Mar '16

01 Mar '16

2 2

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

Wikitext-l