For the last decade we've supported uploading SVG vector images to
MediaWiki, but we serve them as rasterized PNGs to browsers. Recently,
display resolutions are going up and up, but so is concern about
low-bandwidth mobile users.
This means we'd like sharper icons and diagrams on high-density phone
displays, but are leery of adding extra srcset entries with 3x or 4x
size PNGs which could become very large. (In fact currently MobileFrontend
strips even the 1.5x and 2x renderings we have now, making diagrams very
blurry on many mobile devices. See https://phabricator.wikimedia.org/T133496 -
fix in works.)
Here's the base bug for SVG client side rendering:
https://phabricator.wikimedia.org/T5593
I've turned it into an "epic" story tracking task and hung some blocking
tasks off it; see those for more details.
TL;DR stop reading here. ;)
One of the basic problems in the past was reliably showing them natively in
an <img>, with the same behavior as before, without using JavaScript hacks
or breaking the hamlet caching layer. This is neatly resolved for current
browsers by using the "srcset" attribute -- the same one we use to specify
higher-resolution rasterizations. If instead of PNGs at 1.5x and 2x
density, we specify an SVG at 1x, the SVG will be loaded instead of the
default PNG.
Since all srcset-supporting browsers allow SVG in <img> this should "just
work", and will be more compatible than using the experimental <picture>
element or the classic <object> which deals with events differently. Older
browsers will still see the PNG, and we can tweak the jquery.hidpi srcset
polyfill to test for SVG support to avoid breaking on some older browsers.
This should let us start testing client-side SVG via a beta feature (with
parser cache split on the user pref) at which point we can gather more
real-world feedback on performance and compatibility issues.
Rendering consistency across browser engines is a concern. Supposedly
modern browsers are more consistent than librsvg but we haven't done a
compatibility survey to confirm this or identify problematic constructs.
This is probably worth doing.
Performance is a big question. While clean simple SVGs are often nice and
small and efficient, it's also easy to make a HUGEly detailed SVG that is
much larger than the rasterized PNGs. Or a fairly simple small file may
still render slowly due to use of filters.
So we probably want to provide good tools for our editors and image authors
to help optimize their files. Show the renderings and the bandwidth balance
versus rasterization; maybe provide in-wiki implementation of svgo or other
lossy optimizer tools. Warn about things that are large or render slowly.
Maybe provide a switch to run particular files through rasterization always.
And we'll almost certainly want to strip comments and white space to save
bandwidth on page views, while retaining them all in the source file for
download and reediting.
Feature parity also needs more work. Localized text in SVGs is supported
with our server side rendering but this won't be reliable in the client;
which means we'll want to perform a server side transformation that creates
per-language "thumbnail" SVGs. Fonts for internationalized text are a big
deal, and may require similar transformations if we want to serve them...
Which may mean additional complications and bandwidth usage.
And then there are long term goals of taking more advantage of SVGs dynamic
nature -- making things animated or interactive. That's a much bigger
question and has implementation and security issues!
-- brion
Hi everyone,
The ArchCom-RFC status update from our previous Wednesday's ArchCom
meeting is in the mail below. The dedicated wiki page was updated in
a far more timely manner than this mailing list:
<https://www.mediawiki.org/wiki/Architecture_committee/Status>
-----
##### Recent RFC meetings
- ArchCom Planning meeting: 2016-05-11: [Phab:E170][] (E156/6)
- Notes: [Architecture committee/2016-05-11][]
- ArchCom-RFC office hour: 2016-05-11: [Phab:E171][] (E66/34)
- [T113034 RFC: Overhaul Interwiki map, unify with Sites and
WikiMap][]
##### Upcoming RFC meetings
- ArchCom Planning meeting: 2016-05-18: [Phab:E183][] (E156/7)
- Notes: [Architecture committee/2016-05-18][]
- ArchCom-RFC office hour: 2016-05-18: [Phab:E184][] (E66/35)
- [T102476: RFC: Requirements for change propagation][]
- The [ArchCom-RfCs board][] has a "Ready for RFC meeting" column
which should contain an ordered queue of RFCs planned for IRC office
hour
##### Entering Final Comment Period
- (none)
##### Recently Approved
- [T120164 RFC: Institute "last call" period][]
#### RFC inbox
- [ArchCom RFC board][]:
- Inbox zero on 2016-06-11.
##### Shepherd status
- Brion
- (?)
- Daniel
- Software Quality working group?
- Working on Multi Content Rev Spec with Brion
- T113034 [RFC: Overhaul Interwiki map, unify with Sites and
WikiMap][T113034 RFC: Overhaul Interwiki map, unify with Sites
and WikiMap]: (Update?)
- Gabriel
- T39902 [RFC: Implement rendering of redlinks as
post-processor][]: Solutions for highlighting links to
non-existing pages in Parsoid HTML. Plan in place / agreed
between Parsing and Services. Implementation in change
propagation service ready, preparing for deploy possibly next
week.
- T122942 [RFC: Support language variants in the REST API][]:
Waiting for progress on more general question of language
selection granularity / strategy & T114662.
- Roan
- T108655 [RFC: Standardise JavaScript interfaces][]: I need to
start the second part, but the recent comments have me confused.
I'll need to talk to Timo and figure out what the subject of
part two should be.
- RobLa
- Created [RFCstatus][] page.
- Still need to schedule an RFC triage meeting outside of
ArchCom-RFC time
- Tim
- T114444 [RFC: Introduce notion of DOM scopes in wikitext][]:
(Update?)
- Timo
- No update
##### No activity in the last two weeks
- T18691 [RFC: Section headings should have a clickable anchor][]
(Timo)
- T111588 [RFC: API-driven web front-end][] (Timo)
- T123753 [Establish retrospective reports for Security and
Performance incidents][] (RobLa)
- T122825 [Service ownership and minimum maintenance requirements][]
(Gabriel)
- T105766 [RFC: Dependency graph storage][] (Gabriel)
- T124504 [Transition WikiDev '16 working areas into working groups][]
(RobLa)
- T66214 [Use content hash based image / thumb URLs & define an
official thumb API][] (Brion)
- T91162 [RFC: Shadow namespaces][] (Brion)
- T128351 [RFC: Notifications in core][] (Brion)
- T122825 [Service ownership and minimum maintenance requirements][]
(Gabriel)
- T54807 [Identify and remove legacy preferences from MediaWiki
core][] (no shepherd)
- T88596 [Improving extension management][] (Daniel)
##### Useful Phab links
- [Query for shepherd assignments][]
- [Query for all ArchCom RFCs][]
- [ArchCom-RfCs board][]
[Phab:E170]: https://phabricator.wikimedia.org/E170 "phab:E170"
[Architecture committee/2016-05-11]:
https://www.mediawiki.org/wiki/Architecture_committee/2016-05-11
"Architecture committee/2016-05-11"
[Phab:E171]: https://phabricator.wikimedia.org/E171 "phab:E171"
[T113034 RFC: Overhaul Interwiki map, unify with Sites and WikiMap]:
https://phabricator.wikimedia.org/T113034
"phab:T113034"
[Phab:E183]: https://phabricator.wikimedia.org/E183 "phab:E183"
[Architecture committee/2016-05-18]:
https://www.mediawiki.org/wiki/Architecture_committee/2016-05-18
"Architecture committee/2016-05-18"
[Phab:E184]: https://phabricator.wikimedia.org/E184 "phab:E184"
[T102476: RFC: Requirements for change propagation]:
https://phabricator.wikimedia.org/T102476
"phab:T102476"
[ArchCom-RfCs board]: https://phabricator.wikimedia.org/project/board/52/
"phab:project/board/52/"
[T120164 RFC: Institute "last call" period]:
https://phabricator.wikimedia.org/T120164
"phab:T120164"
[ArchCom RFC board]: https://phabricator.wikimedia.org/tag/archcom-rfc/
"phab:tag/archcom-rfc/"
[RFC: Implement rendering of redlinks as post-processor]:
https://phabricator.wikimedia.org/T39902
"phab:T39902"
[RFC: Support language variants in the REST API]:
https://phabricator.wikimedia.org/T122942
"phab:T122942"
[RFC: Standardise JavaScript interfaces]:
https://phabricator.wikimedia.org/T108655
"phab:T108655"
[RFCstatus]: https://www.mediawiki.org/wiki/RFCstatus "RFCstatus"
[RFC: Introduce notion of DOM scopes in wikitext]:
https://phabricator.wikimedia.org/T114444
"phab:T114444"
[RFC: Section headings should have a clickable anchor]:
https://phabricator.wikimedia.org/T18691
"phab:T18691"
[RFC: API-driven web front-end]: https://phabricator.wikimedia.org/T111588
"phab:T111588"
[Establish retrospective reports for Security and Performance
incidents]: https://phabricator.wikimedia.org/T123753 "phab:T123753"
[Service ownership and minimum maintenance requirements]:
https://phabricator.wikimedia.org/T122825
"phab:T122825"
[RFC: Dependency graph storage]: https://phabricator.wikimedia.org/T105766
"phab:T105766"
[Transition WikiDev '16 working areas into working groups]:
https://phabricator.wikimedia.org/T124504
"phab:T124504"
[Use content hash based image / thumb URLs & define an official thumb
API]: https://phabricator.wikimedia.org/T66214 "phab:T66214"
[RFC: Shadow namespaces]: https://phabricator.wikimedia.org/T91162
"phab:T91162"
[RFC: Notifications in core]: https://phabricator.wikimedia.org/T128351
"phab:T128351"
[Identify and remove legacy preferences from MediaWiki core]:
https://phabricator.wikimedia.org/T54807
"phab:T54807"
[Improving extension management]: https://phabricator.wikimedia.org/T88596
"phab:T88596"
[Query for shepherd assignments]:
https://phabricator.wikimedia.org/maniphest/query/xc.j4DEYcjwu/#R
"phab:maniphest/query/xc.j4DEYcjwu/"
[Query for all ArchCom RFCs]:
https://phabricator.wikimedia.org/search/query/lgPo47yENoTl/#R
"phab:search/query/lgPo47yENoTl/"
After a first wave of feedback it is clear that we have two different
discussions that need solving:
1. If we use WMF Technical Collaboration budget for Phabricator
improvements, should we invest it the Calendar application? This is a well
framed discussion with a clear deadline for decisions (very soon, or our
FY2015-16 budget will be gone). I have created a task to discuss this
specifically: https://phabricator.wikimedia.org/T135327
2. If we want to improve the announcement and promotion of Wikimedia tech
events, is the best first step to improve Phabricator Calendar? This is a
complex discussion with many ramifications that we can keep discussing at
https://phabricator.wikimedia.org/T1035
On Sat, May 14, 2016 at 3:06 AM, Legoktm <legoktm.wikipedia(a)gmail.com>
wrote:
> If we're going to be investing money into improving Phabricator upstream
> (a great idea IMO), I think we should start with problem areas that
> affect a large number of users/developers.
Agreed. If I am proposing improvements around Calendar is because our team
believes that we have a problem affecting the announcement and promotion of
technical activities beyond the circle of usual and core contributors. By
improving this area, we believe we could reach better to more casual
technical contributors in our movement, and reach out better to new
contributors out there.
> There's plenty of low-hanging fruit like non-drag-n-drop file uploads[1].
Agreed. This one is being funded already.
> [2] was also mentioned on #wikimedia-tech a few days ago
Good, I wasn't aware. Is there a task in Wikimedia Phabricator reflecting
the level of need, support, consensus? Feel free to propose it as suggested
in https://phabricator.wikimedia.org/T135327
> or some of the UI/UX issues Nemo brought up after the last Phabricator
> upgrade[3].
>
I think https://secure.phabricator.com/T10926 reflects the essence of the
improvements requested after the Phabricator UX update. It looks like the
discussion so far is more about agreement on UX problems/solutions than
complexity of the solution once agreed, but if there is room for funded
prioritization, that also looks like a good candidate for a proposal.
[1] https://phabricator.wikimedia.org/T165#2289766
> [2] https://secure.phabricator.com/T10691#167705
> [3] https://lists.wikimedia.org/pipermail/wikitech-l/2016-May/085489.html
>
--
Quim Gil
Engineering Community Manager @ Wikimedia Foundation
http://www.mediawiki.org/wiki/User:Qgil
So currently, we have two ways of outputting html - $wgWellFormedXml =
true (The default), outputs html that happens to conform with the
rules of XML. $wgWellFormedXml = false on the other hand, uses more
lax html5 rules to save a few bytes.
Having two modes of output, feels rather silly to me. Originally I
think this was meant as a feature flag well $wgWellFormedXml=false
stabilized, but it never got turned on, and here we are 7 years later.
Having $wgWellFormedXml=false increases the complexity of the code,
and not all that many people use it (Notable exception is
translatewiki). I think its important that security critical code be
as simple as possible. Furthermore, there seems to be very little
benefit to having the second mode (After you account for gzip, saving
a few bytes from writing <img> instead of <img/> really doesn't
matter, imo)
With that in mind, I would like to propose killing $wgWellFormedXml =
false; I'm not so much attached to the true mode (Although I do feel
the true mode is significantly more sane), as I just simply want there
to be a single mode. Putting the default to false was vetoed in
T52040, so I think that true would be the best choice to go with going
forward if we are getting rid of one of the modes.
If there are aspects of the other mode that people really want, then I
think we should simply merge that in to the default behavior instead
of having two separate modes.
See gerrit patch https://gerrit.wikimedia.org/r/286495 I would
appreciate everyone's feedback.
Thanks,
Brian
tl;dr: Our beloved scap is changing to use subcommands rather than a
bunch of scripts, but the existing scripts will work for a short time.
Starting with the 3.2.0 release[0], which will hit production in the
next day or so, scap will use subcommands rather than using many
different scripts that all call the same underlying code. The scripts
(e.g., deploy, sync-file, sync-dir, sync-wikiversions.) will continue
to work as usual, but they will issue a deprecation warning until the
next release when they will disappear.
The most notable exception is the `scap` command which must be invoked
as `scap sync [message]`.
The docs are updated[1] and you can see new help output there or on
phabricator[2].
Long story short, you will now run:
scap sync-file <path> [message]
Instead of:
sync-file <path> [message]
This change has been cherry-picked on beta cluster and is currently live there.
<3,
Tyler Cipriani and the Deployment Working Group
[0]. https://gerrit.wikimedia.org/r/#/c/287918
[1]. https://doc.wikimedia.org/mw-tools-scap/
[2]. https://phabricator.wikimedia.org/P3027
Hello Everyone,
Q1 Planning is coming soon. Staff and community are invited to add their
ideas and suggestions for the work to be planned from the upcoming July
till September. Please add your ideas here:
https://www.mediawiki.org/wiki/Reading/Quarterly_planning/FY2016-2017/Q1
It is worth mentioning that this quarter, is most likely to be packed with
pending tasks, so there might not be much room for new ideas, but that
doesn't mean that we can always add and discuss
Happy weekend!
Moushira
TL;DR:
----
* All access to Wikimedia production sites/APIs should use https://
URLs, not http:// -- your bot/tool will break in the near future if it
does not!
* 2016-06-12 - insecure access is unsupported; starting on this date
we plan to break (deny with 403) 10% of all insecure requests randomly
as a wake-up call.
* 2016-07-12 - we plan to break all insecure requests.
----
Hi all,
As you may remember, all production Wikimedia wikis switched to
HTTPS-only for all canonical domainnames nearly a year ago:
https://blog.wikimedia.org/2015/06/12/securing-wikimedia-sites-with-https/
Since way back then, we've been forcing insecure HTTP requests to our
canonical domains over to HTTPS by using redirects and
Strict-Transport-Security, which is effective for the vast majority of
access from humans using browsers and apps.
In the time since, we've been chasing down various corner-case issues
where loopholes may arise in our HTTPS standards and enforcement. One
of the most-difficult loopholes to close has been the "Insecure POST"
loophole, which is discussed in our ticket system here:
https://phabricator.wikimedia.org/T105794 .
To briefly recap the "Insecure POST" issue:
* Most of our humans using browser UAs are not affected by it. They
start out doing GET traffic to our sites, their GETs get redirected to
HTTPS if necessary, and then any POSTs issued by their browser use
protocol-relative URIs which are also HTTPS.
* However, many automated/code UAs (bots, tools, etc) access the APIs
using initial POST requests to hardcoded service URLs using HTTP
(rather than HTTPS).
* For all of the code/library UAs out there in the world, there is no
universally-compatible way to redirect them to HTTPS. There are
different ways that work for some UAs, but many UAs used for APIs
don't handle redirects at all.
* Regardless of the above, even if we could reliably redirect POST
requests, that doesn't fix the security problem like it does with GET.
The private data has already been leaked in the initial insecure
request before we have a chance to redirect it. If we did some kind
of redirect first, we'd still just be putting off the inevitable
future date where we have to go through a breaking transition to
secure the data.
Basically, we're left with no good way to upgrade these insecure
requests without breaking them. The only way it gets fixed is if all
of our API clients in the world use explicit https:// URLs for
Wikimedia sites in all of their code and configuration, and the only
way we can really force them to do so is to break insecure POST
requests by returning a 403 error to tools that don't.
Back in July 2015, I began making some efforts to statistically sample
the User-Agent fields of clients doing "Insecure POST" and tracking
down the most-prominent offenders. We were able to find and fix many
clients along the way since.
A few months ago Bryan Davis got us further when he committed a
MediaWiki core change to let our sites directly warn offending
clients. I believe that went live on Jan 29th of this year (
https://gerrit.wikimedia.org/r/#/c/266958 ). It allows insecure POSTs
to still succeed, but sends the clients a standard warning that says
"HTTP used when HTTPS was expected".
This actually broke some older clients that weren't prepared to handle
warnings at all, and caused several clients to upgrade. We've been
logging offending UAs and accounts which trigger the warning via
EventLogging since then, but after the initial impact the rate
flattened out again; clients and/or users that didn't notice the
warning fairly quickly likely never will.
Many of the remaining UAs we see in logs are simply un-updated. For
example, https://github.com/mwclient/mwclient switched to
HTTPS-by-default in 0.8.0, released in early January, but we're still
getting lots of insecure POST from older mwclient versions installed
out there in the world. Even in cases where the code is up to date
and supports HTTPS properly, bot/tool configurations may still have
hardcoded http:// site config URLs.
We're basically out of "soft" ways to finish up this part of the HTTPS
transition, and we've stalled long enough on this.
** 2016-06-12 is the selected support cutoff date **
After this date, insecure HTTP POST requests to our sites are
officially unsupported. This date is:
* A year to day after the public announcement that our sites are HTTPS only
* ~ 11 months after we began manually tracking down top offenders and
getting them fixed
* ~ 4 months after we began sending warning messages in the response
to all insecure POST requests to the MW APIs
* ~ 1 month after this email itself
On the support cutoff date, we’ll begin emitting a “403 Insecure POST
Forbidden - use HTTPS” failure for 10% of all insecure POST traffic
(randomly-selected). Some clients will retry around this, and
hopefully the intermittent errors will raise awareness more-strongly
than the API warning message and this email did.
A month later (two months out from this email) on 2016-07-12 we plan
to break insecure access completely (all insecure requests get the 403
response).
In the meantime, we'll be trying to track down offending bots/tools
from our logs and trying to contact owners who haven't seen these
announcements. Our Community team will be helping us communicate this
message more-directly to affected Bot accounts as well.
Thank you all for your help during this transition!
-- Brandon Black
Sr Operations Engineer
Wikimedia Foundation
We plan to add more RESTBase endpoints to support the new "Explore feeds"
feature in the apps. The currently proposed names are listed in [1]. It
introduces a new top-level hierarchy, called "project".
If you have issues with the current proposal or ideas to improve them
please comment on the Phab ticket by Thursday, May 19, 2016.
[1] https://phabricator.wikimedia.org/T132597
Thank you,
Bernd Sitzmann
Android app & Mobile Content Service
Thanks Roan & Brad! We'll get back on track with wmf.1 deployments today :D
-Chad
On Wed, May 11, 2016 at 11:08 PM, Roan Kattouw <rkattouw(a)wikimedia.org>
wrote:
> TLDR: the bug is fixed and the errors have stopped.
>
> I started working around this train hold by backporting the entire Echo
> extension from wmf1 to wmf23, assuming that the bug would be in MW core and
> updating Echo wouldn't affect it. Right after I deployed that, these errors
> started being thrown by wmf1 too.
>
> It turned out that one of the Echo changes I backported stores the integer
> -1 in redis under some circumstances. RedisBagOStuff treats integers
> specially, in order to make incr() work: it stores them as plain numbers
> instead of PHP-serialized data. But when retrieving this value, the code
> didn't recognize -1 as a plain number because it didn't consist solely of
> digits ('-' is not a digit), so it thought it was PHP-serialized data and
> passed it to unserialize(), which caused the error. Apparently no one had
> ever tried to store a negative integer in redis (!) until my Echo change
> exposed the bug.
>
> Brad did all the hard work, diagnosing this and writing up a fix on
> Phabricator. I turned that into a patch and deployed it about an hour ago.
> There haven't been any more errors since then.
>
> On Wed, May 11, 2016 at 1:59 PM, Chad Horohoe <chorohoe(a)wikimedia.org>
> wrote:
>
>> Hi,
>>
>> When we deployed the first 1.28 release to the cluster yesterday, we got
>> a new error[0] relating to
>> unserialization of redis data. It's pretty spammy already, so I'm
>> paranoid about deploying wider until
>> we figure out why. Deploying some debugging work soon so we can figure
>> out what's going on.
>>
>> If you've got any information you think would help, please chime in on
>> the bug.
>>
>> -Chad
>>
>> [0] https://phabricator.wikimedia.org/T134923
>>
>> _______________________________________________
>> Engineering mailing list
>> Engineering(a)lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/engineering
>>
>>
>