Hi everyone,
I recently set up a MediaWiki (http://server.bluewatersys.com/w90n740/)
and I need to extra the content from it and convert it into LaTeX
syntax for printed documentation. I have googled for a suitable OSS
solution but nothing was apparent.
I would prefer a script written in Python, but any recommendations
would be very welcome.
Do you know of anything suitable?
Kind Regards,
Hugo Vincent,
Bluewater Systems.
Dear all,
My apologies up front for the long e-mail that follows. In this e-mail you
will find a comprehensive status overview of the recent WebFonts deployment.
On Monday December 12 at 18:00 UTC we deployed the extension WebFonts[1] to
40 wikis in 11 Indic languages and Wikimedia Incubator -- all wikis in
Assamese, Bengali, Gujarati, Hindi, Kannada, Marathi, Nepali, Oriya,
(Eastern) Punjabi, Sankrit and Telugu have WebFonts now. WebFonts was not
deployed on Malayalam and Tamil projects. The reason for this was that
community members had requested us not to. We are confident that in time,
the communities will request that WebFonts is enabled on their projects.
WebFonts aims to resolve the issue that users see incomplete web pages,
because the fonts to properly render the page is not present in the local
system by downloading the font through the browser.
One of our great challenges developing this functionality is the multitude
of scripts and the low availability of freely licensed fonts that may be
modified and redistributed.
Over the past few months we have tried to build out a collection of fonts
in the extension mainly for Indic languages, and we have performed many
tests. We have solicited community involvement through messaging in village
pumps, e-mails on mailing lists, blog posts on personal blogs as well as on
the Wikimedia Foundation blog, at developer events, through personal
e-mails and through our bug tracker, and gotten some feedback, although
unfortunately not for all the languages we would like to have gotten it
for. We will of course continue our efforts in this area. Next to the
community involvement, we have had a two day session with the Red Hat
Localisation team in Pune, India.
Since the deployment, we have been criticised for not communicating enough
-- or not through the right channels, not with the right people, not in
time, or too soon, or not with the right messages. I'm not really sure how
to respond to that, except for uttering a general "mea culpa, mea maxima
culpa". We are working really hard in continuously improving the work that
we do, and the way that we do it. We make mistakes, we are human after all,
and when we become aware of our mistakes, we will do everything in our
power to make it better.
With our team we support the mission of the Wikimedia Foundation to
"imagine a world in which every single human being can freely share in the
sum of all knowledge." I care about that -- a lot. We all care, and I am
pretty certain that we're not ignorant, dismissive or incapable. I
acknowledge that we as the Localisation team are a relatively new entity
within the MediaWiki development community and within the Wikimedia
Foundation, with a very wide scope, and that we are dealing with a lot of
technical details on which we are simply not able to assess the final
quality; there are after all 7.500 languages in this world of over 7
billion people that we theoretically all cover, some 350 of those languages
are supported in MediaWiki, and 280 within Wikimedia.
I accept that we cannot keep everybody happy -- doesn't keep us from
trying, though. I want to try and work with as many people as possible in a
constructive way. With these numbers, that's not always easy to coordinate.
To channel the input on languages, we have set up "Language Support
Teams"[2]. We do not yet have a language support team for every language.
Please sign up if you care about the technical facilitation of your
language in the Wikimedia movement. Let's use the mediawiki-i18n mailing
list[3] to have constructive discussions about language support. Let's use
the #mediawiki-i18n IRC channel[4] on Freenode to have real-time
discussions. Let's use bugzilla.wikimedia.org to report bugs[5]. Link [5]
explains the bug reporting procedure. If you already know how, report
issues quickly using this link: http://ur1.ca/6ov9a .
Since the deployment, we have been made aware of about 17 issues. Some very
serious in nature, others not requiring immediate attention. Yesterday an
issue with web fonts not loading in Firefox was resolved in the
infrastructure. Today around 15:30 UTC, we have deployed fixes for an
additional hand full of issues[6]: functionality disabled in IE6, IE8 on
Windows XP, selection buttons not working properly in IE7 and hiding the
Samyak fonts in the font selector. During our current sprint, we are
working on a framework for multi-lingual and localised user documentation
as well as feature based feedback functionality for WebFonts, Narayam and
Translate. In the future we will also explore what is known as "dark
launch" by some, a kind of hidden live deployment of a feature, only usable
be for example manipulating a URL. This would allow us to deploy a feature
in a live environment, without having the "full deployment" impact.
Thanks for reading through this. I am looking forward to working with you!
Please read on for details on all the issues that were reported on WebFonts
recently.
Cheers!
Siebrand Mazeland
Product Manager Localisation
Wikimedia Foundation
=======================================
Links
=======================================
[1] https://www.mediawiki.org/wiki/Extension:WebFonts
[2] https://translatewiki.net/wiki/Language_support_team
[3] https://lists.wikimedia.org/mailman/listinfo/mediawiki-i18n
[4] https://translatewiki.net/wiki/Special:WebChat
[5] https://www.mediawiki.org/wiki/Bugzilla
[6] https://www.mediawiki.org/wiki/Special:Code/MediaWiki/106204
=======================================
Open issues
=======================================
https://bugzilla.wikimedia.org/33004 -- Old cached pages do not have web
fonts enabled
Priority: HIGH
--------------------------------------------------------------------------------------
Wikimedia is able to serve this many pages with relative few servers
because of very aggressive caching strategies, especially for anonymous
users. WebFonts requires the addition of JavaScript for anonymous users,
which is not being done for pages that are in the squid cache at the moment
WebFonts was enabled. All squid cache objects for wikis on which WebFonts
was deployed need to be purged. An internal RT ticket created for the
Wikimedia Operations team to get anonymous squid caches purged. This may
take up to a week or longer to be resolved.
https://bugzilla.wikimedia.org/33018 -- Firefox 5 on Windows XP has script
time-outs
Priority: MEDIUM
--------------------------------------------------------------------------------------
The Localisation team has tested this report, and was not yet able to
confirm the observation. The reason for using a non-recent version of
Firefox for the report was the alleged lower memory usage. Brion noted that
Mozilla has been actively working on lowering memory usage over the last
year, so the reporter may be better off with the current versions than the
old ones.
https://bugzilla.wikimedia.org/33110 -- Google Crome on Windows XP dispays
gibberish
Priority: LOW
--------------------------------------------------------------------------------------
Observed very rarely on a page on Wikimedia Incubator, and we have not been
able to reproduce this observation, let alone reproduce it reliably. A
screenshot is present in the bug report. Except for reporting upstream, no
action is being taken on this issue at this point in time.
https://bugzilla.wikimedia.org/33054 -- Hinting issues in Lohit fonts
Priority: MEDIUM
--------------------------------------------------------------------------------------
Confirmed in Windows XP. We can do something to the font by adding hinting,
but this is a lot of work if it needs to be done manually. The stem of the
Lohit glyphs could do with more width and darkness. This may not be
desirable for platforms (Linux) which render it perfectly, because it
already has hinting and anti-aliasing on an operating system level. Same
goes got Windows 7.
https://bugzilla.wikimedia.org/33100 -- Page crashes on Webkit browsers
with WebFonts enabled.
Priority: MEDIUM (could be HIGH if we find many occurances)
--------------------------------------------------------------------------------------
A page in Nepali Wikipedia makes a tab on Mac OS X 10.7.2 with Google Crome
crash. This behaviour was also reported for Mac OS X 10.7.2 (11C74) with
Safari 5.1.1 (7534.51.22, r102522) [This is a webkit nightly build] by
thedj. This is most probably related to the WebFonts code, because if, as a
logged in user, web fonts is disabled in preferences, the page does not
crash Chrome.
Developer Derk-Jan Hartman was asked to report this bug in the WebKit.
Please make us aware of any additional pages that would cause this
behaviour in any wiki.
https://bugzilla.wikimedia.org/33102 -- OSX 10.7.2/Opera 11.60 has no
fallback for Latin characters
Priority: MEDIUM
--------------------------------------------------------------------------------------
This is a bug that needs to be reported upstream. No technical measures
have been taken so far to mitigate this issue. One of the Localisation team
members has been in contact with a high level executive of Opera, and will
contact that person again. We're going to wait for a few days for an
outcome -- if there is no expectation of a relatively quick fix, we might
disable WebFonts for Opera completely. Opera unfortunately does not have a
public bug tracker.
https://bugzilla.wikimedia.org/33027 -- Narayam and WebFonts both loading
slows down page
Priority: MEDIUM
--------------------------------------------------------------------------------------
The reporter claims that the functionality is quicker on
translatewiki.netthan it is in Wikimedia wikis. A commenter states
that more functionality
usually means more code, means more data that needs to be transferred, and
without changing bandwidth, that causes longer load times.
This currently isn't our highest priority, but eventually we will look into
this a little deeper. We're inviting volunteers to do some of the data
gathering and analysis for us. What is needed in our opinion is insight in
the data volume added by WebFonts, as well as an assessment of the code
quality with regards to size optimisation. All referenced properly, of
course :). There are alternate EOT conversion tools that have a good
compression ratio. Needs to be explored, but EOT is not required for modern
browsers since they started using WOFF fonts which are compressed OpenType
fonts.
https://bugzilla.wikimedia.org/33085 -- Integration of updated Lohit-Tamil
Font
Priority: MEDIUM
--------------------------------------------------------------------------------------
Request to update WebFonts with a font that is updated upstream. This is
something the Localisation team checks regularly. Will probably be closed
this week, pending issues the have a higher priority.
https://bugzilla.wikimedia.org/32942 -- Provide help page and bug report
link for WebFonts
Priority: HIGH
--------------------------------------------------------------------------------------
More recently developed tools by the Wikimedia Foundation have often
included feedback mechanisms. The Localisation team plans on implementing
these for the functionality of the WebFonts, Narayam and Translate
extension. Besides that, we also want to provide multi-lingual and
localised documentation. This needs some thinking and some work to provide
in a structured and navigable way. We'll keep you posted. It will most
probably involve translatable *user* documentation on MediaWiki.org and
hopefully it is possible to have one feedback location per feature across
the multiple Wikimedia wikis -- this is something we're going to contact
the ArticleFeedback and MoodBar teams for.
=======================================
Closed issues
=======================================
https://bugzilla.wikimedia.org/33025 -- When changing to a non-default web
font, the content does not
--------------------------------------------------------------------------------------
This issue was a side effect of a feature to allow multiple web fonts to be
used using the "lang" attribute. It was resolved in
https://www.mediawiki.org/wiki/Special:Code/MediaWiki/105980 and has been
deployed.
https://bugzilla.wikimedia.org/33034 -- Web fonts not loading in Firefox
--------------------------------------------------------------------------------------
Duplicate reports were 33038 and 33044. This issue originated from
http://www.w3.org/TR/css3-fonts/#same-origin-restriction. Almost all
browsers except for Firefox ignore that specification. A fix was designed
and deployed: https://www.mediawiki.org/wiki/Special:Code/MediaWiki/106092,
https://gerrit.wikimedia.org/r/1501. Thanks to Roan, Brion and Ryan for
their help.
https://bugzilla.wikimedia.org/32775 -- Gibberish in Internet Explorer 8 on
Windows XP
--------------------------------------------------------------------------------------
This is an unexplained phenomenon only observed in Internet Explorer on
Windows XP. It is also hard to reproduce. One of the developers was able to
make something somewhat reproducible on a clean, fully patched installation
of Windows XP with Internet Explorer 8. See bug report for details.
Based on these observations we think it is a bad idea to keep supporting
WebFonts in Internet Explorer 8 on Windows XP and we have disabled it in
https://www.mediawiki.org/wiki/Special:Code/MediaWiki/106172. This fix has
been deployed.
https://bugzilla.wikimedia.org/33096 -- Internet Explorer 6 does not have
font fallback
--------------------------------------------------------------------------------------
IE6 not having font fallback causes Latin characters to display as squares
when a web font is loaded that does not contain glyphs for the Latin
script. A screenshot is available at
http://media.crossbrowsertesting.com/users/34057/screenshots/window/z669002….
Based on this observation, we think it is a bad idea to keep supporting
WebFonts in Internet Explorer 6 and we have disabled it in
https://www.mediawiki.org/wiki/Special:Code/MediaWiki/106172. This fix has
been deployed.
https://bugzilla.wikimedia.org/33024 -- WebFonts menu buttons not working
in IE7
--------------------------------------------------------------------------------------
This was caused by the JavaScript $( '<input type="radio" />' ) . attr(
"name" ,"font"); not working in IE6 and IE7. Updating name attributes once
they have been created is not possible. We think there may be more
occurances of this in our code (one occurance in jQuery has already been
identified: resources/jquery/jquery.validate.js:59). A fix was made in
https://www.mediawiki.org/wiki/Special:Code/MediaWiki/106175. This fix has
been deployed.
https://bugzilla.wikimedia.org/33040 -- Overlap in Samyak font for Hindi
and Sanskrit
--------------------------------------------------------------------------------------
This issue occurs in Windows XP and Windows 7 (possibly also in Windows
Vista) when using Google Chrome. It is not observed when using Chrome with
Mac OS X 10.7.2 or several Linux distributions (Debian and Fedora). Samyak
Devanagari is available as a non-default web font in Hindi, Marathi, and
Sanskrit. Samyak Gujarati is available for Gujarati as a non-default font.
This font needs to be corrected. The maintainers will be notified of the
observed issues, and mean while, the fonts will be removed from the
WebFonts selection list (but can still be used using the font-family
property. A fix was made in
https://www.mediawiki.org/wiki/Special:Code/MediaWiki/106179. This fix has
been deployed.
https://bugzilla.wikimedia.org/33039 -- Overlap in Madan font for Nepali
--------------------------------------------------------------------------------------
This report was invalid. The reporter was not aware of the correct glyph
for the Nepali script.
Comments on this bug report resulted in two odd observations (Crome crash,
Opera font fallback), that have been split off into separate bug reports:
https://bugzilla.wikimedia.org/33100 and
https://bugzilla.wikimedia.org/33102.
https://bugzilla.wikimedia.org/33095 -- WebFonts menu can expand off the
screen
--------------------------------------------------------------------------------------
If the translations for "Select font" and "Login / Register" are really
short, like in http://mr.wiktionary.org, expanding the WebFonts menu for
anonymous users will display a menu that is partially off the screen. It
was resolved in http://www.mediawiki.org/wiki/Special:Code/MediaWiki/106186,
http://www.mediawiki.org/wiki/Special:Code/MediaWiki/106197,
http://www.mediawiki.org/wiki/Special:Code/MediaWiki/106201,
http://www.mediawiki.org/wiki/Special:Code/MediaWiki/106202. These
revisions also depend on a few small UI changes of both WebFonts and
Narayam, and will be deployed on December 19, 2011.
<no bugzilla report> -- WebFonts menu expands under the control for
customised input method in IE6 on transliteration
--------------------------------------------------------------------------------------
There are issues with the z index in IE6. Because of
https://www.mediawiki.org/wiki/Special:Code/MediaWiki/106172, WebFonts is
no longer available in IE6, so this issue is obsolete. Observing that the
Hindi projects Wikipedia and Wiktionary are using an custom input methods
tool, we would like to invite them to test Narayam which contains many
input methods in a MediaWiki extension. We are very open to having the
Hindi input method InScript tested and add a transliteration input method
with some community representatives, as we have done with other Indic
languages. We hope this will eventually lead to Narayam being adopted by
the Hindi community, and the custom input method being abandoned.
A Wikipedia page loads. The last thing to load is the banner. This
pushes the page content down. If you've clicked on a link near the top
of the page, the banner grabs it instead.
This happened last year and it was reported then (and it was
incredibly annoying then too). It also fouls up stats on banner
effectiveness, as banners are clicked on without intention to do so.
Please load the page with a space for the banner to avoid this effect.
- d.
WikiProject Extensions is presenting our first ever "Extension Page Review Drive" - http://www.mediawiki.org/wiki/Project:WikiProject_Extensions/Projects/Page_…
Several extension pages are long overdue for review. Many are lacking the appropriate tags, both header and within the extensions, which cause confusion for other developers and sysadmins. In addition to time wasted on bad installations, there is also time wasted by other developers providing tech support to these bad installations. This will also help with future drives reviewing actual code of extensions (although you're welcome to do thorough code review during this drive if you'd like) and moving extensions based in wikicode to the code repository.
The goal is to review as many of the pages as possible during the 1st quarter of this calendar year (so by March 31st). We've just officially started this and already 1% done. :)
During this drive, all extension pages are marked with an additional category. These will be removed once the drive is completed. Page drive specific template modifications and wikicode will also be removed upon the page drive's completion.
Any interested participants are welcome to sign on as a participant on: http://www.mediawiki.org/wiki/Project:WikiProject_Extensions/Projects/Page_…
Feel free to email me with any questions or feedback. :)
-greg aka varnent
-------
Gregory Varnum
Lead, Aequalitas Project
Lead Administrator, WikiQueer
Founding Principal, VarnEnt
@GregVarnum
fb.com/GregVarnum
Hi,
with regard to bug #24607, I wanted to enable external Stor-
age on my local PostgreSQL-backed wiki. The documentation
at
<URI:http://www.mediawiki.org/wiki/Manual:$wgExternalServers>
reads:
| An array of external MySQL servers.
| e.g.
| $wgExternalServers = array( 'cluster1' => array( 'srv28', 'srv29', 'srv30' ) );
but executing maintenance/storage/compressOld.php with that
configuration yields:
| [...]
| PHP Warning: Invalid argument supplied for foreach() in /var/www/html/w/includes/db/LoadBalancer.php on line 68
| PHP Warning: Invalid argument supplied for foreach() in /var/www/html/w/includes/db/LoadBalancer.php on line 68
| You must update your load-balancing configuration. See DefaultSettings.php entry for $wgDBservers.
| Backtrace:
| #0 /var/www/html/w/includes/db/LoadBalancer.php(571): LoadBalancer->reallyOpenConnection('0rv28', false)
^
| #1 /var/www/html/w/includes/db/LoadBalancer.php(492): LoadBalancer->openConnection(0, false)
| #2 /var/www/html/w/includes/ExternalStoreDB.php(56): LoadBalancer->getConnection(-2, Array, false)
| #3 /var/www/html/w/includes/ExternalStoreDB.php(150): ExternalStoreDB->getMaster('cluster1')
| #4 /var/www/html/w/maintenance/storage/compressOld.php(347): ExternalStoreDB->store('cluster1', 'O:27:"Concatena...')
| #5 /var/www/html/w/maintenance/storage/compressOld.php(94): CompressOld->compressWithConcat(0, 20, '', '', 'cluster1', false)
| #6 /var/www/html/w/maintenance/doMaintenance.php(105): CompressOld->execute()
| #7 /var/www/html/w/maintenance/storage/compressOld.php(408): require_once('/var/www/html/w...')
| #8 {main}
and a look at the code and wmf-deployment's addwiki.php and
renamewiki.php seems to indicate that the structure is a bit
more complex. Could someone update the documentation,
please?
TIA,
Tim
Hi folks,
As you may know, WMF's Platform Engineering group plans to embark on a
major performance initiative this year, and had chosen inline
scripting as having the biggest potential impact given what's
practical now. Tim Starling build a Lua prototype last year which
showed a lot of promise for making things much faster. One major
decision before embarking on this effort was a decision on whether
we'd stick with Lua or try another language such as Javascript or
Victor's WikiScript implmentation. I wanted to make a decision by the
end of the month[1], and I think we've done it.
We've decided to build a deployable version of Lua as a new
alternative to wiki markup for templates, barring some scandalous
revelation about Lua's lurid past or other unforeseen barrier. Tim
will be leading this effort, and will start on the implementation some
time after the dust settles on the 1.19 deployment and the Git
migration. The project page for this is located here:
http://www.mediawiki.org/wiki/Lua_scripting
Rough notes from our meeting yesterday are also available [2]
Rob
[1] http://thread.gmane.org/gmane.science.linguistics.wikipedia.technical/57769…
[2] http://www.mediawiki.org/wiki/Lua_scripting/Meeting_2012-01-25
Hi,
Since we have more and more users with access to wikimedia labs, I think it
would be great to make online conference on irc to discuss, especially with
new users and people interested in getting access there how labs work, and
what all is needed to do. I would be interested in getting a feedback on
what is needed to set up right now. There is a bot project which is slowly
moving forward so I would appreciate some feedback from people who run bots
so that we know what all needs to get implemented before opening
"production" for this part of labs.
Also it would be great to discuss about stuff we are about to set up, write
some new proposals and comment on what we have done. Including:
* Shared SQL server
* Monitoring for services - ganglia and nagios
* Bots cluster
* Deployment cluster
So, if there are people interested in labs, please let me know and we could
make a conference in #wikimedia-labs on freenode - I propose it to happen
this week, on thursday (friday is a day when most of people are leaving
office or heading somewhere far from computers :)) or saturday. (But I
don't know if Ryan and other important people are available for talk on
saturday). Preferably in 18:00 GMT or 22:00 GMT (I guess 22:00 is a best
time for people who live either in US or Europe).
Let me know what you think or if you would prefer a better date
Hi all,
As some of you are probably aware, we've got a test repository converting
phase3 to git up and running on gerrit. You should be able to clone by
`git clone https://gerrit.wikimedia.org/r/p/test/mediawiki/core.git`
Couple of caveats (things I'm gonna try and fix):
* Permissions aren't sorted yet, so it's only supporting anonymous clones,
no pushing yet.
* The revision graph is crazy. svn:mergeinfo is unreliable and we're pretty
much unable to build a cohesive history without a *lot* of manual labor. Right
now I'm thinking of just dropping the mergeinfo so the branches look like linear
graphs cherry picking from master. Not perfect, but less annoying than now.
But yay progress! Clone it. Try it out, see what works (and what doesn't).
I'll try to get the permissions sorted later today so we can go ahead and
try some test pushes (and merges).
Things we still need to do:
* Make a git-setup like we've done for the other git repos. This will setup your
environment/hooks/etc for you.
* Figure out our WMF branching/deployment strategy, since we're very SVN-
centric right now with this.
Thanks for any input, guys.
-Chad
This is a question about an infrastructural detail of ResourceLoader and how it interacts with Internet Explorer. (It's my first post to wikitech-l, so apologies if it's the wrong forum.)
Our MediaWiki 1.17.0 site recently installed a bunch of extensions that use ResourceLoader, such as Extension:WikiEditor. To our surprise, some of our site's unrelated CSS styles stopped working. This was happening only in Internet Explorer. After some detective work, we discovered the problem is Internet Explorer's limit of 31 stylesheets:
http://support.microsoft.com/kb/262161
With so many extensions calling $wgOut->addModule [PHP} and mw.loader.load [JavaScript], the limit of 31 stylesheets is exceeded quickly. I removed a few mw.loader.load calls - it didn't matter which ones - and the problem went away.
Obviously this is an IE problem, not MediaWiki's, but it's going to cause issues on MediaWiki sites. WikiEditor itself loads about 10 stylesheets, for example, taking the site ~30% of the way toward a CSS failure.
So my questions are:
1. Is there a workaround for sites like mine, with many stylesheets from separate extensions loaded by ResourceLoader?
2. Should ResourceLoader address this IE problem? Maybe start combining stylesheets (with @import) automatically?
Thanks,
DanB
Earlier this week
(http://lists.wikimedia.org/pipermail/wikitech-l/2012-January/057638.html),
I wrote about the upcoming 1.19 release and the changes that need to be
made on-wiki to provide a consistent experience.
I've been thinking about this and getting input from on-wiki
administrators, but I'm interested in your thoughts,
I have talked to Krinkle and I realize his concerns -- javascript
dependencies really should be enumerated. Still, I'm concerned about
the experience for users who have installed several gadgets (which we
can test in Krinkle's "Tour de Wiki") as well as possible UserScripts
(which we really can't test, at least not at as easily or as quickly).
I adjusted the [[MediaWiki:Gadgets-definition]] on enwiki.beta and let
the people on enwiki know what I had found, but I think this sort of
adjustment will be needed in more places. For proof, just look at
http://de.wikipedia.beta.wmflabs.org/wiki/Wikipedia:Hauptseite?debug=true
in FireBug.
You'll see (or, at least, I see) two un-fulfilled dependencies on
mw.util.
Some sort of dependency needs to be added on mw.util -- either just
preload it or make it log a message when there is an unenumerated
dependency on it (and other necessary js dependencies).
This, plus a message on WP:VPT or the like, would be a way for users and
gadget authors to update their javascript. It be a great way to notify
users of the deprecation from 1.18 to 1.20 (or 1.21) without providing a
horribly shocking experience after we upgrade the cluster.
Let's ensure that the Wiki* experience is consistent. We can avoid some
of the mistakes like those that happened when we introduced
ResourceLoader. Backwards compatibility is important. If we upgrade
MediaWiki and we know that people are going to complain because a
widespread dependency (like mw.util) disappeared, let's eliminate that
"experience gap".
Mark.
