-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
hello,
i have written a new extension to embed music scores in MediaWiki pages: https://secure.wikimedia.org/wikipedia/mediawiki/wiki/Extension:ABC
unlike the Lilypond extension, this uses a simple input language (ABC) that is much easier to validate for security. ABC is mostly used to transcribe Irish trad and other simple tunes, but it recently gained support for more advanced features, e.g. multiple staves and lyrics. this is supported in the extension using the 'abcm2ps' tool.
unlike the existing ABC extension (AbcMusic), it doesn't support opening arbitrary files as ABC input (which is a potential security issue), and has several additional features:
- - The original ABC can be downloaded easily - - The score can be downloaded as PDF, PostScript, MIDI or Ogg Vorbis - - A media player can be embedded in the page to play the media file
i believe the ABC format is suitable for transcribing the majority of scores currently on Wikimedia projects. although it can't handle all of them, it is better than the current situation. plus, as ABC is simple, and existing ABC scores are easily available, it's easier for novice users to contribute.
i would be interested to hear peoples' thoughts on enabling this extension on Wikimedia.
- river.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
River Tarnell wrote:
unlike the existing ABC extension (AbcMusic), it doesn't support opening arbitrary files as ABC input (which is a potential security issue), and has several additional features:
- The original ABC can be downloaded easily
- The score can be downloaded as PDF, PostScript, MIDI or Ogg Vorbis
- A media player can be embedded in the page to play the media file
Oooooh! Cool!
A few issues to consider, based on some of our experience with media files:
* Can we adapt it to a storage backend which doesn't use a shared filesystem? NFS servers are problematic, and we're planning to remove them from the equation for uploaded media. Output generated during parsing should similarly be able to be stored in a way that fails gracefully when servers are down, or supports better failover.
* Garbage collection for unused output files? Previews or changed pages could leave the system littered with millions of output files which will never be used again.
It'd be nice to solve these for math and timelines as well -- perhaps a unified system?
* Rendering multiple output formats on parse could potentially be quite slow; if there are resource-usage problems, it would be nice to isolate them like we currently do for image thumbnails (rendered on-demand on a sub-cluster).
- -- brion
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Brion Vibber:
- Can we adapt it to a storage backend which doesn't use a shared
filesystem?
yes, but i think it would be very silly to write such a thing for a single extension. is the FileRepo interface generic enough to be used for something like this? (while that would still use NFS for now, it could easily be changed to HTTP or whatever else we decide to use in the future.)
- Garbage collection for unused output files? Previews or changed pages
could leave the system littered with millions of output files which will never be used again.
It'd be nice to solve these for math and timelines as well -- perhaps a unified system?
i'm not sure off hand how this is doable. (and it's another thing that probably shouldn't be done inside a single extension.) but, if a mechanism were to exist to do it, it should be simple to add to the extension.
- Rendering multiple output formats on parse could potentially be quite
slow; if there are resource-usage problems, it would be nice to isolate them like we currently do for image thumbnails (rendered on-demand on a sub-cluster).
i suppose this would need something like thumb-handler.php. i will see about adding it.
- river.
On Tue, Oct 28, 2008 at 2:25 PM, River Tarnell < river@loreley.flyingparchment.org.uk> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Brion Vibber:
- Can we adapt it to a storage backend which doesn't use a shared
filesystem?
yes, but i think it would be very silly to write such a thing for a single extension. is the FileRepo interface generic enough to be used for something like this? (while that would still use NFS for now, it could easily be changed to HTTP or whatever else we decide to use in the future.)
No, it's not. Might be worth looking into...as Brion said, providing a core interface for extensions to generate images on-demand with standardized backend support would be great.
FSRepo might be of some use and might start us in the right direction.
-Chad
On Tue, Oct 28, 2008 at 2:01 PM, Brion Vibber brion@wikimedia.org wrote:
- Can we adapt it to a storage backend which doesn't use a shared
filesystem? NFS servers are problematic, and we're planning to remove them from the equation for uploaded media.
Out of curiosity, what's problematic about them, and what are the better alternatives we're moving to?
On Tue, Oct 28, 2008 at 6:31 PM, Aryeh Gregor Simetrical+wikilist@gmail.com wrote:
On Tue, Oct 28, 2008 at 2:01 PM, Brion Vibber brion@wikimedia.org wrote:
- Can we adapt it to a storage backend which doesn't use a shared
filesystem? NFS servers are problematic, and we're planning to remove them from the equation for uploaded media.
Out of curiosity, what's problematic about them, and what are the better alternatives we're moving to?
Because if the server goes away you get stuck unkillable processes.
If you have cyclic NFS dependencies between systems it becomes impossible to cleanly boot the environment.
NFS is great in the steady state, but during failure it tends to trigger bad and unexpected behaviour in everything around it.
Gregory Maxwell wrote:
Because if the server goes away you get stuck unkillable processes.
NFS is great in the steady state, but during failure it tends to trigger bad and unexpected behaviour in everything around it.
Thinking about it, the problem is not the use of nfs, but that nfs driver performs quite badly on server unreachable conditions. How much would be needed to make them better? Nfs seems to already have several mount options for improving its behavior:
soft Soft-mount the driver. This option causes an I/O error to be reported to the process attempting a file operation when a major timeout occurs timeo=n This sets the time (in tenths of a second) the NFS client will wait for a request to complete. intr Allow signals to interrupt an NFS call. Useful for aborting when the server doesn't respond.
Out of curiosity, what's problematic about them, and what are the better alternatives we're moving to?
Sometimes NFS decides to use a port that is being intercepted by IPMI controller, thats how servers with 900-day uptime die. :-)
Brion Vibber wrote:
- Garbage collection for unused output files? Previews or changed pages
could leave the system littered with millions of output files which will never be used again.
It'd be nice to solve these for math and timelines as well -- perhaps a unified system?
I was thinking about this problem, but for Graphviz (which is based on Timeline, which was the source of the problem).
Instead of opening a tag and dumping graph/timeline code in the article, we have a namespace where articles are parsed as graph/timeline code (some extra syntax needed in order to put some wikitext in the same article). Then, in article pages, you insert a link like [[Graph:Graph title|options|description]] or [[Timeline:Timeline title|...]], and they are inserted just like images are.
This has a few advantages: - Graphs or timelines (or music scores) that appear in multiple pages are maintained in a single central article, changes in it propagates to all pages that include that object, just like it is with images; - It becomes easier to edit article pages, because you have only wikitext to look at, and not a mix of wikitext, dot syntax, timeline syntax, etc... - Perfect garbage collection: we produce no garbage. In-disk files are linked to special article pages; if someone deletes the article, MW just wipes all files with that given hash and you are done. Any file without a matching article in that special namespace is easily marked as garbage.
On Wed, Oct 29, 2008 at 7:42 PM, Juliano F. Ravasi ml@juliano.info wrote:
Instead of opening a tag and dumping graph/timeline code in the article, we have a namespace where articles are parsed as graph/timeline code (some extra syntax needed in order to put some wikitext in the same article). Then, in article pages, you insert a link like [[Graph:Graph title|options|description]] or [[Timeline:Timeline title|...]], and they are inserted just like images are.
This has a few advantages:
- Graphs or timelines (or music scores) that appear in multiple pages
are maintained in a single central article, changes in it propagates to all pages that include that object, just like it is with images;
- It becomes easier to edit article pages, because you have only
wikitext to look at, and not a mix of wikitext, dot syntax, timeline syntax, etc...
- Perfect garbage collection: we produce no garbage. In-disk files are
linked to special article pages; if someone deletes the article, MW just wipes all files with that given hash and you are done. Any file without a matching article in that special namespace is easily marked as garbage.
You could also apply this to SVG graphics to allow on-line editing of them. But everything we discuss here has as basic requirement a common image storage mechanism, which should be implemented before anything else.
Marco
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Juliano F. Ravasi wrote:
Instead of opening a tag and dumping graph/timeline code in the article, we have a namespace where articles are parsed as graph/timeline code (some extra syntax needed in order to put some wikitext in the same article). Then, in article pages, you insert a link like [[Graph:Graph title|options|description]] or [[Timeline:Timeline title|...]], and they are inserted just like images are.
[snip]
- Perfect garbage collection: we produce no garbage. In-disk files are
linked to special article pages; if someone deletes the article, MW just wipes all files with that given hash and you are done. Any file without a matching article in that special namespace is easily marked as garbage.
Previews require generation in some way...
- -- brion
On Wednesday 29 October 2008 19:42:35 Juliano F. Ravasi wrote:
Instead of opening a tag and dumping graph/timeline code in the article, we have a namespace where articles are parsed as graph/timeline code (some extra syntax needed in order to put some wikitext in the same article). Then, in article pages, you insert a link like [[Graph:Graph title|options|description]] or [[Timeline:Timeline title|...]], and they are inserted just like images are.
Yes, my suggestion was [[Image:Blahblah.wiki]]. However, this isn't applicable to all cases, and formulas and music scores are probably best left as they are now.
On Thu, Oct 30, 2008 at 5:42 AM, Juliano F. Ravasi ml@juliano.info wrote:
Brion Vibber wrote:
- Garbage collection for unused output files? Previews or changed pages
could leave the system littered with millions of output files which will never be used again.
It'd be nice to solve these for math and timelines as well -- perhaps a unified system?
I was thinking about this problem, but for Graphviz (which is based on Timeline, which was the source of the problem).
Instead of opening a tag and dumping graph/timeline code in the article, we have a namespace where articles are parsed as graph/timeline code (some extra syntax needed in order to put some wikitext in the same article). Then, in article pages, you insert a link like [[Graph:Graph title|options|description]] or [[Timeline:Timeline title|...]], and they are inserted just like images are.
This has a few advantages:
- Graphs or timelines (or music scores) that appear in multiple pages
are maintained in a single central article, changes in it propagates to all pages that include that object, just like it is with images;
- It becomes easier to edit article pages, because you have only
wikitext to look at, and not a mix of wikitext, dot syntax, timeline syntax, etc...
- Perfect garbage collection: we produce no garbage. In-disk files are
linked to special article pages; if someone deletes the article, MW just wipes all files with that given hash and you are done. Any file without a matching article in that special namespace is easily marked as garbage.
On Wikisource, we would prefer to transcribe the music (and other items) in place in the "Page" namespace. We have started to categorise pages which will need to use this extension when it lands:
http://en.wikisource.org/wiki/Category:Pages_containing_sheet_music
-- John Vandenberg
On Tue, Oct 28, 2008 at 8:18 AM, River Tarnell river@loreley.flyingparchment.org.uk wrote: [snip]
ABC is mostly used to transcribe Irish trad and other simple tunes, but it recently gained support for more advanced features, e.g. multiple staves and lyrics.
I withdraw my prior protest about ABC: the more recent enhancements make it sufficient for Wikipedia use. (It's still short when it comes to doing full works, but most of the gaps aren't feature's we'd likely leave working anyways)
On Tuesday 28 October 2008 13:18:06 River Tarnell wrote:
i have written a new extension to embed music scores in MediaWiki pages: https://secure.wikimedia.org/wikipedia/mediawiki/wiki/Extension:ABC
unlike the Lilypond extension, this uses a simple input language (ABC) that is much easier to validate for security. ABC is mostly used to transcribe Irish trad and other simple tunes, but it recently gained support for more advanced features, e.g. multiple staves and lyrics. this is supported in the extension using the 'abcm2ps' tool.
unlike the existing ABC extension (AbcMusic), it doesn't support opening arbitrary files as ABC input (which is a potential security issue), and has several additional features:
- The original ABC can be downloaded easily
- The score can be downloaded as PDF, PostScript, MIDI or Ogg Vorbis
- A media player can be embedded in the page to play the media file
i believe the ABC format is suitable for transcribing the majority of scores currently on Wikimedia projects. although it can't handle all of them, it is better than the current situation. plus, as ABC is simple, and existing ABC scores are easily available, it's easier for novice users to contribute.
i would be interested to hear peoples' thoughts on enabling this extension on Wikimedia.
One caveat only: in my tests, abc2midi didn't support the same ABC format abcm2ps did. I don't think it is a big problem, and if it turns out to be, it could be solved simply by adding an option to disable sound rendering.
i have written a new extension to embed music scores in MediaWiki pages: https://secure.wikimedia.org/wikipedia/mediawiki/wiki/Extension:ABC
Is there a publicly accessible server where we can play with this extension?
I'm curious how the two formats compare. This: X: 1 T: Tommy Peoples' M: 4/4 L: 1/8 R: reel K: Bmin
doesn't look too intuitive to me. R? L? X? But maybe the learning curve is short.
Also, what's the status of the lilypond extension on Wikipedia? It doesn't seem to be enabled/present atm, but are there plans for it to be?
Steve
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Steve Bennett wrote:
Also, what's the status of the lilypond extension on Wikipedia? It doesn't seem to be enabled/present atm, but are there plans for it to be?
No.
- -- brion
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Steve Bennett:
Is there a publicly accessible server where we can play with this extension?
not currently.
I'm curious how the two formats compare. This: X: 1 T: Tommy Peoples' M: 4/4 L: 1/8 R: reel K: Bmin
doesn't look too intuitive to me. R? L? X? But maybe the learning curve is short.
well, if i were designing my own format, perhaps i wouldn't use single letter keys to embed metadata. still, after you've used it even a little bit, it becomes fairly easy to remember. (and for reading, most of the fields are obvious: R is 'rhythm', K is 'key', etc.)
i've added a link to [0] on the extension page; this is a pretty decent introduction to ABC.
Also, what's the status of the lilypond extension on Wikipedia? It doesn't seem to be enabled/present atm, but are there plans for it to be?
as far as i know, it's unlikely to ever be enabled unless it's significantly reworked, and in the last few years, no one has offered to do that. that's the only reason i wrote this extension; if enabling Lilypond was a serious possibility, i probably wouldn't have bothered.
- river.
[0] http://www.lesession.co.uk/abc/abc_notation.htm
River Tarnell wrote:
doesn't look too intuitive to me. R? L? X? But maybe the learning curve is short.
well, if i were designing my own format, perhaps i wouldn't use single letter keys to embed metadata. still, after you've used it even a little bit, it becomes fairly easy to remember. (and for reading, most of the fields are obvious: R is 'rhythm', K is 'key', etc.)
Remember that it will be used by people from many languages. The abbreviation one language may not make sense on another. They may even not be using letters for naming the notes!
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Platonides:
Remember that it will be used by people from many languages. The abbreviation one language may not make sense on another. They may even not be using letters for naming the notes!
they may not even be using the same notes as Western music, or Western music notation. i'm not particularly interested in creating a notation that can be used to notate every form of music that's ever existed in the world. but since a format already exists that can be used to notate a large repertory, i thought it would be useful to make that format available on Wikipedia.
if you want to change or add to the ABC notation, i suggest taking that up with the people creating the standard ;-)
- river.
On Sun, Nov 9, 2008 at 5:25 AM, River Tarnell river@loreley.flyingparchment.org.uk wrote:
You know what would be useful? A website that lets you input ABC (or LilyPond, for that matter) text, and produces an image as output. Hence avoiding the need to download and install it. Does such a thing exist?
Steve
On Sun, May 24, 2009 at 11:29 PM, Steve Bennett stevagewp@gmail.com wrote:
You know what would be useful? A website that lets you input ABC (or LilyPond, for that matter) text, and produces an image as output. Hence avoiding the need to download and install it. Does such a thing exist?
http://www.concertina.net/tunes_convert.html does a low-res JPEG version; http://www.abc-notation.com/abcapp/list.html?phrase=&system=Online&f... lists others.
2008/10/28 River Tarnell river@loreley.flyingparchment.org.uk:
i have written a new extension to embed music scores in MediaWiki pages: https://secure.wikimedia.org/wikipedia/mediawiki/wiki/Extension:ABC i would be interested to hear peoples' thoughts on enabling this extension on Wikimedia.
If/when this is enabled, please write a post for blog.wikimedia.org announcing it to the world.
- d.
2008/10/28 River Tarnell river@loreley.flyingparchment.org.uk:
i have written a new extension to embed music scores in MediaWiki pages: https://secure.wikimedia.org/wikipedia/mediawiki/wiki/Extension:ABC i would be interested to hear peoples' thoughts on enabling this extension on Wikimedia.
Is there any progress with this extension? There are some interested users in Hungarian Wikisource.
D.
On Fri, May 22, 2009 at 8:50 AM, Tar Dániel bdanee88@gmail.com wrote:
Is there any progress with this extension? There are some interested users in Hungarian Wikisource.
Tim has said he might enable it after the 1.15 release and the next scap.
wikitech-l@lists.wikimedia.org