On Thu, Oct 15, 2009 at 10:16 PM, Gregory Maxwell <gmaxwell(a)gmail.com> wrote: I have the uncompressed history here at 550mb. du -h history/ 556M history/ if I bzip this, it is 29M 2009-10-15 22:35 total.tar.bz 14 mb is still smaller, and the upload is faster! There are ways to optimize all of this. Most users will not want to download the full history. This is just one days of work using git, we will be able to optimize this all. I will be able to find other example of large repositories.. http://laserjock.wordpress.com/2008/05/09/bzr-git-and-hg-performance-on-the… mike

On Thu, Oct 15, 2009 at 11:33 PM, Gregory Maxwell <gmaxwell(a)gmail.com> wrote: I am not most users. I am using git because I think it is the best way forward to implement many of the ideas discussed in the strategy wiki.

It turns out there are a few wikis built on top of git : 1. the git-wiki : http://atonie.org/2008/02/git-wiki http://github.com/jeffbski/git-wiki git-wiki is a wiki that relies on git to keep pages' history and Sinatra to serve them. (ruby) Supports these markups : * Creole= Creole is a Creole-to-HTML converter for Creole, the lightwight markup language (http://wikicreole.org/). * Markdown= Discount Markdown Processor for Ruby http://github.com/rtomayko/rdiscount * Textile = RedCloth is a module for using the Textile markup language in Ruby. http://redcloth.org/ 2. gitit http://hackage.haskell.org/cgi-bin/hackage-scripts/package/gitit gitit: Wiki using happstack, git or darcs, and pandoc. (haskell) 3.ikiwiki http://ikiwiki.info/ Ikiwiki is a wiki compiler. http://ikiwiki.info/ikiwiki/formatting/ 4. wigit : the php git wiki http://el-tramo.be/software/wigit

On Fri, Oct 16, 2009 at 12:45 AM, jamesmikedupont(a)googlemail.com 10 seconds? That's horrible. Have you tried using svn?

On Fri, Oct 16, 2009 at 10:31 AM, Anthony <wikimail(a)inbox.org> wrote: On a reasonably fast network it actually only about 10 seconds to pull the entire edit history from his repo, it would take less if the history has been repacked as I described— but that kind of tight repacking makes it take longer when you only want a portion of the history. Still— much of the neat things that can be done by having the article in git are only possible if you have the complete history, for example: generating a blame map needs the entire history. It would be nice if the git archival format was more efficient for the kinds of changes made in Wikipedia articles: Source code changes tends to have short lines and changes tend to change a significant portion of the lines, while edits on Wikipedia are far more likely to change only part of a very long line (really, a paragraph).... so working with line level deltas is efficient for source code while inefficient for Wikipedia data. On this repository a git fast-export --all | lzma -9 produces a 900kbyte output (505783 bytes if you want to be silly and use PAQ8HP12, which is pretty much the state of the art for English text, instead of LZMA). These methods don't provide fast random access but it's still clear that there is a lot of room for improvement. ;) I'm not sure if anyone is working on improved compression for git for these kinds of documents. Getting the entire history of a frequently edited article like this down to ~1-2mb is roughly where I think it's reasonable for someone doing continued non-trivial work on the article to fetch the entire history and thus gain access to functionality that needs most of the history.

On Sat, Oct 17, 2009 at 4:40 AM, jamesmikedupont(a)googlemail.com <jamesmikedupont(a)googlemail.com> wrote: Char level would probably make it too inefficient to merge deltas. Treating a period followed by a space as a line separator would probably be more efficient. The key to efficiency is to use skip deltas, though. You build a binary tree so accessing any revision requires the application of only log(n) deltas. I asked whether or not you tried svn, because svn already uses skip deltas. Is the idea that the entire file would need to be transferred over the Internet, though? If so, I guess you wouldn't want to use skip deltas - they greatly increase access time to early revisions, but at a slight space penalty.

On Sat, Oct 17, 2009 at 11:04 AM, John Vandenberg <jayvdb(a)gmail.com> wrote: Doesn't mean you can't learn from it.

On Sun, Oct 18, 2009 at 3:39 AM, jamesmikedupont(a)googlemail.com <jamesmikedupont(a)googlemail.com> wrote: Fantastic! If you need more processing power, the toolserver may be willing to give you an account in order to host it, if you can keep the repo small enough, especially if you can provide a wikiblame tool which is usable. http://meta.wikimedia.org/wiki/Toolserver https://wiki.toolserver.org/view/Account_approval_process -- John Vandenberg

Since there are other people out there, perhaps we can start a mediawiki-git discussion list and/or wiki discussion page? I'd love to post the work I'm doing, too as it starts to come together. -Josh

On Mon, Oct 19, 2009 at 5:30 PM, Joshua Gay <joshuagay(a)gmail.com> wrote: Sounds great Josh, lets collaborate. We can setup a google groups in a minute... mike

On Fri, Oct 16, 2009 at 9:45 AM, Denny Vrandecic <denny.vrandecic(a)kit.edu> wrote: I don't know all these wikitools, but if the feature is missing from git, then it will benefit all projects using it. My fascination with using a real distribute version control system is that it provides the features that we are missing from the mediawiki. We can use standard tools to do good things, and not have to reinvent the world all the time. We don't need to have a centralized repository and only one point of view, using a real VCS means that we can multiple hosts, multiple points of view and a failsafe system. My next steps are to work on the reader tool in creating latex output and espeak output of the articles, I am adding in the unicode character support right now. I would like to get that up to speed, to use PDF / Audio rendering of the articles. I will continue to just work with selected articles and improve the import feature. It should be easy to have an import tool feed by an rss feed for some articles that imports them on a regular basis. Mike

I have make two simple vlogs about what and why i did this http://www.youtube.com/watch?v=jc9jo1ZFLqk http://www.youtube.com/watch?v=7WfRuEuvIso Mike

Hoi, After a minute of googling I find http://wikitrust.soe.ucsc.edu/home .. I am sure it is there for you as well. Thanks, GerardM 2009/10/16 jamesmikedupont(a)googlemail.com <jamesmikedupont(a)googlemail.com>

Dear James, you are absolutely right that we were lacking demos: we worked flat out to produce some, and if you visit http://wikitrust.soe.ucsc.edu/ , you can see that there are now a couple of Wikipedias on which you can try this. We wrote our own text analysis engine. The reason is that the typical diff algorithms you find in git, svn, etc, are very fragile for the analysis of wiki text: - They are typically not able to deal with text reordering. If you swap the order of two paragraphs, it will look to them as if you inserted one of the two paragraphs. We wanted to be able to trace text across block moves. - They typically analyze text across the two last revisions only. We wanted to be able to remember which text used to be present, and has subsequently been deleted, so that if the text is later reinserted, we can still correctly attribute it to the original author. Otherwise, if I want to look like the author of text, I can simply delete (or replace) the content of a page, do a few quick-fire edits to confuse the system, and then reinsert the content with some minor changes. We took a lot of pain to make sure that the text attribution system works in a robust way with respect to these kind of phenomena. I am sure it is not perfect yet, and we welcome all feedback. Luca On Fri, Oct 16, 2009 at 5:17 AM, jamesmikedupont(a)googlemail.com < jamesmikedupont(a)googlemail.com&gt; wrote:

Yes, I just sent a message to the quality mailing list. Install the WikiTrust add-on, and visit it.wikipedia.org for instance. pt.wikipedia.org should also work. Please see the message I just sent to the Quality mailing list for more information. Luca On Fri, Oct 16, 2009 at 4:40 AM, jamesmikedupont(a)googlemail.com < jamesmikedupont(a)googlemail.com&gt; wrote: