Hi everyone,
Russ Nelson (svn account: nelson) is a new committer to core MediaWiki. Russ is contracting to Wikimedia Foundation to create a scalable media storage system based on OpenStack's Swift object store, some Swift middleware custom for MediaWiki, and File and FileRepo-derived classes specific to Swift.
User page: http://www.mediawiki.org/wiki/User:RussNelson
Rob
Take a look at riak too. Wikia's sysadmin Artur (crucially) built file storage for Wikia using riak and a custom fuse module.
~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://daniel.friesen.name]
On 11-03-18 04:09 PM, Rob Lanphier wrote:
Hi everyone,
Russ Nelson (svn account: nelson) is a new committer to core MediaWiki. Russ is contracting to Wikimedia Foundation to create a scalable media storage system based on OpenStack's Swift object store, some Swift middleware custom for MediaWiki, and File and FileRepo-derived classes specific to Swift.
User page: http://www.mediawiki.org/wiki/User:RussNelson
Rob
On 3/18/11 9:22 PM, Daniel Friesen wrote:
Take a look at riak too. Wikia's sysadmin Artur (crucially) built file storage for Wikia using riak and a custom fuse module.
This is way past the planning stages -- Russell has been working on this since late last year, along with a handful of WMF people.
Early on we decided we wanted a REST interface more than a FUSE interface, so we had a different set of choices. Swift not only seems to perform well, but it also has some nice management consoles built-in, and it's backed by some large companies.
I don't think Riak ever came up in the discussions -- it might have been eliminated early due to being Erlang based. While Erlang is a wicked cool language, we have zero experience with it in production.
I had a look at Artur's RiakFuse just now. As far as I can tell it works like this:
MediaWiki (PHP) --> FUSE --> RiakFuse (Perl) --> HTTP REST/JSON --> Riak (Erlang)
Artur's a lot smarter than I am... but I look at this and am already a bit alarmed. ;)
Also I wonder why not go directly from a MediaWiki FileRepo using REST calls straight into Riak. That's more or less what Russell is building for us except the backend will be Swift.
Also, in the README Artur mentions his solution isn't good for big files, and the whole point of this project was a backend for Commons.
I can read Perl fine but everybody else hated the idea of using Perl solutions, so perhaps Artur's idea was doubly doomed from our perspective. Still I feel bad that we didn't at least consider this earlier, unless Russell did at some point and rejected it early on. (?)
On 11-03-22 01:22 AM, Neil Kandalgaonkar wrote:
[...] Also I wonder why not go directly from a MediaWiki FileRepo using REST calls straight into Riak. That's more or less what Russell is building for us except the backend will be Swift. [...]
I actually asked Artur about that myself, I was contemplating building a FileRepo store using GridFS (since one of my potential environments would already have distributed MongoDB theoretically). iirc it was something to do with different edge cases popping up and causing bugs for anything not filesystem based. Also, iirc most of the FileRepo related to thumbnailing actions is fairly FS geared. I did check and found that most of the programs can handle fetching images over http but that was probably one of the big sources of edge cases and bugs. That being said, since this is a Wikimedia project having the FileRepo cleaned up so that non-fs based file storage is a viable option without buggyness would be beautiful.
On 19/03/11 00:09, Rob Lanphier wrote:
Hi everyone,
Russ Nelson (svn account: nelson) is a new committer to core MediaWiki. Russ is contracting to Wikimedia Foundation to create a scalable media storage system based on OpenStack's Swift object store, some Swift middleware custom for MediaWiki, and File and FileRepo-derived classes specific to Swift.
Welcome around Russ!
wikitech-l@lists.wikimedia.org