On 3/18/11 9:22 PM, Daniel Friesen wrote:
Take a look at riak too. Wikia's sysadmin Artur (crucially) built file storage for Wikia using riak and a custom fuse module.
This is way past the planning stages -- Russell has been working on this since late last year, along with a handful of WMF people.
Early on we decided we wanted a REST interface more than a FUSE interface, so we had a different set of choices. Swift not only seems to perform well, but it also has some nice management consoles built-in, and it's backed by some large companies.
I don't think Riak ever came up in the discussions -- it might have been eliminated early due to being Erlang based. While Erlang is a wicked cool language, we have zero experience with it in production.
I had a look at Artur's RiakFuse just now. As far as I can tell it works like this:
MediaWiki (PHP) --> FUSE --> RiakFuse (Perl) --> HTTP REST/JSON --> Riak (Erlang)
Artur's a lot smarter than I am... but I look at this and am already a bit alarmed. ;)
Also I wonder why not go directly from a MediaWiki FileRepo using REST calls straight into Riak. That's more or less what Russell is building for us except the backend will be Swift.
Also, in the README Artur mentions his solution isn't good for big files, and the whole point of this project was a backend for Commons.
I can read Perl fine but everybody else hated the idea of using Perl solutions, so perhaps Artur's idea was doubly doomed from our perspective. Still I feel bad that we didn't at least consider this earlier, unless Russell did at some point and rejected it early on. (?)