There are a few issues with how our file/image handling works that make handling remote files a bit harder than it needs to be.
Some things don't work consistently over InstantCommons because a local MediaHandler class is used that doesn't understand the remote data, and the whole FileRepo architecture is pretty awkward for actually storing "local" files outside of the filesystem (eg the Extension:SwiftMedia backend has to jump through a lot of hoops).
I've started collecting some notes for an RFC on the wiki:
https://www.mediawiki.org/wiki/Requests_for_comment/Refactor_on_File-FileRep...
There's not a lot of specific work targets in there yet, as we'll want some more data from the folks working on the alternate backends as well as the frontend issues which are mostly what I've written on.
But I would like to see if we can avoid having to have a local MediaHandler that's compatible with a remotely-hosted file. As long as the necessary metadata can be exposed, and any rendering can be encapsulated as a thumbnail or an iframe, client sites *should* be able to render anything we can push out from Commons without any special support on their end.
-- brion
On Mon, Oct 3, 2011 at 6:52 PM, Brion Vibber brion@wikimedia.org wrote:
There are a few issues with how our file/image handling works that make handling remote files a bit harder than it needs to be.
Some things don't work consistently over InstantCommons because a local MediaHandler class is used that doesn't understand the remote data, and the whole FileRepo architecture is pretty awkward for actually storing "local" files outside of the filesystem (eg the Extension:SwiftMedia backend has to jump through a lot of hoops).
I've started collecting some notes for an RFC on the wiki:
https://www.mediawiki.org/wiki/Requests_for_comment/Refactor_on_File-FileRep...
Yay! I've watchlisted the page and will certainly provide some feedback once I've read thru the whole thing.
There's not a lot of specific work targets in there yet, as we'll want some more data from the folks working on the alternate backends as well as the frontend issues which are mostly what I've written on.
One thing at the very least stands out to me (and has been mentioned in discussions elsewhere)...we *really* need to have a FileStore-esque class that abstracts file system operations. Lots of SwiftMedia duplication could be avoided if all it had to implement was the filesystem logic. By extension, we could possibly end up supporting the "put uploads in the database" enhancement[0].
-Chad
Chad [innocentkiller@gmail.com] writes:
in discussions elsewhere)...we *really* need to have a FileStore-esque class that abstracts file system operations.
Yep. The only real question is whether we tie it to the SwiftMedia project, or whether we split it out into a separate refactoring project? Barring any new issues raising their head, we should have a SwiftMedia 1.0 release prior to the Hack-a-thon. If it had existed *prior* to starting on SwiftMedia, it would have saved me time and effort. I didn't, though, so the work of copying, and understanding what needed to be copied, is done. So I think the driving issue should be: how desperate are we to replace the existing architecture using NFS? Ops would have to tell us, I think. From an architecture basis, now is the time to proceed on FileStore.
On Mon, Oct 3, 2011 at 4:30 PM, Russell N. Nelson - rnnelson < rnnelson@clarkson.edu> wrote:
Chad [innocentkiller@gmail.com] writes:
in discussions elsewhere)...we *really* need to have a FileStore-esque class that abstracts file system operations.
Yep. The only real question is whether we tie it to the SwiftMedia project, or whether we split it out into a separate refactoring project? Barring any new issues raising their head, we should have a SwiftMedia 1.0 release prior to the Hack-a-thon. If it had existed *prior* to starting on SwiftMedia, it would have saved me time and effort. I didn't, though, so the work of copying, and understanding what needed to be copied, is done. So I think the driving issue should be: how desperate are we to replace the existing architecture using NFS? Ops would have to tell us, I think. From an architecture basis, now is the time to proceed on FileStore.
I would *very* strongly recommend doing the internal refactoring before we get anywhere near reviewing and deploying that bad boy; otherwise we'll spend all the code review time pointing out things to refactor to avoid future maintenance problems. :)
-- brion
wikitech-l@lists.wikimedia.org