Hi,
I have been working on getting asynchronous upload from url to work properly[1]. A problem that I encountered was that I need to store data across requests. Normally I would use $_SESSION, but this data should also be available to job runners, and $_SESSION isn't.
As I see there are basically two ways to get a data store. The first is to store the objects in the DB using wfGetCache( CACHE_DB ); I'm not sure though whether it is meant to be used this way.
Alternatively I could revive my staged-upload work. In this branch, all so called stashed uploads, uploads that require user intervention before they can be completed have their meta data stored in the database instead of in the session. That would be still quite a lot of work though.
Or is there any other mechanism to be able to share data between the jobqueue and requests?
Regards, Bryan
[1] http://www.mediawiki.org/wiki/Special:Code/MediaWiki/author/btongminh?offset...
On Thu, Jul 29, 2010 at 1:15 PM, Bryan Tong Minh bryan.tongminh@gmail.com wrote:
I have been working on getting asynchronous upload from url to work properly[1]. A problem that I encountered was that I need to store data across requests. Normally I would use $_SESSION, but this data should also be available to job runners, and $_SESSION isn't.
Isn't this what the job table is for?
On Thu, Jul 29, 2010 at 8:36 PM, Aryeh Gregor Simetrical+wikilist@gmail.com wrote:
On Thu, Jul 29, 2010 at 1:15 PM, Bryan Tong Minh bryan.tongminh@gmail.com wrote:
I have been working on getting asynchronous upload from url to work properly[1]. A problem that I encountered was that I need to store data across requests. Normally I would use $_SESSION, but this data should also be available to job runners, and $_SESSION isn't.
Isn't this what the job table is for?
I need the job to store its result somewhere.
Bryan
On Thu, Jul 29, 2010 at 3:31 PM, Bryan Tong Minh bryan.tongminh@gmail.com wrote:
I need the job to store its result somewhere.
What does it need to store? If it's just success or failure, then why not assume success if the file exists, failure if the job ran (is not still in the job table) but the file doesn't exist?
On Thu, Jul 29, 2010 at 9:52 PM, Aryeh Gregor Simetrical+wikilist@gmail.com wrote:
On Thu, Jul 29, 2010 at 3:31 PM, Bryan Tong Minh bryan.tongminh@gmail.com wrote:
I need the job to store its result somewhere.
What does it need to store? If it's just success or failure, then why not assume success if the file exists, failure if the job ran (is not still in the job table) but the file doesn't exist?
It could be a reupload as well. That would still be possible of course, but makes it a lot more difficult. Furthermore I should be able to pass arbitrary warning messages.
Bryan
Bryan Tong Minh wrote:
Or is there any other mechanism to be able to share data between the jobqueue and requests?
Regards, Bryan
Memcached*
Our $_SESSION simply lives in memcached. So we could do $fake_session = $wgMemc->get( wfMemcKey( 'session', $session_id ) ) ; $fake_session["upload_ok"] = true; $wgMemc->set( wfMemcKey( 'session', $session_id ), $fake_session, 3600 ) ;
One could also use (more portable): session_id( $session_id ); $_SESSION["upload_ok"] = true; session_commit(); but that may be riskier when job is run inside a web request, you would need to switch dynamically sessions (there's a sample of that in http://es.php.net/manual/es/ref.session.php#96309).
If a user web request finishes at the same time a job runs it, or two upload jobs finish at the same time, the result is probably undefined so you may prefer to store in the session pointers to memcached where the result is stored.
Another solution would be to create a general notification framework. File uploads could store its results there, producing a popup to the user, CentralAuth could store there "Your talk on XY has been changed" notifications, tagging a file as copyvio/for deletion could produce a notification on the uploader...
*In the MediaWiki sense of an ObjectCache which can be backed by memcached, a database, an accelerator...
On Thu, Jul 29, 2010 at 6:07 PM, Platonides Platonides@gmail.com wrote:
Memcached*
Our $_SESSION simply lives in memcached. So we could do $fake_session = $wgMemc->get( wfMemcKey( 'session', $session_id ) ) ; $fake_session["upload_ok"] = true; $wgMemc->set( wfMemcKey( 'session', $session_id ), $fake_session, 3600 ) ;
This means that if a memcached server goes down, the information will be lost. The database is the correct place to put this. (Also the correct place to put sessions, for that matter . . .)
On Fri, Jul 30, 2010 at 5:32 PM, Aryeh Gregor Simetrical+wikilist@gmail.com wrote:
On Thu, Jul 29, 2010 at 6:07 PM, Platonides Platonides@gmail.com wrote:
Memcached*
Our $_SESSION simply lives in memcached. So we could do $fake_session = $wgMemc->get( wfMemcKey( 'session', $session_id ) ) ; $fake_session["upload_ok"] = true; $wgMemc->set( wfMemcKey( 'session', $session_id ), $fake_session, 3600 ) ;
This means that if a memcached server goes down, the information will be lost. The database is the correct place to put this. (Also the correct place to put sessions, for that matter . . .)
Also, on places where no memcached or equivalent is available (i.e. CACHE_NONE), this will not work. I think the loss of session data due to memcached breakage was found to be acceptable. Does anybody have some references for this?
Bryan
On Fri, Jul 30, 2010 at 3:57 PM, Platonides Platonides@gmail.com wrote:
Bryan Tong Minh wrote:
Also, on places where no memcached or equivalent is available (i.e. CACHE_NONE), this will not work.
Then you could be using the objectcache table in the database.
No, that's CACHE_DB. CACHE_NONE really means what it says.
-Chad
Chad wrote:
On Fri, Jul 30, 2010 at 3:57 PM, Platonides Platonides@gmail.com wrote:
Bryan Tong Minh wrote:
Also, on places where no memcached or equivalent is available (i.e. CACHE_NONE), this will not work.
Then you could be using the objectcache table in the database.
No, that's CACHE_DB. CACHE_NONE really means what it says.
-Chad
There's no CACHE_NONE environment by itself. Just sysadmins disabling the caching. So if $wgMainCacheType is CACHE_NONE, block the feature or treat it as CACHE_ANYTHING for features that require it.
You know, everybody should have a writable db for running mediawiki...
On Sat, Jul 31, 2010 at 1:07 AM, Chad innocentkiller@gmail.com wrote:
On Fri, Jul 30, 2010 at 3:57 PM, Platonides Platonides@gmail.com wrote:
Bryan Tong Minh wrote:
Also, on places where no memcached or equivalent is available (i.e. CACHE_NONE), this will not work.
Then you could be using the objectcache table in the database.
No, that's CACHE_DB. CACHE_NONE really means what it says.
I could force the usage of the objectcache table even for CACHE_NONE by creating my own cache object with wfGetCache( CACHE_DB ); Are there any objections for (ab)using the objectcache table for this purpose?
Bryan
On 07/29/2010 10:15 AM, Bryan Tong Minh wrote:
Hi,
I have been working on getting asynchronous upload from url to work properly[1]. A problem that I encountered was that I need to store data across requests. Normally I would use $_SESSION, but this data should also be available to job runners, and $_SESSION isn't.
Could the job not include the session_id and upload_session_key .. then in your job handling code you just connect into that session via session_id( $session_id ); session_start(); to update the values ? . That seems like it would be more lightweight than DB status updates. .. I see Platonides suggested this as well.. ( that is how it was done originally done but with a background php process rather than jobs table ) see http://svn.wikimedia.org/viewvc/mediawiki/trunk/phase3/includes/HttpFunction... line 145 ( doSessionIdDownload )
--michael
On Wed, Aug 18, 2010 at 1:13 AM, Michael Dale mdale@wikimedia.org wrote:
On 07/29/2010 10:15 AM, Bryan Tong Minh wrote:
Hi,
I have been working on getting asynchronous upload from url to work properly[1]. A problem that I encountered was that I need to store data across requests. Normally I would use $_SESSION, but this data should also be available to job runners, and $_SESSION isn't.
Could the job not include the session_id and upload_session_key .. then in your job handling code you just connect into that session via session_id( $session_id ); session_start(); to update the values ? .
Thanks, I'll try it that way.
Bryan
wikitech-l@lists.wikimedia.org