--- On Mon, 8/24/09, Chad innocentkiller@gmail.com wrote:
Why skip trying to find the location? If MW_INSTALL_PATH is already missing, what have we got to lose from trying to guess the location? The vast majority of people don't screw with the default structure, so it should be just fine.
That's a reasonable question, stating in another way the useful maxim, "if it ain't broke, don't fix it." The problem is I think it's "broke".
Here is my take on the pros/cons of leaving things unchanged:
Pros:
* Some administrators are used to simply typing the line php <utility>.php. Making them type:
MW_INSTALL_PATH=/var/wiki/mediawiki php <utility>.php
would be inconvenient.
In answer to this, for the MW installations running on unix, it is pretty simple to alias "MW_INSTALL_PATH=/var/wiki/mediawiki php" and put the definition into .bash_profile (or the appropriate shell initialization script). This is a one time effort and so the change isn't as onerous as it might seem. I assume there is a similar tactic available for windows systems.
Cons:
* The use of file position dependent code is a problem during development and much less of a problem during installation and production (as you suggest). Right now there are ~400 sub-directories in the extensions directory. It seems to me reorganization of the extensions directory would help understanding the relationship between individual extensions and the core. For example, having two subdirectories, one for cli utilities and another for hook based extensions would clarify the role each extension plays. However, currently there are 29 extensions where $IP is set using the relative position of the file in the MW directory structure (a couple of other extensions set $IP based on MW_INSTALL_PATH). Reorganizing the directory structure has the potential of breaking them.
* CLI utilities are moved around for reasons other than a reorganization of the extensions directory. For example, as I understand it, DumpHTML was moved from maintenance/ to extensions/. dumpHTML.php sets $IP based on its relative position in the distribution tree. It was a happy coincidence that when it was moved, its relative position didn't change. However, it is unreasonable to think such reclassifications will always be as fortunate.
Since the cons outweigh the pros, I remain convinced that the change I suggested (using die()) improves the code.
Dan
2009/8/24 dan nessett dnessett@yahoo.com:
Pros:
- Some administrators are used to simply typing the line php <utility>.php. Making them type:
MW_INSTALL_PATH=/var/wiki/mediawiki php <utility>.php would be inconvenient.
Here's a question:
These utilities are php. Is there any reason in principle that they couldn't be set up to be accessed from a "MediaWiki control panel" page by WikiSysop?
- d.
Most require far too much execution time to be done over HTTP, you hit timeouts too easily.
-Chad
On Aug 24, 2009 1:37 PM, "David Gerard" dgerard@gmail.com wrote:
2009/8/24 dan nessett dnessett@yahoo.com:
Pros: > * Some administrators are used to simply typing the line php
<utility>.php. Making them t... Here's a question:
These utilities are php. Is there any reason in principle that they couldn't be set up to be accessed from a "MediaWiki control panel" page by WikiSysop?
- d.
_______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia....
2009/8/24 Chad innocentkiller@gmail.com:
Most require far too much execution time to be done over HTTP, you hit timeouts too easily.
Huh, that's annoying. Would a progress ticker be sufficient to keep the connection alive and the admin interested?
(I'm putting this forward as an inchoate feature request not quite up to Bugzilla yet :-) )
- d.
On Mon, Aug 24, 2009 at 1:52 PM, David Gerarddgerard@gmail.com wrote:
2009/8/24 Chad innocentkiller@gmail.com:
Most require far too much execution time to be done over HTTP, you hit timeouts too easily.
Huh, that's annoying. Would a progress ticker be sufficient to keep the connection alive and the admin interested?
(I'm putting this forward as an inchoate feature request not quite up to Bugzilla yet :-) )
- d.
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
There's some extensions that make a stab at it (Maintenance and MaintenanceShell), but I'm not sure what they do to try and overcome the timeout issue. I'm sure it's doable, and it's probably more doable now with the new Maintenance class, rather than trying to shell out to other scripts like we used to :)
-Chad
David Gerard wrote:
2009/8/24 Chad innocentkiller@gmail.com:
Most require far too much execution time to be done over HTTP, you hit timeouts too easily.
Huh, that's annoying. Would a progress ticker be sufficient to keep the connection alive and the admin interested?
(I'm putting this forward as an inchoate feature request not quite up to Bugzilla yet :-) )
Actually SMW has something like this. It has a Special:SMWAdmin page used to set up and upgrade the DB tables, and refresh the data. I believe for the data refresh it splits it into chunks and puts it in the job queue.
On 8/24/09 2:40 PM, Chad wrote:
Most require far too much execution time to be done over HTTP, you hit timeouts too easily.
Yup. I'd love to see a good infrastructure for breaking up a lot of these things into smaller chunks that could be queued up and run more cleanly through a control panel. Nice project for someone? :)
-- brion
2009/8/24 Brion Vibber brion@wikimedia.org:
On 8/24/09 2:40 PM, Chad wrote:
Most require far too much execution time to be done over HTTP, you hit timeouts too easily.
Yup. I'd love to see a good infrastructure for breaking up a lot of these things into smaller chunks that could be queued up and run more cleanly through a control panel. Nice project for someone? :)
Sounds teeeeempting!
(if anyone else does it first good luck to them ;-) )
What sort of environment should be presumed? i.e., how locked down is a locked down server in this context?
This sort of thing will make MediaWiki so much nicer in quite a lot of ways.
- d.
2009/8/24 David Gerard dgerard@gmail.com:
2009/8/24 Brion Vibber brion@wikimedia.org:
Yup. I'd love to see a good infrastructure for breaking up a lot of these things into smaller chunks that could be queued up and run more cleanly through a control panel. Nice project for someone? :)
Sounds teeeeempting! (if anyone else does it first good luck to them ;-) ) What sort of environment should be presumed? i.e., how locked down is a locked down server in this context? This sort of thing will make MediaWiki so much nicer in quite a lot of ways.
I should point out that I asked from personal annoyance, i.e. never quite managing to figure out the requisite environment variables from the command line myself. And it's an obvious nice interface idea. Even if making it technically feasible is more than a little fiddly.
- d.
On 8/24/09 6:09 PM, David Gerard wrote:
2009/8/24 Brion Vibberbrion@wikimedia.org:
On 8/24/09 2:40 PM, Chad wrote:
Most require far too much execution time to be done over HTTP, you hit timeouts too easily.
Yup. I'd love to see a good infrastructure for breaking up a lot of these things into smaller chunks that could be queued up and run more cleanly through a control panel. Nice project for someone? :)
Sounds teeeeempting!
(if anyone else does it first good luck to them ;-) )
What sort of environment should be presumed? i.e., how locked down is a locked down server in this context?
Assume web-only; no shell, no guarantee that you can shell out to a command-line PHP.
-- brion
dan nessett wrote:
--- On Mon, 8/24/09, Chad innocentkiller@gmail.com wrote:
Why skip trying to find the location? If MW_INSTALL_PATH is already missing, what have we got to lose from trying to guess the location? The vast majority of people don't screw with the default structure, so it should be just fine.
That's a reasonable question, stating in another way the useful maxim, "if it ain't broke, don't fix it." The problem is I think it's "broke".
Here is my take on the pros/cons of leaving things unchanged:
Pros:
- Some administrators are used to simply typing the line php
<utility>.php. Making them type:
MW_INSTALL_PATH=/var/wiki/mediawiki php <utility>.php
would be inconvenient.
In answer to this, for the MW installations running on unix, it is pretty simple to alias "MW_INSTALL_PATH=/var/wiki/mediawiki php" and put the definition into .bash_profile (or the appropriate shell initialization script). This is a one time effort and so the change isn't as onerous as it might seem. I assume there is a similar tactic available for windows systems.
Cons:
- The use of file position dependent code is a problem during
development and much less of a problem during installation and production (as you suggest). Right now there are ~400 sub-directories in the extensions directory. It seems to me reorganization of the extensions directory would help understanding the relationship between individual extensions and the core. For example, having two subdirectories, one for cli utilities and another for hook based extensions would clarify the role each extension plays. However, currently there are 29 extensions where $IP is set using the relative position of the file in the MW directory structure (a couple of other extensions set $IP based on MW_INSTALL_PATH). Reorganizing the directory structure has the potential of breaking them.
- CLI utilities are moved around for reasons other than a
reorganization of the extensions directory. For example, as I understand it, DumpHTML was moved from maintenance/ to extensions/. dumpHTML.php sets $IP based on its relative position in the distribution tree. It was a happy coincidence that when it was moved, its relative position didn't change. However, it is unreasonable to think such reclassifications will always be as fortunate.
Since the cons outweigh the pros, I remain convinced that the change I suggested (using die()) improves the code.
Except that the cons are mostly hypothetical. I don't believe anyone except you has actually proposed restructuring the extensions directory. But even if we do, it still wouldn't affect most users because A) There aren't that many extensions that add command line utilities (several extensions also have scripts and hook based extensions so wouldn't neatly fit into such categories) and B) Most users don't checkout the entire extensions directory, just the few extensions they're actually going to use, so they can continue to put them all in the same directory.
It may be, technically, a slight improvement to code quality, but its arguably a degradation to end-user experience. It makes things slightly easier for developers in the event of a hypothetical change (though time-wise, its really only a benefit after the second such change) but makes it so things that have "just worked" for years for users no longer work without changing settings or adding extra command line parameters.
--- On Mon, 8/24/09, Alex mrzmanwiki@gmail.com wrote:
I don't believe anyone except you has actually proposed restructuring the extensions directory.
Perhaps not. But, I don't see why that is relevant. I am making arguments why the extensions directory should be restructured. I may convince no one, but I don't think I should presume that.
A) There aren't that many extensions that add command line utilities (several extensions also have scripts and hook based extensions so wouldn't neatly fit into such categories)
Here are the files in /extensions/ that reference /maintenance/command.inc. There are 65 of them (line number of the reference at the end). I don't know which of these are commonly used and therefore included in installation extension/ directories, but I assume all of them are used by at least a small number of sites (otherwise, why include them in the extensions directory at all?)
/extensions/AbuseFilter/install.php:8 /extensions/AbuseFilter/phpTest.php:8 /extensions/AdvancedSearch/populateCategorySearch.php:9 /extensions/AntiSpoof/batchAntiSpoof.php:6 /extensions/AntiSpoof/generateEquivset.php:4 /extensions/Babel/txt2cdb.php:9 /extensions/BoardVote/voterList.php:6 /extensions/CentralAuth/migratePass0.php:8 /extensions/CentralAuth/migratePass1.php:8 /extensions/CentralAuth/migrateStewards.php:3 /extensions/CentralNotice/rebuildLocalTemplates.php:3 /extensions/CentralNotice/rebuildTemplates.php:3 /extensions/CheckUser/importLog.php:4 /extensions/CheckUser/install.php:8 /extensions/cldr/rebuild.php:11 /extensions/CodeReview/svnImport.php:6 /extensions/CommunityVoice/CLI/Initialize.php:4 /extensions/Configure/findSettings.php:18 /extensions/Configure/manage.php:19 /extensions/Configure/migrateFiles.php:17 /extensions/Configure/migrateToDB.php:16 /extensions/Configure/writePHP.php:18 /extensions/DataCenter/CLI/Import.php:4 /extensions/DataCenter/CLI/Initialize.php:4 /extensions/DumpHTML/dumpHTML.php:61 /extensions/DumpHTML/wm-scripts/old/filterNamespaces.php:4 /extensions/DumpHTML/wm-scripts/queueController.php:6 /extensions/FlaggedRevs/maintenance/clearCachedText.php:13 /extensions/FlaggedRevs/maintenance/reviewAllPages.php:8 /extensions/FlaggedRevs/maintenance/updateAutoPromote.php:8 /extensions/FlaggedRevs/maintenance/updateLinks.php:10 /extensions/FlaggedRevs/maintenance/updateQueryCache.php:8 /extensions/FlaggedRevs/maintenance/updateStats.php:8 /extensions/LiquidThreads/compat/generateCompatibilityLocalisation.php:6 /extensions/LiquidThreads/import/import-parsed-discussions.php:4 /extensions/LiquidThreads/migrateDatabase.php:7 /extensions/LocalisationUpdate/update.php:7 /extensions/MetavidWiki/maintenance/download_from_archive_org.php:4 /extensions/MetavidWiki/maintenance/maintenance_util.inc.php:15 /extensions/MetavidWiki/maintenance/metavid2mvWiki.inc.php:16 /extensions/MetavidWiki/maintenance/metavid_gov_templates.php:2 /extensions/MetavidWiki/maintenance/mv_oneTime_fixes.php:2 /extensions/MetavidWiki/maintenance/mv_update.php:6 /extensions/MetavidWiki/maintenance/ogg_thumb_insert.php:15 /extensions/MetavidWiki/maintenance/scrape_and_insert.inc.php:12 /extensions/MetavidWiki/maintenance/transcode_to_flv.php:13 /extensions/MetavidWiki/maintenance/video_ocr_thumb_insert.php:15 /extensions/OAI/oaiUpdate.php:17 /extensions/ParserFunctions/testExpr.php:4 /extensions/SecurePoll/voterList.php:11 /extensions/SemanticMediaWiki/maintenance/SMW_conceptCache.php:18 /extensions/SemanticMediaWiki/maintenance/SMW_dumpRDF.php:34 /extensions/SemanticMediaWiki/maintenance/SMW_refreshData.php:41 /extensions/SemanticMediaWiki/maintenance/SMW_setup.php:46 /extensions/SemanticMediaWiki/maintenance/SMW_unifyProperties.php:27 /extensions/SemanticResultFormats/Ploticus/SRF_Ploticus_cleanCache.php:24 /extensions/SemanticTasks/ST_CheckForReminders.php:6 /extensions/SpamBlacklist/cleanup.php:9 /extensions/SwarmExport/swarmExport.php:23 /extensions/TitleKey/rebuildTitleKeys.php:3 /extensions/TorBlock/loadExitNodes.php:7 /extensions/TrustedXFF/generate.php:8 /extensions/UsabilityInitiative/PrefStats/populatePrefStats.php:9 /extensions/WikiAtHome/internalCmdLineEncoder.php:6 /extensions/WikiTrust/sql/create_db.php:74
Dan
dan nessett wrote:
--- On Mon, 8/24/09, Alex mrzmanwiki@gmail.com wrote:
I don't believe anyone except you has actually proposed restructuring the extensions directory.
Perhaps not. But, I don't see why that is relevant. I am making arguments why the extensions directory should be restructured. I may convince no one, but I don't think I should presume that.
Most of your argument in favor of changing the method for determining include path seems to revolve around the assumption that we're going to rearrange the directory at some point, possibly multiple times. But if nobody else wants to do that, then its just academic, and even then, it assumes that end-users will also structure their own extensions directory in the same way.
A) There aren't that many extensions that add command line utilities (several extensions also have scripts and hook based extensions so wouldn't neatly fit into such categories)
Here are the files in /extensions/ that reference /maintenance/command.inc. There are 65 of them (line number of the reference at the end). I don't know which of these are commonly used and therefore included in installation extension/ directories, but I assume all of them are used by at least a small number of sites (otherwise, why include them in the extensions directory at all?)
/extensions/AbuseFilter/install.php:8 /extensions/AbuseFilter/phpTest.php:8 /extensions/AdvancedSearch/populateCategorySearch.php:9 /extensions/AntiSpoof/batchAntiSpoof.php:6 /extensions/AntiSpoof/generateEquivset.php:4 /extensions/Babel/txt2cdb.php:9 /extensions/BoardVote/voterList.php:6 /extensions/CentralAuth/migratePass0.php:8 /extensions/CentralAuth/migratePass1.php:8 /extensions/CentralAuth/migrateStewards.php:3 /extensions/CentralNotice/rebuildLocalTemplates.php:3 /extensions/CentralNotice/rebuildTemplates.php:3 /extensions/CheckUser/importLog.php:4 /extensions/CheckUser/install.php:8 /extensions/cldr/rebuild.php:11 /extensions/CodeReview/svnImport.php:6 /extensions/CommunityVoice/CLI/Initialize.php:4 /extensions/Configure/findSettings.php:18 /extensions/Configure/manage.php:19 /extensions/Configure/migrateFiles.php:17 /extensions/Configure/migrateToDB.php:16 /extensions/Configure/writePHP.php:18 /extensions/DataCenter/CLI/Import.php:4 /extensions/DataCenter/CLI/Initialize.php:4 /extensions/DumpHTML/dumpHTML.php:61 /extensions/DumpHTML/wm-scripts/old/filterNamespaces.php:4 /extensions/DumpHTML/wm-scripts/queueController.php:6 /extensions/FlaggedRevs/maintenance/clearCachedText.php:13 /extensions/FlaggedRevs/maintenance/reviewAllPages.php:8 /extensions/FlaggedRevs/maintenance/updateAutoPromote.php:8 /extensions/FlaggedRevs/maintenance/updateLinks.php:10 /extensions/FlaggedRevs/maintenance/updateQueryCache.php:8 /extensions/FlaggedRevs/maintenance/updateStats.php:8 /extensions/LiquidThreads/compat/generateCompatibilityLocalisation.php:6 /extensions/LiquidThreads/import/import-parsed-discussions.php:4 /extensions/LiquidThreads/migrateDatabase.php:7 /extensions/LocalisationUpdate/update.php:7 /extensions/MetavidWiki/maintenance/download_from_archive_org.php:4 /extensions/MetavidWiki/maintenance/maintenance_util.inc.php:15 /extensions/MetavidWiki/maintenance/metavid2mvWiki.inc.php:16 /extensions/MetavidWiki/maintenance/metavid_gov_templates.php:2 /extensions/MetavidWiki/maintenance/mv_oneTime_fixes.php:2 /extensions/MetavidWiki/maintenance/mv_update.php:6 /extensions/MetavidWiki/maintenance/ogg_thumb_insert.php:15 /extensions/MetavidWiki/maintenance/scrape_and_insert.inc.php:12 /extensions/MetavidWiki/maintenance/transcode_to_flv.php:13 /extensions/MetavidWiki/maintenance/video_ocr_thumb_insert.php:15 /extensions/OAI/oaiUpdate.php:17 /extensions/ParserFunctions/testExpr.php:4 /extensions/SecurePoll/voterList.php:11 /extensions/SemanticMediaWiki/maintenance/SMW_conceptCache.php:18 /extensions/SemanticMediaWiki/maintenance/SMW_dumpRDF.php:34 /extensions/SemanticMediaWiki/maintenance/SMW_refreshData.php:41 /extensions/SemanticMediaWiki/maintenance/SMW_setup.php:46 /extensions/SemanticMediaWiki/maintenance/SMW_unifyProperties.php:27 /extensions/SemanticResultFormats/Ploticus/SRF_Ploticus_cleanCache.php:24 /extensions/SemanticTasks/ST_CheckForReminders.php:6 /extensions/SpamBlacklist/cleanup.php:9 /extensions/SwarmExport/swarmExport.php:23 /extensions/TitleKey/rebuildTitleKeys.php:3 /extensions/TorBlock/loadExitNodes.php:7 /extensions/TrustedXFF/generate.php:8 /extensions/UsabilityInitiative/PrefStats/populatePrefStats.php:9 /extensions/WikiAtHome/internalCmdLineEncoder.php:6 /extensions/WikiTrust/sql/create_db.php:74
Of those 65 files, they appear to be in ~30 extensions (of around 400 total) and as far as I can tell, only 2 are CLI-only extensions (SwarmExport and DumpHTML).
On 8/24/09 5:49 PM, Alex wrote:
Most of your argument in favor of changing the method for determining include path seems to revolve around the assumption that we're going to rearrange the directory at some point, possibly multiple times. But if nobody else wants to do that, then its just academic, and even then, it assumes that end-users will also structure their own extensions directory in the same way.
Exactly; I'd appreciate if we don't spend more time on this list on the subject. :)
-- brion
wikitech-l@lists.wikimedia.org