Hi,
I plan to try and use a big (~67K entries) table as a "database" in a LUA script, with a basic lookup function. One call of the module would be well within the time and memory limits, but I'm worried for pages that might invoke the module several hundred times, so I did an experiment with only a single entry in the table and the results are encouraging:
1 call: Lua time usage 0.006/10.000 seconds Lua memory usage 513 KB/50 MB
10 calls: Lua time usage 0.010/10.000 seconds Lua memory usage 873 KB/50 MB
100 calls: Lua time usage 0.074/10.000 seconds Lua memory usage 1,35 MB/50 MB
There are obviously some optimizations there and I would be curious what they are? Do you use some CoW or other trick to prevent loading that table 100 times? Do you have the loading/parsing process documented somewhere?
Thanks, Strainu
On Fri, Feb 12, 2016 at 6:07 AM, Strainu strainu10@gmail.com wrote:
There are obviously some optimizations there and I would be curious what they are? Do you use some CoW or other trick to prevent loading that table 100 times?
Regarding memory usage, much of the actual data generated during an #invoke is no longer referenced afterward and can be garbage collected, so the peak memory usage won't necessarily increase.
Regarding loading, Lua has a two-step process for loading a package: the searcher returns a loader function, then the loader function returns the actual package table. Scribunto's searcher caches the loader functions inside Lua across #invokes, which can speed up subsequent loads. For modules loaded from the wiki, the PHP code that fetches the text of the module page also caches that text across #invokes.
If your table is static data, you can go one better by using mw.loadData() to load it. This caches the actual data table inside Lua across #invokes.
Do you have the loading/parsing process documented somewhere?
The standard Lua package module is documented at http://www.lua.org/manual/5.1/manual.html#5.3
mw.loadData() is documented from an end-user standpoint at https://www.mediawiki.org/wiki/Extension:Scribunto/Lua_reference_manual#mw.l...
If you want to look at the actual code, the package module is implemented in engines/LuaCommon/lualib/package.lua, the searcher is near the top of engines/LuaCommon/lualib/mw.lua (look in makePackageModule()), the PHP code behind the searcher starts in engines/LuaCommon/LuaCommon.php (Scribunto_LuaEngine::loadPackage()), and mw.loadData is near the end of engines/LuaCommon/lualib/mw.lua.
wikitech-l@lists.wikimedia.org