On Fri, Feb 22, 2013 at 4:41 AM, Johnuniq wp.johnuniq@gmail.com wrote:
On Feb 20, 2013 at 3:54 pm, Tim Starling wrote:
The idea of storing a database in a large string literal could be made to be fairly efficient and user-friendly if a helper module was written to do parsing and a binary search.
I have implemented the above suggestion with some promising results. Packing a large table in a string and unpacking it on demand appears to work well, and the data is accessed as if it were stored in a standard table. Using the table from Wiktionary Module:Languages mentioned earlier in this thread, testing shows that accessing the packed data is 20 times faster. Info is at
http://test2.wikipedia.org/wiki/User_talk:Johnuniq#Big_tables
Note that https://gerrit.wikimedia.org/r/#/c/50299/ added a mw.loadData() function that should solve the problem for normal tables. It works like require, but can only handle simple data (no functions, tables with metatables, or tables with tables as keys), the returned data structure is made read-only, and it avoids having to re-execute the module chunk on every #invoke.
Speaking of which, I need to update the documentation.