On Thu, 5 Nov 2020 at 18:35, Dan Andreescu <dandreescu(a)wikimedia.org> wrote: LGMT3. -- Timo

I don't know enough about the parser cache to give Daniel good advice on his question: But I see some people hiding in the back who might have some good ideas :) This is just a bump to invite them to respond.

The page properties table is already loaded into Hadoop on a monthly basis (wmf_raw.mediawiki_page_props). I haven't played with it much, but Hive also has JSON-parsing goodies, so give it a shot and let me know if you get stuck. In general, data from the databases can be sqooped into Hadoop. We do this for large pipelines like edit history <https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Edit_data_loading> and it's very easy <https://github.com/wikimedia/analytics-refinery/blob/master/python/refinery/sqoop.py#L505> to add a table. We're looking at just replicating the whole db on a more frequent basis, but we have to do some groundwork first to allow incremental updates (see Apache Iceberg if you're interested).

On Tue, Nov 3, 2020 at 1:59 AM Daniel Kinzler &lt;dkinzler(a)wikimedia.org&gt; wrote: I'm not sure I understand the problem. Binary data can be trivially represented as JSON, by treating it as a string. Is it an issue of storage size? JSON escaping of the control characters is (assuming binary data with a somewhat random distribution of bytes) an ~50% size increase, UTF-8 encoding the top half of bytes is another 50%, so it will approximately double the length - certainly worse than the ~33% increase for base64, but not tragic. (And if size increase matters that much, you probably shouldn't be using base64 either.) * Don't write the data to page_props, treat it as extension data in It would also mean that fetching template data or some other page property might require a parse, as parser cache entries expire. It would also also mean the properties could not be searched, which I think is a dealbreaker. * Apply compression for page_props, but not for the data in ParserOutput. IMO we don't want to. page_props is for page *properties*, not arbitrary structured data. Also it's somewhat problematic in that it is per-page data but it represents the result of a parse, so it doesn't necessarily match the current revision, nor what a user with non-canonical parser options sees. New features should probably use MCR for structured data. * Introduce a dedicated database table for templatedata. Cleaner, but That seems like a decent solution to me, and probably the one I would pick (unless there are more extensions in a similar situation). This is secondary data so it doesn't really need to be migrated, just make TemplateData write from the new table and fall back to the old one when reading. Creating new tables should also not be time-consuming. * Put templatedata into the BlobStore, and just the address into Doesn't BlobStore support batch loading, unlike ParserCache? * Convert TemplateData to MCR. This is the cleanest solution, but would MCR has fairly different semantics from parser metadata. There are many ways TemplateData data can be generated for a page without having a <templatedata> tag in the wikitext (e.g. a doc subpage, or a template which generates both documentation HTML and hidden TemplateData). Switching to MCR should be thought of as a workflow adjustment for contributors, not just a data migration.