On Tue, Nov 3, 2020 at 4:38 AM Daniel Kinzler dkinzler@wikimedia.org wrote:

Am 02.11.20 um 19:24 schrieb Daniel Kinzler:

T262946 https://phabricator.wikimedia.org/T262946 *"Bump Firefox version in basic support to 3.6 or newer"*: last call ending on Wednesday, November 4. Some comments, no objections.

Since we are not having a meeting on Wednesday, I guess we should try and get quorum to approve by mail.

I'm in favor.

+1

I saw in the patch for https://phabricator.wikimedia.org/T266200 a strategy was devised to base64-encode page prop values that aren't strictly UTF-8. If I understand correctly, this means TemplateData extension code and page props interfaces require no change while the JSONification of Parser Cache output proceeds. Is that right? It's a clever solution.

Now, one thing I've been wondering about: might there be ways to query the database component of Parser Cache with relatively fresh results at the command line without deployer rights? And will it be possible, if not encouraged, to drop stringified JSON into the Parser Cache values?

The page props table tends to be useful for content analysis for UX interventions, and part of its usefulness has stemmed from being able to do simple MySQL queries (when the payload is encoded for JSON and even if it were compress()'d, it can also be trivial to use MySQL JSON built-ins). The more, shall we say, creative, uses of page props I'm told aren't great for scaling, but I'm wondering, how can we get some of the capabilities of querying derived data via another straightforward SQL mechanism on a replicated persistence store off the serving code path?

I hope those questions made sense! Maybe something exists already in Hadoop or the replicas, but I couldn't quite figure it out. I do look forward to other application layer and firehouse mechanisms in the works from different teams, although am most interested right now in the content analysis use case for some of our forthcoming Wikifunctions / Wikilambda and Abstract Wikipedia work.

Thanks! -Adam

On Fri, Nov 6, 2020 at 3:24 PM Dan Andreescu dandreescu@wikimedia.org wrote:

I don't know enough about the parser cache to give Daniel good advice on his question:

...

That's another issue I wanted to raise: Platform Engineeing is working on switching ParserCache to JSON. For that, we have to make sure extensions only put JSON-Serializable data into ParserOutput objects, via setProperty() and setExtensionData(). We are currently trying to figure out how to best do that for TemplateData.

TemplateData already uses JSON serialization, but then compresses the JSON output, to make the data fit into the page_props table. This results in binary data in ParserOutput, which we can't directly put into JSON. There are several solutions under discussion, e.g.: [...(see Daniel's original message for the list of ideas or propose your own)...]

But I see some people hiding in the back who might have some good ideas :) This is just a bump to invite them to respond. _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Dan Andreescu dandreescu@wikimedia.org wrote:

Maybe something exists already in Hadoop

...

The page properties table is already loaded into Hadoop on a monthly basis (wmf_raw.mediawiki_page_props). I haven't played with it much, but Hive also has JSON-parsing goodies, so give it a shot and let me know if you get stuck. In general, data from the databases can be sqooped into Hadoop. We do this for large pipelines like edit history https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Edit_data_loading and it's very easy https://github.com/wikimedia/analytics-refinery/blob/master/python/refinery/sqoop.py#L505 to add a table. We're looking at just replicating the whole db on a more frequent basis, but we have to do some groundwork first to allow incremental updates (see Apache Iceberg if you're interested).

Yes, I like that and all of the other wmf_raw goodies! I'll follow up off thread on accessing the parser cache DBs (they're in site.pp and db-eqiad.php, but I don't think those are presently represented by refinery.util as they're not in .dblist files).

On Tue, Nov 10, 2020 at 5:50 PM Gergo Tisza gtisza@wikimedia.org wrote:

On Tue, Nov 3, 2020 at 1:59 AM Daniel Kinzler dkinzler@wikimedia.org wrote:

...

TemplateData already uses JSON serialization, but then compresses the JSON output, to make the data fit into the page_props table. This results in binary data in ParserOutput, which we can't directly put into JSON.

I'm not sure I understand the problem. Binary data can be trivially represented as JSON, by treating it as a string. Is it an issue of storage size? JSON escaping of the control characters is (assuming binary data with a somewhat random distribution of bytes) an ~50% size increase, UTF-8 encoding the top half of bytes is another 50%, so it will approximately double the length - certainly worse than the ~33% increase for base64, but not tragic. (And if size increase matters that much, you probably shouldn't be using base64 either.)

The binary aspect here refers to the gzip output buffer. While these are represented in PHP as a string, the string is not encodable as UTF-8 or indeed as JSON. Attempting to do so results in a PHP json error with boolean false returned.

Condensed example: https://3v4l.org/cJttU

On Wed, Nov 4, 2020 at 12:23 AM Krinkle krinklemail@gmail.com wrote:

*RFC: Expiring watch list entries* https://phabricator.wikimedia.org/T124752

This just missed the triage window, but it looks like this was implemented and deployed meanwhile (it was in Phase 3). I'm proposing we put this on Last Call for wider awareness and so that the team can answer any questions people might have, and to address any concerns that people might have based on reviewing the proposal we now know the team wanted/has chosen.

+1 to this as well