There is a logging_logindex table. I incorporated all suggested changes into https://quarry.wmflabs.org/query/23829
Since I still think they query will take more than 20 minutes, I think quarry is not the way to go. I saved the query on the grid and tried to run it using this command:
jsub -once -N "custom_sql" -mem 1g -o /data/project/huji/err/custom_sql.out -e /data/project/huji/err/custom_sql.err sql fawiki_p < /data/project/huji/query.sql
But even though when I run this I get the response that my job was submitted, running qstat immediately thereafter shows no results. Is it okay, or am I running the job incorrectly? (Note that both the .out and the .err files are empty).
On Sat, Dec 30, 2017 at 2:10 PM, Maximilian Doerr < maximilian.doerr@gmail.com> wrote:
Isn’t there a logging_logindex table to use that should optimize this?
Cyberpower678 English Wikipedia Account Creation Team English Wikipedia Administrator Global User Renamer
On Dec 30, 2017, at 14:07, Brad Jorsch (Anomie) bjorsch@wikimedia.org wrote:
On Sat, Dec 30, 2017 at 1:07 PM, John phoenixoverride@gmail.com wrote:
Use the logging_userindex table instead of logging
That won't make much difference, since the select on the logging table isn't targeting any user columns.
On Sat, Dec 30, 2017 at 1:09 PM, John phoenixoverride@gmail.com wrote:
I would also find the first log of 2017 and use that instead of the timestamp
That would make it worse, since there's no index on (log_type, log_id). It'll either have to use the primary key and filter out all rows with a different log_type, or use one of the indexes that begins with log_type and filter out all the rows with an earlier log_id.
On Sat, Dec 30, 2017 at 1:32 PM, Dennis Tobar dennis.tobar@gmail.com wrote:
Replace count(*) with count(1) in the subquery. It could help (?) to improve the performance.
"count(*)" and "count(1)" should be treated equivalently. The "*" in "count(*)" does not cause the database to fetch all fields.
If anything, "count(*)" might be ever so slightly faster since it's literally staying "count the number of rows" rather than "count the number of rows where the constant 1 is not null". But the DB probably optimizes counting of a constant to make them identical.
-- Brad Jorsch (Anomie) Senior Software Engineer Wikimedia Foundation _______________________________________________ Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud
Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud