Hi Adrian,
2017-05-12 14:55 GMT+02:00 Adrian Bielefeldt < Adrian.Bielefeldt@mailbox.tu-dresden.de>:
Hello everyone,
I have a problem on stat1002.eqiad.wmnet using https://github.com/Wikidata/QueryAnalysis/blob/master/ tools/hourlyFieldValue.py on two files (1.1 GB and 2.4 GB respectively); the processed ends with Killed. My guess is that my script uses too much memory. However, it was my understanding that csv.DictReader reads line-by-line, so the file sizes should not matter.
If anyone can tell me why my script is taking up so much memory or if there is any other reason for the script getting killed I'd be grateful.
I checked dmesg on stat1002 and the Kernel OOM killer is the one that ended your process. I didn't check very carefully but maybe the problem are the size of the structs in https://github.com/Wikidata/QueryAnalysis/blob/master/tools/hourlyFieldValue... ?
I'd also check the usage of zip, since from https://docs.python.org/2/library/functions.html#zip it seems that it unpacks all the items of your csv dictionaries in one go.
Hope that helps!
Luca