JustinO linked these in IRC, and i thought they were quite interesting:
http://romip.ru/russir2015/pdf/kanoulas-onoff-evaluation-russir-part-I.pdf
http://romip.ru/russir2015/pdf/kanoulas-onoff-evaluation-russir-part-II.pdf
It's also nice sometimes to have the talk that goes with the paper. This
isn't for the same paper, but is from the same presenter about offline
evaluation of search engines:
https://www.youtube.com/watch?v=SgrfLmqfR_0