JustinO linked these in IRC, and i thought they were quite interesting:

http://romip.ru/russir2015/pdf/kanoulas-onoff-evaluation-russir-part-I.pdf
http://romip.ru/russir2015/pdf/kanoulas-onoff-evaluation-russir-part-II.pdf

It's also nice sometimes to have the talk that goes with the paper. This isn't for the same paper, but is from the same presenter about offline evaluation of search engines:

https://www.youtube.com/watch?v=SgrfLmqfR_0