JustinO linked these in IRC, and i thought they were quite interesting:
http://romip.ru/russir2015/pdf/kanoulas-onoff-evaluation-russir-part-I.pdf http://romip.ru/russir2015/pdf/kanoulas-onoff-evaluation-russir-part-II.pdf
It's also nice sometimes to have the talk that goes with the paper. This isn't for the same paper, but is from the same presenter about offline evaluation of search engines: