After strugling some time with the profiler (my debugfile never showed any content) I tried the Eaccelerator knowing this might (not) be the answer. But it was. It increased the speed significantly, whatever the cause was. Thanks!
You're probably just hiding the problem, though. Eaccelerator will just speed up viewing pages that have already been parsed, it won't speed up viewing pages for the first time (and that's first time for each logged in user, I expect, so the difference may be minimal if you test with other accounts).