What's your configured memory_limit of php? If a request surpassed it, it would be cancelled, but not the total system memory shouldn't be surpassed (many concurrent requests?). I assume "the server had completely maxed out its memory" refers to the apache process?
Note that a php like MediaWiki cannot produce a memory leak in the server, as php frees all the memory used by the request and the end of it. Are you sure there were absulutely no requests that night? Perhaps it was crawled by e.g Googlebot.