Couldn't that logic be applied almost equally to allowing everybody to edit?
It can, but it's a matter of degree. The amount of damage, and the time for cleanup, is much less with regular editing, and (very importantly) the gain is much greater. Inexperienced editors do a lot of good for Wikipedia, inexperienced admins wouldn't do much good at all.