But seriously, even if there is a secret channel, and some on-wiki action is taken as a result of that discussion, you have every right to demand to know why. And if the acting administrator didn't clarify his reasons for the action he would stand a good chance of getting desysoped.
That's the key distinction - actions are different from discussions. Actions should be transparent and fully justified (give or take exceptional circumstances - we have IAR for that). Discussion is a completely different matter. You can talk freely in the admin channel in a way you can't in public. In public admins are scrutinised and accused of being part of various cabals for sneezing at the wrong moment, in the admin channel we can talk to each other without worrying about that. You remove the ability to talk in private and you won't get the talk happening in public, you'll just stop people talking. People will no longer sanity check their actions (they'll either be too scared to act at all and nothing will get done, or they'll act unilaterally and ArbCom will have a series of nervous breakdowns from overwork), and morale will suffer.