Daniel, I happen to think that any Arb who is asked to excuse themselves from a case should do so, within reason.
But in particular I think women who see certain Arbs as sexist should be able to require recusal. Otherwise the case is hobbled before it begins. Ditto for anyone with concerns about racism or homophobia.
I would like to see a jury system replace the committee, with small groups chosen to resolve particular issues. The committee has not worked for a long time. It isn't the fault of any individual or group. It's a combination of the way Arbs are nominated and elected, and the way they end up cloistered away from the community. It creates a "thin blue line" mentality. I would like to see a grassroots approach, at least as an experiment.