There's no way they can be voted against, until
their three year term ends.
Even then, they don't lose the extra privs granted to them.
They don't lose the checkuser bit, but they do lose their seat on the
committee that handles complaints about checkusers, which is the
important thing.