I don't think "unloading" modules will ever be a thing. While ResourceLoader could be made to keep track of modules' CSS to be able to remove it, we can't "unload" JavaScript modules at all (because their code has already executed), so I'm not sure if we'd want to add such special case.
If you want a slightly less horrible hack than you do now – you could generate an appropriate load.php URL yourself (with &only=styles) and add that in a <link> tag to the page, much like we did back in the old days with importStylesheetURI(). ResourceLoader won't even know the module is loaded. And this way, you can remove the <link> tag later without messing with ResourceLoader's internals.