Right now, we build some of that JS based on configuration parameters that are needed by the WikiLabels service back-end. It seems that it would be easiest to generate those files and cache them so that we can serve with them extension code.
I'm not sure how much sense this makes long-term. If the answer is "no sense -- at all", then I think our next step would be getting a good description of which parts of that JS are configured and then working on a proposal for how we'd change things within the extension. I can do the description if we get that far.