The problem isn't with the template itself, but the way it's used. Somebody (possibly a script or bot) will add the template wherever they see ==Trivia==, and then somebody else (possibly another script or bot) will patrol the relevant category and blank the section, without any attention as to whether the content is related to that of other sections, or properly referenced, or outright remarkable.
If people/bots are doing that, they should be stopped. Last time I read the appropriate policy it said to integrate trivia into the rest of the article, not remove it.
"First they came for the spoiler warnings, but I didn't care because I thought plot sections shouldn't be labelled. Next they came for the trivia sections...."