We have just updated our guide on transferring data between wikibase instances. It is now complete. https://wikibase.consulting/transferring-wikibase-data-between-wikis/

That does not allow you to create new entities with a specific ID via the API though. Wikibase definitely supports creating such entities internally. We have some scripts for our customers that do exactly that. It seems like you already figured out where changes would be needed, and I agree that introducing a new option with a default value that does not change the current behavior makes sense. Some extra work would be needed to ensure creation of new items where no ID is specified (like on Special:NewItem) still works. But overall quite feasible. Still, someone has to make it happen :)

As to custom IDs (no Q or P prefix), there now is a phabricator ticket: https://phabricator.wikimedia.org/T271723

Best

--
Jeroen De Dauw | www.EntropyWins.wtf