MySQL 5.0 does not reject the surrogate characters between U+D800 and
U+DFFF. This means we can store characters above the BMP either by setting
the character set to UTF-8 and inserting CESU-8, or by setting the character
set to UCS-2 and inserting UTF-16.
It might be a hack, but at least it's a standard hack. After all, this is
exactly what UTF-16 and CESU-8 were designed for. According to a certain
online encyclopedia that we all know and love, the exact same problem is
observed in Oracle, and CESU-8 is used to solve it.
Implementing it in MediaWiki would be a similar task to implementing
PostgreSQL bytea support, which I was talking about in #mediawiki the other
day. We could convert from UTF-8 to CESU-8 in Database::strencode(), and
decode by altering the result rows returned by mysql_fetch_object() before
they are returned by Database::fetchObject().
Binary data would have to be flagged both on write and read. On write, we
could use a function other than strencode() where there is a need to
construct raw SQL, and for the wrapper functions we could pass binary data
in a blob object. When reading from the database, we could probably use
mysql_field_type() or mysql_field_flags() to detect binary columns and skip
the conversion accordingly.
It'd probably be easiest to do conversion offline, in the process of
switching to MySQL 5.0. We wouldn't be converting the bulk text, just the
metadata.
-- Tim Starling
