As far as I understand drupal's interaction with the database (mysql), it always sends and expects data in UT8 format, but without specifying a character set in the database connection. This means that if the database server's is not set up with utf8 as the default encoding, then the data will not be displayed properly when looking directly in the database, although it does work fine when accessed back from drupal. More importantly, this makes it difficult to import existing non-ascii data in a way that will be displayed correctly by drupal, since you have to "lie" about its character set (ie convert data from latin1 to utf8, then ut8 to binary).
Ideally, since drupal is handling utf8 data, it should set the connection character set to utf8. Is there a deliberate decision not to do so? Of course, the transition to such a system will need to be handled carefully to preserve the data, but I hope this could be done automatically. The resulting situation would be much cleaner thant the current one.
Comments
Comment #1
killes@www.drop.org commentedThe reason not to do so is support for mysql 3 which has formally no utf-8 support. I don't know when we will drop mysql3 support, but at that time, this should be addressed.
Comment #2
dbr commentedI see, thanks for the answer. I guess it would be possible to start adding mysql4 as if it was a separate database (alongside postgre and mysql3), but I'm not sure if that's worth the trouble.
Comment #3
dbr commentedComment #4
dbr commentedWith Drupal 4.7 moving to utf8, I guess this issue is solved. Anyone confirms?
Comment #5
Steven commentedComment #6
(not verified) commented