Non-english characters are lost with UTF-8

This is due to the content for form items being filtered in some way.
If you look, it breaks on the preview form too.
I'm off to patch this up right now. Will keep you posted.

Right, the issue is that the form output is going through check_form() which calls drupal_specialchars().
This in turn calls htmlspecialchars() (complete with "utf-8" charset argument). However, despite what the PHP manual[1] says, this is messing up the unicode. If I call htmlspecialchars() _without_ the "utf-8" charset argument, it actually seems to work fine, which is deeply stupid. Am investigating if this breaks Korean... will post patch shortly.

[1] http://www.php.net/manual/en/function.htmlspecialchars.php

Doesn't break Korean. Yay.
Patch is in contributions/for_review (0033).
Hopefully this should sort things once and for all.

I can _not_ reproduce this behavior (without Al's patch). I'm using PHP 4.3.2 with Apache 1.3.27. I tried with both Opera and Firebird so this is not a browser issue.

I've just noticed this very same behaviour (all my German Umlauts where gone from both node content as well as locale translations) after updating CVS. Digging down I noticed that the variable site_charset (used to default to "iso-8859-1") was replaced with hard-coded "utf-8" throughout includes/common.inc when this file was updated to r1.213. The log for revision 1.213 (date: 2003/06/08 16:50:04) says

- Charset simpliciations. Patch #46 by Al.

Al, could you please put the original (pre-1.213) behaviour back in?
--
Olav

Ah... It's a bug in PHP.
I think the PHP boyz have bugfixed things for 4.3, but on older versions, calling htmlspecialchars() with a "utf-8" charset rewrites things.

This is why Dries can't reproduce it, but I can on my dev box.

If we just call htmlspecialchars() without the "utf-8" argument, everything works fine for German and Korean. It *might* make Japanese behave less sensibly, but I have no way of testing that.

Dries - please apply the patch - it doesn't do any harm, and we don't want to make PHP 4.3 a requirement for Drupal 4.2... ing" alt="Smil
ing" />