CSS Optimization breaks with non-UTF-8 .css files

Same problem here. Thanks for posting it, but unfortunately using the old version of the function doesn't fix my situation.
Disabling CSS optimization partially solves the problem, as some style sheets are loaded, while others are not, but at least the site is usable.

Same problem...

The problem seems to be a combination of non-latin chars in CSS comments and the PCRE configuration.
I am able to reproduce on one server where a CSS comment has an umlaut , i.e.

/* Anfänge */

With that line in the file, this yields an empty String:

<?php
$contents = preg_replace_callback(
        "<$double_quot|$single_quot|$comment>Sus",  // Match all comment blocks along
        "_process_comment",                         // with double/single quoted strings
        $contents);                                 // and feed them to _process_comment().
?>

EDIT: correction - the problem was that the CSS file was not UTF-8 encoded, but in ANSI/Latin1. Converting the file to UTF-8 resolved the issue.

The same problem

Subscribe...

I can confirm that the regular expression fails when non UTF-8 characters are fed to it. Not sure why we need the Unicode modifier here. It seems that it might be an oversight.

Bumping to D7 for further study.

I've passed all my ANSI css files to UTF-8 and it works perfectly. Thanks husdhof.

While this comments tlaking anout Drupal 7, the same Problem exist under Drupal 6.
Solution here is also to convert the CSS-Files to utf8-format.

been having the same problem since updating to D6.19 ... subscribing

D7 patch attached. Haven't tried, but I suspect it'll apply to D6 too. It basically removes the u modifier from the regular expression. There may be a reason that the pattern had it in the first place, though…

Tracking.

adding tag

Same problem in Drupal 6.19 after update!

If I enable the Css Compression in "Performance" page, then my style.css file result blank!!! :(

The problem is cross-browser: tested on IE, Firefox, Opera

Dret: disable CSS optimization to fix the problem quickly. Then, re-save your CSS files in utf8, or try the patch above which Garrett Albright notes may work for D6.

It's marked as need backport to D6, so you don't need to change the version.

Ok ... so the "u" modifier was first added here http://drupal.org/node/444228#comment-2625578 without any explanation (plenty of explanation about the "s" modifier though).

In fact there is no mention of it anywhere in that entire issue... That said, it doesnt look like it was added as a result of an obvious problem that is caused without it.

I've tested the patch in #12 on both D7 and D6.19 and I cant see any problems. I can say that this does fix my borked (read missing) style.css from the aggregated css file

subscribe

I can confirm the problems mentioned in #4 with german "Umlauten" "Ä,Ö,Ü" after i have replaced them, everything was fine... thank you!

I solve the problem saving the css in utf-8 coding.

Now the compression works fine again.

I've got way too many css files across multiple sites to save each of them in utf-8 coding. I tried to apply the patch but in 6.19 but it failed. I reviewed the patch manually and don't see the corresponding lines in the common.inc file that should be edited (thus explaining the failure).

Edit: I also tried manually editing the common.inc per the code in the OP and that had unexpected results (part of the page displayed correctly, part not). Also, in my case on one site I did try to save as utf-8 and I still had issues.

@Lloyd just to clarify, the patch in #12 was created against D7 and does not apply against D6 ... once this gets committed to D7 we can create a patch for D6.

Making the change manually should have worked. Can you try these steps:
1) change that one line of code in common.inc manually
2) turn off css aggregation
3) clear cache
4) turn aggregation back on
5) clear cache
6) make sure you hard refresh your browser so you dont get locally cached css files

I really sounds like this is working, but your getting some cached css somewhere.

Sorry for the misunderstanding. As soon as I looked at the patch manually it was obvious it wasn't for D6.

And yes, when I made the changes to the one line in common.inc that needed to be changed everything worked just fine. I thought I posted that back here but I guess I didn't save it or something. But yes, easy fix.

D6 patch (untested, but it's pretty much the same change as the D7 one…), plus a new D7 one which makes the comments line up again whitespace-wise.

White space fix in D7 patch == cool beans.
This patch still works as advertised and I still cannot find any adverse effects to removing the "u" modifier.

The D6 patch == highly neato.
As with D7 patch this change solves the aggregation problem with no visible side effects. Whitespace looks good too.

RTBC++

Did anyone test with CSS containing multibye UTF-8 sequences? Rather, Did anyone analyze the preg whether it'll break? Since when does Drupal support non-UTF-8 anything? There are heavy questions to be answered.

I reverted the patch, then typed some Japanese into a CSS file and saved it as Shift-JIS. The same issue occurred. Saving the file as UTF-8 (hence a multibyte UTF-8 sequence), or re-applying the patch, fixed the problem as expected.

Since when does Drupal support non-UTF-8 anything?

Well, apparently we did before #444228: Optimize CSS option causes php cgi to segfault in pcre function "match" made it into core. Now we have people with broken sites. If we don't want to support non-UTF-8 anything, we can just ignore fixing this issue in D7, but I think we at least need to fix this in D6.

Here's a quick way to look through your site to find and type your css files if you want to fix this by moving them all to utf-8. First create a file that holds a list of all your css files. In your Drupal root:
find . -name *.css -fprint css_list
This causes the find command to traverse your site and write the path to each css file into the file css_list. Then use the file command to produce a list of the files with the encoding shown:
file -i < css_list
Then you can re-encode as needed. I only needed to re-encode the files in my custom theme.

Thanks for the suggestion. For me,

file -i < css_list

doesn't work. The css_list file definitely contains all of the css files, but that syntax doesn't work on my server.

Any suggestions?

Also, to confirm, do we need to convert into UTF-8 WITHOUT BOM, or just UTF-8? Or does it not make a difference?

I'd still like to see a better explanation as to why this change fixes the problem.

Can someone point us to the documentation for 'u' on php.net?

subscribe

@dries: I cannot find much...
http://www.phpro.org/tutorials/Introduction-to-PHP-Regex.html mentions the "u" flag towards the very bottom of the page

I also found this info being quoted from PHP.net but I cant find it on php.net:

u (PCRE_UTF8)
This modifier turns on additional functionality of PCRE that is incompatible with Perl. Pattern strings are treated as UTF-8. This modifier is available from PHP 4.1.0 or greater on Unix and from PHP 4.2.3 on win32. UTF-8 validity of the pattern is checked since PHP 4.3.5.

Other than that I'm stumped to find more docs...

I don't see it documented so we can only conclude from comments and observed behaviour:

http://www.php.net/manual/en/reference.pcre.pattern.modifiers.php, comment by hfuecks (emphasis added) describes behaviour:

1. If the pattern itself contains an invalid UTF-8 character, you get an error (as mentioned in the docs above - "UTF-8 validity of the pattern is checked since PHP 4.3.5"

2. When the subject string contains invalid UTF-8 sequences / codepoints, it basically result in a "quiet death" for the preg_* functions, where nothing is matched but without indication that the string is invalid UTF-8

I've conducted some tests with invalid bytes in UTF-8 subject strings; preg_replace always returns an empty string on such subject strings.

Any byte > 128 that's not in a multibyte sequence is invalid in UTF-8. That means that ASCII/latin-1 encoded CSS files containing > 128 bytes result in an empty preg_replace result.

As the characters matched by preg_replace are all in the < 128 range where UTF-8 and ASCII/latin-1 correspond and none of those single byte sequences are a subset of multibyte sequences, the UTF-8 mode is not needed.

Removing UTF-8 mode restores old behaviour. I do however encourage to declare only UTF-8 supported. After all, the fact that "it works" is purely coincidental and other character encodings are not supported at all.

Subscribe.

Removing UTF-8 mode restores old behaviour. I do however encourage to declare only UTF-8 supported. After all, the fact that "it works" is purely coincidental and other character encodings are not supported at all.

Your idealism is admirable, but we've broken peoples' sites with a point release. I reiterate my above idea; perhaps we can remove the "u" for D6, but leave it in for D7 and force people to convert their stylesheets to UTF-8 encoding when they upgrade.

Also, note that we haven't heard of anybody complaining about this same problem happening with JS aggregation…

Heine is just saying that because we remove 'u' doesn't mean that we support non-UTF-8 stylesheets. We don't.

Let's get #25 in.

@Garrett, as DamZ points out; I'm in favor of restoring the old behaviour, which is why I went to the trouble to explain why we can remove the 'u' modifier without any expected problems.

The old behaviour then "works" again, but is still not supported (there's a difference between those).

After I upgraded to 6.19, my stylesheet is still loading, however, my background color is not rendering anymore... Showing as white even though it's defined as #413d34. Website is:
http://www.bailliegrohman.com/test/
Again, no other issues with CSS loading properly - only the background color.
Any help would be greatly appreciated as the people who sold me this theme won't respond to my e-mails. Not sure if it's the CSS file not compatible with the new upgrade etc.
Thanks!
Attached the CSS file.

Looks correct to me. Maybe a cache problem? But the correct background is definitely showing.

This issue is about a specific, confirmed bug, not about all random CSS problems you may encounter. Please keep it focussed.

Committed to CVS HEAD. Moving to Drupal 6.

Subscribe

I ran the D6 patch and there was no change. Cleared Cache and no change. I opened the file and verified the lines edited by the patch.

Is there another step I am missing? If not, can you redirect me to a beginner tutorial on how to convert the css files to utf-8? I have seen a line referring to UTF-8 at the top of a regular HTML file, but not a css file.

Ashford: Are you sure the patch was applied correctly? Try applying it again - if you don't see a message saying "it looks like this patch was already applied," then it probably wasn't applied correctly the first time.

As for how to convert the CSS files manually, it will vary depending on which text editor you're using… Surf the menus (presuming you're using a GUI one) for an item that says something like "Text encoding." Most passable text editors will have an option like this.

Converting css files to utf-8 fixed my webs

I fixed #29 (finding non-utf8 files) like this:

find . -name *.css -exec file -i {} \; |  grep -v us-ascii | grep -v utf-8

note: my utf-8 converted files appeared as us-ascii but still were utf-8.

If there are too many files to handle, advanced users could play with line above and this one

ex some_file "+set ff=unix fileencoding=utf-8" "+x"

Subscribe

I've removed offtopic comments from this issue and created a FAQ page. If you have trouble with CSS and require support, post a new forum topic. Only post here to comment on the pending patch.

@Ashford; regarding "nothing changes"; are you sure you do experience the problem described in this issue, not some other problem?

"@Ashford; regarding "nothing changes"; are you sure you do experience the problem described in this issue, not some other problem?"

Sorry for the delay in replying. I have been out of town and unplugged.

Before leaving town I put the site back to the saved backup of Drupal 6.17. It will be a couple of weeks before I have time to try the upgrade and the patch again. I am a volunteer webmaster for a nonprofit group, so I only get to work on the web site as time permits. I will report back as soon as possible.

------------------------------------------------------------------
12/7/2010 update
reference: http://drupal.org/node/905140#comment-3743416

We are moving our site to another web host. I have a test site running now. I did figure out how to convert the files using my editor, HTML-Kit.

I had this bug in Drupal 6.16 and applying #25 fixed my problem. (thanks!)

Heavy bug which cost me a lot of time to fix.

I just removed the UTF-8 signs (especially German ä,ü,ö,ß) and now everything works fine again.

When will the problem be solved in D6?

#25 fixed this for me as well on Drupal 6.19.

subscribing

subscribing

subscribing

subscribing

Sadly neither the D6 patch, nor trying to convert ASCII css files to UTF-8 (they have no multi-byte characters so both are the same) worked for me. Am running with CSS optimisation off for now.

@Fonant, then you likely did not encounter the bug here, but another issue with CSS aggregation.

subscribe

subscribe

puh - hours of research -> you should warn, warn, warn at a prominent place - this is near to a bug and typical drupal time waste ...

re#61
Whiny responses like yours don't help nobody. Did you use the patch? Did it help? See #48!

had this same issue on a 6.14 and 6.19 site.
I changed it from fast/cgi to mod_php and the issues went away.
Not sure if this will help anyone.

Thanks, committed. This will be in the upcoming Drupal 6.20 release.

Hi All,

We are dealing with this issue on a 6.22 site, when we turned on CSS compression the sites all die where the CSS is there but none of the browsers are able to read it in. Was this fix actually rolled into 6.20?

Cheers,
Gene Bernier

CSS is there but none of the browsers are able to read it

this is a different issue.

Check if the CSS file is really created on the disk, you can be having permissions issues.

I found a similar problem was caused by a un-closed comment at the bottom of a CSS file, causing everthing below it in the aggregated file to be ignored.

I'm closing this issue as it has been fixed.

A problem is not 'similar' just because it has CSS in the title.