JsMin: Browser console gives errors like SyntaxError: illegal character

I have not experienced what you're describing with console errors, thus it is unknown to me. The stack overflow code is used in our outbound XML service; we were getting some XML parsing errors on the receiving end due to copy paste characters getting in text fields. That stack overflow code is great for testing things as it can generate every single UTF-8 character and it can also filter out any UTF-8 character you want to.

I already filter out BOM

How can I repo this bug?

Look at this issue for inspiration? #2576381: CSS aggregation is broken by files encoded in UTF8 with BOM

I've got this issue on my site at the moment, using advagg version 7.x-2.16.

I seem to get the 'ILLEGAL character' at the end of files that are being concatenated together when using the jsmin C extension.

On this specific file, I've got a semi-colon and another character:

;

Hex:

3B 0A

But in the aggregate that the file is built into I have:

;;;
/* S

Hex:

3B 01 0E 3B 3B 0A 2F 2A 20 53

(I've included up to the added source licensing comment)

Oh, so upgraded to 2.17 and now have some of these in my log:

The jsmin function does not handle cyrillic characters. Using JSqueeze.

:(

I'll check installed versions of jsmin, I'm trying to use the C extension version.

Yeah, jsmin 2.0.1.
They had gone away, but just flushed all caches, and the weird characters are back!

Ran into this myself on a production site (D8) that I was updating AdvAgg on which already had JS minification active. I couldn't determine any reason it changed nor why it didn't show up on some of my test sites. However did confirm that is is not directly in AdvAgg - both JSqueeze and JSMin (2.0.1 at least) can do it including when called from a minimal PHP test script.

So added an extra check & fix to 8.x-2.x and back ported that to 7.x-2.x.

actually doesn't catch all cases...

There that gets all cases I've managed to generate.

New 7x patch.

Awesome! Thanks for the patch!

Seems to be working for me. Thanks!

Guessing we should try and get this fixed upstream in jsmin?

Fix for js ending with }.

I wonder if we cutoff anything after the ; or } for every file, as in get rid of the if statement from the patch in #28.

Browsing around other js file and I do see some of them also end in ). Here's the idea from #32

Guessing #34 fixes it?

#34 has been committed

@osopolar
Let me know after a week or two if the patch in #40 is still working. It's a little risky but it should solve the issue, thus why I'm waiting before committing it.

@osopolar
Is #40 still good?

Hi all and thanks for the patch :),

I have updated from 2.17 (stable) to latest dev version due to this issue (unsuccessfully), the patch in #40 fixes it.
Moving to Major, as JS is broken on pages with advagg enabled.
Marking #40 as RTBC as it is a working fix.

I think this needs a stable in near future, as the 2.17 broken when using recommended settings (JSMin).

BR, Nikolay Dobromirov.

@ndobromirov
Thanks for the feedback! Jsmin issue is here: https://github.com/sqmk/pecl-jsmin/issues/46

I've committed #40 with the addition of linking to the github issue as well.

hi,
when using curl "patchurl" | git apply -v

i'm getting this error

Checking patch advagg_js_compress/advagg_js_compress.advagg.inc...
error: while searching for:
return;
}

// Under some unknown/rare circumstances, JSMin and JSqueeze can add a
// couple of extraneous/wrong chars at the end of the string. Check and
// remove if necessary.
if (substr(trim($contents), -2) == "\xE3\x7F") {
$contents = substr($contents, 0, -2);
}

// Ensure that $contents ends with ; or }.

error: patch failed: advagg_js_compress/advagg_js_compress.advagg.inc:364
error: advagg_js_compress/advagg_js_compress.advagg.inc: patch does not apply

what I'm doing wrong?

@andsigno82
Patch was rolled from dev, you're trying to patch 7.x-2.17. This has been committed (#44) so using the latest dev will have this patch already applied to it. Reason this conflicts is I committed a fix that worked in most cases in #24 and then #39. So to get this patch to apply to 2.17 you'll need to apply #22, #34 and then #40.

I installed the latest dev version but the problem persists. I'm getting this error:

Uncaught ReferenceError: V is not defined

And this is like the offending code looks like:

$('form.ctools-auto-submit-full-form',context).add('.ctools-auto-submit',context).filter('form, select, input:not(:text, :submit)').once('ctools-auto-submit').change(function(e){if($(e.target).is(':not(:text, :submit, .ctools-auto-submit-exclude)')){triggerSubmit.call(e.target.form);}});var discardKeyCode=[16,17,18,20,33,34,35,36,37,38,39,40,9,13,27];$('.ctools-auto-submit-full-form input:text, input:text.ctools-auto-submit',context).filter(':not(.ctools-auto-submit-exclude)').once('ctools-auto-submit',function(){var timeoutID=0;$(this).bind('keydown keyup',function(e){if($.inArray(e.keyCode,discardKeyCode)===-1){timeoutID&&clearTimeout(timeoutID);}}).keyup(function(e){if($.inArray(e.keyCode,discardKeyCode)===-1){timeoutID=setTimeout($.proxy(triggerSubmit,this.form),500);}}).bind('change',function(e){if($.inArray(e.keyCode,discardKeyCode)===-1){timeoutID=setTimeout($.proxy(triggerSubmit,this.form),500);}});});}}})(jQuery);;~	V;

If I disable the JS compression it works fine.

I deleted manually the folder sites/default/files/advagg_js and now it is working. I thought it should be deleted when clearing caches.

@maxtorete ([#49]) - no that is not normally cleared, although AdvAgg does provide an easy way to clear it - under the settings - admin/config/development/performance/advagg/operations under 'drastic operations'

@maxtorete
Clearing the caches does not delete files; you have to go to admin/config/development/performance/advagg/operations and under Drastic Measures click the button under "Clear All Files". This is something that is usually not needed.

@osopolar
File in question - http://cgit.drupalcode.org/devel/tree/devel_krumo_path.js?h=7.x-1.x. Super strange that ;); was added to it; seems like a parser bug in jsmin.

@mikeytown2: Where you able to reproduce the problems with JSMin?

I've never had this issue so I'm shooting in the dark.

@osopolar
Sounds like that is something I could check for on the status report page... good to know. We're using CentOS and you're using Ubuntu; this could explain a lot.

@osopolar
what happens when you run locale on your sever? I get everything coming back as en_US.UTF-8.

Here's some code that should detect if the locale is not UTF-8

  if (strtoupper(substr(PHP_OS, 0, 3)) !== 'WIN') {
    $return = shell_exec('locale');
    $settings = parse_ini_string($return);
    foreach ($settings as $key => $value) {
      if (strpos($value, 'UTF-8') === FALSE) {
        echo "$key = $value is not UTF-8.<br>\n";
      }
    }
  }

Haven't tested it yet; been focused on #2451801: Empty JS on multilingual's default language pages; sometimes binary (gzip) or 504 as well. 307 issue.

Hi again,

The fix from 40 was working until today.
After a new code was deployed to a staging environment the issue is back with different broken symbols.

The script from #60 outputs the following on that server:

  LANG = C is not UTF-8.
  LC_CTYPE = C is not UTF-8.
  LC_NUMERIC = C is not UTF-8.
  LC_TIME = C is not UTF-8.
  LC_COLLATE = C is not UTF-8.
  LC_MONETARY = C is not UTF-8.
  LC_MESSAGES = C is not UTF-8.
  LC_PAPER = C is not UTF-8.
  LC_NAME = C is not UTF-8.
  LC_ADDRESS = C is not UTF-8.
  LC_TELEPHONE = C is not UTF-8.
  LC_MEASUREMENT = C is not UTF-8.
  LC_IDENTIFICATION = C is not UTF-8.
  LC_ALL = is not UTF-8.

ndobromirov
What's your linux distro and its version?

cat /proc/version

  Linux version 2.6.32-504.12.2.el6.x86_64 (mockbuild@c6b9.bsys.dev.centos.org) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-11) (GCC) ) #1 SMP Wed Mar 11 22:03:14 UTC 2015

cat /etc/*-release

  CentOS release 6.6 (Final)

Using PHP 5.5 with the REMI packages, nothing is compiled from source, so nothing is that custom.

@ndobromirov
What do you get when you run locale -a?

Hi it appears that it's utf8 and not UTF-8. See in the attached file (ndobromirov-locale--a-info.txt).

Good to know! My server is also CentOS 6.6, so it seems the UTF8 naming scheme is caused by something else.

  if (strtoupper(substr(PHP_OS, 0, 3)) !== 'WIN') {
    $return = shell_exec('locale');
    $settings = parse_ini_string($return);
    foreach ($settings as $key => $value) {
      if (stripos($value, 'UTF-8') === FALSE && stripos($value, 'UTF8') === FALSE) {
        echo "$key = $value is not UTF 8.<br>\n";
      }
    }
  }

Have a proof of concept code for checking the required min php version and the current jsmin version on github.

  $request = drupal_http_request('https://cdn.rawgit.com/sqmk/pecl-jsmin/master/package.xml');
  $string = $request->data;
  $dom = new DOMDocument();
  $dom->loadXML($string);
  $items = $dom->getElementsByTagName('php');
  foreach ($items as $item) {
    echo $item->nodeValue . "\n";
  }
  $items = $dom->getElementsByTagName('release');
  foreach ($items as $item) {
    echo $item->nodeValue . "\n";
  }

I gave a go at this with xpath queries but I didn't have any luck; I think this will work for now.

So now it's only this:
LC_ALL = is not UTF 8.

I am with PHP 5.5 so I do not think it's PHP's version-related.

Is it possible the internal file encoding to cause the issue?
see this from the JSmin Home-page (http://crockford.com/javascript/jsmin):

Character Set:

JSMin requires, but does not verify, that the character set encoding of the input program is either ASCII or UTF-8. It might not work correctly with other encodings.

So if some of the source files internal encoding in the aggregates is different than UTF-8 it will be undefined/invalid input for JSMin.
I will try to see whether this is the problem is my case sometime today or tomorrow and keep you posted.

Recently discovered http://php.net/setlocale. Does this fix the issue for people who are having jsmin issues?

Here's the status report page patch.

Committed part of #73; the locale parts are now in this patch.

I need someone with this issue to test #72 and advagg-2627468-74-status-report-locale.patch in #75. Looking like once I get #2376849: ReferenceError: krumo is not defined & fix js errors inside of krumo.js done I'll be tagging the next release of AdvAgg; I would like to include a better fix for the jsmin issue if at all possible.

Hi, This is very strange...

Last time, I was able to reproduce this on all environments (local, integration, stage, life).
I can not reproduce this anywhere now, two weeks later, JSMin works... JMin is still 2.0.1. (4 month old stable release).

No changes were done on the server configurations, just some new code was deployed / developed.
It is custom module logic in pure PHP (cron process), in no way related to JS resources handling.

Keep you posted if this comes up in future and if the patch in#72 & #75 have any effect on fixing it.
At the moment everything works without the patch. Code is in the same state as it was in #40.

I was experiencing these "Uncaught SyntaxError: Unexpected token ILLEGAL" (Chrome) and SyntaxError: Invalid character '\u0008' (Safari) errors after moving a site from an Apache server to NGINX on Ubuntu 14.04. I was able to fix the problem by downgrading the PECL JSMin PHP extension from version 2.0.1 to 1.1.0. The PECL release notes mention adding UTF-8 support in version 2.0.0:

https://pecl.php.net/package-changelog.php?package=jsmin&release=2.0.0

@noahott
If you go back to using 2.0.1 and apply #72 does that fix the issue for you? If not what does the status report say when you apply the 2nd patch of #75 (advagg-2627468-74-status-report-locale.patch)?

I was really hoping that setlocale would have fixed it. Changing the servers locale seems like a non starter so at this point I'm thinking of commenting out the latest github version check and pushing out a new advagg version.

This code has been removed from the advagg_js_compress.install file as the 2.0.1 version seems to cause issues so recommending that people upgrade to it would be a bad idea at the moment.

  // Check locale and jsmin version.
  if ( function_exists('jsmin')
    && defined('PHP_VERSION_ID')
    && constant('PHP_VERSION_ID') >= 50310
  ) {
    // Check versions.
    $request = drupal_http_request('https://cdn.rawgit.com/sqmk/pecl-jsmin/master/package.xml', array(
      'timeout' => 10,
      'headers' => array('Connection' => 'close'),
    ));
    if (!empty($request->data) && $request->code == 200) {
      $string = $request->data;
      $dom = new DOMDocument();
      $dom->loadXML($string);
      $items = $dom->getElementsByTagName('release');
      $github_version = 0;
      foreach ($items as $item) {
        if (!empty($item->nodeValue) && preg_match("/[0-9]+/", $item->nodeValue)) {
          $github_version = $item->nodeValue;
          break;
        }
      }
      $jsmin_version = phpversion('jsmin');
      if (!empty($jsmin_version) && !empty($github_version) && version_compare($jsmin_version, $github_version, '<')) {
        $requirements['advagg_js_compress_jsmin_version'] = array(
          'title' => $t('AdvAgg JS Compressor'),
          'severity' => REQUIREMENT_WARNING,
          'value' => $t('jsmin should be upgraded.'),
          'description' => $t('There is a newer version of <a href="@url">jsmin</a> available. Server version: @server, Git hub version: @github', array(
            '@url' => 'https://github.com/sqmk/pecl-jsmin',
            '@server' => $jsmin_version,
            '@github' => $github_version,
          )),
        );
      }
    }
  }

Similar to @noahott in #79, running Nginx with Ubuntu 14.04 and PHP 5.5. Had to downgrade from PECL JSMin 2.0.1 to 1.1.0, otherwise JS Compression is broken trying to use JSMin.

@osopolar, I'm using 7.x-2.18. It's odd... JS compression was causing errors all over the site, so I downgraded to JSMin 1.1.0 and that fixed things. Now upgraded back to 2.0.1, and the site's JavaScript problems are no longer present.

Still getting errors at the command line when running drush updatedb or drush cc and a full cache clear is being done:

[notice] WD advagg_js_compress: The jsmin function does not exist. Using JSqueeze.                                                                                        [notice] WD advagg_js_compress: The jsmin function does not exist. Using JSqueeze.                                                                                        [notice] WD advagg_js_compress: The jsmin function does not exist. Using JSqueeze.                                                                                        [notice] WD advagg_js_compress: The jsmin function does not exist. Using JSqueeze.  

Downgrading to JSMin 1.1.0 fixes the command line error. I'll have to monitor and will post an update if I start seeing the JS errors return. (or maybe the JS errors won't return because the site is actually using JSqueeze at this point as a fall back?)

Some additional information... I just put a few dpm's in advagg_js_compress_prep to see what is happening. If I have JSqueeze as the chosen compressor in /admin/config/development/performance/advagg/js-compress, I get that $compressor = 5 and function_exists('jsmin') = TRUE (as I would expect).

However, if I set JSMin as the chosen compressor, it looks like advagg_js_compress_prep is never run? I never get any result that the function is called. I haven't tracked up to see what might be causing that result, but just thought I would mention.

Edit: the function is being called, just looks like it required several cache clears to get it to run.

Yeah was going to say that I cache css/js compression pretty heavily as they are expensive.

Done a rather large degree of testing, manipulation and smashing my head into brick walls.
My results are so far not incredibly nice:
- yes the JsMin PHP extension has the problem without Drupal/AdvAgg
- it is close to random - at least I can't find any specific factor that ties them together
- The most freakish: sometimes just re-running on the same file directly after itself will fix it after 2 iterations or 4 or x.
- It can actually add pretty much any number of characters less than 5 - and they can be any character pretty much.

Despite those nasty results I have figured out changes to the post processing - a bit yucky but it works in all cases I've so far found/tested. I'll pop up a patch for 7x too shortly.

Actually, I missed commit 8a896b8 in the 7x branch, @osopolar, did that sufficiently fix it for you?

@NickWilde, is there a 7x branch commit for this issue? I'm looking for 8a896b8 and not seeing it... can you post a link? Thanks!

#44, #40

Well going to postpone this for now, not much else we can do...

Hi as far as I can see the fix is to downgrade to JsMin 1.1.0. I am testing on a new project now with the old version - no problems so far.

It seems JsMin 2.0.1 generates this problem.
Downgrading to JsMin 2.0.0 solves the problem: no more "illegal character" errors.

I tried to downgrade to JsMi 2.0.0 but the "Uncaught SyntaxError: Unexpected token )" is stil there (and breaks views ui).
Chrome shows that the error is in my case in views-admin.js.
Looking at the compressed code, it seems that jsmin is making up some characters:

Original code:

jQuery(function() {
  jQuery(window).bind('resize', Drupal.viewsUi.resizeModal);
  jQuery(window).bind('scroll', Drupal.viewsUi.resizeModal);
});

Compressed code:
jQuery(function(){jQuery(window).bind('resize',Drupal.viewsUi.resizeModal);jQuery(window).bind('scroll',Drupal.viewsUi.resizeModal);});;);;

Notice the end of the compressed code.
Any workaround?

I had a similar problem on some files with JSMin 2.0.1, and it had two different origins :

• Some non visible characters (non-breaking spaces instead of regular spaces, made with "ALT-GR + SPACE" by developpers). These characters forced JSMin to fallback to JSSqueeze. Browsers seem to ignore non-breaking spaces and treat them as normal spaces, but JSMin didn't liked them.
• "Uncaught SyntaxError: Unexpected token )". The javascript code came from an external lib (social-likes), and has code like this :

            if (rect.top < offset)
                elem.css('top', offset - rect.top + top);
            else if (rect.bottom > window.innerHeight - offset)
                elem.css('top', window.innerHeight - rect.bottom - offset + top);

Even if this code is valid, JSMin does like the fact that there is no "{" and "}" after "if" and "else".

Maybe it can help...

Related #2896671: UTF-16LE encoded js files are not being added to aggregates correctly; mixed encoding issue.

I'm thinking this issue is the result of non UTF-8 js files not being converted correctly. I'll need to search thought the code base and have a wrapper for where file_get_contents is; making sure the encoding is correct.