In http://drupal.org/node/119441#comment-221260 and http://drupal.org/node/119441#comment-221572, while developing the patch to implement JS aggregation and whitespace compression, m3avrck changed the patch and said:

Also it fixes the JS files so that they use correction notation, e.g.,:

var foo = function bar() {
};

Need that ";" when you define variable functions like that.

Note, you only need those *optional* semi-colons when you're compressing (e.g., removing whitespace) from JS files, it prevents variables from running into each other by being defined on the same line.

In other words, the JS compressor breaks on legal Javascript. This is a mistake. Requiring valid JS to be changed to be compatible with the compressor is a wrong decision, especially since the $cache argument is TRUE by default (but even if it were not). Oh, and this new requirement is also not documented. It just cost me half a day of work to figure this out and not all JS/module developers will even be capable of grokking the packer compression logic to figure out what is going on. Not acceptable for core.

In my particular case, it was the D5 CCK Link module that has the close-brace not followed by a semi-colon. I'm sure the D6 version will, too, along with numerous other modules, and we should not force them to change.

I've attached a patch which fixes this problem by making sure a newline appears after close-braces. This patch is against D6 HEAD.

I predict this is not the last time that this JS compression code will bite us. I recommend removing it completely; just concatenating the files into a single request is the big win of this patch. But the patch to fix the particular bug I found is smaller and less likely to be controversial, so that is what I'm attaching.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

bjaspan’s picture

Forgot to add:

1. I marked this 'critical' because the bug allows valid JS in any enabled module to disable all JS from all modules. It's a pretty severe loss of function and most sites will have no way to debug or fix the problem.

2. There is a bug in _packer_apply(). See http://drupal.org/node/183871.

ChrisKennedy’s picture

I agree with your synopsis that it is better to remove this requirement, which adds significant burdens without much benefit.

I've tried the patch but from my testing it doesn't solve the issue, although if I just comment out the white-space removing line the original javascript does function correctly. Is this patch being held up by the _packer_apply() bug, or is my testing perhaps flawed?

My test module inserts this javascript code in an external file (middle case omits the terminating semicolon):

$(document).ready(function() {
  $("#header").prepend("<p>test1</p>");
});
$(document).ready(function() {
  $("#header").prepend("<p>test2</p>");
})
$(document).ready(function() {
  $("#header").prepend("<p>test3</p>");
});
bjaspan’s picture

My patch only ensures that close-braces are followed by newlines because that is the particular issue I encountered. Your demonstration code has a close-paren that needs to be followed by a newline, so my patch does not help with that. It would be very easy to fix my patch to include close-parens, but this is why I said:

"I predict this is not the last time that this JS compression code will bite us. "

Unless someone can say confidently that the compression code will work for ALL valid JS, I think we should yank it and just concatenate the files. And "valid JS" in this case has to mean JS that works in all popular browsers; we can't break working code that happens to violate the strict JS specifications unless we want people to blame Drupal for it not working.

moshe weitzman’s picture

is there a patch out there which removes the compressor?

dmitrig01’s picture

Why not add a semi-colon if one is missing?

bjaspan’s picture

#4: Sure, here is a patch that removes the compression while leaving the aggregation in place.

dmitrig01’s picture

Maybe have a setting for compression and another one for aggregation on the admin page?

moshe weitzman’s picture

Status: Needs review » Reviewed & tested by the community

IMO a new setting makes little sense. This feature simply isn't ready.

Gábor Hojtsy’s picture

I tried to drive some attention here by sending a mail to the development list. I agree separate settings for compression and aggregation are not a good idea, especially not to mask a broken feature. I am looking into other opinions on this before rolling this feature back though.

bjaspan’s picture

We could probably safely remove comments and eliminate empty lines and leading and trailing whitespace on lines, just not newlines. The question, though, is why bother? This is a static file that should only be delivered once to each client and then result in "Not Modified" responses until it next changes (assuming we have our cache headers set correctly; if we don't that is a separate and important issue).

Custom-compressing the aggregated JS introduces extra code to write, maintain, review, and debug and the programmer time already spent on it probably exceeds the total time that will ever be saved by users in downloading the compressed version versus the non-compressed version. And if that is not true, if the compression is actually providing important value, then we would be better served delivering a gzip-compressed file instead which would save even more space and be much more likely to be bug-free.

ChrisKennedy’s picture

I think the comment-removal might still be worthwhile. According to ohloh.net 38% of Drupal's code is comments - programmatically removing those is a pretty solid amount of savings. And with all the jQuery and AHAH functionality being added to Drupal, we will continue to see significant growth in Drupal's JavaScript codebase. Removing comments is different from gzip compression and will likely even enhance gzip's compression rate above and beyond the reduction in filesize (since comments and executing code probably have very different word distributions).

arthurf’s picture

While I have encountered the problems of javascript compression mentioned here already, I do think it is worth pursuing. On high traffic sites compression could make a significant difference. On a 5.3 site with the aggregation patch, the total JS went from 70k + to slightly under 50k. If you factor this out over the course of the month, it can be significant.

On the other hand, getting aggregation working required significant effort- lots of hacking on modules that'd I'd rather not have hacked and making my upgrade paths more dubious. On the other, it really improved my yslowscores :)

I think the bottom line is that if aggregation is kept for d6, we need to improve the packer's ability to handle the case mentioned in this post. We may want to change drupal_add_js's default behavior to not aggregate unless explicitly told to. That may help ease the transition.

hass’s picture

Uhh - is removing this compression code really the right decision?

I don't like this... compression is very cool stuff and developers like to develop a D6 module with JS should add some semicolons or set the pre-process param to FALSE... no big thing. Why removing this cool stuff if it only needs some minor thinking about JS compression... :-(((

bjaspan’s picture

The compression code is not cool in a production system if it is wrong. If it breaks legal JS, it is wrong. If we can fix it so we are confident it will not break legal JS, then I have no serious objection to it (I personally am not going to put more time into fixing it but if other people want to, fine with me---this is a volunteer project.)

Gábor Hojtsy’s picture

Note that I kept this RTBC, so that unless someone comes around with actual fixes for JS compression, this feature gets removed. No matter how cool this sounds, if we only talk about fixing it and never do it, we need to live without it.

catch’s picture

May well not be possible, but any way to leave in a hook so compression could be done in contrib? That way it'd get more testing during the D7 release cycle and could move back into core next version.
I'm assuming the answer will be 'no', but better to ask than not.

Crell’s picture

I just want to point out that "legal Javascript" allows dropping the ending semi-colon in the same way that "legal HTML" has half of its closing tags optional. That's been well-understood as very bad practice for a very long time now, and I don't know anyone in the JS community who doesn't encourage using semi-colons everywhere, even when not strictly required. Making the code compressor-friendly is one key reason for that.

So are we supporting "all legal Javascript", or are we supporting "Javascript not written stupidly?" It seems to me the proper solution is to consider JS code in a Drupal module that can't compress due to missing semi-colons to be a bug to be fixed.

hass’s picture

@Crell:

So are we supporting "all legal Javascript", or are we supporting "Javascript not written stupidly?" It seems to me the proper solution is to consider JS code in a Drupal module that can't compress due to missing semi-colons to be a bug to be fixed.

That's totally true. Ever time a developer create compression friendly JS code the developer MUST add semicolons. I know many projects outside Drupal that added them for compression compatibility only, because users requested this feature. JSMIN and other compressors will not work if code is not compression friendly, too. If the JS code is not compression friendly and i cannot write JS code myself, but add this to my D6 project - i could leave this out of compression - for Drupal - preprocess = FALSE.

Why not go this pre-process way?

hass’s picture

If we should remove this good JS compressor feature we must remove the CSS compressor, too. See http://drupal.org/node/139117 about a core bug that breaks the CSS compressor. And removing this CSS compressor would be a wrong decision, too.

Wim Leers’s picture

I tend to agree with Crell I think.

How do Javascript compressors in general handle this? I think it's best to do like the rest in this case.

In follow-up #10, bjaspan says that gzip compression is not yet included. I wasn't aware of that. I guess that's something for D7...

Wim Leers’s picture

Thanks hass, you already answered my question. Then I agree that Drupal should not worry about this either.

hass’s picture

Let's document this in the module and theme upgrade docs and do not commit this removal patch, please. I tend to mark this as CNW for documentation...

ms2011’s picture

coming in a little late in the discussion here, but wanted to be sure no one had overlooked the YUI Compressor + GZip solution as an alternative if Dean Edwards' Packer is causing trouble:
http://www.julienlecomte.net/blog/2007/08/21/gzip-your-minified-javascri...

nedjo’s picture

Priority: Critical » Normal
Status: Reviewed & tested by the community » Active

As a developer who probably maintains as much JS module code as anyone, I welcome the JS compression and the opportunity it presents for improving consistency in our Javascript codebase. "Valid JavaScript" is almost a meaningless phrase, as the language is so loosely defined. A few well justified and minimal standards on our part can only be for the good. Supporting compression is an important goal. Without compression, aggregation will increase overall download bandwidth (since the same code is downloaded once per unique combination of .js files on a page, rather than only once per site).

I've drafted the missing documentation, http://drupal.org/node/114774#javascript-compression. Is more needed?

RobRoy’s picture

I'd hate to see compression go and would like to see missing semicolons added to any offending JavaScript. I think compression is a valid feature and it works great. Compressing code and removing comments is great to reduce size in a heavy traffic environment. All you have to do is make your JavaScript a little stricter by adding some semicolons.

My $0.02.

hass’s picture

@nedjo: docs extension looks good to me. We should add this to themes http://drupal.org/node/132442, too. Let's change this case to fixed after this...

Owen Barton’s picture

I am not sure how I feel about this one yet. I don't like the route of having a massively more complex regexp (it is already complex enough!). I would be happy with adding separate 'aggregate' and 'aggregate and compress' options. I would be happy removing it and adding a hook so it can live in contrib.

I discovered some interesting test cases at http://js-crunchy.googlecode.com/svn/trunk/crunchy/tests.js

RobRoy’s picture

Having three options is fine with me: Disabled, Aggregate, Aggregate and Compress. As long as I can easily get the benefits of compression, I'm happy. And the new option sounds like it may be requested by the community. Patch anyone?

nedjo’s picture

Status: Active » Closed (won't fix)

Having three options is fine with me

The issue that's been identified is that some JS files may not comply with the new (minimal) coding standards. This should not be addressed through configuration--it's a code-level problem.

And we already have the solution in place. Developers who wish to use non-complying code can set the $preprocess argument to FALSE when calling drupal_add_js(). Problem solved so far as I can see, so marking this won't fix.

bjaspan’s picture

Status: Closed (won't fix) » Reviewed & tested by the community
FileSize
314.5 KB

I used the Mozilla Javascript Test Library (http://www.mozilla.org/js/tests/library.html) to test the Drupal JS compression code and found substantially more failures with compression than without.

Specifically, I ran the ecma, ecma_2, and ecma_3 tests against a current CVS build of Mozilla (I needed a debugging build in order to get xpcshell to run the tests; it was a bit of doing to set up the build system, etc.) First I ran the tests as they exist in CVS. Then I applied _drupal_compress_js() to each test .js file and ran the same tests again.

Without compression, 77 out of 954 tests failed. With compression, 159 out of 954 tests failed. I'm attaching the test results to this and the next comment.

Why did the extra tests fail? Are the bugs all related to missing semi-colons, or are there other problems with the compression? I have no idea and do not feel like taking the time to figure it out. IMHO, delivering a slightly larger aggregated JavaScript file to each visitor once each time the aggregated file changes is just not that much extra bandwidth and is not a worthwhile savings compared against the drawback of breaking an unknown quantity of valid code.

If someone wants to rewrite the compression patch just to strip comments and other really obviously safe things, I'll re-run the test suite to see how it does. Otherwise, I really think we should serve uncompressed aggregated JS.

Un-compressed test results attached. It's an HTML file renamed to be attachable.

bjaspan’s picture

FileSize
665.83 KB

Test results against compressed code attached.

RobRoy’s picture

Status: Reviewed & tested by the community » Needs review

I'm setting back to review as from reading the thread it sounds like we can safely keep comment removal and whitespace removal even if the other compression stuff is broken. But, I think it would help to come to some clear consensus on *what* exactly is broken about the compression. If it's just the semicolon thing, than I agree with nedjo and think it should stay in, but it sounds like there may be some additional error cases, they just haven't accurately been identified. Am I off base?

moshe weitzman’s picture

Status: Needs review » Reviewed & tested by the community

Barry did careful analysis and demonstrated that the compression is buggy. if someone wants to fix it, they are encouraged to do so. in the meanwhile, bugs need to be fixed or removed and thats what this issue currently does. setting it to 'for review' is not appropriate IMO unless one adds an improved patch at same time.

RobRoy’s picture

My point was, if all the "bug" reports that came from that analysis really just had to do with missing semicolons, then that would mean something different than if there are other valid problems with the compression. That is why I set it to review. It would be nice to see what specifically is broken. Maybe we can get some m3avrck love on this issue.

m3avrck’s picture

I agree with Nedjo and RobRoy.

This patch is removing something that really speeds up the load time of sites. I have been using this on MothersClick.com and MomBlogNetwork.com for 6+ months now, with absolutely no problems. My JS coders have had almost 0 problems with this, writing tons of Javascript.

The missing ";" while not there still produces runnable JS is incorrect and technically invalid. The jQuery teams STRESSES this in their docs for all plugins to conform to the full syntax, using all optional ";" for consistency. Since Drupal relies on jQuery and 99% of all JS written will be in jQuery, it makes the most sense to follow the jQuery standards and enforce this.

Just because you write bad JS code, why should we support your bad code in core? We don't do this for any PHP code and hooks, why should we start now with JS?

I'll see if I get John Resig to comment as welll...

Owen Barton’s picture

I think we need to analyse the test cases some more before we decide here. If it is simply the ';' issue then I think we can live with it - that is a pretty reasonable request to make to coders.

However if there are *other* actual bugs in the code (unless the 82 failing tests are caused by compression are all due to missing semicolons) then I think it is very important that we find and fix these (or remove/simplify the compression). Otherwise we end up with a bunch or weird edge cases that are a major pain to debug (try debugging compressed code if you don't know what I mean here!).

Anyone knowledgeable on javascript syntax standards want to help go through these test cases?

I had a quick look at the test logs and it does look like there are a lot of errors coming up with "SyntaxError: missing ; before statement:". There were other errors though, like "SyntaxError: missing variable name:" - I haven't checked to see if those are in the set of existing failing tests, or if they are side effects of missing semicolons.

nedjo’s picture

Category: bug » feature
Status: Reviewed & tested by the community » Fixed

Barry's latest tests add nothing new but only reiterate his original note that compression won't support any old (valid) Javascript. Yes, we know this. Yes, we have introduced some minimal coding standards in Javascript. This is by design.

Is there a list of active Drupal Javascript developers who are not willing to accept minimal coding standards in Javascript? Are there errors that compression produces in our core Javascript files or major jQuery plugins? (If so, this would reflect problems in that code, not in the compression routine.)

Again, aggregation and compression are optional. There will be no Javascript errors if developers use the current methods properly.

Hence, this clearly is not a bug.

Maybe it's a feature request that aggregation be available for non-compliant Javascript. Marking as such.

Either that or it's a bug report and should be marked "by design".

nedjo’s picture

Status: Fixed » Active

Fixing status.

Owen Barton’s picture

Here is a list of syntax errors found in the test logs that are *not* either:
a) In the uncompressed logs (i.e. preexisting errors/warnings)
b) Not "SyntaxError: missing ; before statement:" errors (i.e. caused by missing semicolons)
- I have removed any lines corresponding to those 2 cases. Line numbers are included for your convenience.

2569:./ecma/LexicalConventions/7.1-1.js:1: SyntaxError: illegal character:
2639:jsTestDriverEnd ReferenceError: options is not defined./ecma/LexicalConventions/7.1-2.js:1: SyntaxError: missing variable name:
2709:jsTestDriverEnd ReferenceError: options is not defined./ecma/LexicalConventions/7.1-3.js:1: SyntaxError: missing variable name:
4640:jsTestDriverEnd ReferenceError: options is not defined./ecma_2/LexicalConventions/regexp-literals-002.js:1: SyntaxError: invalid flag after regular expression:
6113:jsTestDriverEnd ReferenceError: options is not defined./ecma_3/Function/regress-58274.js:1: SyntaxError: illegal character:

Another note is that these are only *syntax* errors, tests may be failing for reasons caused by our compression that _don't_ cause syntax errors. However these will be quite a bit more work to root out.

If we can track down these 5 errors and determine that they are all caused indirectly by missing semicolons then I think we are looking good as we are. If they are caused by other bugs then we need to fix our compression code (or somehow make it so that these errors don't cause people lost nights).

Owen Barton’s picture

Category: feature » bug

I think this is still in the category of (suspected) bug report, at least until we rule out the failing tests that are not obviously caused by missing semicolons.

nedjo’s picture

I've reread http://drupal.org/node/119441, the original JS aggregation issue, to see if somehow it could have been left unclear that we were introducing some minimal coding standards in Javascript that uses aggregation and compression, and that we provided an opt out. I'm not seeing it though. IMO it was perfectly clear. It was discussed and reviewed. The specific changes we required were modified along the way.

So this is by design.

Barry, Moshe, do you agree this far--that what we have is not a bug but is rather by design?

The objection that's been made, as far as I can see, is that people may wish to aggregate Javascript code that for whatever reason hasn't been upgraded to meet these minimal standards, and this isn't possible (will lead to Js errors). In practical terms, this position comes down to: developers who fail to upgrade their code (or, e.g., use external code that isn't compliant) shouldn't have to leave their code out of aggregation.

Personally, I don't find this a very compelling issue. I support the original design. We should be actively promoting minimal standards that facilitate the important goal of compression. And we should allow opting out as we've done.

But let's pursue the logic. If we accept that this is by design and not a bug per se, we're left with two possible arguments.

The first is that the design decision was bad and should be rolled back, removing JS compression altogether. Probably forever, since the argument isn't really with the implementation--it's with the idea of requiring any standards (which, let's agree, is a prerequisite for any meaningful compression algorithm).

The second is that we need additional flexibility, specifically, a new option in drupal_add_js() that will trigger aggregation without compression for a particular .js file.

Barry, Moshe, or someone else who thinks this is an important issue, which is it? Or can we just mark this "by design" and move on?

RobRoy’s picture

I think Grugnog2 got what I'm saying. If compression has some non-semicolon bugs, then those should be fixed but as far as I can tell, no one has displayed any cases where compression breaks Drupal-valid JavaScript (meaning you have your semicolons in order according to Nedjo's link above). If the only known "bug" is the semicolon issue, then IMHO this is by design.

Owen Barton’s picture

@nedjo - I think there is a consensus among most people so far (including me) that the semicolon issue is a reasonable compromise, and good practice anyway. I don't think this is the issue any more.

However I think that there is the possibility of other bugs in the compression code, as suggested by Barry's test results. If this is the case then I don't think it is reasonable at all to just expect people to work around them. Instead we either need to (a) fix the bugs or, failing that (b) document them exhaustively, and present people easy ways to work around each one.

Of course if it turns out that there are not actually any other bugs/side effects caused by the compression code (apart from the accepted semicolon requirement) then great, we can close this ticket and forget about it! As a side note, we might still want to add more documentation in drupal_add_js and the handbook on the semicolon requirement, and perhaps still consider making preprocess default to false.

One way of confirming the results of Barry's test (without having to individually debug each case) would be to temporarily alter the compression code so that the semicolon effect is no longer an issue - such as the patch in this initial post to insert linebreaks after each }. Alternatively we could alter the test cases and add the appropriate semicolons (I prefer the first method I think). Then, we run the battery of tests again. If the before and after numbers match, then we know the semicolon requirement is all that we are dealing with (and our code can be considered done). If there is a difference in these numbers then we have an actual bug, that we totally need to nail down and either fix, or exhaustively document - before we can consider this code done.

@bjaspan - do you fancy having a shot at this, since you have the test environment all set up?

bjaspan’s picture

FileSize
3.31 KB

I was concerned that perhaps the compression was causing tests to fail for some reason other than flaws in the way code is compressed so I decided to identify at least one test where that was not the case to prove I am not just blowing smoke. The first many failures are all "missing ; before statement" but then I came up ecma/LexicalConventions/7.1-1.js. The original test case is attached to this comment and the compressed test case is attached to the next comment.

In the final new TestCase() block, the compression has changed the constant strings in the compressed version, causing the test to fail.

I draw three conclusions:

1. The test suite depends on statements being separated by newlines, not semi-colons.
2. At least one test case is broken for reasons other than missing semi-colons.
3. The compression code is proved buggy in at least one way that cannot be fixed by "minimal coding standards."

bjaspan’s picture

FileSize
678 bytes

Here is the compressed version of ecma/LexicalConventions/7.1-1.js.

nedjo’s picture

Can we agree that:

1. (Optional) compression is good.
2. Not all valid code will compress, no matter what the compression algorithm (unless perhaps the algorithm is so minimal as to be practically useless).

?

I've added the following to the upgrade instructions at http://drupal.org/node/114774#javascript-compression :

Aggregation and compression is optional. It is important to test code and ensure it works compressed. In certain cases where complex or unusual coding approaches are used it may be difficult to identify exactly why a piece of code fails under compression. If the compressed code generates Javascript errors, you can try to edit it so that compression is supported. Alternately, you can call drupal_add_js() with the $preprosess argument set to FALSE, meaning that this file will be left out of aggregation and compression. See http://api.drupal.org/api/function/drupal_add_js.

Does this documentation adequately cover the case of the (relatively obscure) remaining issues? If not, how can it be improved?

catch’s picture

I still like the idea of three options:

Disabled
Aggregate
Aggregate and compress.

That ought to satisfy most of the concerns right? No patch for that though.

hass’s picture

#1 you mean "very good" :-).

#2 this example looks very special, but couldn't we try to take a look on the regex's if we are able to solve this example? if not possible - sh** happens, but maybe there is an easy solution... if not preprocess = FALSE is the very last solution... nevertheless no point to remove the compressor.

hass’s picture

@catch: this is tooo technical for beginners... :-) developers should simply use preprocess = FALSE or we need to change this drupal_add_js API for D7, but for now i think it is too late for an API change like this.

catch’s picture

@hass, no worries, I'll leave you to it on this one, although Grugnog2 and RobRoy might be offended to be called beginners.

fwiw any random site admin such as me looking at the performance options will understand the difference between aggregate and compress, but if there's no need for the option, all the better.

bjaspan’s picture

I am really astounded by this thread. nedjo and hass are arguing that a minor optimization is more important than correct functionality. I do not see how we are going to reach agreement.

Replying to #46, "can we agree that":

1. "(Optional) compression is good." I agree that aggregation is a big benefit. In this case, I'd say "compression is a minor additional benefit." Again, we're talking about a single static file that will be delivered once per visitor and then be cached by the browser.

2. "Not all valid code will compress, no matter what the compression algorithm (unless perhaps the algorithm is so minimal as to be practically useless)." Well, yes and no. To be more precise:

2a. "Not all valid code will compress *using this kind of technique*." Yes, absolutely I agree with that.
2b. "unless the algorithm is so minimal as to be useless." No, I disagree. I think if we strip comments we'll get a big percentage of the gain of this kind of compression with almost no risk (but not zero risk) of breaking valid code.
2c. Not all valid code will compress *using any technique." Disagree. We could presumably deliver gzip-compressed files quite reliably to browsers that accept them. Using a little mod_rewrite magic, this can be accomplished as a static file serve, not requiring launching PHP at all. Probably not in D6, though.

Your change to the docs does not help. You might just as well have written: "Drupal's JavaScript compression code is buggy in unknown ways. Try it and see if it works for you (and hope you notice any failures before you release your module). If not, then you'll have to disable aggregation for your module, making it less efficient for everyone who uses it, or try to debug how Drupal is breaking your code. Have fun!"

I think we are all just repeating ourselves now. I've made my case. I will not restate these positions of mine again and encourage others to do the same. If someone has new information to contribute, by all means do so. Otherwise, I suggest we leave this up to the core committers.

Owen Barton’s picture

@nedjo - So we are essentially saying "compression may introduce random bugs in your code, and we don't know what they are (apart from the semicolon issue) and we can't be bothered to find out what these bugs are and fix them or tell you how to work around them"? OK, so that is not a very diplomatic way of putting it, but it still just feels far too sloppy to me for something in core :)

hass’s picture

@bjaspan: it sounds like a religious war for me... you found one small piece of code that does not work. Maybe you found this 1% that will break... why are we discussing about a possibility that no one will see in real live or do we know better?

JS libraries are often marked as "compressor compatible" or something like this. I don't know why we should care about buggy or ultra special JS regression test code from Mozilla or others. They can fix their code and all is fine. Others using their code will tell them the same or will request for compressor compatibility...

Why are you not simply going to fix the regex if there is a bug or you think there is a bug not related to semicolons?

nedjo’s picture

I understand the argument that it's wrong to introduce a feature (even an optional one) that can break valid Javascript code. I acknowledge that this as a valid position. But I don't share it, for the reasons given.

I'm not arguing that our particular (fairly minimal) compression algorithm is perfect. If people have the energy for tracking down and documenting or preventing possible edge cases (whether or not there's any sign that they represent real problems for actual Drupal development) I think that's commendable and would be happy to see further improvements.

I'm also open to the idea of enabling developers to trigger aggregation without compression. This doesn't look like a lot of work, and wouldn't break any existing code (since the optional $preprocess argument happens to be the last one in drupal_add_js()). Attached is a first untested draft of the minimal patch needed to make compression optional, if anyone wants to push that direction forward.

Let's focus on the positive energy needed for reaching agreement and identifying and addressing issues.

chx’s picture

Status: Active » Reviewed & tested by the community

that's one fine patch that can satisfy everyone.

bjaspan’s picture

I whittled away at the test case I identified yesterday to make a smaller example of some valid JavaScript that breaks under compression. Input:

/* \X */
x("\t\v\f");

Output:

x("\X\t\v");

Something about the "\" in the comment is confusing the regexps. If you remove it, the code compresses correctly. So, "minimal coding standards" apparently need to include "don't put backslashes in your comments."

hass’s picture

Only as a side note - the docs about D6 coding standard für JS has been created on "31/08/2007 - 20:34 by pwolanin" at http://drupal.org/node/172169.

Dries’s picture

I must say that this issue makes me a bit nervous. The more I think about it, the more I dislike the fact that we're imposing limitations. The past weeks, I've talk to people who would like to use or integrate additional Javascript libraries (not jQuery plugins).

I'm going to think about this some more. (I'm still traveling so it might take me a day or two to get back to this.)

nedjo’s picture

I hear agreement from everyone that we need to support external Javascript without changes. The remaining questions seem to be:

  1. Is it acceptable to limit the new aggregation feature to files compressible by our compressor, as is the current situation?
  2. If no to (1), which option or options are best?
    1. drop compression (patch at #6).
    2. fix (or minimize) compression to avoid possible issues, e.g., suggestion in #51 to strip only comments
    3. make it possible to have aggregation without compression (patch at #54).

A personal concern with dropping compression altogether, as I commented in the original JS aggregation issue, is that

without compression, the obvious advantages of fewer files loaded could possibly be offset by increased overall download size [since] a given js file is [now] downloaded to the client once per page visited with a unique combination of js files, rather than once total

Tangentially related: I realized we didn't have any documentation to help module authors decide when aggregation is and is not appropriate. With CSS aggregation we note that

you should *not* preprocess every file as this can lead to redundant caches

I've added some notes to the update instructions, http://drupal.org/node/114774#javascript-aggregation-compression, improvements would be welcome, and I suppose we should also have this in the code documentation.

Owen Barton’s picture

@nedjo - either option 2 or 3 sounds fine to be. Option 1 would be OK too, as long as we made it possible to hook up some compression as a contrib module.

I pointed out the issue of redundant caches several times in the original CSS aggregation thread(s), but couldn't get anyone to see the issue. Hopefully people are now getting what is happening and why this is a problem! For D7, I would actually like us to move to a hook_resources or a .info declaration that returns an array of all the css and js that should be aggregated on all paths. The drupal_add_js/css functions would then be reserved for including the specialized or admin css/js on just the pages where it is needed (and that hence should not be aggregated - we should drop the aggregate parameter in this case). Moving to a more explicit model makes this impossible (or at least very hard) to screw up.
The alternative is to improve documentation and education such that it is exceedingly clear that aggregated css/js should ONLY ever be added in hook_menu nocache, with NO conditional logic (i.e. on every path etc). Personally I think making the API a more bulletproof is a simpler path here :)

hass’s picture

Grugnog2: This is OT here, but drupal_add_js shouldn't be moved into static .info files. I'm changing this files dynamically in themes... and i'd like to use aggregate and compression together with dynamic added JS and CSS files. The D6 change moving theme files into themes .info files is not good and very limiting.

Owen Barton’s picture

@hass - moving css/js includes into .info files (or anywhere else) does not stop you messing with the list in your theme (although I think the files would already be aggregated at this point, so I don't see how you can do this now). The css or js array would still be built as normal (from both sources), it would just be architecturally constrained so that aggregated files were always global, which means we don't get lots of (mostly) duplicate caches wasting bandwidth.

hass’s picture

@Grugnog2: Sorry i don't understand you... to shorten this - you may take a look into http://www.yaml-fuer-drupal.de/de/download (D5 or D6 Theme). See one of the "template.php" files how this is done today and i assure you it will become a much more configurable theme (with custom theme settings) in the next release and therefor have more and more "dynamic" CSS files.

Owen Barton’s picture

@hass, what I am saying is that the css files would be added from the hook or info file using drupal_add_css(), just in the same way they are now, except by a core function. The difference is that the 'global' css files would all be added via a centralized global mechanism (not each module calling drupal_add_css() in random places, which is what leads to duplicate caches), and hence could be safely aggregated. There would be nothing stopping you adding or removing the files in your theme as you do now using drupal_add_css() and $css = drupal_add_css(); (to remove a file). The css 'entering' the theme would look and feel exactly the same as it does now. The same approach could apply to JS too, of course.

nedjo’s picture

Status: Reviewed & tested by the community » Needs work

So far in this thread we've focused on the (valid) use case where we want aggregation but not compression for a given JS file. But there's also the opposite case: some files will appear only infrequently (not good candidates for aggregation) but will benefit from compression.

In fact we have several examples of this opposite case in core. E.g., in color.module we currently have this call:

drupal_add_js('misc/farbtastic/farbtastic.js');

farbtastic.js is seldom used and presumably shouldn't be aggregated, but does support and could benefit from compression.

The conclusion seems to be, JS aggregation and compression are distinct needs and so we should make them independent from each other. We would keep the distinct $aggregate and $compress arguments in drupal_add_js() from the patch at #54, but evaluate and respond to them separately, such that a given file could be aggregated but not compressed, compressed but not aggregated, or neither or both.

Crell’s picture

Status: Needs work » Reviewed & tested by the community

nedjo: That would be an API change, and it is way too late IMO to start worrying about all permutations of compression and aggregation in Drupal 6.

(Setting back to RTBC because it was before, although I have not tested the most recent patch myself.)

bjaspan’s picture

Status: Reviewed & tested by the community » Needs work
FileSize
7.42 KB

This issue has been marked RTBC for a while but with no clear consensus about what to do. The only patch that received a review and got marked RTBC was #6 but it no longer applies due to whitespace changes in common.inc.

The currently committed JS compression code is demonstrably wrong in a serious way: backslashes in comments can modify text in constant strings elsewhere in the code. This could easily lead to serious but unnoticed bugs and is not something that can be addressed by coding standards. There are also other bugs revealed by Mozilla's test suite that we have not isolated. I maintain that this situation has to be fixed, either by fixing the packer or removing it; we just can't knowingly be introducing unknown bugs into Javascript.

So:

1. I have re-rolled my patch to remove JS compression but leave in JS aggregation. Since it was marked RTBC before I am leaving it in that state.

2. If someone fixes the two known problems in the compression (newlines/semi-colons and backslashes in comments) I will run the new patch against the Mozilla test and help identify any additional problems. If/when the compression passes the test suite, I'll support its inclusion.

catch’s picture

Status: Needs work » Reviewed & tested by the community

I think barry meant to leave this RTBC. I'm not expressing an opinion, just sweeping the queue.

bjaspan’s picture

Yes, I did; sorry 'bout that.

bjaspan’s picture

I decided to change the compression code not to remove any newlines and see what other errors popped up. The problem I reported in #56 (backslashes in comments seem to break things) is still there. Here's another: The code

AddTestCase(
  "// A regular expression literal represents an object of type RegExp.",
  "true",
  (/x*/ instanceof RegExp).toString() );

got turned into

AddTestCase(
"// A regular expression literal represents an object of type RegExp.",
"true",
(/x*/instanceof RegExp).toString());

The space after the regexp /x*/ got removed resulting in an "invalid modifier" error.

nedjo’s picture

Can we can live without JS compression for D6? Sure, just as we can live without any particular new feature. What feels important is that we're mapping out a sound direction.

Should we remove JS compression permanently? Should we support only "100% safe" compressors (if such exist, which I still tend to doubt, at least if they're trying to do more than the minimum)? Should we be enabling compression implementations in contrib?

Some relevant sites and posts:

http://blogs.pathf.com/agileajax/2006/08/compressing_you.html, now somewhat old, compares various compression solutions, most or all of which at that time introduced JS errors.

http://developer.yahoo.com/yui/compressor/, includes the lofty phrase "100% safe"

Dojo Shrinksafe, http://alex.dojotoolkit.org/shrinksafe/,

Instead of relying on brittle regular expressions, ShrinkSafe is based on Rhino, a JavaScript interpreter. This allows ShrinkSafe to transform the source of a file with much more confidence that the resulting script will function identically to the file you uploaded.

JSLint, http://www.jslint.com/, sometimes used to analyze and prepare code for compression support.

bjaspan’s picture

My answers to your questions:

1. Should we remove JS compression permanently? No.

2. Should we support only "100% safe" compressors? In core, yes, absolutely. If that means just concatenating files, so be it. Correctness trumps performance.

3. Should we be enabling compression implementations in contrib? Yes, I would support that. We can add a js_alter hook or somesuch. Sites that want to install riskier JS compressors would then be free to do so.

FYI, I discovered another problem with this compression: a custom jQuery script that works fine when compressed on FF but not on Safari. No syntax errors, but event handlers seemed to be disabled. I didn't look into the details, I just converted the compressor to "append only" and the problem went away.

Dries’s picture

Status: Reviewed & tested by the community » Needs work

I propose that we continue to support JS aggregation, but that we remove JS compression from core until we have a 100% safe compressor. As illustrated by our early testers, correctness and compatibility matter a great deal.

A 100% safe compressor, btw, might be a gzip compressor that simply gzips the aggregated JS. I'd expect it to generate files that are smaller than a high-level JS compressor. (Not validated but probably worth exploring).

catch’s picture

Status: Needs work » Reviewed & tested by the community
FileSize
7.17 KB

In that case bjaspan's patch in #67 remains RTBC, removing compression but leaving aggregation. I got some "removing trailing CRs" messages so have just converted to unix format. Obviously don't credit me if this gets committed

Gábor Hojtsy’s picture

Status: Reviewed & tested by the community » Needs work

The settings form in system.admin.inc also needs modification, as it says JS is compressed/optimized.

Gábor Hojtsy’s picture

Anyone up for fixing the remaining TODOs so we can get this into RC1?

keith.smith’s picture

Status: Needs work » Needs review
FileSize
8.76 KB

Late to the party, here, but this changes the text under "Bandwidth Optimizations" to read: Drupal can automatically aggregate external resources like CSS and JavaScript into a single cached file. This can help reduce the number of requests made to your website, and ultimately, may reduce server load, bandwidth requirements, and page loading times.

keith.smith’s picture

That last one wasn't quite right (since CSS is compressed and aggregated, while JS is only aggregated). New version attached.

Gábor Hojtsy’s picture

Status: Needs review » Fixed

Thanks. I am a bit sad that there were no big efforts to fix the bugs identified and do more extensive testing, but as Dries noted, we are better with something which works still, instead of shipping with a broken solution. Thanks Keith for stepping in and helping fix the descriptions.

Anonymous’s picture

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for two weeks with no activity.

ximo’s picture

This tool might be useful: http://compressorrater.thruhere.net/

It tests JavaScript code in a number of packers/minifiers, with and without gzip compressions.

(I didn't find any better place to post this but on this closed issue)

nedjo’s picture

I opened a new feature request, http://drupal.org/node/210447, for adding back in an improved approach to Javascript compression.

I also removed the compression instructions I'd previously added to the module updating instructions.

RobRoy’s picture

Just for reference, I just found some JS that is broken by the compressor so this was actually worthwhile to take out.

$.ui.intersect = function(oDrag, oDrop, toleranceMode) {
    if (!oDrop.offset)
      return false;
    var x1 = oDrag.rpos[0] - oDrag.options.cursorAt.left + oDrag.options.margins.left, x2 = x1 + oDrag.helperSize.width,
        y1 = oDrag.rpos[1] - oDrag.options.cursorAt.top + oDrag.options.margins.top, y2 = y1 + oDrag.helperSize.height;
    var l = oDrop.offset.left, r = l + oDrop.item.element.offsetWidth,
        t = oDrop.offset.top,  b = t + oDrop.item.element.offsetHeight;
    switch (toleranceMode) {
      case 'fit':
        return (   l < x1 && x2 < r
          && t < y1 && y2 < b);
        break;
      case 'intersect':
        return (   l < x1 + (oDrag.helperSize.width  / 2)        // Right Half
          &&     x2 - (oDrag.helperSize.width  / 2) < r    // Left Half
          && t < y1 + (oDrag.helperSize.height / 2)        // Bottom Half
          &&     y2 - (oDrag.helperSize.height / 2) < b ); // Top Half
        break;
      case 'pointer':
        return (   l < oDrag.rpos[0] && oDrag.rpos[0] < r
          && t < oDrag.rpos[1] && oDrag.rpos[1] < b);
        break;
      case 'touch':
        return (   (l < x1 && x1 < r && t < y1 && y1 < b)    // Top-Left Corner
          || (l < x1 && x1 < r && t < y2 && y2 < b)    // Bottom-Left Corner
          || (l < x2 && x2 < r && t < y1 && y1 < b)    // Top-Right Corner
          || (l < x2 && x2 < r && t < y2 && y2 < b) ); // Bottom-Right Corner
        break;
      default:
        return false;
        break;
    }
  };

It doesn't remove the comments correctly in the 'touch' and 'intersect' cases and results in...

$.ui.intersect=function(oDrag,oDrop,toleranceMode){if(!oDrop.offset)return false;var x1=oDrag.rpos[0]-oDrag.options.cursorAt.left+oDrag.options.margins.left,x2=x1+oDrag.helperSize.width,y1=oDrag.rpos[1]-oDrag.options.cursorAt.top+oDrag.options.margins.top,y2=y1+oDrag.helperSize.height;var l=oDrop.offset.left,r=l+oDrop.item.element.offsetWidth,t=oDrop.offset.top,b=t+oDrop.item.element.offsetHeight;switch(toleranceMode){case'fit':return(l<x1&&x2<r&&t<y1&&y2<b);break;case'intersect':return(l<x1+(oDrag.helperSize.width/ 2) //Right Half&&x2-(oDrag.helperSize.width/ 2) < r //Left Half&&t<y1+(oDrag.helperSize.height/ 2) //Bottom Half&&y2-(oDrag.helperSize.height/ 2) < b ); //Top Half break;case'pointer':return(l<oDrag.rpos[0]&&oDrag.rpos[0]<r&&t<oDrag.rpos[1]&&oDrag.rpos[1]<b);break;case'touch':return((l<x1&&x1<r&&t<y1&&y1<b)||(l<x1&&x1<r&&t<y2&&y2<b)||(l<x2&&x2<r&&t<y1&&y1<b)||(l<x2&&x2<r&&t<y2&&y2<b));break;default:return false;break;}
158};})($);;

which is broken.

hass’s picture

Is this the same like http://drupal.org/node/211915 what i've found in devel_themer?

Flying Drupalist’s picture

This is still a problem, though I'm not sure if it's the same problem.

If there is a comment at the start of your js file it will break. I discovered this only after hours of testing.