In http://drupal.org/node/119441#comment-221260 and http://drupal.org/node/119441#comment-221572, while developing the patch to implement JS aggregation and whitespace compression, m3avrck changed the patch and said:
Also it fixes the JS files so that they use correction notation, e.g.,:
var foo = function bar() {
};Need that ";" when you define variable functions like that.
Note, you only need those *optional* semi-colons when you're compressing (e.g., removing whitespace) from JS files, it prevents variables from running into each other by being defined on the same line.
In other words, the JS compressor breaks on legal Javascript. This is a mistake. Requiring valid JS to be changed to be compatible with the compressor is a wrong decision, especially since the $cache argument is TRUE by default (but even if it were not). Oh, and this new requirement is also not documented. It just cost me half a day of work to figure this out and not all JS/module developers will even be capable of grokking the packer compression logic to figure out what is going on. Not acceptable for core.
In my particular case, it was the D5 CCK Link module that has the close-brace not followed by a semi-colon. I'm sure the D6 version will, too, along with numerous other modules, and we should not force them to change.
I've attached a patch which fixes this problem by making sure a newline appears after close-braces. This patch is against D6 HEAD.
I predict this is not the last time that this JS compression code will bite us. I recommend removing it completely; just concatenating the files into a single request is the big win of this patch. But the patch to fix the particular bug I found is smaller and less likely to be controversial, so that is what I'm attaching.
Comment | File | Size | Author |
---|---|---|---|
#78 | js-aggregation-183940-67_2.patch | 8.94 KB | keith.smith |
#77 | js-aggregation-183940-67_1.patch | 8.76 KB | keith.smith |
#74 | js-aggregation-183940-67.patch | 7.17 KB | catch |
#67 | js-aggregation-183940-67.patch | 7.42 KB | bjaspan |
#54 | drupal-js-compression-option.patch | 3.23 KB | nedjo |
Comments
Comment #1
bjaspan CreditAttribution: bjaspan commentedForgot to add:
1. I marked this 'critical' because the bug allows valid JS in any enabled module to disable all JS from all modules. It's a pretty severe loss of function and most sites will have no way to debug or fix the problem.
2. There is a bug in _packer_apply(). See http://drupal.org/node/183871.
Comment #2
ChrisKennedy CreditAttribution: ChrisKennedy commentedI agree with your synopsis that it is better to remove this requirement, which adds significant burdens without much benefit.
I've tried the patch but from my testing it doesn't solve the issue, although if I just comment out the white-space removing line the original javascript does function correctly. Is this patch being held up by the _packer_apply() bug, or is my testing perhaps flawed?
My test module inserts this javascript code in an external file (middle case omits the terminating semicolon):
Comment #3
bjaspan CreditAttribution: bjaspan commentedMy patch only ensures that close-braces are followed by newlines because that is the particular issue I encountered. Your demonstration code has a close-paren that needs to be followed by a newline, so my patch does not help with that. It would be very easy to fix my patch to include close-parens, but this is why I said:
"I predict this is not the last time that this JS compression code will bite us. "
Unless someone can say confidently that the compression code will work for ALL valid JS, I think we should yank it and just concatenate the files. And "valid JS" in this case has to mean JS that works in all popular browsers; we can't break working code that happens to violate the strict JS specifications unless we want people to blame Drupal for it not working.
Comment #4
moshe weitzman CreditAttribution: moshe weitzman commentedis there a patch out there which removes the compressor?
Comment #5
dmitrig01 CreditAttribution: dmitrig01 commentedWhy not add a semi-colon if one is missing?
Comment #6
bjaspan CreditAttribution: bjaspan commented#4: Sure, here is a patch that removes the compression while leaving the aggregation in place.
Comment #7
dmitrig01 CreditAttribution: dmitrig01 commentedMaybe have a setting for compression and another one for aggregation on the admin page?
Comment #8
moshe weitzman CreditAttribution: moshe weitzman commentedIMO a new setting makes little sense. This feature simply isn't ready.
Comment #9
Gábor HojtsyI tried to drive some attention here by sending a mail to the development list. I agree separate settings for compression and aggregation are not a good idea, especially not to mask a broken feature. I am looking into other opinions on this before rolling this feature back though.
Comment #10
bjaspan CreditAttribution: bjaspan commentedWe could probably safely remove comments and eliminate empty lines and leading and trailing whitespace on lines, just not newlines. The question, though, is why bother? This is a static file that should only be delivered once to each client and then result in "Not Modified" responses until it next changes (assuming we have our cache headers set correctly; if we don't that is a separate and important issue).
Custom-compressing the aggregated JS introduces extra code to write, maintain, review, and debug and the programmer time already spent on it probably exceeds the total time that will ever be saved by users in downloading the compressed version versus the non-compressed version. And if that is not true, if the compression is actually providing important value, then we would be better served delivering a gzip-compressed file instead which would save even more space and be much more likely to be bug-free.
Comment #11
ChrisKennedy CreditAttribution: ChrisKennedy commentedI think the comment-removal might still be worthwhile. According to ohloh.net 38% of Drupal's code is comments - programmatically removing those is a pretty solid amount of savings. And with all the jQuery and AHAH functionality being added to Drupal, we will continue to see significant growth in Drupal's JavaScript codebase. Removing comments is different from gzip compression and will likely even enhance gzip's compression rate above and beyond the reduction in filesize (since comments and executing code probably have very different word distributions).
Comment #12
arthurf CreditAttribution: arthurf commentedWhile I have encountered the problems of javascript compression mentioned here already, I do think it is worth pursuing. On high traffic sites compression could make a significant difference. On a 5.3 site with the aggregation patch, the total JS went from 70k + to slightly under 50k. If you factor this out over the course of the month, it can be significant.
On the other hand, getting aggregation working required significant effort- lots of hacking on modules that'd I'd rather not have hacked and making my upgrade paths more dubious. On the other, it really improved my yslowscores :)
I think the bottom line is that if aggregation is kept for d6, we need to improve the packer's ability to handle the case mentioned in this post. We may want to change drupal_add_js's default behavior to not aggregate unless explicitly told to. That may help ease the transition.
Comment #13
hass CreditAttribution: hass commentedUhh - is removing this compression code really the right decision?
I don't like this... compression is very cool stuff and developers like to develop a D6 module with JS should add some semicolons or set the pre-process param to FALSE... no big thing. Why removing this cool stuff if it only needs some minor thinking about JS compression... :-(((
Comment #14
bjaspan CreditAttribution: bjaspan commentedThe compression code is not cool in a production system if it is wrong. If it breaks legal JS, it is wrong. If we can fix it so we are confident it will not break legal JS, then I have no serious objection to it (I personally am not going to put more time into fixing it but if other people want to, fine with me---this is a volunteer project.)
Comment #15
Gábor HojtsyNote that I kept this RTBC, so that unless someone comes around with actual fixes for JS compression, this feature gets removed. No matter how cool this sounds, if we only talk about fixing it and never do it, we need to live without it.
Comment #16
catchMay well not be possible, but any way to leave in a hook so compression could be done in contrib? That way it'd get more testing during the D7 release cycle and could move back into core next version.
I'm assuming the answer will be 'no', but better to ask than not.
Comment #17
Crell CreditAttribution: Crell commentedI just want to point out that "legal Javascript" allows dropping the ending semi-colon in the same way that "legal HTML" has half of its closing tags optional. That's been well-understood as very bad practice for a very long time now, and I don't know anyone in the JS community who doesn't encourage using semi-colons everywhere, even when not strictly required. Making the code compressor-friendly is one key reason for that.
So are we supporting "all legal Javascript", or are we supporting "Javascript not written stupidly?" It seems to me the proper solution is to consider JS code in a Drupal module that can't compress due to missing semi-colons to be a bug to be fixed.
Comment #18
hass CreditAttribution: hass commented@Crell:
That's totally true. Ever time a developer create compression friendly JS code the developer MUST add semicolons. I know many projects outside Drupal that added them for compression compatibility only, because users requested this feature. JSMIN and other compressors will not work if code is not compression friendly, too. If the JS code is not compression friendly and i cannot write JS code myself, but add this to my D6 project - i could leave this out of compression - for Drupal - preprocess = FALSE.
Why not go this pre-process way?
Comment #19
hass CreditAttribution: hass commentedIf we should remove this good JS compressor feature we must remove the CSS compressor, too. See http://drupal.org/node/139117 about a core bug that breaks the CSS compressor. And removing this CSS compressor would be a wrong decision, too.
Comment #20
Wim LeersI tend to agree with Crell I think.
How do Javascript compressors in general handle this? I think it's best to do like the rest in this case.
In follow-up #10, bjaspan says that gzip compression is not yet included. I wasn't aware of that. I guess that's something for D7...
Comment #21
Wim LeersThanks hass, you already answered my question. Then I agree that Drupal should not worry about this either.
Comment #22
hass CreditAttribution: hass commentedLet's document this in the module and theme upgrade docs and do not commit this removal patch, please. I tend to mark this as CNW for documentation...
Comment #23
ms2011 CreditAttribution: ms2011 commentedcoming in a little late in the discussion here, but wanted to be sure no one had overlooked the YUI Compressor + GZip solution as an alternative if Dean Edwards' Packer is causing trouble:
http://www.julienlecomte.net/blog/2007/08/21/gzip-your-minified-javascri...
Comment #24
nedjoAs a developer who probably maintains as much JS module code as anyone, I welcome the JS compression and the opportunity it presents for improving consistency in our Javascript codebase. "Valid JavaScript" is almost a meaningless phrase, as the language is so loosely defined. A few well justified and minimal standards on our part can only be for the good. Supporting compression is an important goal. Without compression, aggregation will increase overall download bandwidth (since the same code is downloaded once per unique combination of .js files on a page, rather than only once per site).
I've drafted the missing documentation, http://drupal.org/node/114774#javascript-compression. Is more needed?
Comment #25
RobRoy CreditAttribution: RobRoy commentedI'd hate to see compression go and would like to see missing semicolons added to any offending JavaScript. I think compression is a valid feature and it works great. Compressing code and removing comments is great to reduce size in a heavy traffic environment. All you have to do is make your JavaScript a little stricter by adding some semicolons.
My $0.02.
Comment #26
hass CreditAttribution: hass commented@nedjo: docs extension looks good to me. We should add this to themes http://drupal.org/node/132442, too. Let's change this case to fixed after this...
Comment #27
Owen Barton CreditAttribution: Owen Barton commentedI am not sure how I feel about this one yet. I don't like the route of having a massively more complex regexp (it is already complex enough!). I would be happy with adding separate 'aggregate' and 'aggregate and compress' options. I would be happy removing it and adding a hook so it can live in contrib.
I discovered some interesting test cases at http://js-crunchy.googlecode.com/svn/trunk/crunchy/tests.js
Comment #28
RobRoy CreditAttribution: RobRoy commentedHaving three options is fine with me: Disabled, Aggregate, Aggregate and Compress. As long as I can easily get the benefits of compression, I'm happy. And the new option sounds like it may be requested by the community. Patch anyone?
Comment #29
nedjoThe issue that's been identified is that some JS files may not comply with the new (minimal) coding standards. This should not be addressed through configuration--it's a code-level problem.
And we already have the solution in place. Developers who wish to use non-complying code can set the
$preprocess
argument toFALSE
when callingdrupal_add_js()
. Problem solved so far as I can see, so marking this won't fix.Comment #30
bjaspan CreditAttribution: bjaspan commentedI used the Mozilla Javascript Test Library (http://www.mozilla.org/js/tests/library.html) to test the Drupal JS compression code and found substantially more failures with compression than without.
Specifically, I ran the ecma, ecma_2, and ecma_3 tests against a current CVS build of Mozilla (I needed a debugging build in order to get xpcshell to run the tests; it was a bit of doing to set up the build system, etc.) First I ran the tests as they exist in CVS. Then I applied _drupal_compress_js() to each test .js file and ran the same tests again.
Without compression, 77 out of 954 tests failed. With compression, 159 out of 954 tests failed. I'm attaching the test results to this and the next comment.
Why did the extra tests fail? Are the bugs all related to missing semi-colons, or are there other problems with the compression? I have no idea and do not feel like taking the time to figure it out. IMHO, delivering a slightly larger aggregated JavaScript file to each visitor once each time the aggregated file changes is just not that much extra bandwidth and is not a worthwhile savings compared against the drawback of breaking an unknown quantity of valid code.
If someone wants to rewrite the compression patch just to strip comments and other really obviously safe things, I'll re-run the test suite to see how it does. Otherwise, I really think we should serve uncompressed aggregated JS.
Un-compressed test results attached. It's an HTML file renamed to be attachable.
Comment #31
bjaspan CreditAttribution: bjaspan commentedTest results against compressed code attached.
Comment #32
RobRoy CreditAttribution: RobRoy commentedI'm setting back to review as from reading the thread it sounds like we can safely keep comment removal and whitespace removal even if the other compression stuff is broken. But, I think it would help to come to some clear consensus on *what* exactly is broken about the compression. If it's just the semicolon thing, than I agree with nedjo and think it should stay in, but it sounds like there may be some additional error cases, they just haven't accurately been identified. Am I off base?
Comment #33
moshe weitzman CreditAttribution: moshe weitzman commentedBarry did careful analysis and demonstrated that the compression is buggy. if someone wants to fix it, they are encouraged to do so. in the meanwhile, bugs need to be fixed or removed and thats what this issue currently does. setting it to 'for review' is not appropriate IMO unless one adds an improved patch at same time.
Comment #34
RobRoy CreditAttribution: RobRoy commentedMy point was, if all the "bug" reports that came from that analysis really just had to do with missing semicolons, then that would mean something different than if there are other valid problems with the compression. That is why I set it to review. It would be nice to see what specifically is broken. Maybe we can get some m3avrck love on this issue.
Comment #35
m3avrck CreditAttribution: m3avrck commentedI agree with Nedjo and RobRoy.
This patch is removing something that really speeds up the load time of sites. I have been using this on MothersClick.com and MomBlogNetwork.com for 6+ months now, with absolutely no problems. My JS coders have had almost 0 problems with this, writing tons of Javascript.
The missing ";" while not there still produces runnable JS is incorrect and technically invalid. The jQuery teams STRESSES this in their docs for all plugins to conform to the full syntax, using all optional ";" for consistency. Since Drupal relies on jQuery and 99% of all JS written will be in jQuery, it makes the most sense to follow the jQuery standards and enforce this.
Just because you write bad JS code, why should we support your bad code in core? We don't do this for any PHP code and hooks, why should we start now with JS?
I'll see if I get John Resig to comment as welll...
Comment #36
Owen Barton CreditAttribution: Owen Barton commentedI think we need to analyse the test cases some more before we decide here. If it is simply the ';' issue then I think we can live with it - that is a pretty reasonable request to make to coders.
However if there are *other* actual bugs in the code (unless the 82 failing tests are caused by compression are all due to missing semicolons) then I think it is very important that we find and fix these (or remove/simplify the compression). Otherwise we end up with a bunch or weird edge cases that are a major pain to debug (try debugging compressed code if you don't know what I mean here!).
Anyone knowledgeable on javascript syntax standards want to help go through these test cases?
I had a quick look at the test logs and it does look like there are a lot of errors coming up with "SyntaxError: missing ; before statement:". There were other errors though, like "SyntaxError: missing variable name:" - I haven't checked to see if those are in the set of existing failing tests, or if they are side effects of missing semicolons.
Comment #37
nedjoBarry's latest tests add nothing new but only reiterate his original note that compression won't support any old (valid) Javascript. Yes, we know this. Yes, we have introduced some minimal coding standards in Javascript. This is by design.
Is there a list of active Drupal Javascript developers who are not willing to accept minimal coding standards in Javascript? Are there errors that compression produces in our core Javascript files or major jQuery plugins? (If so, this would reflect problems in that code, not in the compression routine.)
Again, aggregation and compression are optional. There will be no Javascript errors if developers use the current methods properly.
Hence, this clearly is not a bug.
Maybe it's a feature request that aggregation be available for non-compliant Javascript. Marking as such.
Either that or it's a bug report and should be marked "by design".
Comment #38
nedjoFixing status.
Comment #39
Owen Barton CreditAttribution: Owen Barton commentedHere is a list of syntax errors found in the test logs that are *not* either:
a) In the uncompressed logs (i.e. preexisting errors/warnings)
b) Not "SyntaxError: missing ; before statement:" errors (i.e. caused by missing semicolons)
- I have removed any lines corresponding to those 2 cases. Line numbers are included for your convenience.
Another note is that these are only *syntax* errors, tests may be failing for reasons caused by our compression that _don't_ cause syntax errors. However these will be quite a bit more work to root out.
If we can track down these 5 errors and determine that they are all caused indirectly by missing semicolons then I think we are looking good as we are. If they are caused by other bugs then we need to fix our compression code (or somehow make it so that these errors don't cause people lost nights).
Comment #40
Owen Barton CreditAttribution: Owen Barton commentedI think this is still in the category of (suspected) bug report, at least until we rule out the failing tests that are not obviously caused by missing semicolons.
Comment #41
nedjoI've reread http://drupal.org/node/119441, the original JS aggregation issue, to see if somehow it could have been left unclear that we were introducing some minimal coding standards in Javascript that uses aggregation and compression, and that we provided an opt out. I'm not seeing it though. IMO it was perfectly clear. It was discussed and reviewed. The specific changes we required were modified along the way.
So this is by design.
Barry, Moshe, do you agree this far--that what we have is not a bug but is rather by design?
The objection that's been made, as far as I can see, is that people may wish to aggregate Javascript code that for whatever reason hasn't been upgraded to meet these minimal standards, and this isn't possible (will lead to Js errors). In practical terms, this position comes down to: developers who fail to upgrade their code (or, e.g., use external code that isn't compliant) shouldn't have to leave their code out of aggregation.
Personally, I don't find this a very compelling issue. I support the original design. We should be actively promoting minimal standards that facilitate the important goal of compression. And we should allow opting out as we've done.
But let's pursue the logic. If we accept that this is by design and not a bug per se, we're left with two possible arguments.
The first is that the design decision was bad and should be rolled back, removing JS compression altogether. Probably forever, since the argument isn't really with the implementation--it's with the idea of requiring any standards (which, let's agree, is a prerequisite for any meaningful compression algorithm).
The second is that we need additional flexibility, specifically, a new option in
drupal_add_js()
that will trigger aggregation without compression for a particular .js file.Barry, Moshe, or someone else who thinks this is an important issue, which is it? Or can we just mark this "by design" and move on?
Comment #42
RobRoy CreditAttribution: RobRoy commentedI think Grugnog2 got what I'm saying. If compression has some non-semicolon bugs, then those should be fixed but as far as I can tell, no one has displayed any cases where compression breaks Drupal-valid JavaScript (meaning you have your semicolons in order according to Nedjo's link above). If the only known "bug" is the semicolon issue, then IMHO this is by design.
Comment #43
Owen Barton CreditAttribution: Owen Barton commented@nedjo - I think there is a consensus among most people so far (including me) that the semicolon issue is a reasonable compromise, and good practice anyway. I don't think this is the issue any more.
However I think that there is the possibility of other bugs in the compression code, as suggested by Barry's test results. If this is the case then I don't think it is reasonable at all to just expect people to work around them. Instead we either need to (a) fix the bugs or, failing that (b) document them exhaustively, and present people easy ways to work around each one.
Of course if it turns out that there are not actually any other bugs/side effects caused by the compression code (apart from the accepted semicolon requirement) then great, we can close this ticket and forget about it! As a side note, we might still want to add more documentation in drupal_add_js and the handbook on the semicolon requirement, and perhaps still consider making preprocess default to false.
One way of confirming the results of Barry's test (without having to individually debug each case) would be to temporarily alter the compression code so that the semicolon effect is no longer an issue - such as the patch in this initial post to insert linebreaks after each }. Alternatively we could alter the test cases and add the appropriate semicolons (I prefer the first method I think). Then, we run the battery of tests again. If the before and after numbers match, then we know the semicolon requirement is all that we are dealing with (and our code can be considered done). If there is a difference in these numbers then we have an actual bug, that we totally need to nail down and either fix, or exhaustively document - before we can consider this code done.
@bjaspan - do you fancy having a shot at this, since you have the test environment all set up?
Comment #44
bjaspan CreditAttribution: bjaspan commentedI was concerned that perhaps the compression was causing tests to fail for some reason other than flaws in the way code is compressed so I decided to identify at least one test where that was not the case to prove I am not just blowing smoke. The first many failures are all "missing ; before statement" but then I came up ecma/LexicalConventions/7.1-1.js. The original test case is attached to this comment and the compressed test case is attached to the next comment.
In the final new TestCase() block, the compression has changed the constant strings in the compressed version, causing the test to fail.
I draw three conclusions:
1. The test suite depends on statements being separated by newlines, not semi-colons.
2. At least one test case is broken for reasons other than missing semi-colons.
3. The compression code is proved buggy in at least one way that cannot be fixed by "minimal coding standards."
Comment #45
bjaspan CreditAttribution: bjaspan commentedHere is the compressed version of ecma/LexicalConventions/7.1-1.js.
Comment #46
nedjoCan we agree that:
1. (Optional) compression is good.
2. Not all valid code will compress, no matter what the compression algorithm (unless perhaps the algorithm is so minimal as to be practically useless).
?
I've added the following to the upgrade instructions at http://drupal.org/node/114774#javascript-compression :
Does this documentation adequately cover the case of the (relatively obscure) remaining issues? If not, how can it be improved?
Comment #47
catchI still like the idea of three options:
Disabled
Aggregate
Aggregate and compress.
That ought to satisfy most of the concerns right? No patch for that though.
Comment #48
hass CreditAttribution: hass commented#1 you mean "very good" :-).
#2 this example looks very special, but couldn't we try to take a look on the regex's if we are able to solve this example? if not possible - sh** happens, but maybe there is an easy solution... if not
preprocess = FALSE
is the very last solution... nevertheless no point to remove the compressor.Comment #49
hass CreditAttribution: hass commented@catch: this is tooo technical for beginners... :-) developers should simply use
preprocess = FALSE
or we need to change this drupal_add_js API for D7, but for now i think it is too late for an API change like this.Comment #50
catch@hass, no worries, I'll leave you to it on this one, although Grugnog2 and RobRoy might be offended to be called beginners.
fwiw any random site admin such as me looking at the performance options will understand the difference between aggregate and compress, but if there's no need for the option, all the better.
Comment #51
bjaspan CreditAttribution: bjaspan commentedI am really astounded by this thread. nedjo and hass are arguing that a minor optimization is more important than correct functionality. I do not see how we are going to reach agreement.
Replying to #46, "can we agree that":
1. "(Optional) compression is good." I agree that aggregation is a big benefit. In this case, I'd say "compression is a minor additional benefit." Again, we're talking about a single static file that will be delivered once per visitor and then be cached by the browser.
2. "Not all valid code will compress, no matter what the compression algorithm (unless perhaps the algorithm is so minimal as to be practically useless)." Well, yes and no. To be more precise:
2a. "Not all valid code will compress *using this kind of technique*." Yes, absolutely I agree with that.
2b. "unless the algorithm is so minimal as to be useless." No, I disagree. I think if we strip comments we'll get a big percentage of the gain of this kind of compression with almost no risk (but not zero risk) of breaking valid code.
2c. Not all valid code will compress *using any technique." Disagree. We could presumably deliver gzip-compressed files quite reliably to browsers that accept them. Using a little mod_rewrite magic, this can be accomplished as a static file serve, not requiring launching PHP at all. Probably not in D6, though.
Your change to the docs does not help. You might just as well have written: "Drupal's JavaScript compression code is buggy in unknown ways. Try it and see if it works for you (and hope you notice any failures before you release your module). If not, then you'll have to disable aggregation for your module, making it less efficient for everyone who uses it, or try to debug how Drupal is breaking your code. Have fun!"
I think we are all just repeating ourselves now. I've made my case. I will not restate these positions of mine again and encourage others to do the same. If someone has new information to contribute, by all means do so. Otherwise, I suggest we leave this up to the core committers.
Comment #52
Owen Barton CreditAttribution: Owen Barton commented@nedjo - So we are essentially saying "compression may introduce random bugs in your code, and we don't know what they are (apart from the semicolon issue) and we can't be bothered to find out what these bugs are and fix them or tell you how to work around them"? OK, so that is not a very diplomatic way of putting it, but it still just feels far too sloppy to me for something in core :)
Comment #53
hass CreditAttribution: hass commented@bjaspan: it sounds like a religious war for me... you found one small piece of code that does not work. Maybe you found this 1% that will break... why are we discussing about a possibility that no one will see in real live or do we know better?
JS libraries are often marked as "compressor compatible" or something like this. I don't know why we should care about buggy or ultra special JS regression test code from Mozilla or others. They can fix their code and all is fine. Others using their code will tell them the same or will request for compressor compatibility...
Why are you not simply going to fix the regex if there is a bug or you think there is a bug not related to semicolons?
Comment #54
nedjoI understand the argument that it's wrong to introduce a feature (even an optional one) that can break valid Javascript code. I acknowledge that this as a valid position. But I don't share it, for the reasons given.
I'm not arguing that our particular (fairly minimal) compression algorithm is perfect. If people have the energy for tracking down and documenting or preventing possible edge cases (whether or not there's any sign that they represent real problems for actual Drupal development) I think that's commendable and would be happy to see further improvements.
I'm also open to the idea of enabling developers to trigger aggregation without compression. This doesn't look like a lot of work, and wouldn't break any existing code (since the optional
$preprocess
argument happens to be the last one indrupal_add_js()
). Attached is a first untested draft of the minimal patch needed to make compression optional, if anyone wants to push that direction forward.Let's focus on the positive energy needed for reaching agreement and identifying and addressing issues.
Comment #55
chx CreditAttribution: chx commentedthat's one fine patch that can satisfy everyone.
Comment #56
bjaspan CreditAttribution: bjaspan commentedI whittled away at the test case I identified yesterday to make a smaller example of some valid JavaScript that breaks under compression. Input:
Output:
Something about the "\" in the comment is confusing the regexps. If you remove it, the code compresses correctly. So, "minimal coding standards" apparently need to include "don't put backslashes in your comments."
Comment #57
hass CreditAttribution: hass commentedOnly as a side note - the docs about D6 coding standard für JS has been created on "31/08/2007 - 20:34 by pwolanin" at http://drupal.org/node/172169.
Comment #58
Dries CreditAttribution: Dries commentedI must say that this issue makes me a bit nervous. The more I think about it, the more I dislike the fact that we're imposing limitations. The past weeks, I've talk to people who would like to use or integrate additional Javascript libraries (not jQuery plugins).
I'm going to think about this some more. (I'm still traveling so it might take me a day or two to get back to this.)
Comment #59
nedjoI hear agreement from everyone that we need to support external Javascript without changes. The remaining questions seem to be:
A personal concern with dropping compression altogether, as I commented in the original JS aggregation issue, is that
Tangentially related: I realized we didn't have any documentation to help module authors decide when aggregation is and is not appropriate. With CSS aggregation we note that
I've added some notes to the update instructions, http://drupal.org/node/114774#javascript-aggregation-compression, improvements would be welcome, and I suppose we should also have this in the code documentation.
Comment #60
Owen Barton CreditAttribution: Owen Barton commented@nedjo - either option 2 or 3 sounds fine to be. Option 1 would be OK too, as long as we made it possible to hook up some compression as a contrib module.
I pointed out the issue of redundant caches several times in the original CSS aggregation thread(s), but couldn't get anyone to see the issue. Hopefully people are now getting what is happening and why this is a problem! For D7, I would actually like us to move to a hook_resources or a .info declaration that returns an array of all the css and js that should be aggregated on all paths. The drupal_add_js/css functions would then be reserved for including the specialized or admin css/js on just the pages where it is needed (and that hence should not be aggregated - we should drop the aggregate parameter in this case). Moving to a more explicit model makes this impossible (or at least very hard) to screw up.
The alternative is to improve documentation and education such that it is exceedingly clear that aggregated css/js should ONLY ever be added in hook_menu nocache, with NO conditional logic (i.e. on every path etc). Personally I think making the API a more bulletproof is a simpler path here :)
Comment #61
hass CreditAttribution: hass commentedGrugnog2: This is OT here, but drupal_add_js shouldn't be moved into static .info files. I'm changing this files dynamically in themes... and i'd like to use aggregate and compression together with dynamic added JS and CSS files. The D6 change moving theme files into themes .info files is not good and very limiting.
Comment #62
Owen Barton CreditAttribution: Owen Barton commented@hass - moving css/js includes into .info files (or anywhere else) does not stop you messing with the list in your theme (although I think the files would already be aggregated at this point, so I don't see how you can do this now). The css or js array would still be built as normal (from both sources), it would just be architecturally constrained so that aggregated files were always global, which means we don't get lots of (mostly) duplicate caches wasting bandwidth.
Comment #63
hass CreditAttribution: hass commented@Grugnog2: Sorry i don't understand you... to shorten this - you may take a look into http://www.yaml-fuer-drupal.de/de/download (D5 or D6 Theme). See one of the "template.php" files how this is done today and i assure you it will become a much more configurable theme (with custom theme settings) in the next release and therefor have more and more "dynamic" CSS files.
Comment #64
Owen Barton CreditAttribution: Owen Barton commented@hass, what I am saying is that the css files would be added from the hook or info file using drupal_add_css(), just in the same way they are now, except by a core function. The difference is that the 'global' css files would all be added via a centralized global mechanism (not each module calling drupal_add_css() in random places, which is what leads to duplicate caches), and hence could be safely aggregated. There would be nothing stopping you adding or removing the files in your theme as you do now using drupal_add_css() and $css = drupal_add_css(); (to remove a file). The css 'entering' the theme would look and feel exactly the same as it does now. The same approach could apply to JS too, of course.
Comment #65
nedjoSo far in this thread we've focused on the (valid) use case where we want aggregation but not compression for a given JS file. But there's also the opposite case: some files will appear only infrequently (not good candidates for aggregation) but will benefit from compression.
In fact we have several examples of this opposite case in core. E.g., in color.module we currently have this call:
drupal_add_js('misc/farbtastic/farbtastic.js');
farbtastic.js is seldom used and presumably shouldn't be aggregated, but does support and could benefit from compression.
The conclusion seems to be, JS aggregation and compression are distinct needs and so we should make them independent from each other. We would keep the distinct $aggregate and $compress arguments in
drupal_add_js()
from the patch at #54, but evaluate and respond to them separately, such that a given file could be aggregated but not compressed, compressed but not aggregated, or neither or both.Comment #66
Crell CreditAttribution: Crell commentednedjo: That would be an API change, and it is way too late IMO to start worrying about all permutations of compression and aggregation in Drupal 6.
(Setting back to RTBC because it was before, although I have not tested the most recent patch myself.)
Comment #67
bjaspan CreditAttribution: bjaspan commentedThis issue has been marked RTBC for a while but with no clear consensus about what to do. The only patch that received a review and got marked RTBC was #6 but it no longer applies due to whitespace changes in common.inc.
The currently committed JS compression code is demonstrably wrong in a serious way: backslashes in comments can modify text in constant strings elsewhere in the code. This could easily lead to serious but unnoticed bugs and is not something that can be addressed by coding standards. There are also other bugs revealed by Mozilla's test suite that we have not isolated. I maintain that this situation has to be fixed, either by fixing the packer or removing it; we just can't knowingly be introducing unknown bugs into Javascript.
So:
1. I have re-rolled my patch to remove JS compression but leave in JS aggregation. Since it was marked RTBC before I am leaving it in that state.
2. If someone fixes the two known problems in the compression (newlines/semi-colons and backslashes in comments) I will run the new patch against the Mozilla test and help identify any additional problems. If/when the compression passes the test suite, I'll support its inclusion.
Comment #68
catchI think barry meant to leave this RTBC. I'm not expressing an opinion, just sweeping the queue.
Comment #69
bjaspan CreditAttribution: bjaspan commentedYes, I did; sorry 'bout that.
Comment #70
bjaspan CreditAttribution: bjaspan commentedI decided to change the compression code not to remove any newlines and see what other errors popped up. The problem I reported in #56 (backslashes in comments seem to break things) is still there. Here's another: The code
got turned into
The space after the regexp /x*/ got removed resulting in an "invalid modifier" error.
Comment #71
nedjoCan we can live without JS compression for D6? Sure, just as we can live without any particular new feature. What feels important is that we're mapping out a sound direction.
Should we remove JS compression permanently? Should we support only "100% safe" compressors (if such exist, which I still tend to doubt, at least if they're trying to do more than the minimum)? Should we be enabling compression implementations in contrib?
Some relevant sites and posts:
http://blogs.pathf.com/agileajax/2006/08/compressing_you.html, now somewhat old, compares various compression solutions, most or all of which at that time introduced JS errors.
http://developer.yahoo.com/yui/compressor/, includes the lofty phrase "100% safe"
Dojo Shrinksafe, http://alex.dojotoolkit.org/shrinksafe/,
JSLint, http://www.jslint.com/, sometimes used to analyze and prepare code for compression support.
Comment #72
bjaspan CreditAttribution: bjaspan commentedMy answers to your questions:
1. Should we remove JS compression permanently? No.
2. Should we support only "100% safe" compressors? In core, yes, absolutely. If that means just concatenating files, so be it. Correctness trumps performance.
3. Should we be enabling compression implementations in contrib? Yes, I would support that. We can add a js_alter hook or somesuch. Sites that want to install riskier JS compressors would then be free to do so.
FYI, I discovered another problem with this compression: a custom jQuery script that works fine when compressed on FF but not on Safari. No syntax errors, but event handlers seemed to be disabled. I didn't look into the details, I just converted the compressor to "append only" and the problem went away.
Comment #73
Dries CreditAttribution: Dries commentedI propose that we continue to support JS aggregation, but that we remove JS compression from core until we have a 100% safe compressor. As illustrated by our early testers, correctness and compatibility matter a great deal.
A 100% safe compressor, btw, might be a gzip compressor that simply gzips the aggregated JS. I'd expect it to generate files that are smaller than a high-level JS compressor. (Not validated but probably worth exploring).
Comment #74
catchIn that case bjaspan's patch in #67 remains RTBC, removing compression but leaving aggregation. I got some "removing trailing CRs" messages so have just converted to unix format. Obviously don't credit me if this gets committed
Comment #75
Gábor HojtsyThe settings form in system.admin.inc also needs modification, as it says JS is compressed/optimized.
Comment #76
Gábor HojtsyAnyone up for fixing the remaining TODOs so we can get this into RC1?
Comment #77
keith.smith CreditAttribution: keith.smith commentedLate to the party, here, but this changes the text under "Bandwidth Optimizations" to read: Drupal can automatically aggregate external resources like CSS and JavaScript into a single cached file. This can help reduce the number of requests made to your website, and ultimately, may reduce server load, bandwidth requirements, and page loading times.
Comment #78
keith.smith CreditAttribution: keith.smith commentedThat last one wasn't quite right (since CSS is compressed and aggregated, while JS is only aggregated). New version attached.
Comment #79
Gábor HojtsyThanks. I am a bit sad that there were no big efforts to fix the bugs identified and do more extensive testing, but as Dries noted, we are better with something which works still, instead of shipping with a broken solution. Thanks Keith for stepping in and helping fix the descriptions.
Comment #80
(not verified) CreditAttribution: commentedAutomatically closed -- issue fixed for two weeks with no activity.
Comment #81
ximo CreditAttribution: ximo commentedThis tool might be useful: http://compressorrater.thruhere.net/
It tests JavaScript code in a number of packers/minifiers, with and without gzip compressions.
(I didn't find any better place to post this but on this closed issue)
Comment #82
nedjoI opened a new feature request, http://drupal.org/node/210447, for adding back in an improved approach to Javascript compression.
I also removed the compression instructions I'd previously added to the module updating instructions.
Comment #83
RobRoy CreditAttribution: RobRoy commentedJust for reference, I just found some JS that is broken by the compressor so this was actually worthwhile to take out.
It doesn't remove the comments correctly in the 'touch' and 'intersect' cases and results in...
which is broken.
Comment #84
hass CreditAttribution: hass commentedIs this the same like http://drupal.org/node/211915 what i've found in devel_themer?
Comment #85
Flying Drupalist CreditAttribution: Flying Drupalist commentedThis is still a problem, though I'm not sure if it's the same problem.
If there is a comment at the start of your js file it will break. I discovered this only after hours of testing.