See #11416-23: Please provide *.zip downloads. for more.

Soon, we'll have both .zip and .tar.gz packages for any new releases or -dev rebuilds. I have two versions of the patch for the packaging script awaiting review. One of them just does the .zip packaging. The other (based on prior work I had already done) also handles DOS vs. UNIX line ending conversion on all text files in the directory tree. So, the .zip files would include copies of all the code with DOS line endings, while the .tar.gz would include the UNIX line endings versions.

This issue is basically an RFC for the other issue. Unfortunately, no one's replied to my questions over there, so I'm opening this infra issue to hopefully get more of the interested parties to see it and weigh in. We might as well just discuss it here, then I'll circle back over there and test/commit/deploy whatever approach we agree on.

Here's the relevant portion of the other issue to consider:

The second patch introduces a bigger change: UNIX vs. DOS line ending conversion. The basic idea is that before we create a .tar.gz we ensure that all the text files in the directory tree have UNIX line endings, and before we create a .zip, we ensure all the text files have DOS line endings. This is mostly code I wrote for TNT, but merged into the d.o packaging script (especially all the packaged distribution stuff). I'm not 100% sure we want this. Here are the implications of the change:

A) Platform-specific packages will have platform-appropriate line endings (probably a good thing).

B) Lots more file I/O during packaging runs. I'll try to get nnewton to take a look at this to see how he feels about it. ;)

C) All the text files (.php, .inc, .css, whatever) in the packages will be packaged with modification times from when the packaging script ran, not from the last time those files were committed in CVS. The conversions all happen after we do the logic to find the most recent change in the directory tree when deciding if a -dev tarball needs to be rebuilt, so it won't break that. I'm not sure anyone will notice or care, but maybe they will.

Now that I think about it, it should be a relatively easy change to the convert_file() function in here to stat the target file before it does anything, do the conversation, and then use touch() to set the modification and access times back to how they were before the conversion. Other than adding even more file I/O to the whole process (making B a bit worse), I think this would eliminate any potential trouble from C.

Thoughts?

Thanks,
-Derek

CommentFileSizeAuthor
#5 Selection_2003.png28.94 KBgreggles

Comments

gerhard killesreiter’s picture

gerhard killesreiter commented

I think that this is overengineering. Did anybody request this feature?

Most people will not look at the files at all.

greggles’s picture

greggles
he/him
Primary language English
Location Denver, Colorado, USA
commented

Having had to suffer trying to read Drupal's txt files on windows in notepad, yes please. This is a great feature.

I think it's more important for the files to be easily read in a typical text editor than it is for them to have the exact right timestamp. It would be possible to use set the right timestamp (if nothing else, touch -t can do that) but that adds even more complexity and file IO.

webchick’s picture

webchick
she/they
Primary language English
Location Vancouver 🇨🇦
commented

This sounds like an absolutely horrible idea to me. Am I missing a cluestick?

It means that some people on my site building team who check in new modules to our version control would have completely screwed-up line endings depending on if the code they're checking in came from a .zip file or a .tar.gz file? It also means that even though .zip is a far more commonplace file extension that almost everyone is familiar with, we're assuming Mac and Linux people won't choose it?

Let's just get .zip files there, and not monkey with the line endings, IMO.

deviantintegral’s picture

deviantintegral commented

webchick's summarized all of my concerns. I also wonder about the situation where users use Windows to download, but are uploading to a *nix FTP server.

Let's keep *nix line endings as the "one true line ending".

greggles’s picture

greggles
he/him
Primary language English
Location Denver, Colorado, USA
commented
StatusFileSize
new Selection_2003.png28.94 KB

#3/#4, Hopefully development teams can standardize. I think "folks who use version control to manage their sites" are most likely to be savvy enough to figure out the line ending issue.

The zip downloads are aimed at new and/or less savvy users and we want to provide them something as useful as possible.

I've attached a screenshot from what a typical windows user will see when they double click on the INSTALL.txt. It's really not user friendly. Unfriendly enough that I imagine a fair number of folks have just said "Well, this is weird. I guess Drupal's wrong for me."

webchick’s picture

webchick
she/they
Primary language English
Location Vancouver 🇨🇦
commented

Yeah, I know that's a problem for Windows users. I remember it well. Which is why about 30 seconds into my life as a web developer, I downloaded Notepad2/Notepad++/whatever, like every other Windows-based web developer.

And when I did use Windows, it was so I could put my stuff on a Debian web server. The kind that shows ugly "^M" everywhere when you upload a file with Windows line endings, making it insanely difficult to edit configuration files like settings.php.

But mostly, I reject is the notion that .zip == Windows. I download .zip files all the time and haven't used Windows since 2006. If we want to make two separate .zip downloads, one of which is marked "Windows", and I can instruct anyone I ever work with (and anyone they ever work with) never, ever to download them so our version controlled files don't get screwed up, then I guess I could get behind that. But I think that's adding a bit too much emphasis to such a simple little feature, personally.

Josh The Geek’s picture

Josh The Geek commented

Subscribing.

unzip is easier to remember that tar xzf, and they are Linux commands. -1

moshe weitzman’s picture

moshe weitzman commented

-1 from me. Introduces issues in the process of solving one that isn't *that* painful.

pwolanin’s picture

pwolanin commented

we should standardize - I agree with #4 that we should force unix line endings in every file.

The code should be identical regardless if how it's packaged. The possibly weirdness and confusion this will introduce seems to far outweigh any benefit.

dww’s picture

dww
we/he/they
commented

Sounds like a fairly overwhelming rejection of the idea of including DOS line endings in any files (for various reasons, some of which I think are more compelling than others).

However, the proposal in #9 to force UNIX line endings is new. We don't do that now. We just package whatever's in CVS. If we wanted to force UNIX line endings, we'd need to use a variant of the bigger and more complicated patch at #11416-23: Please provide *.zip downloads.. If we want to do that, cool, but it still makes the packaging more complex, I/O intensive, etc.

If we want the minimal possible change and the highest consistency, we'd do neither. We'd always package whatever's in version control just like we do now. That's the simpler patch in #11416-23.

Cheers,
-Derek

webchick’s picture

webchick
she/they
Primary language English
Location Vancouver 🇨🇦
commented

No, leave whatever was in CVS alone. If the module/theme developer develops in Notepad for some ungodly reason, it would be a huge WTF if suddenly the next time they opened their module up it looked akin to greggles's #5. Us Unix-types can always file patches against the module to convert the line endings in bulk and inform them about better alternatives.

webchick’s picture

webchick
she/they
Primary language English
Location Vancouver 🇨🇦
commented

Also, is there a particular reason that these two discussions need to be held at once? Those seem like two completely separate features to me: a) .zip downloads for files (a no-brainer) and b) manipulating the contents of files prior to them being packaged for some undefined purpose (contentious). I don't understand why a) is being held up by talking about b)... couldn't it always be added at any time in the future if enough people felt it was super important and were able to sway the others?

dww’s picture

dww
we/he/they
commented
Status: Active » Closed (fixed)

Re: #13: They're separate features, but they're definitely related. If you make the (perhaps false) assumption that .zip files are primarily going to benefit windows users, and we're mostly providing them to lower Drupal's barrier to entry for people on that platform, it's worth considering what else we should be doing with those packages while we're at it.

The other issue is postponed on the outcome of this discussion because I had already written the code for both approaches and needed to decide which one to proceed with, yet no one was reviewing the patch. So, I wanted to start a new issue in a more visible place to get the necessary feedback on which patch to move forward (at least for now), but I didn't want the conversation to happen in two places. Therefore, this one is active, and the other is postponed, so it was clear to everyone I wanted their comments here, not there.

That said, it seems we have strong agreement that line ending conversion isn't a good idea (at least not now), so I can safely and happily proceed with the simpler patch. If/when we want to revisit this question, we can reopen this issue (or start a new one).

Thanks to everyone who contributed meaningfully to this discussion.

dww’s picture

dww
we/he/they
commented

p.s. I meant to add: If we *were* going to start doing DOS line ending conversion, it'd certainly make a lot more sense to introduce it when .zip files first appear. It's going to be even more of a WTF if we started doing that at some later date (unless we add a 3rd *-dos.zip download for every release). So the fact we decided not to do this means we're probably never going to do it. That's fine. But that's a big part of why I wanted to get more feedback on the proposal before I just decided on my own to do it or not, hence, having the two conversations at once.

And, to be really picky about it, there's really only one conversation happening (rather, that just happened). ;) There's absolutely no question that we're providing .zip files ASAP. There's nothing else to discuss about [ #11416] -- that's completely decided.

However, I'd like to encourage anyone interested in moving this whole effort forward to look at one other implementation detail that's still under debate:

#652566-16: Support multiple download files per release in release history XML

Thanks,
-Derek