There are two basic problems with not shipping Drupal with a files directory:

- as Chris Messina noted (http://factoryjoe.pbwiki.com/FeedbackForDrupal6) this makes Drupal 6 return with big red error boxing right after installation, as the file system is not set up yet (so JS and CSS aggregation are not possible for example)
- JavaScript localization is not possible to be used in the installer, and also not possible to be used until the files directory is set up, as it is using static JS files composed of strings from the Drupal database. So a localized Drupal will have a broken localization, until the files directory is properly set up.

JirkaRybka cooked up the solution, I am just fixing small issues with his patch and breaking it out from our installer localization issue (http://drupal.org/node/190283), as it is for broader discussion. The attached patch does a few simple things:

- adds a files directory with a README.txt
- modifies files directory check to work nicely in installer applying special wording for the installer (this one I modified a bit from the original patch and also fixed two small bugs with building the resulting array)
- modifies the installer requirements display to be able to display error messages without version numbers (as this part was originally intended for version compatibility checks)
- also modifies the page title to that effect

Patch and screenshot attached.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Gábor Hojtsy’s picture

Uh, of course exactly that error will not show up, as there will be a files directory shipped. The "not writable" error will show up which is very similar.

joshk’s picture

I think shipping with a /files directory is a ++

I think the solution for the user-experience is to have slightly separate logic in the installer and not display this as an error, but rather a TODO. "You should enable file uploads and other features by making your files directory accessible to drupal. For more information on how to do this, click here." We should have a better handbook page on what this entails.

The user will need to alter filesystem permissions, which is an unavoidable part of the install, but we don't need to present it to them as an error or warning, especially not at this stage when we know it will still be un-done.

cwgordon7’s picture

Category: bug » task
Priority: Critical » Normal

Although this may be an interesting idea, I hardly think it qualifies as a "critical bug report."

Personally, I don't see a reason not to include the "files" directory: what was the original reasoning behind that? If I understood that, maybe I'd change my mind, but I think it's a good idea.

Gábor Hojtsy’s picture

Category: task » bug
Priority: Normal » Critical

We should not release D6 without this == critical. It is also a locale bug as explained in the post. Argue with the specific points instead of the principle if you think otherwise.

wmostrey’s picture

Patch applies cleanly, things work as expected on a new install. The readme is clear but the line-up might be more in check with the sites/all/README.txt:

This is the default directory used for all your uploaded files,
as well as some temporary files created by Drupal. It is highly
recommended to consider whether this location is matching your needs,
especially if you are going to run multiple sites from the same
codebase. You can adjust the settings at the following url:

  admin/settings/file-system

After doing so, you will be able to remove this directory. Make sure you
move all files, if any, to the new directory. Changing the setting later
might cause problems.

Gábor Hojtsy’s picture

Modified README.txt as suggested.

keith.smith’s picture

This is all clear, except for perhaps the first paragraph here:

This is the default directory used for all your uploaded files,
as well as some temporary files created by Drupal. It is highly
recommended to consider whether this location is matching your needs,
especially if you are going to run multiple sites from the same
codebase. You can adjust the settings at the following url:

  admin/settings/file-system

After doing so, you will be able to remove this directory. Make sure you
move all files, if any, to the new directory. Changing the setting later
might cause problems. 

What about something like this?

The files directory is the default file system path used to store
all uploaded files, as well as some temporary files created by Drupal.

If you wish to store uploaded files in a different location, modify
the settings for the file system path at the following url:

  admin/settings/file-system

If your site runs multiple Drupal installations from a single codebase, 
you may wish to set the file system path of each installation
to a different directory so that uploads do not overlap.

Changing the file system path after files have been uploaded may cause 
unexpected problems on an existing site. If you modify the file system path
on an existing site, remember to copy all files from this directory to the
new location.

This directory may be removed if the file system path has been modified
to point to another location. 
keith.smith’s picture

Status: Needs review » Needs work
Gábor Hojtsy’s picture

Good suggestions. I would also say URL instead of url.

catch’s picture

This may be dealt with elsewhere, but why not sites/all/files?

Gábor Hojtsy’s picture

This is already dealt with elsewhere. See http://drupal.org/node/166169 and http://drupal.org/node/98824 for in-depth coverage.

People are moving away from using the "sites" folder for files as it complicates moving domains or moving from a staging site to a production site. The solution suggested in the above issues is to use "files/site-specific-alias", which alias would not change with moving from a domain to another or moving from staging to production. So the root files folder seems to be here to stay. The first time you visit the files setup page, if PHP has permisisons, it creates the root "files" folder for you automatically right there, not in "sites", and this is the default files location. So we just move the same "creation" earlier, not changing how Drupal works in any way.

catch’s picture

Thanks! I'd missed those two. fwiw Keith's updates to the text look good.

Gábor Hojtsy’s picture

Status: Needs work » Needs review
FileSize
5.76 KB

New patch rolled with text suggestions from keith.smith.

JirkaRybka’s picture

Status: Needs review » Needs work
FileSize
14.37 KB
12.33 KB

Oh, I'm working on this the whole evening, got out of sync a bit :-(

Anyway, attaching my patch now, with the above text changes NOT included yet. Hope to merge that soon.

My patch, besides of more changes to the text, which will be probably further changed with the above, have these new things:

--- Updating also INSTALL.txt, which is highly needed here.

--- Changing the installer's "Verify requirements" phase a bit:
Currently, every single requirements error (i.e. anything found by drupal_check_profile() OR missing modules OR settings.php not writable) ends up with it's own error page. So basically, if you have more problems of different sorts, you are only able to see one of them. After resolving it and reloading the page, you'll see another ONE problem. And so on... This is not very nice.
Even worse, with this patch, if you just upload the tarball without caring about write permissions, you'll have TWO problems (files AND settings.php not writable). The process of resolving these by showing you errors one-by-one is horrible, you'll need to go into your filesystem permissions in a two-pass process before proceeding.
Additionally, the problem with settings.php not writable doesn't show under "Verify requirements" phase - it goes under Database setup.

So I changed the error handling, so that the single functions don't just print an error page and exit, but only just set error messages. This also meant removing install_missing_modules_error() - in fact a "theme function" for one of such isolated error pages - and moving the settings.php check from install_change_settings() to install_check_requirements(), to avoid proceeding to the db-settings form if there are other problems. The error page is then printed if any error messages are pending, which means a single consistent page, with all the errors collected nicely. Screenshot attached.

The text changes to follow soon.

JirkaRybka’s picture

Status: Needs work » Needs review
FileSize
12.67 KB

It went faster than I thought...

This patch is identical to the above, apart from the README.txt wording:

- If you're going to change the files path setting, you need to create that directory manually first. Now mentioned.

- You're only going to be able of such changes AFTER you've installed Drupal. This is README.txt file, people are supposed to read it before they start. And they may get scared by the notices about trouble with moving it later.

- The provided path is NOT a usable URL, especially to a new user who didn't even start the installation yet. Since this is a static textfile, we can't provide a valid URL (domain / directory / clean URL - depending), and instructions about that path would be pointlessly verbose.
So I included rather a "walk path" how to get there, which is in line with already existing occurence in INSTALL.txt

- Added a mention about the requirement of files directory being writable. This is maybe the biggest point here, and got lost somehow.

Corrections welcome, I'm not a native speaker.

Gábor Hojtsy’s picture

http://drupal.org/node/145335 was marked a duplicate of this bug, as it tried to deal with the settings.php error on a smaller scale. You might be able to pick up some good stuff from there too. Although the verbose explanation on permission settings as used there was voted down so far there.

JirkaRybka’s picture

Ok, I examined the latest patch on that issue, and I see that the only bits not covered here are the proposed changes in UI messages, specifically:
All necessary changes to %dir and %file have been made, so you should remove write permissions to them now in order to avoid security risks. To do this, you'll need to change permissions of %dir to 'read' and 'execute' only for 'owner', 'group' and 'other' and %file to 'read' only for 'owner', 'group' and 'other'. This can be done via a control panel on shared hosting or by typing 'chmod 555 %dir' (without quotation marks) and 'chmod 444 %file' in the shell command line.
As for this one, I'm not a fan of anything past the first sentence. It's quite too long, and applies only to one of possible operating systems. (Also lees knowledgable users are likely to be on shared webhosts with limited access to the system, so they rather need instructions on FTP clients for file system, PhpMyAdmin for SQL tweaks, and poormanscron.module perhaps, rather than *nix commands examples they're unable to put anywhere. I came through the confusion already when I started with Drupal a year ago, took me a while to realize I should ignore all examples and google for workarounds instead.)

I think this all may be covered on a linked manual page (as suggested in the other issue), so I would put in just the first sentence and a link to manual.

The @drupal installer requires special permissions to %file during the installation process. In order to continue, you'll need to change permissions of %file to 'read' 'write' and 'execute' for 'owner', 'group' and 'other'. This can be done via a control panel on shared hosting or by typing 'chmod 777 %file' (without quotation marks) in the shell command line.
Well... The same as above. I'll put in just a manual link, somewhere.

Let's discuss this a bit, before re-rolling. Opinions?

Gábor Hojtsy’s picture

Title: Ship with a files directory and check in install » Installer usability: better errors and shiping with a files dir

As stated there, I think a link is better.

keith.smith’s picture

FileSize
11.96 KB

Attached is the same patch as the one in #15, but I tweaked the language in README.txt and substantially added to the language in files/README.txt. I believe that the information now in files/README.txt should be sound advice (as to when someone might want to change their file system path), but please double check.

Since I reference the files/README.txt file in the INSTALL.txt, I removed the portion from the files/README.txt that said the directory (and also, then, the README file) could be deleted if it wasn't being used.

I re-worded one message in system.install to move a clause to the front of the sentence, but other than that, did not mess with the original patch.

I should also note that with the patch applied, everything functioned as I expected it to -- multiple errors were shown on a single page as expected, along with a "try again" link. After changing the appropriate permissions and clicking the "try again" link, installation proceeded as normal.

keith.smith’s picture

Title: Installer usability: better errors and shiping with a files dir » Installer usability: better errors and shipping with a files dir
snufkin’s picture

FileSize
11.03 KB

It seems that files/README.txt is missing from the last patch. Rerolled with it (copied it from #15, where it was still there), and tested: installation works as expected, files directory now throws no errors. If there is nothing more i can test on this, its RTBC.

keith.smith’s picture

Thanks, snufkin.

I think your patch in #21 has some merge conflicts.

I'm not where I can really re-roll a patch right now; in #19, I meant for the files/README.txt to read:

// $Id$

The files directory is the default file system path used to store
all uploaded files, as well as some temporary files created by Drupal. To
successfully install Drupal, the files directory must exist and be
writable by the file server process.

After installation, the settings for the file system path may be modified
to store uploaded files in a different location. Ensure that this new
location exists, is accessible, and is writable by the file server process.
The file system path settings can be accessed by selecting these menu items
from the Navigation menu:

  administer > site configuration > file system

You may wish to modify the file system path if:

  * your site runs multiple Drupal installations from a single codebase
    (modify the file system path of each installation to a different
    directory so that uploads do not overlap between installations); or,

  * your site runs a number of web server front-ends behind a load
    balancer or reverse proxy (modify the file system path on each
    server to point to a shared file repository).

Changing the file system path after files have been uploaded may cause
unexpected problems on an existing site. If you modify the file system path
on an existing site, remember to copy all files from the original location
to the new location.

Sorry for the confusion, and when I get near a better computer tomorrow I'll attempt to actually add this file into the patch from #19.

There are still JirkaRybka's comments in #17 to sort out as well.

snufkin’s picture

FileSize
11.29 KB

Hmm I tested with a clean install, and there was no merge problem with the patch. I rerolled it with your modifications to files/README.txt.

I would agree with JirkaRybka's comment, the less detailed instruction the better.

catch’s picture

Status: Needs review » Needs work

Snufkin - patch will need to be in unified diff format to get committed.

JirkaRybka’s picture

Status: Needs work » Needs review
FileSize
14.69 KB

Yes, unified diff needed...

So rerolling:

- Patch from #19

- files/README.txt pasted from #22 without any changes (+1 to this wording)

- Included modifications originally carried in from http://drupal.org/node/145335 and mentioned in #17, now modified per #18, #23 and my own opinions:
All necessary changes to %dir and %file have been made, so you should remove write permissions to them now in order to avoid security risks. If you are unsure how to do so, please consult the on-line handbook. The link goes to http://drupal.org/Troubleshooting-FAQ - the general troubleshooting help. The first sentence comes from #17 now, as I really don't like the word 'failure' in there.
The @drupal installer requires write permissions to %file during the installation process. If you are unsure how to grant file permissions, please consult the on-line handbook. This is just a small add-on for consistency, I didn't take 'special permissions' from #17, because it sounds quite mysterious to me.
To proceed with the installation, please change the %directory directory permissions to allow the installer to write to it. If you are unsure how to do so, please consult the on-line handbook. Another small add-on to Gábor's original text.

The links to the Troubleshooting FAQ page are giving enough room to explain how to change all these permissions from command line, via FTP, on Linux, on Windows, on other systems, explain terms like file/dir ownership and that Drupal(Apache) acts as different user on the system (and so is unable to change permissions on others' files, and that's why the user needs to intervene manually, AND that's why the user might be unable to remove Drupal-created settings.php - also seen in support requests), the necessity to grant "execute" permission on *nix directories to make them searchable....

There's a lot of stuff like that (also discussed at the partially-parent issue http://drupal.org/node/145335 ) which I might call a follow-up documentation task after this gets committed. The Troubleshooting page currently doesn't say much to this subject, but I think it's definitely the right place where it should be added.

Gábor Hojtsy’s picture

I think a more precise link to an exact page where this explained would be more helpful. The Troubleshooting-FAQ seems to be a good idea, a subpage there could be created and the link pointed to that page.

JirkaRybka’s picture

Yes, definitely. But right now we have no such page, and I'm not in time-condition to suggest one, let alone the amount of topics needed to cover. Also this issue needs to move on, as installer localization fix depends on it (it seems), so I think the more precise link may be a follow-up patch once the page is created. The current link should work too (with one extra click), once that page will be added.

Gábor Hojtsy’s picture

I think it is pretty important that we link to a page with actual information or at least a generic page which focuses stuff better suited for the problems at hand. So I looked over the handbooks, and realized the Getting started guide (which could possibly explain such things) is not really wordy about permissions, in fact it does not mention any permission needs, although Drupal 5 needs permission alignments too. Anyway, linking to that would be a possibility, exactly http://drupal.org/getting-started/6 will become the getting started guide for Drupal 6 (no, there is no such page yet, but that should not stop us from linking to it :)

Otherwise I also took over an empty "Installation/Configuration" titled page (it was strange) at the Troubleshooting FAQ and repurposed it: http://drupal.org/node/1213 This could also be a possibility, but the more I think about it, the more the getting started guide looks appropriate

JirkaRybka’s picture

FileSize
14.68 KB

OK, now pointing to http://drupal.org/getting-started - given that the 6.x page isn't there yet, AND the link doesn't need to be updated per-version this way. Further improvements welcome.

Gábor Hojtsy’s picture

Looked through the patch and only have some string change suggestions (otherwise all looks fine including the link). Let's discuss the README weirdness I see, and I can always fix the code comments on commit. So we hold off this for patch as short time as possible.

So "writable by the file server process" is misleading. We are not dealing with a "file server", but a "web server". I understand "file server" refers to the Drupal file serving process or something. Anyway, it needs to be writable by PHP at the end, whether it runs as a server module or under a different user/process.

JirkaRybka’s picture

Yes, that's not ideal wording, probably. I'm not very sure on English terms in this area, but I would also write "web server". Also note, that existing INSTALL.txt is currently saying: "The sub-directory requires "read and write" permission by the Drupal server process." - this is perhaps most accurate, stating that read AND write permissions are needed, for the server process related to Drupal (however we might call it).

As for myself - feel free to change this bit to any of these, I will not scream :)

keith.smith’s picture

FileSize
14.67 KB

"file server process" was probably something I did by accident. I'm sure I meant "web server process". Oops.

sepeck’s picture

When did people move away from using the /sites directory? Where has the practice / trend been noted or documented / discussed?

The /files/site1 files/site2 was opted out in the past in favor of the sites directory approach to make long term backups and such more easy as all files specific to a site were consolidated in one location/directory. Something more common then moving sites to different domains which is encountered more in development rather maintenance.

If you are going to make plans to rely on documentation hard links in the handbook please kick me an email so I don't miss these issues and the suggested starter text that goes with them.

Gábor Hojtsy’s picture

sepeck: The issues I linked in http://drupal.org/node/191310#comment-627434 give you some background. There is no reason one cannot symlink their files/sitekey folder to sites/.../files so the backup would work nicely. But breaking links to valuable content is not something people want, as discussed in the issues I pointed to above.

sepeck’s picture

I will follow up on that. Note, symlink assumes *nix OS. There is no reason to assume *nix OS and Links in Windows system not as widely exploited / experienced on Win32 platforms, so 'can sym-link' is not a good answer.

JirkaRybka’s picture

Well, we can always move the directory anywhere, and the advice to consider another location is in there already. This issue deals with the default directory non-existence only, as a fix for error messages and localization problems during install and post-install phases. Additionally, it'll make the default really working for small setups with no need to tweak the path, i.e. usability.

Otherwise, this is no change - we had default path 'files' and that's unchanged.

keith.smith’s picture

One possibility to address sepeck's concerns would be to add some variation of a third bullet point to the information in files/README.txt. Sample text, very rough, below:

You may wish to modify the file system path if:

* your site runs multiple Drupal installations from a single codebase
(modify the file system path of each installation to a different
directory so that uploads do not overlap between installations);

* your site runs a number of web server front-ends behind a load
balancer or reverse proxy (modify the file system path on each
server to point to a shared file repository); or,

* your site policies specify that all site-related files are stored under
the sites directory in order to simplify backup and restore
operations (modify the file system path to point to a newly-created
directory underneath sites).

catch’s picture

I'd agree with any default files directory to get rid of that error message. Given there's disagreement about /files sites/all/files /files/sitestring - it seems best to leave as /files which has been around for years, then revisit changing it for D7.

In the meantime Keith Smith's suggestions for files/README.txt look very sensible.

JirkaRybka’s picture

Agreed with catch and keith.smith. (Plus, people were also discussing sites/default/files and sites/sitename/files, each having own pros and cons. This goes out of scope for this issue here.)

Gábor Hojtsy’s picture

Status: Needs review » Needs work

Great, get those text changes into the patch, so I can do some final testing, before getting this in.

JirkaRybka’s picture

Status: Needs work » Needs review
FileSize
14.92 KB

Rerolled with #37 added third bullet.

Gábor Hojtsy’s picture

Status: Needs review » Fixed

Great. Tested, works fine. Committed.

xmacinfo’s picture

I wish I would have seen this sooner, because I would have suggested to have the installer let the user name the "files" folder to something else. Usually I create this folder using a French name (e.g. fichiers) on French only sites.

Now I know that I will still be able to create a new files folder, but if the installer already installs things in "files" it will be almost to late to change the folder name. Unless I miss something.

We may also split the files folder in two: a "files" folder needed at install time, like the JavaScript localization files, and an administrator created "download" folder, for files downloads, attachments, etc. The download folder would be created only after the installation, like currently done in Drupal 5.

Should I create a new issue for this?

JirkaRybka’s picture

I think you may continue at http://drupal.org/node/166169 or http://drupal.org/node/98824 - both about files directory _location_.

Installer doesn't put in there anything important, I believe, we just fixed it for the sake of warnings shown and a few temporary files, which - I believe - are OK to remove. Unless you've uploaded something already, you should be able to move the directory without difficulty.

KentBye’s picture

Glad to see that this issue got acted on since it's an annoying usability issue when you're installing Drupal for the first time.

Does having a files directory ship with Drupal make it easier for people to accidentally wipe out their existing files directory?
I double checked the UPGRADE.txt, and it does say that people should make back-ups, but I didn't know if there needs to be extra warnings anywhere for all of those people who have gotten used to dragging and dropping.

1.  Backup your database and Drupal directory - especially your "sites"
    directory which contains your configuration file and added modules and
    themes, any contributed or custom modules in your "modules" directory,
    and your "files" directory which contains uploaded files.
...
8.  Copy your backed up "files" and "sites" directories to the Drupal
    installation directory. 
Gábor Hojtsy’s picture

What operating system/ftp program deletes the existing files directory with all its contents and replaces it with the one with only the README if people drag and drop? (Note that we also ship with a sites/all folder, which in the same situation would remove all the non-core modules and themes installed there). I do think that if this is a problem, it is much more widespread, and should have already materialized with the other existing dirs. Another example is sites/deault, where the settings.php is created and is not removed when you copy over the new tarball with only the default.settings.php in this folder.

JirkaRybka’s picture

Component: install system » documentation
Category: bug » task
Priority: Critical » Normal
Status: Fixed » Active

Re-opening, because we also need to set up the documentation page we're trying to link to. Let me post my starter wording here; definitely needs more work:

FILE SYSTEM PERMISSIONS

Throughout the Drupal installation and maintenance, you're likely to encounter situations, when you'll need to grant write permissions for Drupal to certain files and/or directories. Typical case is the files storage directory (with any subdirectories if created separately), and for the initial installation also the sites/default directory.

There are also situations where you want to REMOVE write permissions, to avoid security risks. This is for example the sites/default directory and settings.php file AFTER the installation is done. It's a good idea to also remove write permissions to all the Drupal code.

LINUX

Linux is the most common system used around on web servers.

To change file/directory permissions, you need to log in to the system, browse to the file/directory in question and change permissions. The exact way to do so depends on the way how you access the system:

From command line:
- Browse to the parent directory using cd [path]
- Change permissions using chmod .......[TODO: exact syntax]

Through FTP client:
After connecting to the server and browsing to the parent directory, you usually need to right-click the file/directory and use some sort of "change permissions" option, depending on the exact application you're using.

There are also other ways, such us simply using the desktop GUI on a test install, if available.

Files ownership
You need to know, that each file/directory have it's owner. If you created the file yourself, you're the owner, and you can do virtually anything to the file, including change of permissions. Drupal, in the other hand, is an automated script acting on the system as "apache", "webserver" or some different identity, which is not the same as yours. So to allow Drupal to write, you need to grant write permissions to everyone, which can't be done by Drupal automatically. If removing the write permissions, it's only necessary to remove for everyone; you may safely keep the write permissions for yourself, for the ease of further maintenance.

The same also works the other way: Any files created by Drupal (such as settings.php, or any uploaded and/or temporary files) are owned by the webserver process, and you may be unable to change/remove them if Drupal didn't give the permission. Sometimes it helps to manipulate the parent directory, but mostly such a case needs to remove the files as root user, if you've access to that. Otherwise, you might want to use some helper php scripts to execute the operation through php, and so from the webserver process perspective owning the file. Don't forget to remove any such helper scripts immediately after use, to avoid security risks!

Which permissions?
You'll probably need to care only about write permissions, but to allow Drupal for searching a directory, you also need to grant 'execute' permission to it. You probably don't want to remove read permission anywhere.

WINDOWS
[TODO]

MAC
[TODO]

[TODO: Correction by a native speaker]

catch’s picture

Title: Installer usability: better errors and shipping with a files dir » Documentation for files directory
xmacinfo’s picture

What operating system/ftp program deletes the existing files directory with all its contents and replaces it with the one with only the README if people drag and drop?

Mac OS X does this.
GUI-based FTP applications will do it too, at least the one I'm using.

When I began using Drupal I often deleted by mistake the sites folder. I had to reenter my database name, password manually.

The only folder that I did not delete by mistake was files, since it was not shipping by default.

All the Drupal folders can always be replaced in case a user delete the wrong files, but if the user delete the "files" folder accidentaly, he loses the content permanently.

If anything can go wrong, it will go wrong.

Now that we ship with files folder, backup will be more important than ever.

This is why I was suggestings having two files folder, one for the installer and the other for the web site content.

KentBye’s picture

I agree with xmacinfo about if it can go wrong, it will go wrong.

I'm on a both a local Mac environment and use a GUI FTP program -- and yes, I've accidentally wiped out my sites folder and know that it is a definite possibility for doing the same with the files folder.

I think the biggest danger of this would come from moderately experienced people who have successfully performed an upgrade before and are unlikely to read the fine print of the UPGRADE.txt. These are the people who may not realize that there is now a empty files folder that is going to wipe out their existing one when they used to dragging and dropping all folders except the sites folder.

I realize that this is probably not a best practice for people to be doing, but I'm pretty sure there are folks out there updating with drag and drop. So I guess I'm just throwing out the possibility that there might need to be a shiny red warning / reminder of this change somewhere in the release announcement. We can't hold people's hands every step of the way, but this seems to be a significant change that admins need to be aware of.

JirkaRybka’s picture

As for myself, I never drag/copy anything over my data unless explicitly wanting to change. Approaches seem to differ. But a warning in release announcement - yes, +1 definitely.

Also this page: http://drupal.org/upgrade/downloading-drupal-gui is quite relevant. It basically covers the right practice (steps II. and V.), but perhaps needs a bit of update in step III. to make things absolutely clear. Anyway, as the page stays now, it will work (move to backup, copy back after uploading new Drupal).
The command-line version: http://drupal.org/upgrade/downloading-drupal-command-line seems to be clear already.

BTW - as you're both on Mac, can you please (pretty please) contribute to the #47 proposed text, adding some basic paragraph about file permissions on that system, and how to handle it with Drupal (especially while starting with Drupal, or some common pitfalls if any)... Thanks in advance.

I might add a relevant link (probably not ideal one, but still) to the helper scripts for removing php-created stuff:
http://www.110mb.com/forum/cant-delete-files-that-were-created-with-a-ph...

Gábor Hojtsy’s picture

Title: Documentation for files directory » Shipping with a files folder screws up Mac and certain FTP users
Category: task » bug
Priority: Normal » Critical

I am also on a Mac (just since the end of August), but never experienced this overwrite behavior, possibly because I am not using the finder for moving files over each other :) Anyway, I just tested and it indeed does replace the old contents with the new. IMHO as people noted already, because the sites folder is restorable without a backup (if you only have your modules and theme there, not uploaded files :), the files folder is not. Of course if you have your files in the sites/default or sites/all folder as some people suggest, you are screwed already.

Anyway, we can just as well not ship with a default files folder, and keep bugging the user in the installer to create and write enable the files folder. If people ask why we do it, we can always answer that Apple is in fault :) IMHO this "replace on copy" feature is just broken, but I might be just too much used to the "merge on copy" behavior after years of Windows and then years of Linux desktop experience.

So in short: let's not pose such a high possibility to screw up, if we can live with the installer asking people to set up the files folder instead.

JirkaRybka’s picture

Component: documentation » install system

Then we're discussing the install system here again. If removing files directory from the package again (apart from the oddly looking requirement to create manually something that is possible to ship with), we also need to put the info from our new README somewhere, possibly into INSTALL.txt.

But personally, as I said in #51, I believe that copying ANYTHING over your valuable data is a very bas idea, and the documentation already have it right.

Edit: BTW, isn't the copy-is-merge behavior also needed for per-module translatios? This is maybe broader, if Mac behave differently.

xmacinfo’s picture

Anyway, we can just as well not ship with a default files folder, and keep bugging the user in the installer to create and write enable the files folder.

Would it more simplier to have two files folders?

ONE for the installer and it's need for JavaScript localization files. This one can be named 'files' or 'installerfiles' or whatever.

TWO in the /admin/settings/file-system page, let the user define a second storage area. This one would be used for storing pictures, upload files, etc.

So the current commit in #42 would stay. However, a modification to the sile-system page would be needed for the user to set an alternate file area. The default name could be 'download'.

Dries’s picture

I'm not sure this is a critical issue. Please reconsider the priority.

KentBye’s picture

Well, I'm torn on this because it is an annoying usability issue on first install, but as JirkaRybka says in #51 & #53 the documentation states that it's a worst practice & then we'd be creating an inconvenience to create a worst-case-scenario of wiping out the files folder.

It's definitely a trade-off of giving a good first impression versus preventing upgrade horror stories.

But how do other CMSs handle this very issue?
Does anyone have any update experience with how Wordpress or Joomla! handles this?

eaton’s picture

I'm definitely seeing this on my setup. Previously, I saw a warning on the /admin page after I installed, and had to create a files directory. Now, I get a validation error on the first stage of installation and can't proceed until I chmod the included folder.

What operating system/ftp program deletes the existing files directory with all its contents and replaces it with the one with only the README if people drag and drop?

As others have noted, this is the default behavior in OSX's Finder file browser. http://lowendmac.com/misc/05/1031.html includes a peek at the debate about that behavior in the Mac community. Regardless, it seems that we've replaced an annoyance late in the installation cycle with an annoyance early in the installation cycle -- and one that potentially eats data on some platforms.

Rather than shipping with an actual folder, couldn't we trigger the 'make a folder if permissions exist, tell the user to create one if we can't do it automatically' behavior that's used on admin/settings/files? Localization at install time seems like something that should be special-cased if it's that critical, like the installer theme.

JirkaRybka’s picture

Priority: Critical » Normal

Thinking of it:
- You should always have backups if upgrading: http://drupal.org/upgrade/tutorial-introduction already says it's 'Important!'
- Documentation suggests ( http://drupal.org/upgrade/downloading-drupal-gui ) to move your old files out of the way, and after unpacking new release, copy your site's data - manually and explicitly - back over the defaults.
- If you've ignored both the above suggestions (which is very bad itself), AND your system have the directory-replace behavior, you'll damage not only your 'files' contents, but also 'sites/all' including any custom modules and themes (!), 'sites/default' with your settings.php, any customized robots.txt and .htaccess ...
- You are free to move the files directory elsewhere on your (new) install, as suggested on multiple places now, to avoid the problem.
- I'm entirely not sure if supporting something I might call a "legacy-bad-practice-workflow" (please forgive me, no offence meant!) is a good reason to screw the experience of a struggling first-time-installing new user. As far as I can tell, you should NEVER even attempt to copy something over your data, even less without a backup.

So I agree with Dries, that this is at the very least not critical.

I agree with #50 - let's just put out a reminder, somewhere.

Edit: Eaton - you already can't proceed without chmoding sites/default, so it's not much of a difference IMO.

eaton’s picture

- I'm entirely not sure if supporting something I might call a "legacy-bad-practice-workflow" (please forgive me, no offence meant!) is a good reason to screw the experience of a struggling first-time-installing new user. As far as I can tell, you should NEVER even attempt to copy something over your data, even less without a backup.

Fair enough. I think that in the future, though, it might be beneficial for us to agree on some sort of 'user-friendliness hierarchy'. Is 'having the Javascript error messages in your native language' friendlier than 'not destroying your files directory by overwriting it with an empty one?' I'm not sure.

Edit: Eaton - you already can't proceed without chmoding sites/default, so it's not much of a difference IMO.

I just checked and realized that I'd automated the chmodding of sites/[foo] when I create a new install, so I only noticed the files directory. In my case, adding /files to the list of auto-chmodded directories will solve the problem. In other users' cases, they have to do a bit of chmodding already.

Good point.

xmacinfo’s picture

If you've ignored both the above suggestions (which is very bad itself), AND your system have the directory-replace behavior, you'll damage not only your 'files' contents, but also 'sites/all' including any custom modules and themes (!), 'sites/default' with your settings.php, any customized robots.txt and .htaccess.

There are good documentation, and bad documentation.

As well, there are good behavior, and bad behavior.

If a user deletes all his 'sites/all', his settings.php, modules, themes and robots.txt and even .htaccess files, no harms is done. It easy to reconstruct all these files. And the web site will still run even if somes modules are missing.

If a user deletes 'files', now this is a big issue. All the files inside the 'files' folder can not be recreated.

I know, backing up data should always be done first. However, even if a good backup is done, we should not let in any way the user replace the 'files' folder. That would be conter productive.

If we cannot create a second set of 'dowload' or 'files' folder, I suggest that we roll back the commit as it was before.

JirkaRybka’s picture

I disagree. There are dozens of other ways to damage your site, as well as database, through commonly used filesystem and database tools. Not documented in Drupal handbooks even. This is documented well.

Elementar safety precautions are always needed when manipulating your files, and when upgrading in general.

If some people are just used to something else, that's not a reason to roll back.

catch’s picture

In terms of experience, someone installing for the first time is 99% likely to be less experienced than someone upgrading for the first time. So I think making things easier for that first user is more important. Data loss is worse yes, but if you've never ever done a backup and you don't do one when you upgrade software, well, you're going to lose data at some point and blaming Drupal would be a bit harsh.

One thing we could do is add a sentence to the big flaming red warning when Drupal core goes out of date to run a backup before you upgrade maybe?

Gábor Hojtsy’s picture

Eaton: this is not about 'having the Javascript error messages in your native language' friendlier than 'not destroying your files directory by overwriting it with an empty one?'.

- First, we are solving a "usability issue". Chris Messina pointed out that right after the system is installed, it should not say you have some errors still to fix. We just moved a post-install annoyance to an install-annoyance. We aim for a Drupal install, which says your system is healthy right after the install, and it has no errors. Yes, this might mean you need to do more in installation.
- The installer itself does not use any JS, which needs to be translated the "Drupal way". The installer uses hackish JS translations (injected from the PHP translation API) with or without this patch which works there.
- Any install profile could try to use the normal JS translation though, which would not work in the installer, without this patch.
- Also read on for the last point :)

xmacinfo: We cannot do an "installerfiles" and "sitefiles" folder. For starters, it would break our API, introduce new features, we'd need to alter the file API, etc. Not for Drupal 6. Still (this is the last point for the above list), one of the problems solved is that without the files directory set up, any Drupal runtime operations which require JS translation would not be translated. Ie. you setup a localized Drupal 6 site without a files folder, go to the blocks page, and everything is localized, except the warning message to submit the form when done, or the poll submission page has an untranslated button, or all node forms for that matter have the split summary button untranslated. So the issue at hand is actually not to have a files folder which the installer could use, but to have the files folder set up by the first time Drupal runs, so *Drupal can use it*, not that the installer can use it.

All-in-all, in most hosting environments, I think you still need to chmod your files dir, even if shipped with Drupal, so another mkdir requirement before the chmod would not be a big thing to ask IMHO. I don't think anything in this patch needs to be rolled back to support the files directory protectors, except the files directory itself. My concern is to have a Drupal instance at the end of the installer, which is really setup, no further nags to do. Whether we ship with a files folder or not does not change that Drupal requires it and it needs to be done on install. Our files folder check ensures that if it is possible to create that folder, it is created, if it is there, but needs some chmod (and it is possible to chmod), then it is done. An error is only thrown on last chance.

xmacinfo’s picture

Whether we ship with a files folder or not does not change that Drupal requires it and it needs to be done on install. Our files folder check ensures that if it is possible to create that folder, it is created, if it is there, but needs some chmod (and it is possible to chmod), then it is done.

Thanks for all the extra information Gábor. I've tried the head with the shipping 'files' folder today. It is still giving out errors:

  • I had to chmod the 'files' folder to give write permission.
  • I had to chmod the 'sites/default' folder too.

So the patch is working fine. But the goal to have an error-free install is not reached. At least on some platforms.

My only concern, though, is: Do we want Drupal to ship with a 'sites' folder or not.

http://drupal.org/cvs?commit=88148

in your last comment you did not tell us if Drupal 6 should ship or not with a 'files' folder. I’d personnaly remove it.

As for creating an "installerfiles" and "sitefiles" folder, I will wait for the final version of Drupal 6 to try it out on a few install before suggesting modifications targeted for D7.

Gábor Hojtsy’s picture

xmacinfo: again, our goal is not to have a problem free install process, but to have a problem free Drupal instance, once the installation is done. One of Chris Messina's main points was that *after* you install the system fine, it should not say there are errors. We should take care that all requirements are fulfilled in install time. That's what we do, regardless of whether there is a filed dir or not in the tarball, we require it to be there in the Drupal instance. That you need to chmod your files and sites/default folder is an expected natural consequence, not a problem. That fact that Drupal does not show errors once installed is our goal.

I am fine with removing the files folder. Again, as stated above, I don't think that as you need to chmod your files dir anyway, an extra mkdir before the chmod makes any difference, and it would protect blind FTP and Mac users. Although some people don't think the files folder is any more valuable then the sites folder, I got convinced from the above that most stuff can be restored from the sites folder, but not from the files (unless you store your files in 'sites' of course, which we cannot help).

JirkaRybka’s picture

Reminder: Being that the case, we should move the useful info from files/README.txt somewhere. Also we need to have the linked manual page (see also #47).

Gábor Hojtsy’s picture

Title: Shipping with a files folder screws up Mac and certain FTP users » Document files directory requirement and placement
Component: install system » documentation
Category: bug » task

OK, removed the files folder with the README. Let's move that documentation to the install docs sensibly. This was the file's contents:

The files directory is the default file system path used to store
all uploaded files, as well as some temporary files created by Drupal. To
successfully install Drupal, the files directory must exist and be
writable by the web server process.

After installation, the settings for the file system path may be modified
to store uploaded files in a different location. Ensure that this new
location exists, is accessible, and is writable by the web server process.
The file system path settings can be accessed by selecting these menu items
from the Navigation menu:

  administer > site configuration > file system

You may wish to modify the file system path if:

  * your site runs multiple Drupal installations from a single codebase
    (modify the file system path of each installation to a different
    directory so that uploads do not overlap between installations);

  * your site runs a number of web server front-ends behind a load
    balancer or reverse proxy (modify the file system path on each
    server to point to a shared file repository); or,

  * your site policies specify that all site-related files are stored
    under the sites directory in order to simplify backup and restore
    operations (modify the file system path to point to a newly-created
    directory underneath sites).

Changing the file system path after files have been uploaded may cause
unexpected problems on an existing site. If you modify the file system path
on an existing site, remember to copy all files from the original location
to the new location.
keith.smith’s picture

Assigned: Unassigned » keith.smith

I'll roll a patch to move this into INSTALL.txt.

keith.smith’s picture

Assigned: keith.smith » Unassigned
Status: Active » Needs review
FileSize
5.82 KB

First draft of consolidating files/README.txt into INSTALL.txt.

Gábor Hojtsy’s picture

Doh, we also need an error message fix, as the current error message says the following on install:

The directory files does not exist. To proceed with the installation, please change the files directory permissions to allow the installer to write to it. If you are unsure how to do so, please consult the on-line handbook.

While the first sentence is correct, the second should suggest creating the directory first :)

keith.smith’s picture

FileSize
7.25 KB

Draft change to that error message integrated into the INSTALL.txt changes from #69.

I may have made this warning too complicated. I assume that this is also the same warning that shows up in the installer if the directory does exist, but is not writable, so one can't assume the first sentence will always be the "The directory files does not exist."

catch’s picture

Error message looks good to me now.

keith.smith’s picture

I'm bumping this back up near the top in the hopes that someone will review it or RTBC. As it stands now, this really needs to be resolved before a new beta, since shipping things as they currently stand will result in (a) a misleading error message in the installer and (b) a reference in INSTALL.txt to files/README.txt, which does not exist.

Patch in #71 purports to change these things, and still applies.

catch’s picture

Reading through, it all looks good to me. So RTBC.

catch’s picture

Status: Needs review » Reviewed & tested by the community
Gábor Hojtsy’s picture

Status: Reviewed & tested by the community » Fixed

Great, thanks, committed.

KentBye’s picture

I just wanted to ping the subscribers of this to point to an issue submitted by FiReaNG3L regarding the install usability & files folder.
http://drupal.org/node/194369

JirkaRybka’s picture

Project: Drupal core » Documentation
Version: 6.x-dev »
Component: documentation » User Guide
Status: Fixed » Active

Another ping - #47 still waits for another set of eyes, before it'll be able to come into the on-line handbook. Please don't forget, that we added a link into core (installer) to explain that stuff in more detail than we'll be ever able in a help-text, but we don't have that page yet. There are bits about filesystem here and there in "Getting started" and "Troubleshooting FAQ", but no one address this directly or completely.

After fixing this, please reset issue status back to the patch committed here earlier.

Bevan’s picture

Gábor Hojtsy’s picture

No, that's not a duplicate.

catch’s picture

mlncn’s picture

Project: Documentation » Drupal core
Version: » 6.x-dev
Component: User Guide » install system
Category: task » bug
Priority: Normal » Critical
Status: Active » Closed (fixed)

Acknowledging that my issue is a duplicate - http://drupal.org/node/197690 - as this issue was closed and got reopened, I'm closing it again and opening mine to be a little nicer and more targeted to GHOP folk.

Please follow over there. Thanks. Their task is to finish up JirkaRybka's documentation from #47: http://drupal.org/node/191310#comment-631567