Problem/Motivation

Users who download Drupal routinely add new modules directly to the /modules or themes to the /themes directory. The correct location of "sites/all/modules" is not at all clear and the number of files and directories in the Drupal core download makes it an exploratory effort to find the correct location. This issue's goal is to simplify the Drupal root directory.

Proposed resolution

This issue has been going on for 6 years, please take into consideration the compromise that has already taken place. The scope of this issue is clear. These exact things have been agreed upon as what we're handling in this issue:

  • Move all directories other than "sites" and "profiles" into a "core" directory.
  • Move all files other than .gitignore, .htaccess, index.php, README.txt, robots.txt, and web.config into the "core" directory also.
  • Adjust code and documentation throughout core to reflect the new locations.

These things specifically are not covered by this issue:

  • Moving the "profiles" directory to "core/profiles".
  • Moving "sites/all/*" to the root directory.
  • Any other changes to the "sites" directory, such as removing "sites/default" or relocating settings.php.
  • The addition of a "docs" directory.

See comment #121 for information about the "profiles" directory and other omitted changes.

Remaining tasks

This issue was RTBC at one point but now is postponed to let all the bugs get worked out of D7/D8. This issue is on hold until November 1, 2011 per Dries' comment in #254. In late October, this patch needs a reroll to prepare it for commit.

User interface changes

None, other than the directory structure changing.

API changes

A side-effect is documented in #124, where paths such as "profiles/sites/all/modules" or "modules/sites/all/modules" are no longer parsed. The addition of the "core/" prefix brought this issue to light, it could be considered a minor bug-fix.

Original report by kbahey

Problem:

Currently, themes, modules and configuration files are all under separate directories in the directory where Drupal is installed (root directory). Also, several files exist in that directory, such as various .txt files.

This causes several issues:

1. Clutter in the root directory.

2. Base Drupal is intermixed with files that site admins install (e.g. themes, theme engines, modules, directory for images, sites directory, ....etc.).

3. Upgrades are problematic because of that intermixing.

This issue was discussed in the forums, then on the developers mailing list.

Here is a summary of what was discussed so far:

Ideally, core Drupal will be in one directory called drupal under the root directory, with only the index.php and .htaccess at the same level.

Another directory called custom will hold the configuration directory, the site admin installed modules, and site admin installed themes.

So, the end result would be:

public_html (root directory)
public_html/index.php
public_html/.htaccess
public_html/drupal
public_html/drupal/includes
public_html/drupal/modules
public_html/drupal/...
public_html/drupal/sites/
public_html/drupal/sites/default
public_html/drupal/sites/default/settings.php

All the above would be included in the Drupal tarball, and should never need to be changed by a site admin in a normal install. The settings.php provided will contain localhost.

The following directory will be included in the tarball, with an empty structure (directories, and no files).

public_html/custom
public_html/custom/themes
public_html/custom/modules
public_html/custom/sites
public/html/custom/sites/default

Users then start by copying the drupal/sites/default/settings.php to:
public_html/custom/sites/default/settings.php

Drupal core will search the custom/sites directory first for configuration files, then if it cannot find any, it will go to the drupal/sites/default/settings.php. We should discourage (or better: prevent) adding any directories under drupal/sites other than the default directory.

Site admins then add their public/html/custom/sites/hostname/settings.php as needed, and can even add site specific themes and modules under each site directory as they do today.

Upgrades then consist of simply copying a new drupal directory to replace the existing one (and for backup, people can rename the old one to drupal-4.6 in case they want to go back), and copying the new index.php over the old one. They can then upgrade their own modules in separate directories.

The user's content (images, movies, flat files of any kind) is not affected in an upgrade, and the site admins are advised not to change the 'drupal' directory in any way.

The .txt files can be moved into doc directory under the drupal directory.

Files: 
CommentFileSizeAuthor
#299 drupal.core_move_22336_299.patch495.85 KBquicksketch
PASSED: [[SimpleTest]]: [MySQL] 33,771 pass(es).
[ View ]
#297 drupal.core_move_22336_297.patch494.4 KBquicksketch
FAILED: [[SimpleTest]]: [MySQL] 33,759 pass(es), 1 fail(s), and 0 exception(es).
[ View ]
#292 drupal.core_move_22336_292.patch494.25 KBquicksketch
PASSED: [[SimpleTest]]: [MySQL] 33,762 pass(es).
[ View ]
#291 drupal.core_move_22336_291.patch494.74 KBquicksketch
PASSED: [[SimpleTest]]: [MySQL] 33,759 pass(es).
[ View ]
#288 drupal.core_move_22336_288.patch659.75 KBquicksketch
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch drupal.core_move_22336_288.patch. This may be a -p0 (old style) patch, which is no longer supported by the testbots.
[ View ]
#227 drupal.core_move_22336_227.patch481.74 KBquicksketch
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch drupal.core_move_22336_227.patch.
[ View ]
#224 drupal.core_move_22336_223.patch961.91 KBquicksketch
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch drupal.core_move_22336_223.patch.
[ View ]
#208 move.patch356.58 KBtstoeckler
FAILED: [[SimpleTest]]: [MySQL] Drupal installation failed.
[ View ]
#204 drupal.core_move_22336_204.patch7.18 MBrfay
PASSED: [[SimpleTest]]: [MySQL] 31,434 pass(es).
[ View ]
#202 drupal.core_move_22336_202.patch7.14 MBrfay
FAILED: [[SimpleTest]]: [MySQL] 31,418 pass(es), 12 fail(s), and 5 exception(es).
[ View ]
#200 drupal.core_move_22336_200.patch7.12 MBrfay
FAILED: [[SimpleTest]]: [MySQL] 31,398 pass(es), 29 fail(s), and 8 exception(es).
[ View ]
#197 drupal.core_move_22336_197.patch7.02 MBrfay
FAILED: [[SimpleTest]]: [MySQL] 31,393 pass(es), 31 fail(s), and 9 exception(es).
[ View ]
#193 drupal.core_move_22336_192.patch7.02 MBrfay
FAILED: [[SimpleTest]]: [MySQL] 28,655 pass(es), 92 fail(s), and 95 exception(es).
[ View ]
#180 drupal.core_move_22336_180.patch7.02 MBrfay
PASSED: [[SimpleTest]]: [MySQL] 0 pass(es).
[ View ]
#172 drupal_core_move.patch345.86 KBboombatower
FAILED: [[SimpleTest]]: [MySQL] Drupal installation failed.
[ View ]
#168 dashboardcssandprofiles.patch518 bytesRobLoach
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch dashboardcssandprofiles.patch.
[ View ]
#165 drupal_core_move.patch474.1 KBquicksketch
FAILED: [[SimpleTest]]: [MySQL] Drupal installation failed.
[ View ]
#158 drupal_core_move.patch472.63 KBquicksketch
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch drupal_core_move_4.patch.
[ View ]
#154 drupal_core_move.patch472.63 KBquicksketch
FAILED: [[SimpleTest]]: [MySQL] Drupal installation failed.
[ View ]
#145 drupal_core_move.patch473.71 KBquicksketch
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch drupal_core_move_2.patch.
[ View ]
#140 drupal_core_directory.patch18.72 MBquicksketch
FAILED: [[SimpleTest]]: [MySQL] Drupal installation failed.
[ View ]
#138 22336_138_git_apply.patch.txt472.93 KBscor
#134 22336_132_1.patch18.73 MBscor
FAILED: [[SimpleTest]]: [MySQL] Drupal installation failed.
[ View ]
#125 drupal_core_move.patch158.11 KBquicksketch
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch drupal_core_move_1.patch.
[ View ]
#122 drupal_core_move.patch145.58 KBquicksketch
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch drupal_core_move_0.patch.
[ View ]
#121 drupal_core_move_commands.txt502 bytesquicksketch
#121 drupal_core_move.patch159.2 KBquicksketch
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch drupal_core_move.patch.
[ View ]

Comments

Zen’s picture

I would prefer:

public_html (root directory)
public_html/index.php
public_html/.htaccess

public_html/drupal
public_html/drupal/core

public_html/drupal/config
public_html/drupal/user

config containing sites and any other config files
user containing all user additions including modules, themes etc.

using 'local' is not very intuitive IMO, and I think config should be separate.

I am however unsure how this would pan out for multi-site configurations, as a big plus point is having modules unique to certain "sub-sites" etc.

Regards,
-K

kbahey’s picture

Adrian's installer will take care of upgrade issues and avoid the inconvenience that presently exists.

This is scheduled for 4.7.

Whether we still need a reorg of directory for aesthetics is yet to be determined.

naudefj’s picture

Some of Drupal's top level directories, like "scripts", are clashing with directories already on my site. Hope this will be implemented soon!!!

sergio@www.s3006.com.br’s picture

The solution for me to install everything of drupal in another directory and make it 'feels' like it is in the main page is this:

<?php
  chdir
("drupal-4.6.3/");
  include(
"index.php");
?>

Works like a charm... And you don't have to worry about your other directories...

kbahey’s picture

naudefj: It is generally safe to delete the scripts directory of Drupal. It is needed for some development work, but not for running a site.

Success’s picture

<?php
  chdir
("drupal-4.6.3/");
  include(
"index.php");
?>

I tried this method, but when the index appear, css didn't load and the node links don't work.

Success’s picture

Update, the above method work. But you need to enter your base url in your settings.php

But in our nodes url, the /drupal/ directory still appears.

Any way to remove that without causing any problem?

nickistre’s picture

Version:x.y.z» 4.7.4

I've been looking everywhere for this functionality. The above code works very nicely for redirecting from the root directory to the frontpage under the /drupal directory, but it still leaves the URLs looking like and accessible only at http://www.example.com/drupal/links, for example. I would really prefer if it could be accessed directly from the root directory (I.E., http://www.example.com/links).

nickistre’s picture

Version:4.7.4» x.y.z

Oops, fixed version field back to cvs.

Frando’s picture

Version:x.y.z» 6.x-dev
Success’s picture

Another method that work via htaccess.

http://drupal.org/node/135206

But we still need to know how to remove the /drupal/ portion from the urls.

CheckeredFlag’s picture

I solved this problem by using a RewriteRule in the .htaccess file.

I also found that the key to removing "/drupal/" from the URL is in setting $base_url appropriately.

See my comments here and here for how I did it.

Marco

linuxpowers’s picture

Component:base system» update system

I have a somewhat unique situation that seems to be a bit direct but, I would like someones input on this. I currently have Drupal 6.1 (w/no additionally installed modules or themes - only written nodes and some edited CSS files for themes) installed and running in a sub-domain, ie: http://test.mydomian.com, whereas test, being the sub-domain. I am looking to install/upgrade Drupal 6.2, but eventually I will have it located in my main domain, ie: http://www.mydomain.com, whereas mydomain, being the main domain.

I have an option, I believe, that would allow me to just "install" a fresh copy of 6.2 in my main domain and move effected files and such to the new installation from my sub-domain version. I thought this would be less extensive than updating 6.1 first, then moving it into my main domain later. Would this be the most appropriate choice? (I am ready to have it go public in my main domain at this point!)

Also, I have a concern with the existing database information. Will the new installation over-right any db entries? Will the current users still be there? etc. (I have backed up my db and such already) I hope this is not off topic but, the thread that contained all these concerns directed me to this node for any new posts.

Thanks folks....I'm really impressed with this Drupal website maintenance environment!

Roger

StevenPatz’s picture

Component:update system» base system

Please don't hijack issues.

linuxpowers’s picture

Component:base system» update system

Never mind....again!

Damien Tournoud’s picture

Status:Active» Closed (won't fix)

I guess this issue is outdated anyway. Drupal cleanly separate core files from custom ones (that lie in /sites). So marking that as won't fix.

StevenPatz’s picture

Component:update system» base system
CheckeredFlag’s picture

Status:Closed (won't fix)» Active

This is not an outdated issue. Drupal 6 still expects everything to be installed in the root. This request is to allow everything to be installed in a /drupal subdirectory instead and potentially move /drupal/sites outside the core installation (e.g. /drupal_sites) -- all WITHOUT including "drupal" in the URL.

While this can be done currently, it is less than straight-forward and requires hacking the .htaccess rewrite rules and settings.php. It would be ideal if Drupal would support this configuration natively instead of hacks that work to obscure its relocation.

Damien Tournoud’s picture

Status:Active» Closed (won't fix)

No, Drupal does not requires you to have a "drupal" subdirectory or to include "drupal" in the URL. You can put everything in the root directory of your website (do you see any "/drupal", here on drupal.org?). The separation between Drupal Core and custom code is *very* clean (every thing custom is put under /sites), so there is no point in that issue anymore. If you really want to physically separate the two, you could use a link at the operating system level (symbolic link or anything else).

I'm closing that issue again (won't fix, because you can't fix what ain't broken). I see no point in hacking rewrite rules to do something that is done by default.

Please explain your needs more clearly if you want to reopen that issue.

CheckeredFlag’s picture

Status:Closed (won't fix)» Active

The original post explains the issue very well.

I believe you are misunderstanding that the primary issue is to ALLOW Drupal to be installed transparently in a subdirectory. Currently, Drupal expects everything to be installed in the document root, causing problems described by the original poster.

This issue was discussed at length in the developers forum (link in the original post), with a general consensus that this is a good idea that should be implemented.

Please don't disregard this issue as being pointless without reading that thread.

sepeck’s picture

Status:Active» Closed (works as designed)

then we'll mark it by design.

Damien Tournoud’s picture

Does it make sense to mark a feature request as "by design"?

About the original feature request, just linking the /sites/[your site] to another directory would allow complete separation of Drupal Core and custom files.

kbahey’s picture

Status:Closed (works as designed)» Active

Damien

This is not about separating core from contrib. That part has been addressed in D5 and D6 by using sites/all/themes, sites/all/modules, and having a default.settings.php shipped by default.

This is about two things:

1. Moving certain parts to above public_html

It is about taking stuff that does not need to be under document root and move it outside of that directory. This includes stuff like all code that is executed, and configuration stuff (settings.php).

Only index.php, update.php, install.php, and other files of type .js, .css, .ico, and media files need to be under Document Root. The theme tpl.php does not even need to be under document root, or perhaps we make an except for those, and have the themes under document root, since it is the presentation layer.

Everything else will do:

<?php
include_once('../drupal/includes/bootstrap.inc');
...
?>

Or we even do a chdir http://ca3.php.net/chdir before anything else, and we are still good.

This way, all that stuff is protected from any flaws in configuring the web server.

This is a common way to setup other applications.

The objection that has been voiced is that on some shared hosts, this would not work. I have not seen this in shared hosts that I used, since you get /home/username/public_html. So Drupal can be in /home/username/drupal, and the rest of it under public_html.

2. Better organization

Moving everything under drupal in one directory, so document root is not cluttered.

/home/username
/home/username/drupal/
/home/username/drupal/site.php
/home/username/drupal/includes
/home/username/drupal/modules
/home/username/public_html
/home/username/public_html/.htaccess
/home/username/public_html/index.php
/home/username/public_html/drupal
/home/username/public_html/drupal/site/all/themes/themename

The site.php file, or some other mapping file, will tell index.php where to find the rest of the stuff. Crell already has a working patch for something similar for the "sites" directory, so we don't have to have all the files under the domain name.

This way, we don't have clutter, and in each place (home directory, and document root), we have ONE directory with everything under it.

If there are reasons not to do that, then let us hear them.

maartenvg’s picture

Version:6.x-dev» 7.x-dev

First off, new features go in 7.x.

I'm against putting parts of drupal outside of the document root, mainly because in a lot of situations where access to the directory below public_html is not straightforward. My reasons are as follows:

1) It adds clutter: Having 2 locations depending on each other, but keeping them at separate locations is counter intuitive and is prone to errors when moving installations around.
2) There are cases such behavior will cause annoyance and perhaps even breaking. All of my websites are on servers which use virtualdocumentroot where folders in /home/username are sub domains of the main domain, and don't have access below that directory.
e.g.
/home/domain.com/www = www.domain.com
/home/domain.com/blog = blog.domain.com

If drupal would need /home/domain.com/drupal/ for includes and stuff and /home/domain.com/www/drupal/ for the rest of the files, that would prohibit me from using drupal.domain.com as a functional subdomain.
3) There are a lot of situations (including the above) where you don't have a directory directly below the document root that is accessible to the users. For example if you have a server on which you have your document root (user-writable) in /var/www, you would need to give users write-access to /var to make up for the missing public_html structure.
4) On some servers Apache (and PHP) is prohibited to look outside their document root for files to run.

Thus, if this feature gets into core it will break a lot of websites on shared and access-limited servers, and will seriously hinder platform independence of Drupal.

I'm urging to mark this won't fix.

joshuajabbour’s picture

maartenvg, I don't think we'd ever require that the Drupal core files live outside of the webroot, just allow it. Ideally, part of this re-org would be adding a variable with the path to the drupal directory, so that it could be moved out of the webroot.

My take on this thread is that it would be great to consolidate all Drupal core files in a "drupal" subdirectory, which I whole-heartedly support. My ideal layout would be something like this:

.htaccess
index.php
robots.txt
drupal
sites

This would enforce a clear separation between core/system files (drupal) and user files (sites). Everything that currently lives in the webroot would be moved into the "drupal" directory (except those listed above). This would also have the side effect of discouraging users from editing/hacking files in core (or dropping contrib modules into the top-level "modules" dir, etc). And it would make upgrading (conceptually) easier, just replace the "drupal" directory.

Finally, Drupal is currently quite rude in that it assumes it's the only script living in the webroot -- making it difficult to integrate with other systems that might be in use. This re-org would make this task a little bit easier (in that there would be less potential conflicts).

andreiashu’s picture

discouraging users from editing/hacking files in core (or dropping contrib modules into the top-level "modules" dir, etc).
Yup, putting new modules into the top-level "modules" dir was the first thing I did as a newbie.

So a big +1 to this because of this and of all the reasons stated in this issue.

maartenvg’s picture

I like your (and the original) take on it; it seems the next step in separating core from contribution. I will review & support any nice patch that floats by.

On a side note: Making it possible for D to live outside the document root is okay, but definitely shouldn't be the default behavior. Perhaps leave it almost undocumented because otherwise it will introduce a wagon load of cleanurl- and multisite-like questions which limit the user experience and increase the support load.

wretched sinner - saved by grace’s picture

I've marked #311783: Directory called "modules" is not for modules as a duplicate. The focus there was just on the modules directory, but I think this would be the better option.

drifter’s picture

Agree with #25, and also keep install.php, update.php, cron.php and xmlrpc.php in the root directory for backward compatibility. My initial suggestion would have been to move /modules and /themes into /core/modules and /core/themes, but this one is more comprehensive and makes more sense.

Renaming files and folders is difficult in CVS, how do you propose we roll a patch for this issue?

joly’s picture

Hi,

I am really happy to find this thread for D7.

It seems not needed to have drupal outside the webroot. But i think it can be very handy.
Then a SERVER-side install of Drupal will become possible!! This would be a major feature, and is not so difficult to make.

To get this done for our install I had to hack Drupal 6 very much. Sorry :(
I wrote an article to explain what we did. I will be happy to write more info, if that is needed to get D7 to adopt this . ;-)

If Drupal 7 would support variables to all the important paths anyone can decide for himself how he wants to handle its installation. And Drupal 7 can boost that it can be installed SERVER-SIDE.

What is needed:

  • configuration to set the paths (also to /modules, /themes, /includes etc.)
  • configuration to set symlink-paths
  • a class/script/functions to
    • search the global and client paths and decide what to send back
    • it should also test if the path exists, handling files and dirs
    • give back root-paths and url-paths
    • give back a relative path to store in the database
    • keep a list of paths, so each path is searched for only once
  • change drupal scripts to request paths by this class (ohoh.....)
  • no hardcoded paths ever; even the class gets its info only from the init-settings

Greetz, Joly

joly’s picture

Ohoh, I cant get used to this kind of forum ;)
I keep editing settings and tags, thinking it is only for my comment.
I do hope I learned it from now on.

Mike_Waters’s picture

By just symlinking ../sites/[client site name] to another location on the server, you get separation of server/client files. Then provide the client with access to that area only.
e.g.
$ cp -R /var/www/drupal/sites/* /var/www/public_html && rm -rf /var/www/drupal/sites
$ ln -s /var/www/public_html/ /var/www/drupal/sites
$ chown -R client1:client1 /var/www/public_html/client1-site.com/
or something like that. This could be done across NFS mounts, if you have multiple servers.

It's what I do, though I am a small shop with under a dozen clients.

The individual site directories can contain themes, modules, and files - pretty much the only things that you cannot separate out are theme engines and core modules; if you need to separate these out for some reason, then you'll just have to give each client their own copy of core. Which is really not a big problem if you are using revision control (svn/cvs/bzr etc.)to keep every install updated.
For site-specific apache configuration, use per-site directives in htaccess or apache config file. Sure, you'll actually have to do the work of changing the config files, but if you are a service provider your worth is determined by your toolchain, and so you probably would write a script to change the apache config (as well as create and populate the new site directory) for each new site.

I just find it very curious that you hacked core so much (Ive read your article), and wonder exactly what circumstance would require you to do so given that drupal + linux provides much of this functionality by default. Call me curious ;)

EDIT: also be sure to check out Aegir, a hosting platform for Drupal. http://groups.drupal.org/aegir-hosting-system

dww’s picture

Note: this would make life easier for #606592: Allow upgrading core with the update manager

joshmiller’s picture

Version:7.x-dev» 8.x-dev
Issue tags:+Needs usability review, +Bikeshed, +DX (Developer Experience), +DrupalWTF

This is a long standing issue I've had with Drupal since I started back at D6.1 -- and continue to wrestle with on a daily basis. All of the core files are "up front" and all of the "you should edit these files" are hidden under that strange name "sites."

For the usability experts, tell me which folder you are likely to drop your very first module into...

  • modules
  • sites/all/ —> mkdir modules —> sites/all/modules/
  • scripts

Exactly. This is a huge Drupal-WTF. The best proposal so far from the comments above is this:

.htaccess
index.php
robots.txt

core/
sites/

What if we did something more along the "Drupal as a framework" lines...

.htaccess
index.php
robots.txt

drupal_framework/
drupal_config/

At least then we are not asking a new user to understand the difference between 'modules' and 'sites.' In fact, "Config" is 1000% more understandable without documentation than "sites." I also recommend that we move all non-essential *.txt files to drupal_framework/ as well.

Ideally, this folder structure should be very lean and very controlled. Sadly, I don't think drieschick would consider this kind of change so late in D7. But that's ok because I wouldn't even know how to roll a patch with this ;P

Please, can we have others weigh in on this issue? Tagging for exposure.

Mike_Waters’s picture

drieschick == dries + webchick ?

deviantintegral’s picture

I'm not sure I like the idea of hardcoding the name "drupal" into the folder structure. It makes things more confusing for those repacking Drupal.

/core and /sites seem really good to me. However, I think all of the *txt files, and special files like authorize and update.php would need to remain in the root, unless we require mod_rewrite?

joshmiller’s picture

Current D7 List:

includes/
misc/
modules/
profiles/
scripts/
sites/
themes/

.htaccess
CHANGELOG.txt
COPYRIGHT.txt
INSTALL.mysql.txt
INSTALL.pgsql.txt
INSTALL.sqllite.txt
INSTALL.txt
LICENSE.txt
MAINTAINERS.txt
UPGRADE.txt
authorize.php
cron.php
index.php
install.php
robots.txt
update.php
xmlrpc.php

New proposed structure based on #37 and my own thoughts:

core/
    includes/
    misc/
    modules/
    profiles/
    scripts/
    themes/

    cron.php          <-- move files that admins don't have to type up
    authorize.php         into the core/ folder

readme/ <-- folder that contains txt files like changelog.txt, etc.
sites/

index.php
install.php
update.php
robots.txt
xmlrpc.php <-- I don't think we could move this, not sure though

Edit: Sorry, I meant to click preview originally, I just changed some formatting and moved a few files.

metzlerd’s picture

WHy not also move install.php and update.php into core? If we can move cron, why not those?

Like this structure, but also like config better than sites, but I'm a bit on the fence on that one.

Spiked’s picture

I'd recommend naming "readme" "docs" instead. And don't forgot the .htaccess!

antgiant’s picture

Building on #38, #39 & #40 I would propose the following

core/
    docs/   <-- Everything proposed for readme/ seems to be core related.
    includes/
    misc/
    modules/
    profiles/
    scripts/
    themes/

    cron.php
    authorize.php
    install.php
    update.php

custom/  <-- It should be clear that this folder is for more than just config files.
    modules/  <-- What is now contained in sites/all/modules
    themes/    <-- What is now contained in sites/all/themes
    sites/        <-- For multi-site specific files

    settings.php  <-- What is now in sites/default/settings.php
    readme.txt    <-- A file explaining not to hack core and to put your files here.

.htaccess
index.php
robots.txt
xmlrpc.php <-- If possible I'd prefer in core/

My thoughts are, while config is clearer than sites, it also encourages new users to think only configuration files go there not modules, themes, etc. I think that custom is clearer. I also believe that a single instance install is much more common than a multi-site install. If that is true I think Drupal should be optimized for that. Merging the sites/default and the sites/all folders to the root of custom I think accomplishes that, and leaving the sites folder there notifies people of that capability but doesn't require that they understand it just to get a single site install running.

That is my two cents worth.

merlinofchaos’s picture

.htaccess
CHANGELOG.txt
core/
  css/
  images/
  includes/
  js/
  modules/
  scripts/
  themes/
index.php
install.php
INSTALL.txt
README.txt
sites/
  all/
  default/
update.php

I am in favor of leaving the *.txt files at the top level where they will be easy to find.

IMO we should consider this strongly as an early entry into D8 so that we can get it done and out of the way while there are still few patches against D8 (since it will invalidate, oh, every patch. And then some).

I don't believe the .php files can be safely moved out of the top level directory, because technically we do not rely on .htaccess. We only rely on that for clean urls.

merlinofchaos’s picture

I'm not sure I like 'custom' because it suggests that it's custom to a site. I don't know that it's worth changing it from sites/all, especially since we now actually include sites/all/modules

antgiant’s picture

I was envisioning custom as in my custom files, modules, themes, etc.. However, I'm not set on that particular name.

I'm really opposed to sites/all and sites/default. I've used Drupal for quite a while now and the distinction between them only matters for multi-site. Thus the current stucture requires understanding the multi-site capabilities just to "properly" get a single instance site up. I feel that is a major usability problem. I have also witnessed firsthand the problems it creates for new users.

Crell’s picture

Actually it's documented on drupal.org somewhere (I'm sure it's moved since the page was written) to use sites/all for contrib modules and sites/default for custom-written modules. That makes a perfectly good distinction even for non-multi-site sites.

Of course, the D7 update system breaks that horribly by putting all modules in sites/default, even though drush puts its downloaded modules in sites/all. I think that's a critical core bug, but I was overruled. :-(

merlinofchaos’s picture

That doesn't even work for multi-site, since sites/default is for just one site. =(

merlinofchaos’s picture

#45: Yes, but would this usability problem exist if 1) 'modules' wasn't at the top level, and 2) 'sites/all/modules' already exist? Particularly if the INSTALL.txt instructions told you precisely where to put modules you downloaded? With a separated 'core' directory, it's easier to say "DO NOT MODIFY THIS STUFF" AKA "DON'T HACK CORE".

aaron’s picture

to throw more bikeshedding into the custom folder name debate, when we build sites, we use:

sites/all/modules/contrib
sites/all/modules/custom

'contrib' for contributed modules and 'custom' for modules written specifically for a site.

and subscribe, and +1 to this great idea.

antgiant’s picture

#48

1) Helps a lot.
2) Helps some.

However I really like the idea of the sites/all/ content being at the root of some custom/ (or your favorite name) folder. It seems to me that logicly the contents of those folders would be avaliable to all the deeper folders. It is using the folder structure to reinforce the actuallity of what is happening. For instance.

Currently.

sites/
    all/          <-- Avaliable to all domains.
    default/      <-- Avaliable only to all "Default" domains.
    example.com/  <-- Avaliable only to the example.com domain.

My Proposal

custom/         <-- Insert favorite name here
    modules/          <-- Avaliable to all domains including
    themes/                    multi-sites living in the sites folder
    ...
    sites/
        default/      <-- Avaliable only to all "Default" domains.
        example.com/  <-- Avaliable only to the example.com domain.
        ...

This also makes getting a single-site up clearer. However, I can see your point that it will also discourage proper multi-site setup over time. (My use case of Drupal has multi-site installs causing far more problems than they solve, so I am a bit biased I suppose.)

Having said that any of these proposals are significantly better than what there is currently.

Crell’s picture

Let us not forget settings.php. That has to be per-site by nature. And there's also now sites.php. Multi-site is rather intrinsic to Drupal's architecture at this point, like it or not.

sdsheridan’s picture

Throwing my two bits in, having default / multi-site-wide modules and themes under sites/all is a bit of a pain from a backup standpoint. It would be better if 'sites' was just that... sites... and modules and themes (the contrib and custom kind) were held elsewhere. I'd frankly prefer something like

core/
contrib/ (or custom, although i think i like contrib better, as in my mind it covers custom too)
sites/
   default/
   sitea.com/
   siteb.com/
   ...

I'm working with multisite setups, and this seems much cleaner to me.

wretched sinner - saved by grace’s picture

What about a (slightly) different approach. Are we completely wedded to the sites folder, or could we have something like:

core/         <- everything Drupal that shouldn't be modified ever, and can be completely replaced when upgrading
.htaccess
index.php  
README.txt
robots.txt

files/
    all/
    example.com/
    example.net/
modules/
    all/
    default/  <- do we really need this? Isn't it too confusing for most users?
    example.com/
    example.net/
settings/
    default-settings.php
    default-sites.php
    example.com/
        settings.php
        sites.php
    example.net/
        settings.php
        sites.php
themes/
    all/
    default/   <- do we really need this? Isn't it too confusing for most users?
    example.com/
    example.net/

My thoughts behind this layout is that it becomes instantly obvious to a new user where to put their modules and themes. A slightly simpler option for single-site installs would be to drop the subfolders of modules and themes, and allow multi-site to add example.com/{modules|themes|files} folders for site-specific files.

Do we really need a default folder for modules and themes? Can someone let me know where you would use default in place of example.com, especially now with the ability of sites.php to provide a different domain and/or path for development.

As an aside, maybe the index.php should as minimal and free from change as possible. Maybe even to the point of simply being a list of include statements, so that when upgrading users do not have to think about copying the index.php file into the root. That way, the Drupal tarball could be extracted into the folder and renamed core and the upgrade performed.

Crell’s picture

For the normal case of single-site, having a consistently named directory is very helpful. Especially when you're spinning off a lot of non-multisite sites on a regular basis (like many dev shops do).

I'm really not wild about splitting settings files from site-specific modules. That only makes backup more of a challenge since your not-from-d.o files are now spread out to 2 or more locations. (Remember the files directory.)

kbahey’s picture

Let us not over complicate this.

Let us keep it clean and simple:

The Drupal tarball should ship with two top level directories and just 2 files:

/sites
/core
index.php
.htaccess

The /sites directory should have the following empty directories
/sites/all/modules/contrib
/sites/all/modules/custom
/sites/all/themes
/sites/all/addons
/sites/default/default.settings.php (renamed in D7, I know)
/sites/README.txt

Anything you download should go in contrib
Anything you develop should go in custom

The README.txt explains that ...

/core should not be touched. It is the part that can be safely overridden during updates.

Don't hack core now means something tangible ...

deviantintegral’s picture

I agree with #55 in general, though it's missing robots.txt, UPGRADE.txt, INSTALL.txt, CHANGELOG.txt, and hopefully #607028: Add README.txt to core.

What's sites/all/addons? The same as libraries?

kbahey’s picture

We should add robots.txt to the files in the root directory (alongside index.php and .htaccess).

The rest can go into /core.

Yes, addons is the same as libraries (where things that are not drupal should go, like WYSIWYG editors, XML parsers, ...etc.). I am not hung up on the name addons. It can be anything else.

Crell’s picture

Side note, as previously mentioned on the dev list I am now working with the PHP Standards Working Group (such as it is) on naming conventions as they pertain to classes and namespaces. It's almost certain that such standards will influence file paths. I am trying to keep this thread in mind as I discuss the point. We'll see what happens. :-)

Just trying to keep folks in the loop.

jwilson3’s picture

I agree that the Drupal root folder needs cleaning, but going from bloat to the minimalist 2-folders 3-files (a la #55 + #57) seems also overly archaic for noobs as well as seasoned developers. Here's a disclaimer: I guess I'm kinda new-ish to Drupal, but ... anyone can dream, and think outside the box. I hope not to offend anyone with my galactic translucent bikeshed paint :) I'm coming at this from an end-user friendly p.o.v.

As developers and web architects (noob or veteran) using, testing, and building with Drupal, we all live with three components not simply two: core AND contributions AND our site(s). After installing a fresh Drupal instance, the first thing you have to do is add your favorite modules... so burying module code four folders deep inside the sites/all/modules directory (#55 suggests five folders deep!) just doesn't add up to a "sensible default". Neither does five folders deep (!?!) for my own cherished custom code (sites/all/modules/custom/my_module) I want to SEE my code, and know its there, and allow others on the team to readily see and know that i have custom plugins in any given Drupal install. While we are dreaming up ways to improve Drupal folder structures lets go ahead and improve sites/ as well!

In addition, as people move towards automating their update mechanisms (be it through RCS update commands, scripts like drush, or web-based Update modules) EVERYONE needs to deal with and distinguish between "core" updates and "contrib" updates. Lets make a local folder structure agree with the server repository logical division in order to foster usability and instantaneous grokability with what people will be checking out and keeping up to date. "As above, so below."

I hope my structure makes sense, its heavily doted with footnotes.

core/          <--- contents already hashed out in prior comments.
  .../              (certainly core modules, themes, profiles, and php and txt files
  .../               generally EVERYTHING that could potentially be touched by a drupal minor version update.
  -misc/       <--- misc folder needs serious cleanup, move visual elements to theme.
                    move base jquery to a module handler which can be overridden by contrib
plugins/       <--- replaces sites/all/ to embrace the idea of "core"  and "contrib"  (*) (***)
  3rdparty/    <--- makes official and replaces the unofficial convention of 'sites/all/libraries', aka 'sites/all/lib'  (**) 
  custom/      <--- a second-level folder that users can create to single out their own code (**) (***)
  modules/     <--- the sensible default place for all your favorite "contrib" modules; replaces sites/all/modules.  (***)
  themes/      <--- the sensible default place for base themes from "contrib"; replaces sites/all/themes
    engines/   <--- the sensible default place for theme engines from "contrib"
    zen/       <--- an example base theme
  profiles/    <--- the sensible default place for installation profiles from contrib
  translations/<--- the sensible default location for translation packs
  .../         <--- leave room for other futuristic contributions here  (widgets, or frameworks, or feature-sets)
  Readme.txt
sites/        <---  bundled with packaging script
  default/     <--- yuck, but oh well, i don't have a better solution for this :/ (****)
  example.com/
    files/
    modules/
    themes/
      my_zen_subtheme/
  Readme.txt
index.php      <---  Simple one-liner that points to core. (*****)
.htaccess
robots.txt

(*) Despite the idea of introducing an organization that mimics the 'core' and 'contrib' structure, I propose 'plugins' to reenforce the idea that contributions not necessarily limited to the official Drupal "contrib" repository; Take the 3rdparty folder special case as an example (details below). The advantage of having a plugins folder is that it leaves room for future growth, and welcomes healthy additions on down the road, for things that dont classify as modules or themes or profiles or translations. Introducing and really embracing the term "plugins", and noting that modules are really a subset of the plugin cosmos is something Drupal could use to reduce the initial 'nomeclature' barrier.

(**) The 3rdparty folder is THE special case inside the plugins folder that wouldn't necessarily contain elements that sync with drupal contrib repository. Also, the name '3rdparty' makes more sense than 'libraries' to non-programers, because it truly distinguishes this section for what it is: ANYTHING non-drupal that various modules might put to use, that may or may not be added to the Drupal repository (for Licensing reasons or other). Also, very often, the external code packages that go in there aren't technically "code libraries" in the strictest sense, as they are often full-blown applications and may be portrayed online as 'packages', 'components', 'libraries', frameworks, etc. If we stick with the generic term '3rdparty plugin' we can use it everywhere.

(***) Since this folder will potentially house drupal modules and sub-modules, leave it up to the user to determine folder structure inside the directory, and recurse the folder (if it exists). Update: after meditating on the structures here. This point to me is the one that sells the idea of a plugins directory. Essentially, plugins will be a better modules/ directory. One that has sensible defaults, but since its recursed, its internal structure could be customized to anyone's desired structure. For example, if you're going to distribute a drupal+specialsauce solution (eg OpenAtrium), you could isolate your special sauce into plugins/specialsauce/modules plugins/specialsauce/theme.

(****) The sites/default folder should go away if at all possible and be replaced with sites.php or other type of general config file. Disclaimer: I'm not up to speed on D7 changes in the works here.

(*****) A user may choose to customized 'index.php' to point to core outside the webroot. So as not to "overwrite" these changes upon a core update, this file 1) should not be in the repository 2) should never be included in any update scripts and 3) should be added by the packaging script. It's contents should contain a link to more information on how to move "drupal core" out of the webroot.

bsherwood’s picture

My take on the setup:

core/          <--- I like the idea of everything core related going into one directory. This also helps with the "don't hack core" ideal.
  .../          
  .../              
  -misc/      
------------------          
plugins/       <--- 'plugins' is a bad naming scheme since it is used nowhere in the drupal community. "contrib" would be better than "plugins"
  3rdparty/   
  custom/    
  modules/   <-- If we are going to continue to use the /sites directory, I think we should use sites/all instead of this.
  themes/           //
    engines/        //
    zen/             //   All same as above. Let's either break /sites up or consolidate all site specific stuff with it.
  profiles/          //   We don't need /core/modules, /plugins/modules, /sites/all/modules. Same for themes, etc..
  translations/   //
  Readme.txt
-----------------
sites/        <---  Easy to understand if using Drupal in a multi-site setup, not as user friendly in a one site install
  default/     <--- "default" is an awful name. One way to make it more intuitive would be to change the directory ->
  example.com/     name when install.php runs (i.e force $base_url or $_SERVER['PHP_SELF', DOC_ROOT, etc...]) 'default' => 'mynewsite.com'
    settings.php    
    files/
    modules/
    themes/
      my_zen_subtheme/
  Readme.txt
-------------------
index.php      // Nice and clean root directory. I don't really have an opinion on moving the README, CHANGELOG, etc.. into a 'docs' directory. Both 'root' or 'core/docs' seem like logical places.
.htaccess
robots.txt

I also looked at Wordpress and Joomla to see how they handled their 'core/contrib' separation (from a file system perspective only). Wordpress keeps all plugins and themes in 'wp-content', which doesn't seem all that intuitive. Joomla pretty much throws everything into the root directory (i.e administrator, components, plugins, modules, logs, etc...). Joomla seems the most intuitive yet it fails to distinguish between multi-site setups (if Joomla can even do multi-site setups?).

Spiked’s picture

Doesn't this raise the question that contributed modules/themes should have a different filesystem?
Something like..

mymodule/
mymodule/mymodule.info
mymodule/mymodule.module
mymodule/mymodule.install
mymodule/inc/
mymodule/inc/include1.inc.php
mymodule/css/
mymodule/css/style.css
mymodule/lib/
mymodule/lib/jquery.js

..and so on. I understand that this is partially up to the developer, but a little encouragement never did anyone harm.

For the D7 part; I'm still with #43. Why do we think this is a usability issue? Why don't we document it better? Show explanations of the directories in the installer or admin/help, include a readme, refer to the readme, etc. Someone who uploads his D7 folder using ftp isn't looking for .txt files in the directory, he just uploads everything and clicks 'Install'. .txt files are not (new-)userfriendly.
Look at http://drupal.org/node/43816 for example. It's the first google hit for "Multisite drupal", but as soon as I load that page, I get blasted with irrelevant information. I only see "4.x" and "5.7" in the first comment. Where can I find 6.x? 7.x? Why isn't the first link on the page working?
D.O needs a cleanup. Bigtime. Don't get me wrong, there are tons of useful information available here, but there is even more irrelavant data around it, which makes it hard to find.

The problem isn't in the filesystem, it's in the documentation (of the filesystem, in this matter).

chx’s picture

http://buytaert.net/8-steps-for-drupal-8 let's stop saying smallcore.

joshmiller’s picture

Title:Core Drupal in its own directory, configuration, modules and themes in custom» Reconfigure Drupal folder structure to improve usabilitiy and upgrades

@chx - does that mean you're opposed to creating a folder called "core" ? Because, no one has mentioned "smallcore" until now.

Ok, folks, I just read through this whole issue for first time in a while -- we have some well meaning posts, but everyone's ideas are getting mangled and hard to follow. Let's create a wishlist of changes so Dries can weigh in on each idea.

@Crell or @merlinofchaos or @chx -- you guys have big names in the community, please weigh in and try to personally ask Dries to buy in to Earl's idea that this should be one of the first D8 commits.

Here's my first crack at our wishlist (we'll start with letters, please identify your remarks using letters or assigning your thoughts to new letters)...

A) Move all folders except sites (misc, modules, themes) to a single parent folder called "core" or "admin" or "something".
Pro: We can now let core upgrades work: #606592: Allow upgrading core with the update manager, our documentation that says "don't hack core" now has a real folder correlation.
Con: The only downside is the perceived amount of work involved. No one has yet argued against it.
Impact: Less "first time user" accidental core hacks, makes "sites" folder instantly recognizable as the only "non core" folder, may solve "D" issue right away
B) Move as many core PHP files (update.php, install.php, xmlrpc.php, etc.) as possible from root to parent "core" folder as possible. Keeping critical files (like robots.txt, index.php) in the root directory is implied.
Pro: Moving non-critical php files outside of the root directory "hides" the mess and makes the "core" folder concept much more powerful.
Con: People used to typing in "/install.php" or "/update.php" will get 404 or WSOD. Adds cruft to common administratable urls in the name of "clean up."
Impact: At best, we will have improved the concept of the "core" folder and made it that much harder for a first time user to tweak core php files. At worst, long time developers will feel alienated and frustrated. Middle ground might be to .htaccess some of those common admin files to their new locations.
C) Move documentation files (README.txt, INSTALL.txt) out of root
Pro: Having sooo many *.txt documentation files alongside critical core files is confusing and amateur
Con: The argument for keeping all doc files in the root is that they are right there when you install, obvious documentation, and already in the standard place for most software
Impact: At best, this change would not confuse first time users. At worst, first time users would be very confused. Middle ground might be to have one documentation file (readme.txt?) that points out a folder somewhere in the installation to check out further tips, like installation instructions, etc.
D) Rework "sites" to mean something to people who need to know where to drop custom files. (See comment #53)
Pro: A new user should be able to instantly know where to put their modules and themes
Con: The "sites" convention is well understood for all serious Drupal developers and this would pose a major WTF for people familiar with NOT adding files to the folder called "modules," etc.
Impact: At best, new users would have instant understanding. At worst, 99% of Drupal developers are facing major re-learning.
E) Keep "sites" but create a third "non-core" folder called "contrib" to replace "sites/all" (see #59)
Pro: Helps new users get started without the issues of D, Provides a place for installation profiles to hide their "special sauce"
Con: This creates a strange special case that may be so different that most people won't "get it." Adrian will probably hate the multi-site implications
Impact: At best, this helps new users and installation profiles. At worst, this confuses everyone.
F) Move current "index.php" to "core" folder and make the new "index.php" be a one-line include.
Pro: It's likely includes won't need upgrading. It's also likely this *might* be the first step in moving core off root.
Con: We're promoting indirection for the sake of clean-up and upgradability
Impact: At best, this makes upgrading that much easier. At worst, we've added a layer of indirection that causes many issues.
G) Keep "profiles" as "non-core" folder
Pro: It should be possible to upgrade an installation profile without hacking core.
Con: Creates confusion as to the difference between "sites" and "profiles"
Impact: At best, all major installation profiles gain some upgradability outside of core updates. At worst, we've added additional confusion to the whole "less is more" argument

Simplest set of folders I think is possible:

core/
sites/
profiles/
index.php
robots.txt
.htacess
wretched sinner - saved by grace’s picture

H) Create modules, themes and files directories in root and leave sites for advanced users.
These 3 directories could be direct replacements for the existing sites/all/* directories, and operate as such. For users who want a multi-site setup, they can create a sites hierarchy in the same for as we currently use, with the same rules. settings.php would then be placed in the webroot.
Pro: It is very obvious for users who run a single website from a codebase (the majority case) as to where to put their downloaded files. It would also allow G to make sense.
Con: Create confusion for experienced users as to why the sites folder is missing.
Impact: At best, new users know exactly where to put their modules and themes. At worst, experienced multi-site users are confused as to the lack of a sites folder.
Crell’s picture

Problem with H: Where do we then put settings.php et al?

Honestly I'm -1 on H. Even when I started with Drupal /sites/ was never confusing to me. Really, the only confusion I had about file placement was fixed by adding sites/all.

drifter’s picture

There are two advantages to H:

1. less confusing for newbies, immediately visible where themes and modules should go,

but also, and while it sounds silly I think it's a major point, it

2. SAVES TWO CLICKS in your editor/IDE/FTP client every time you edit or upload a module or theme.

most custom work is under sites/[all|default]/[themes|modules]. If it was just /themes and /modules, it would probably save me 20 clicks or 10 seconds per day. But estimating 10.000 Drupal developers, it could save us over 5000 man-hours/year, better spent on hacking Drupal :)

As for settings.php, it can still live under sites/default, nothing wrong with that, rarely needs to be touched.

mcrittenden’s picture

Title:Reconfigure Drupal folder structure to improve usabilitiy and upgrades» Reconfigure Drupal folder structure to improve usability and upgrades

Fixing typo.

jwilson3’s picture

Since in actuality the module and themes folders already exist in the webroot folder, the net effect of H is not to "create" those directories, but to repurpose them for simple site installs and have settings.php default to webroot. I assume then that H would have to imply A (moving the contents of the modules and themes folders in the webroot to a 'core' folder).

Problem with H: Where do we then put settings.php et al?

it can still live under sites/default, nothing wrong with that, rarely needs to be touched.

Note that in #66, H's OP says: settings.php would then be placed in the webroot. Its a plus that it could exist under sites/default, for say, upgradability from the existing sites folder system, but I understand that the point of H is to remove the need for sites unless you intend to develop in a multi-site environment. If thats true, then everything in sites/default would (for lack of a better term) default to the public webroot folder for regular default site installs.

I can certainly identify with #68.2; Anecdotally, how many of you added contrib modules to the /modules folder the first couple times you played with Drupal? Repurposing the modules, themes and files folder is a lofty idea, but if I had to choose, I'd say retaining and imposing the sites structure is easy to learn with good documentation, and not a problem if the aforementioned folders aren't there in the webroot to confuse new users in the first place (in other words, if A gets approved for consideration).

Perhaps the best ideas to keep in mind in the thread are to aim for a webroot folder cleanup that permits core "updatablity" while maintaining site/other/custom/stuff "portability" independent of core; and to not confuse the heck out of newcomers, existing users, and important contrib maintainers.

joshmiller’s picture

This is the best comment to date and I'm so ready to rally around it:

Perhaps the best ideas to keep in mind in the thread are to aim for a webroot folder cleanup that permits core "updatablity" while maintaining site/other/custom/stuff "portability" independent of core; and to not confuse the heck out of newcomers, existing users, and important contrib maintainers."

We have to remember that if we change folder structure it will totally create more confusion than necessary. But a little pain in the transition (if done in the name of "clean-up" and "making it simpler") could be just what the doctor ordered.

Crell’s picture

To that end, and for general maintainability and performance, we want to have a clear, small number of places that "stuff" can live. That's one reason I don't think settings.php in / is a good idea.

Currently, Drupal safely assumes that everything it worries about in the installer is in /sites, usually /sites/default. A lot of Drupal management software (Drush, Aegir, and custom stuff that I and others have written) assumes the same pattern. If that pattern becomes more complicated, that makes Drupal and all of that auxiliary code more complicated as well.

As one of my professors in school once said, "overly complex code leads to overly complex bugs". :-)

That is, KISS, both from a documentation perspective, a performance perspective, and a bug-avoiding perspective.

jwilson3’s picture

KISS must be most definitely THE guiding principle for this thread having any chance of getting accepted. I agree with Crell that technically H is introducing more entropy than order and simplicity.

We have a good number of ideas all summarized now, is it time to start analyzing the deeper questions? It sounds like options A+B+C are generally accepted by everyone participating as a really logical thing to do in the name of cleanup, usability, upgradability.

But Option D, Option H and various previous posts (eg #53) are all attempts at addressing specifically one main issue: question #73.1: Is sites/ structure really that confusing or overly complex?

Purely speaking in terms of a perceived complexity, for which this thread exists to confront, and rash disregard for the necessary code changes that may be implied by some of the proposals, it seems like everything is boiling down to two (well, three) camps. Those who think sites/ structure as it exists today is the golden ticket, those that don't, and those that are still riding the fence.

Newcomers, non-developers and perhaps even some developers in the community probably think its not an optimal (not intuitive?) structure.

Here's a quick run-down of some excellent and important features of sites/ as it works today.
(Note that I'm using the word "project" to generalize themes/modules/profiles/etc, afterall, its not drupal.org/module/devel nor is it drupal.org/theme/zen)

* permit multiple site configurations: example1.com-settings.php|example2.com-settings.php,
* inherently distinguish site-specific projects: all|default|example.com,
* inherently distinguish project types: themes|modules|libraries, (what about profiles? are they the black-sheep? technically they are "projects" in contrib repository too)
* inherently distinguish project sources: contrib|custom|fservers,
* inherently distinguish project groupings: features|kits|distros|modules-within-modules,
* permit nearly any structure you can come up with, within the "modules" folder (ie, recursive folder exploration for *.info;)

Important maintainers may argue that sites/ works really well and that any issue regarding complexity is outweighed by the perceived effort and risk in changing existing code, which is to be sure, a natural fear of a paradigm shift that could potentially affect code integrity, as does the whole premise of this thread. (woah, did I really say paradigm shift?) But I remind we of the goal: usability; and the guiding principle: simplicity.

A goal of the Drupal project is to systematically embrace the concept of permitting people to easily override someone else's groundwork of sensible defaults, and BUILD on top of a great foundation. As a Builder, its generally accepted now as reeeely baaaad to have to come in contact with the man behind the curtain (ie hack core), thus, the guiding principle is that Drupal defaults shouldn't get in the way of site customizations. This principle should apply both in code as well as in the filesystem. This is why most people are for moving 'core' out of webroot.

The next logical step, then, is to draw a line between the needed drupal contrib projects, 3rd party contrib projects/profiles/features, and custom or glue code that all work together to hold a particular website together. But the way things seem to be heading with "distros", feature servers, kits, base-themes, etc. I'm seeing things standardizing on a system of yet more and more subfolders to handle these latest inventions of the community. Its almost like, the goal is to see how deep of a folder structure we can create to handle all these inventions. The recursive exploratory nature of the modules folder structure is pushing along nicely the adoption of this depth-first strategy. Following emerging trends, its not far-fetched to envision a larger single-site project with a structure like this:

core/...omg_easy_peasy_upgradable_core_in_a_single_folder_yay

sites/all/modules/contrib/...a_standard_array_of_contrib_modules
sites/all/modules/custom/...a_dab_of_glue
sites/all/modules/development/...the_tools_for_the_job
sites/all/modules/features/...exported_gui_settings
sites/all/modules/migration/...custom_drush_build_and_upgrade_scripts

sites/all/themes/base_admin_theme_from_contrib
sites/all/themes/base_site_theme_from_contrib
sites/all/themes/custom_skin_theme_based_on_base_site_theme

sites/all/libraries/...third_party_goodies

profiles/distro/modules/contrib/...
profiles/distro/modules/features/...
profiles/distro/themes/base_theme
profiles/distro/themes/skin

Do I understand this structure completely? Sure. But, I'm still scratching my head a bit... Is this an elegant solution? While it may be great for (paraphrasing Crell) "bug-avoiding, documentation, and performance", architecturally (perhaps visually), it appears to be about the farthest thing I can think of from the KISS methodology.

One perhaps philosophical concern I'm seeing being brought up by a few people now is 'Why does this important stuff need to be buried so deeply?'

What if we step outside the box and think a second, be a kid again, ahem, i mean.... a software architect. Temporarily forgetting the lines of code written on existing assumptions, could there be something better?

Thus question #73.2: is there a more optimal folder structure that can accommodate for and distinguish between the existing features and attributes of sites/ (some mentioned above) but has some sort of inherent simplicity or other added value that prepares Drupal for the future/long haul?

Its cool if there there isn't a better way, I'm just thinking out loud. anyway, fwiw, I apologize that I don't have any good answers... just food for thought, and, sorry for being so damn longwinded. This will probably be my last post in this thread, meaning ya'll can either ignore all this blahblahblah, or hopefully, if i said something worth while, someone will run with it.... so I'll leave with just a few more thoughts that crossed my mind while writing this up.

Lots of web projects have an intriguing security feature that amounts to module code residing outside of webroot. The whole 'core in its own folder' is a move towards that, but then what about contrib modules, which arguably are more susceptible to attacks than core? Leaving sites/ 'as is' certainly wouldn't permit that. Maybe this calls for introduction of a distinguishing between public web files (css/js/files/theme) and stuff that could potentially exist outside webroot.

Also, the special case that exists today of "profile" type projects being okay to exist in root but modules and themes not seems like a strange aberration in the of order and logic argument for the sites/ structure. If Option A were to be adopted, then what? 'profiles' will be left behind (outside of core/ and outside of sites/). A move towards order or special cases? Or are 'profiles' THE special case (versus features/modules/themes/X-new-invention)?

Is it not at least partially true that the momentous move towards adopting the sites/ structure was born at least partially out of 1) a need to have a place for people to put site-specific files, modules, and themes, 2) and so that folks don't 'dirty up' those inconvenient Drupal core folders in webroot, 3) in order to promote upgradability and code sanity among other things. Again, assuming we adopt option A, and get core off by itself, does that leave us with a really robust solution for a need that simply no longer exists? The only remaining case for a 'sites/' structure would then be exclusively for mult-site support (a la option H)... And around we go again :/

kbahey’s picture

I still see too much discussion for something that can be taken gradually.

The first step is to separate core from what you download/create.

Hence we have:

Only these files in the web root directory:

www/index.php
www/.htaccess
www/robots.txt

Then we have core, as it is downloaded, left untouched, easy to upgrade.
www/core
www/core/modules
www/core/themes
www/core/includes
www/core/whatever/else/...

Then we have everything else under sites. Profiles that are not with core can be there too
www/sites
www/sites/all/themes
www/sites/all/modules
www/sites/all/profiles
www/sites/default/settings.php
www/sites/whatever/else/...

Let us not try to solve everything at once, and focus on whether this is doable, what drawbacks it has. By drawbacks, I mean just bugs, not "wouldn't it be nice if we can have X ....".

If we keep bikeshedding this, it will never make it into core, so let us take reasonable baby steps ...

merlinofchaos’s picture

I am completely in agreement with kbahey.

IMO if you want to stick to the KISS principle then you have to stick to fixing the problems that are identified. The big problem that needs to be fixed is upgrading and preventing people from putting their files in the wrong place. Making upgrades as painless as possible is, IMO, the primary goal here.

wretched sinner - saved by grace’s picture

Title:Reconfigure Drupal folder structure to improve usability and upgrades» Move all core Drupal files under a /core folder to improve usability and upgrades

As a proponent of discussing the value of the sites hierarchy, I vote to get the core move in, and discuss that later. To that end, I have opened up #760992: Where should site specific modules, themes and settings be kept? to discuss the sites issue, so we can focus here on moving core Drupal to a subfolder.

chx’s picture

I also agree with kbahey. That sounds simple and doable for now.

deviantintegral’s picture

Do we want to have DRUPAL_ROOT as '/' or '/core'? It seems to be a bit simpler to redefine DRUPAL_ROOT, but I'm wondering if we need WWW_ROOT, DRUPAL_CORE, and DRUPAL_SITES as global constants in index.php and so on.

kbahey’s picture

@deviantintegral

Hmmm ...

I wanted to initially say that we leave DRUPAL_ROOT as it is, and then base things off it, for example

DRUPAL_ROOT . '/core/' . $something

But it will become tedious after a while.

Perhaps we need DRUPAL_ROOT to stay the same, then DRUPAL_CORE is really DRUPAL_ROOT . '/core' and DRUPAL_SITES is DRUPAL_ROOT . '/sites'. Those can be the first 2 lines in bootstrap.inc (and friends, such as install.php ...)

wmostrey’s picture

I like the structure as proposed in #74, along with DRUPAL_ROOT, DRUPAL_CORE and DRUPAL_SITES.

arlinsandbulte’s picture

Priority:Normal» Critical

+++++++++++1!!

I agree with #74
Marking as critical because this needs to be done as early as possible in development or it becomes impossible.
(sticks his neck out...)

mcrittenden’s picture

Priority:Critical» Normal

This isn't critical. Definition of critical: "When a bug renders a module, a core system, or a popular function unusable." Source

joshuajabbour’s picture

I agree with kbahey in #74 (and not just because it's what I proposed in #25 :)

Anyway, can moving these files around be done in a patch, or is it something that can/should be done on the server with crazy archaic cvs commands to save histories (I seem to recall this being a possibility)? Obviously there will still need to be a patch to update the paths _within_ modules, etc.

I'm willing to tackle the patch part, just wondering if it should be a giant monolithic thing that does everything, and if it should actually move the files (remove/add)?

Crell’s picture

A CVS patch can't do this. There's some hacky back door that only dww knows how to use that involves mucking about in the CVS history directly that I try to not even think about.

A git commit object, on the other hand, can move files around like this without even blinking. :-)

joshuajabbour’s picture

oh, trust me, I know. I can't wait for the future.

so best way to proceed? how's this:

- map out dir structure
- create patch to update paths to new structure within drupal (inc, misc, etc)
- bribe dww to actually do the work to move the files in cvs

Crell’s picture

Alternatively:

- Map out dir structure.
- Wait for git.
- Bribe Dries to actually do the work of moving stuff around.

We still need to finish step 1, however.

kbahey’s picture

During the DrupalCon San Francisco pre/post code scripts, deviantintegral and myself worked on a script that would do this conversion for us. This way, we will not chase the ever changing HEAD.

It is half working now, with the installation going half way through with the new directory structure. There is a lot of work to be done though.

It is committed to my sandbox here.

If anyone wants to improve the code further, please check it out and go hack on it further.

[Edit: link was not visible. Made it visible.]

dww’s picture

a) At the presentation I gave about this at the Drupal Core Developer Summit, I warned that the single greatest threat to this issue is the high likelihood of a monster bikeshed, not any technical problems. Please, let's not "map out dir structure". It's already mapped. If we keep debating it, we'll never see any improvement at all.

b) http://drupal.org/node/369445 tells all about CVS renaming. Yes, doing this in Git would be better, but we don't have to block this on Git. If Git is ready when we create the D7 branch and open up HEAD (aka "trunk") for D8, lovely. If not, at the break-out session on this at the core dev summit, we all agreed it would be better to just make this change, regardless of CVS vs. Git. The history is preserved, and we even know that our CVS -> Git migration handles it properly (since we've moved around core files in CVS before).

So, please, no more FUD in here. ;)

Thanks,
-Derek

joshuajabbour’s picture

by "map out dir structure", I absolutely mean to use what's been mapped out in #74.

what I really meant was more "map out what changes need to be made in code to point to new directory structure". poor wording on my part. and maybe not even necessary, as those changes will need to be made by patch anyway. however I wrote "map out" because it will still be a massive, far-reaching commit, and keeping it up-to-date may be a pita (hence the "map out", i.e. keep track of what _types_ of changes we're making)...

and, yes I absolutely think git is not a blocker. it's also nice to know there's a script to do cvs renames--every other time I've seen it brought up, it was said you were the only person with those magical capabilities. :) hope I wasn't spreading FUD, because my intent was actually to get this thing moving...

EvanDonovan’s picture

Agreed with @kbahey, @dww, @merlinofchaos, @chx, et al.: /core & /sites sounds great. I think there was another issue along these lines which should be flagged duplicate of this one. (Or has it already been?)

izmeez’s picture

subscribing, I had no idea this discussion has been going on since 2005 !

Like the original feature request I have wondered about the clutter.

When I first started using Drupal beginning with 6.1 the directories were confusing.

As a newbie, I was surprised that Drupal was not typically installed in its own sub-folder of the web root. Instead Drupal is typically installed in the web root, public_html folder where a number of files are placed and several folders with files, etc.

It would be great to have all the Drupal related files and folders in one directory.

From a usability perspective I like "core" and "sites" and would like the Drupal folder to have these two main sub-folders and all the other files currently in public_html folder.

From the various threads I have come across, including this one, I realize there are many considerations and issues including multisites, clean url, duplicate urls, seo etc.

And a variety of solutions including using symlinks, .htaccess, and possibly $base_url in settings.php although I have wondered if configuring $base_url in D6 causes filenames to be stored as a full filepath name rather than relative which can cause problems when migrating.

All of this is confusing to a newbie who is just trying to have a tidy server, but I have come to learn is part of the Drupal journey.

The features requested in the original post in 2005 resonate with my old newbie Drupal dreams,

1. Less clutter in the root directory.

2. Separation of core and sites files without intermixing.

3. This would simplify upgrades.

jspotswood’s picture

Subscribing. Also extremely surprised that this discussion has been going on for sooooo long.

I am planning on using a work around described at http://drupal.org/node/144643, to get around this short coming of Drupal. But like all work-arounds, I am sure it will bite.

juan_g’s picture

Yes, this issue is active since 2005. The topic of this issue was my first thought when I downloaded Drupal 5 and Drupal 6 a couple of years ago. In Drupal 7.0-alpha7 we still have the following.

Folders in web root:

includes
misc
modules
profiles
scripts
sites
themes

Files in web root:

.htaccess
authorize.php
CHANGELOG.txt
COPYRIGHT.txt
cron.php
index.php
INSTALL.mysql.txt
INSTALL.pgsql.txt
install.php
INSTALL.sqlite.txt
INSTALL.txt
LICENSE.txt
MAINTAINERS.txt
robots.txt
update.php
UPGRADE.txt
web.config
xmlrpc.php

Having the Drupal admin folders and files together in one or two folders (excepting .htaccess, index.php and robots.txt) would make maintenance, permissions, updates, etc. easier. In our case, we have a lot of other folders and files in addition to the Drupal ones, and they are now mixed, which increases errors and security risks.

It would be indeed great to take this issue into account now that Drupal 8 is near to the beginning of its development cycle. Later it would be not so easy as now.

juan_g’s picture

merlinofchaos wrote:

IMO we should consider this strongly as an early entry into D8 so that we can get it done and out of the way while there are still few patches against D8 (since it will invalidate, oh, every patch. And then some).

Yes, it looks like the best moment for this task is coming soon. D7 alpha7 has been released today, the first D7 beta will be probably out one of these days (there are just two remaining issues blocking D7 beta right now), and developers will soon be able to start concentrating more and more in D8.

Patches for this will affect multiple files. The following seems a good way:

kbahey wrote:

Perhaps we need DRUPAL_ROOT to stay the same, then DRUPAL_CORE is really DRUPAL_ROOT . '/core' and DRUPAL_SITES is DRUPAL_ROOT . '/sites'. Those can be the first 2 lines in bootstrap.inc (and friends, such as install.php ...)

Also, as mentioned in this issue, this kind of folder tree change seems much easier for Git than for CVS, but doable, especially if the change is made just at the beginning of the Drupal 8 development.

Damien Tournoud’s picture

Status:Active» Postponed

Seems that we have an agreement here.

Let's postpone on the Git migration.

juan_g’s picture

I don't think this is an issue to be be postponed until a total CVS -> Git migration of all the historical code. If we postpone it until 2011, it will be probably postponed again until Drupal 9. I think all Drupal 8 development, with the new folder tree, should be done from the start with Git only, even during the migration of the old versions.

As said recently in Git Transitional Workflow Discussion (2010-09-06):

Even though the Drupal CVS to Git migration will take a few more months, it's possible to start using git now for nearly everything you used to do.

So, what is the consensus on this task: postponed until 2011 (CVS ending) or later (D9), or active for D8 and transitional Git (2010)?

Damien Tournoud’s picture

No, it's not possible to start developping in Git and migrate old revisions later. The Drupal 8 tree is not open yet, and will not open before the 7.0 release. This can easily wait that we switch the Drupal Core to Git.

antgiant’s picture

Status:Postponed» Active

See comments #84 and #88. The CVS vs GIT issue has already been discussed and decided. It was already explictly decided that this issue should not be postponed due to GIT. Let's please keep it that way.

juan_g’s picture

Damien Tournoud wrote:
> No, it's not possible to start developing in Git and migrate old revisions later.

Well, now I think probably that point of mine was a too unpractical idea, sorry. There is now a Git mirror where it seems everything works except committing. So it will be simpler to disable committing on CVS and enable it on Git for everything at the same time, rather than first for D8. Let's hope this will happen sooner than I expected, hopefully before Drupal 8 is open for development.

antgiant wrote:
> It was already explictly decided that this issue should not be postponed due to GIT.

You are right, dww -who for example designed the current Drupal release system- has said on this issue:

Yes, doing this in Git would be better, but we don't have to block this on Git. If Git is ready when we create the D7 branch and open up HEAD (aka "trunk") for D8, lovely. If not, at the break-out session on this at the core dev summit, we all agreed it would be better to just make this change, regardless of CVS vs. Git.

So it seems this is coming for D8.

dww’s picture

Note: the fact the version says "8.x-dev" means this is effectively "postponed" anyway. Y'all can get in a pointless pushing match about it, but the facts remain that: a) we can't do anything about this right now, b) we know what we want to do (at least to start), c) we all want to see this happen in D8. Please let's not generate excess noise.

Thanks!
-Derek

joshuajabbour’s picture

now that we're officially in d8 development, and very close to TGGM, I thought I'd bump this topic so it's not forgotten about. Not sure if it still needs to happen asap after head is opened up (what with git's easy moving and all), however...

juan_g’s picture

Category:feature» task
Priority:Normal» Major

@joshuajabbour

The Drupal 8.x-dev page is still just a placeholder for D8 issues like this one. So, although I can be wrong, it looks like D8 is maybe waiting some days until the CVS -> Git migration on February 17.

Yes, there was an agreement about this issue being a task with significant repercussion -that is, major I think- needing to be done at the beginning of the D8 development. As dww said, it could be done on CVS -even if it's simpler on Git- but, since Git is coming so soon and people is focusing on the migration now...

However, I haven't seen yet an official word on a D8 opening date for actual commits, with dev releases, etc. Just the placeholder for this waiting issue queue.

Dave-B’s picture

Rather than, e.g.,

/core
/sites
index.php
.htaccess

surely it'd be a good idea to prefix the directory names so they don't interfere with paths that might be needed for clean urls, e.g. use

/d_core
/d_sites
index.php
.htaccess

If I'm right in thinking that we get 403 errors on any alias matching a folder in the drupal root, then even a reduced number of folders at that level would still prevent a webmaster using e.g. www.drupal.org/core or www.drupal.org/sites.

I found this issue due to getting a 403 Forbidden error from a page using the url alias "sites", which led me to http://drupal.org/node/121362, and then here.

If there's some way of viewing pages using aliases matching those drupal folders (sites, misc, themes, etc) that I've missed, I'd be very happy to hear about it...

EDIT:

  1. I agree with @joshmiller - sorting the drupal directories by only using an underscore for the prefix would be good.
  2. I've found that I can use aliases on existing directories (e.g. "sites") by amending the clean URLs rewritecond (http://drupal.org/node/246093).
joshmiller’s picture

Not to continue the bikeshed that we've been warned about, but @Dave_B makes a good point in the above comment.

What if we were to name the folders with one character instead of two characters. In most modern operating systems that would make these folders group together and be on top of a listing.

/_core
/_sites

Just a thought...

juan_g’s picture

Some kind of prefix seems indeed a good practice. For example, I think another well-known open source CMS (WordPress) has three folders: wp-admin, wp-content, wp-includes. The suggestions on previous posts (d_..., _...), or something like "drupalcore" and "drupalsites", could be also good.

danillonunes’s picture

Subscribe.

And also +1 for just move things to /core first and deal with sites/custom folder after.

joachim’s picture

Good point made here, though I think d_core and _core are kinda ugly, and we don't do ugly round here ;)

'drupal_core' will do just as well, surely. Also, the standard form for menu paths is to use a hyphen to break words (eg admin/settings/date-time), so if someone does want a page to talk about drupal core, they'd probably alias it 'drupal-core' :)

danillonunes’s picture

I created a new sandbox trying to start #65 A.

There will be a lot of work, so my idea is to divide it into small moves for each directory (misc, includes, profiles and "modules and themes"), and branch it. Thanks to Git will be possible to merge all them easily later.

It's in http://drupal.org/sandbox/danillonunes/1098200. Help will be welcome.

scor’s picture

I think this approach of multiple branches is cumbersome, you need to have all the changes made in the same branch to test whether it works or not.

Most importantly, instead of making manual changes into the core codebase, I think we should work on a script for automatizing this, like what kbahey started to work on in #65. I've imported his script from the legacy RO CVS into http://drupal.org/sandbox/scor/1102276 and started to make changes so that it works with git (move files around and do string replace across core). The script is about 120 loc so far, which will be much easier to review and agree on than an actual branch diff (which currently spans about 2000 lines).

joachim’s picture

Another thing to bear in mind is that in order to preserve file history, the commit that moves a file must not change it at all.

Hence we ultimately will need a commit on the 8.x branch that moves all the files and *breaks* Drupal, as no paths in code will be changed.

arlinsandbulte’s picture

Another thing to bear in mind is that in order to preserve file history, the commit that moves a file must not change it at all.

Why? What if we took this gradual approach:

  1. add a new folder to root: \drupal_core\
  2. add relevant subfolders to the \drupal_core\ folder:
    \drupal_core\includes\
    \drupal_core\misc\
    \drupal_core\modules\
    \drupal_core\profiles\
    \drupal_core\scripts\
    \drupal_core\themes\
  3. start, slowly & one-by-one, moving the files & folders to the corresponding \drupal_core\*\ folders, making making sure each 'push' is tested and functional.
  4. lastly, rename the sites folder to drupal_sites

I don't think steps 1 & 2 would actually break anything. Those empty folders could simply be added without any ill effects (might need a dummy placeholder file).
Then, the real work could be split up and done in #3 in steps or stages. Thus files and folders could gradually migrate into the new folder system.
Lastly, step #4 should be fairly trivial (maybe optional, even).

Does that makes sense, or am I talking about things of which I know nothing about?

joachim’s picture

What I mean is that AFAIK if you rename or move a file in git (which is the same thing), the file must not change during that move if you want git history to be preserved. I seem to be repeating what I said in my earlier comment, I'm afraid, but I don't know how to phrase it differently.

If we want, say, the file system.module to preserve its history, then commit X must move it from modules/system/system.module to core/modules/system/system.module and NOT change its contents in that commit.

Hence after that commit, Drupal lies broken until the next (jolly quick!) commit fixes all the paths.

scor’s picture

Status:Active» Needs work

Another thing to bear in mind is that in order to preserve file history, the commit that moves a file must not change it at all.

--follow will give you all the history, including the changes the file had during the move. for a file moved and changed during the same commit, git log -p --follow core/authorize.php will say:

commit ddebf3a9f8ceadb703bddbcf4ed95e938ec4e80b
Author: scor <scorlosquet@gmail.com>
Date:   Wed Mar 23 11:17:47 2011 -0400

    move Drupal core files into /core folder

diff --git a/authorize.php b/core/authorize.php
similarity index 91%
rename from authorize.php
rename to core/authorize.php
index cd3df50..bc1ed26 100644
--- a/authorize.php
+++ b/core/authorize.php
@@ -23,7 +23,10 @@
/**
  * Root directory of Drupal installation.
  */
+chdir('..');
define('DRUPAL_ROOT', getcwd());
+define('DRUPAL_CORE', DRUPAL_ROOT . '/core');
+define('DRUPAL_SITES', DRUPAL_ROOT . '/sites');

/**
  * Global flag to identify update.php and authorize.php runs, and so

juan_g’s picture

About the points on folder naming in #107, if d_core and _core are kinda ugly, and hyphens aren't convenient, then other suggestions from above comments are:

  1. core, sites
  2. drupalcore, drupalsites
  3. drupal_core, drupal_sites

I think probably any of these can be good, maybe better with the "drupal" prefix.

jwilson3’s picture

I was under the impression that the first step was moving to "core", the cleanest and simplest of the above options. Then, the second step would be to make the location of the core folder over-ridable (eg through a variable defined in index.php), so if someone needed the url "/core" or wanted to move core completely out of the webroot, they could achieve that.

I'd propose to leave "sites" alone for now, as 1) it seems out of scope for the issue as it stands now, and 2) it will add confusion and make upgrading a lot more challenging (see Earl #75).

quicksketch’s picture

Just joining in to say that putting everything in "core" that we can sounds great. Eventually I'd like to eliminate "sites/all" and put modules and themes directories right in the root (option H from the long discussion above), but I understand that we've had enough bike-shedding for now. Let's make the core directory and update the code. We'll save further restructuring for later.

To move forward we should outline the git commands needed to move the files from one location to another. That way Dries actually has some way of implementing the recommended changes. As scor said, we do NOT need to break core to have these changes made. We'll just use one series of move commands, do the git commit of the moves, apply a patch to the moved files, commit the patched files. Then push the series of commits back up to origin. No problem, no tests ever broken.

I also think this should be the first thing we handle in Drupal 8 (the nice thing being the "sites/all" directory has absolutely zero impact on further core development). After #1097100: Remove all 7xxx update functions and tests (D6 to D7 upgrade path) is completed (it's only a matter of hours away), Drupal 8 development will open and we'll need to move on this immediately.

mlncn’s picture

+1 subscribe to the core, sites, profiles move for simplicity and beauty reasons (and i think this get us closer to my sysadmin's dream of having core installed in /usr/local/bin or somewhere i'd never find ;-) )

scor’s picture

most of the work will be writing the script to replace all the paths... the latest git-aware version is at http://drupalcode.org/sandbox/scor/1102276.git/blob_plain/HEAD:/22336.php

quicksketch’s picture

Status:Needs work» Needs review
StatusFileSize
new159.2 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch drupal_core_move.patch.
[ View ]
new502 bytes

I don't think a script will be able to handle all the use-cases we need (though it might help if this task takes a long time). I didn't use the script for this patch, I just did it "the old manual way" with a search and reading every place we do an include of Drupal core; call update.php, install.php, or cron.php; and all calls to DRUPAL_ROOT (plus some other searches).

RE: #78 and #79: I do not think we should make a new constant for DRUPAL_CORE, because there are many places where we need the string "core/" without DRUPAL_ROOT attached to the beginning of it. While we could make a constant for just the string "core/" and then another for DRUPAL_ROOT . DRUPAL_CORE, I find it would not actually save us any typing or be any more clear than just using the actual path string. Moreover we'd need to redefine that constant in update.php, install.php, authorize.php, index.php, and several tests. Generally just not worth it.

Another interesting hold-up is what we should do with the "profiles" directory. Some people have noted above that the "profiles" directory could be modified by distributions. Unfortunately profiles do not yet have the flexibility of modules and themes to be located in multiple places. Since we don't want distributions to "hack core" (which is what they'd be doing if they had to change /core/profiles), the profiles directory must stay in the Drupal root until we add more flexibility to install profile locations. This isn't too bad because you can still upgrade core just by replacing the "core" directory, since you'll never need to update the Drupal core install profiles after you've installed Drupal anyway. Unless there are large objections, let's save the "fixing" of profiles for a later issue.

This patch is already quite large, which goes to show just how big of a change this is even without further changes. This approach follows the recommendations of kbahey, merlinofchaos, chx, and others; so I'd like to see us start with this and fix all of the following later:

Things for later:
- Moving the "profiles" directory to "core/profiles".
- The "Option H" described in the comments above, moving "sites/all/*" to the root directory.
- Any other changes to the "sites" directory, such as removing "sites/default" or relocating settings.php
- The addition of a "docs" directory.

All of these things are great ideas and I'm personally interested in contributing (thoughts or code) to all of them, but this patch is big enough as long as we all agree on the direction being suitable. I've attached two files, one that provides a list of git commands to move the files to their new homes, and then a patch to be applied on top of the new structure that makes all necessary code changes. What say you guys?

quicksketch’s picture

StatusFileSize
new145.58 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch drupal_core_move_0.patch.
[ View ]

Reroll of the patch against the proper Git branch (last one was against 7.0 apparently). Passes SimpleTests! Very cool. Regarding an upgrade path from Drupal 7, I think the change should be pretty simple. The only thing that looks like it's absolutely necessary is to rebuild the file/object registry immediately upon visiting update.php, since all the existing paths will be out of date. We'd probably need to do this eventually anyway if we ever added any new files to the database system.

scor’s picture

@@ -1040,7 +1040,7 @@
   // Escape the periods in the extension.
   $regex = '/' . str_replace('.', '\.', $extension) . '$/';
   // Get a listing of all template files in the path to search.
-  $files = drupal_system_listing($regex, $path, 'name', 0);
+  $files = file_scan_directory($path, $regex, array('key' => 'name'));

   // Find templates that implement registered theme hooks and include that in
   // what is returned so that the registry knows that the theme has this
@@ -1200,7 +1200,7 @@

Is this necessary for this patch?

quicksketch’s picture

-  $files = drupal_system_listing($regex, $path, 'name', 0);
+  $files = file_scan_directory($path, $regex, array('key' => 'name'));

Is this necessary for this patch?

Yes this is an intentional change. When the theme system goes through and tries to find all the template files for a particular theme, it passes in the entire structure, such as "themes/garland" or "sites/all/theme/mytheme". It's mostly by accident that drupal_system_listing() matches files against "sites/all/themes/mytheme", since what it's actually looking for when it successfully matches is a theme in core. It also causes unnecessary checks at the location "sites/all/themes/sites/all/themes/mytheme" and in "profiles/sites/all/themes/mytheme". Clearly this isn't intentional behavior.

In any case, the reason it's needed for this patch is because trying to find the templates for a core theme needs to look in "core/theme/garland", not "core/core/theme/garland". I think the use of drupal_system_listing() entirely incorrect, since you really should only find template files in the same directory as the theme, not in one of several directories. Note that this only affects discovery of template files, not the discovery of themes themselves.

quicksketch’s picture

StatusFileSize
new158.11 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch drupal_core_move_1.patch.
[ View ]

One more fix in system.module I found due to a SimpleTest failure of AJAX requests.

EDIT: Difference in patch size is due to rolling with different diff options (this time with -p to show functions).

chx’s picture

quicksketch, on #121 and another constant -- it would worth it! Put the constants in constants.php and include it always that saves you the work of defining of it N times. And then aspiring people can put their Drupal where they please. Like, outside of the document root. Put the php files being used in docroot (just index.php in production, most of the time) edit constants.php and have fun. The point, of course, is somewhat heightened security.

quicksketch’s picture

quicksketch, on #121 and another constant -- it would worth it!

@chx: I really don't think new constants are worth it. There are literally hundreds of places in core where the path is already relative to the DRUPAL_ROOT variable, such as variables for cache.inc, password.inc, menu.inc are all defined as such: variable_get('cache_inc', 'includes/cache.inc'). Would changing this to variable_get('cache_inc', DRUPAL_CORE . '/includes/cache.inc really be an improvement? Like I said above we'd actually need not just one constant this way, we'd actually need two, (say DRUPAL_CORE and DRUPAL_CORE_ROOT), or we'd just end up using both constants all the time like this:

include_once DRUPAL_ROOT . '/' . DRUPAL_CORE . '/includes/database/prefetch.inc';

Or at best:

include_once DRUPAL_CORE_ROOT . '/includes/database/prefetch.inc';

I don't find this to be an improvement at all. The ideas of having a swappable "constant.inc" or moving the core directory outside of the Drupal root aren't necessarily bad, they're just not part of the issue that we're trying to address here, which is to improve the structure of new Drupal installs. The patch is already 160KB and it doesn't change anything beyond moving the files to their new locations. Let's keep it that way and keep re-architecting things in a different issue. At the very least, this patch makes future find/replace tasks much easier, because it adds the very-searchable string "core/" everywhere.

Status:Needs review» Needs work

The last submitted patch, drupal_core_move.patch, failed testing.

juan_g’s picture

So, following quicksketch's (#128), chx's (#126), and others' suggestions, the current issue creates an unified core folder, and a follow-up issue could develop chx's really tempting proposal, for example #1116144: Customizable locations for Drupal core folders (constants DRUPAL_CORE, DRUPAL_SITES) (just opened for D8; there seemed to be no issue yet on that specific task).

quicksketch’s picture

Oh right... of course this patch isn't going to work if it's against the new file structure and not the old one. I'll need to roll a patch like scor demonstrated in #115 in order to get testbot to move the files to their new locations and then apply the changes.

Thanks juan_g for the new issue, I've posted some of my thoughts over there.

scor’s picture

The patch including all the file move instructions is 19MB, which is over the 3MB limit allowed on d.o :(

rfay’s picture

And we really need to use an ordinary git patch that has moves. #1107552: Make git patch application more robust and use git for patch application

scor’s picture

Status:Needs work» Needs review
StatusFileSize
new18.73 MB
FAILED: [[SimpleTest]]: [MySQL] Drupal installation failed.
[ View ]

thanks DamZ

Status:Needs review» Needs work

The last submitted patch, 22336_132_1.patch, failed testing.

scor’s picture

filed #1116914: failed to set database information error during installation, most likely due to the fact that the testbot excepts to find install.php next to index.php...

danillonunes’s picture

@scor the patch is big because it's a copy of all moved files. Also, if some file is changed, the patch will be broken. Instead of a patch, just send instructions to move directories like git mv modules core/modules

scor’s picture

StatusFileSize
new472.93 KB

@danillonunes we're trying to get the testbot to test this. the testbot does not understand git mv instructions, and will likely never will. At best it will support git patches.

Uploading a git patch (to be applied with git apply or git am only) for when the testbot supports it. The file size goes down to 473KB. instructions on how to generate and keep this file size low are at #1116900: git format-patch displays file moves as whole verbose diffs.

chx’s picture

After consulting with quicksketch it seems I already have what I want 'cos all PHP files are already DRUPAL_ROOT bound so please disregard #126.

quicksketch’s picture

Status:Needs work» Needs review
Issue tags:-Bikeshed
StatusFileSize
new18.72 MB
FAILED: [[SimpleTest]]: [MySQL] Drupal installation failed.
[ View ]

@scor's #138 is definitely the way to go if we can get testbot to apply patches with git apply or git am, but this patch is just another huge patch using the old format to see if testbot will pass with a few redirects set up manually (install.php to core/install.php) in htaccess.

Status:Needs review» Needs work

The last submitted patch, drupal_core_directory.patch, failed testing.

boombatower’s picture

Confirmed bug in redirect if running in sub-directory which is preventing testbot form working since it does install.php?profile=standard&locale=en.

quicksketch’s picture

Status:Needs work» Needs review
StatusFileSize
new473.71 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch drupal_core_move_2.patch.
[ View ]

Okay boombatower figured out the issue being that my new redirects simply didn't work when Drupal was installed under a sub-directory of the web root. Since this is what pifr does, the redirect didn't work. Here's a new patch in the new format (two separate commits) instead of the giant format. We're going to see if pifr can handle it.

Status:Needs review» Needs work

The last submitted patch, drupal_core_move.patch, failed testing.

boombatower’s picture

Status:Needs work» Needs review

SimpleTest browser chokes on /core during install (pifr uses), so I tweaked pifr to force it to work. Another reason why we need the good old browser rewrite in core. Anyway here is the issue to track #1120030: Support moving of core files to core/ directory.

boombatower’s picture

Status:Needs review» Needs work
rfay’s picture

Status:Needs work» Postponed

I fully support this initiative. Absolutely.

Would it be possible to postpone it until the huge backlog of D7 patches gets at least a bit cleared? This will break every patch, of course, and since we have about 200 patches that need to be cleared through D8 and then back to D7, it will make the process easier if we defer this one.

Marking postponed. Just move it back if you disagree.

BTW: Deploying PIFR code today that will allow git copies and moves to be allowed in patches (it now uses git to apply). #1107552: Make git patch application more robust and use git for patch application

quicksketch’s picture

Status:Postponed» Needs review

#145: drupal_core_move.patch queued for re-testing.

quicksketch’s picture

@rfay: I would say we shouldn't postpone this issue since what we're attempting to do is get it to pass test bots at this time (which we can't do if it's marked postponed). I definitely agree that we should try to clean out the RTBC queue for D8/D7 first, but we likely still have a ways to go on this patch I'd like to continue pushing forward until it's RTBC, then we can give other RTBC issues priority above this one.

Thanks for the heads up on test-bot, I marked the last patch for a new review to try it out.

Status:Needs review» Needs work

The last submitted patch, drupal_core_move.patch, failed testing.

rfay’s picture

@quicksketch, right-ee-o. I'm all for getting this all ready. Jus twould hope it shouldn't get committed until the backlog is cleared.

quicksketch’s picture

Status:Needs work» Needs review
StatusFileSize
new472.63 KB
FAILED: [[SimpleTest]]: [MySQL] Drupal installation failed.
[ View ]

Heh, apparently it honestly needed a reroll this time. New patch off of HEAD.

RobLoach’s picture

So glad we have git during this move :-) .

Status:Needs review» Needs work

The last submitted patch, drupal_core_move.patch, failed testing.

scor’s picture

Would it be possible to postpone it until the huge backlog of D7 patches gets at least a bit cleared? This will break every patch, of course, and since we have about 200 patches that need to be cleared through D8 and then back to D7, it will make the process easier if we defer this one.

I worry that the backlog will not really go down that much: as some RTBC patches get committed, more NR patches will become RTBC. How many weeks would it take do you think? maybe we will be half way through the D8 release cycle by then... From what I can tell looking at the RTBC queue for D7 backports, we're only at 39 issues. Yes, this pacth would break many patches, but all it would take to fix these patches is:

s:--- a/:--- a/core/:
s:+++ b/:+++ b/core/:

I'd be happy to help rerolling these patches.

quicksketch’s picture

Status:Needs work» Needs review
StatusFileSize
new472.63 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch drupal_core_move_4.patch.
[ View ]

Another sad, destined-to-fail reroll. Just keeping it up-to-date while Pifr gets sorted out.

Status:Needs review» Needs work

The last submitted patch, drupal_core_move.patch, failed testing.

rfay’s picture

Oops. I confirm that #158 does not apply manually.

rfay@rfay-gazelle:/tmp/drupal$ git apply drupal_core_move_4.patch
error: patch failed: core/includes/theme.inc:1760
error: core/includes/theme.inc: patch does not apply
error: patch failed: core/modules/simpletest/simpletest.pages.inc:82
error: core/modules/simpletest/simpletest.pages.inc: patch does not apply
error: patch failed: core/modules/update/update.report.inc:57
error: core/modules/update/update.report.inc: patch does not apply

I know you'll apply the patch and run at least a couple of tests before the next one :-)

RobLoach’s picture

Would a git fork and pull be a better solution for this?

quicksketch’s picture

Weird, yeah right now I can git pull and it merges successfully. I then do a git format-patch per scor in #138, but then immediately it doesn't apply to a fresh install. I think I'm probably missing something on the git side of things. I'll investigate and see if I can get an adequate patch.

@Rob Loach: It probably would be a very good scenario for a pull request, though we'd need to get testbot testing that branch (though I suppose you can theoretically have testbot enabled on other projects, it might work). In any case we have a conflict (or something) right now that just requires manual work, I don't think it's a problem with our tools at the moment.

boombatower’s picture

Yea, it would make it much more reliable to test that the testbot can work with /core...we can deal with patch later.

donSchoe’s picture

+1

quicksketch’s picture

Status:Needs work» Needs review
StatusFileSize
new474.1 KB
FAILED: [[SimpleTest]]: [MySQL] Drupal installation failed.
[ View ]

New updated patch that I've tested against the 8.x branch and I'm sure it should apply. It seems that merging my conflicts increased the number of commits in this patch from 2 to 3, plus I forgot robots.txt in my commit so now there are 4 commits/patches total included in this file. If this happens every time we get conflicts this could make for quite a few merge commits being included in every reroll. I'll set up a sandbox we can enable testing on.

Status:Needs review» Needs work

The last submitted patch, drupal_core_move.patch, failed testing.

quicksketch’s picture

Well, dang. Testbot still can't install. No different from #134 and #140.

I've pushed my new sandbox up at http://drupal.org/sandbox/quicksketch/1147512. Perhaps we can get testing on it somehow?

RobLoach’s picture

StatusFileSize
new518 bytes
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch dashboardcssandprofiles.patch.
[ View ]

It looks like drupal_core_directory_move/modules/dashboard/dashboard-rtl.css and drupal_core_directory_move/profiles are still there. Seems like these should be in drupal_core_directory_move/core too.

Is this all that's required to move the profiles?

git clone http://git.drupal.org/sandbox/quicksketch/1147512.git drupal_core_directory_move
cd drupal_core_directory_move
git checkout 8.x
git mv modules/dashboard/dashboard-rtl.css core/modules/dashboard/dashboard-rtl.css
git mv profiles core/profiles
wget http://drupal.org/files/issues/dashboardcssandprofiles.patch
git apply dashboardcssandprofiles.patch
quicksketch’s picture

Thanks Rob Loach for the heads up on the dashboard CSS file. Regarding profiles, see my comment in #121 about why they need to stay where they are (for now).

RobLoach’s picture

Yeah, sounds reasonable. After we do this move, we should consider moving profiles to sites/all/profiles and/or core/profiles, with the ability to enable/disable certain profiles in settings.php, or something. I've always wondered why we don't have a sites/all/profiles.

boombatower’s picture

well by that I would assume you mean profiles directory in any sites directory which seems logical and is something I have actually looked for on a few projects.

boombatower’s picture

Status:Needs work» Needs review
StatusFileSize
new345.86 KB
FAILED: [[SimpleTest]]: [MySQL] Drupal installation failed.
[ View ]

Fixed up run-tests.sh.

git diff -C -M core/8.x origin/8.x

Status:Needs review» Needs work

The last submitted patch, drupal_core_move.patch, failed testing.

pwolanin’s picture

We had another issue about sharing a set of modules across multiple profiles, which is something core doesn't support right now. It would be good to consider that problem space as part of this re-organization.

quicksketch’s picture

@boombatower: So testbot should in theory be able to handle this now? Any idea why #172 is still failing to install Drupal during testing? Once testbot is able to install I'm all-over getting 100% of tests passing. Right now it still seems like we're somehow hung-up on our tools.

@pwolanin: This patch does absolutely nothing outside of moving files around and adjusting the code to support the new locations. There are plenty of things we can do to improve our file structure, but we're saving them for follow-ups. As a 300-400KB patch, this is already a huge issue. See #121 for other reorganization ideas that are also as follow-ups.

boombatower’s picture

We just need to deploy it now. With the redirect it gets farther to the point where it got stuck locally. Rfay has some things he'd like to get in as well so hopefully we can push a deployment this week.

pillarsdotnet’s picture

rfay’s picture

FYI, the code to support this in PIFR is committed and I'll try to test it in a real testbot environment probably next week.

rfay’s picture

So sad to say this... This patch is stale again and does not apply.

git apply ~/tmp/drupal_core_move_6.patch
error: patch failed: modules/node/node.module:1455
error: modules/node/node.module: patch does not apply
error: patch failed: modules/simpletest/tests/batch.test:366
error: modules/simpletest/tests/batch.test: patch does not apply

That means I can't test the testbot against it. I want to get that stuff committed to PIFR so will go forward without testing this part of the release unless a reroll shows up shortly. It's possible that with an updated patch here it will work fine.

@quicksketch: Just noticed your request to test your sandbox. We can't test sandboxes, but can test full projects. Just promote it and let boombatower or me know and we'll turn on testing. (We've never tested a drupal fork before, so that may cause its own problems... I'll bet it will...)

rfay’s picture

StatusFileSize
new7.02 MB
PASSED: [[SimpleTest]]: [MySQL] 0 pass(es).
[ View ]

In sandbox:

git remote add drupal git://git.drupal.org/project/drupal.git
git fetch drupal
git rebase drupal/8.x
git diff -C -M drupal/8.x 8.x

It seems there was just one merge collision in common.test; I think I got it right.

This is committed to the sandbox (didn't know I had commit rights, but to keep things going I'll commit)

quicksketch’s picture

This is committed to the sandbox (didn't know I had commit rights, but to keep things going I'll commit)

Yep, I gave both you and boombatower maintainer access when I set it up. Please feel free to push code as necessary! Thanks for your help on this, I was really beginning to believe we'd just be stuck forever on testbot failing.

rfay’s picture

Actually, I did fail to commit, as you can't rebase a sandbox.... But I am in the process of deploying this today, so maybe we'll be done!

boombatower’s picture

That or you could push to a new branch?

rfay’s picture

Status:Needs work» Needs review

OK, changes to testbot deployed. What will happen? Fingers crossed.

Status:Needs review» Needs work
Issue tags:-Needs usability review, -DX (Developer Experience), -DrupalWTF

The last submitted patch, drupal.core_move_22336_180.patch, failed testing.

rfay’s picture

Status:Needs work» Needs review
Issue tags:+Needs usability review, +DX (Developer Experience), +DrupalWTF

#180: drupal.core_move_22336_180.patch queued for re-testing.

rfay’s picture

IT PASSED! YAY!

Note that I did *not* commit this patch. I think in the future the way to go forward will be to create new branches, since that's the drupalish way to rebase (since you can't push rebased stuff)

rfay’s picture

Status:Needs review» Needs work

Ooops. That was a "0 passes". Yuck. We're going to have to work this one.

My current theory is that these are caused by the testbot timing out or something, although in this case it didn't have time to time out (90 minutes).

webchick’s picture

rfay’s picture

boombatower’s picture

Why did the patch change from 345.86 KB to 7.02 MB? Yes it is creating the /tmpfs problem, but seems like patch just needs a reroll or something to get the size back down?

rfay’s picture

I was surprised at the size change as well. Maybe you can get it down.

However, I don't think the size of the patch has anything to do with the tmpfs fail.

rfay’s picture

Status:Needs work» Needs review
StatusFileSize
new7.02 MB
FAILED: [[SimpleTest]]: [MySQL] 28,655 pass(es), 92 fail(s), and 95 exception(es).
[ View ]

This patch is still large, but it applies and I believe is correct. I don't know why it's so large.

In sandbox...

git checkout --track origin/8.x -b sandbox_8.x # New branch to rebase on
git fetch drupal # the actual commits from core
git rebase drupal/8.x
# Fix the js merge conflict in common.test again since I lost that commit
git rebase --continue
git push origin sandbox_8.x

We'll either need to delete the sandbox_8.x and recreate it each time we rebase, or create new branches for every rebase.

This will be pretty painful if it takes months. Since we touch so many files (all of them) merge conflicts happen awfully easily.

Status:Needs review» Needs work

The last submitted patch, drupal.core_move_22336_192.patch, failed testing.

rfay’s picture

Status:Needs work» Needs review

Yay, we got a test result through!

In addition to the test #fails, it seems that simpletest is not deleting its tables as it goes. We got up to 2500 tables in the drupaltestbotmysql database before this was over. I had increased the available memory for the mysql databases. It had in fact hit the limit in earlier tests.

BTW, Fabianx suggests that doing a git merge will be every bit as effective as the rebase, and still allow pushing the branch without renaming it. He's right.

rfay’s picture

Status:Needs review» Needs work

Needs work, of course.

rfay’s picture

Status:Needs work» Needs review
StatusFileSize
new7.02 MB
FAILED: [[SimpleTest]]: [MySQL] 31,393 pass(es), 31 fail(s), and 9 exception(es).
[ View ]

Several of the failures were due to the lack of core/ on the front of includes/language.inc.

tstoeckler’s picture

Sorry if this is a stupid question, but did you forget the "-M" flag when rolling the patch? The earlier patches were actually renaming the files while the newer patches are deleting the old ones (thus, printing their entire content) and adding new ones (thus, printing their entire content again).

Status:Needs review» Needs work

The last submitted patch, drupal.core_move_22336_197.patch, failed testing.

rfay’s picture

Status:Needs work» Needs review
StatusFileSize
new7.12 MB
FAILED: [[SimpleTest]]: [MySQL] 31,398 pass(es), 29 fail(s), and 8 exception(es).
[ View ]

A couple of more low-hanging fruit. There are some items in here that are harder, but these are a couple of easy ones. Commit f1fd0963d844ed1e39d33ada8b440f54e748ed9a from the sandbox.

@tstoeckler, yes, git diff -C -M. If you look at the patch I think you'll see lots of renames... just not as many as we wished.

Everybody, of course, can check out the sandbox and see if they can wrangle the patch creation better.

git clone git://git.drupal.org/sandbox/quicksketch/1147512.git --branch sandbox_8.x
git remote add core git://git.drupal.org/project/drupal.git
git fetch core
git diff -C -M core/8.x

Status:Needs review» Needs work

The last submitted patch, drupal.core_move_22336_200.patch, failed testing.

rfay’s picture

Status:Needs work» Needs review
StatusFileSize
new7.14 MB
FAILED: [[SimpleTest]]: [MySQL] 31,418 pass(es), 12 fail(s), and 5 exception(es).
[ View ]

This version resolves the xmlrpc tests and also allows drupal_system_listing() to peer into the profiles directory, which inexplicably is not in core/profiles. Is that OK? It sure seems like there should be a core/profiles directory and also a sites/all/profiles.

Sandbox commit 5605366425cad75abfc2c49687c07aa77c8eb96c

I believe that only the https tests will still be broken after this one. Can't test those until I set up https on my (nginx) local :-)

Status:Needs review» Needs work

The last submitted patch, drupal.core_move_22336_202.patch, failed testing.

rfay’s picture

Status:Needs work» Needs review
StatusFileSize
new7.18 MB
PASSED: [[SimpleTest]]: [MySQL] 31,434 pass(es).
[ View ]

This one fixes up the https url used in the session testing. It finds its way directly into the test directory, so needs core.

Commit 7aa7403d33275c253cfa83a27390417d7e27241f

rfay’s picture

Well YAY! Clean tests.

I encourage you all to move quickly on this, as it's quite difficult to maintain. Go lobby those maintainers quick!

One thing I noticed (since I run nginx on my local) is that the built-in assumption you can use /install.php and friends is invalid, since nginx doesn't respect .htaccess. Of course, we've always looked the other way with configurations that didn't use .htaccess, so perhaps this is nothing new.

And restating: Do we really want the core profiles directory in /profiles?

donSchoe’s picture

  • Applied cleanly, except 2 warnings:
    warning: squelched 17 whitespace errors
    warning: 22 lines add whitespace errors.
  • Fresh installation: Site is running. (Didn't check the update process though.)

Good work!

tstoeckler’s picture

StatusFileSize
new356.58 KB
FAILED: [[SimpleTest]]: [MySQL] Drupal installation failed.
[ View ]

I tried to split the above patch in 2 commits. One purely for renaming and one for all other changes. This gets the patch down to about 300KB, and I'm crossing my fingers that it passes. If not disregard this patch in favour of #204.

Status:Needs review» Needs work
Issue tags:-Needs usability review, -DX (Developer Experience), -DrupalWTF

The last submitted patch, move.patch, failed testing.

donSchoe’s picture

Status:Needs work» Needs review
Issue tags:+Needs usability review, +DX (Developer Experience), +DrupalWTF

#204: drupal.core_move_22336_204.patch queued for re-testing.

Crell’s picture

This strikes me as an issue that should not be done via patches at all, but via a git sandbox and just rebased/merged when we're ready. The patch workflow is just not suited to this sort of change.

rfay’s picture

@Crell... It's been in a sandbox the whole time. But we needed to get it tested.

dww’s picture

Then I'd vote for getting the test bot working nicely on sandbox clones of core... Doesn't that already Just Work(tm)? If not, is/are there an(y) issue(s) about why not? We'll need that more and more, especially with D8 initiative sandboxes, so we might as well invest the effort in getting that part working, instead of rerolling this monster patch...

Furthermore, as much as I'm excited to see this move forward, this is exactly the kind of patch that's going to make life miserable for #1050616: Figure out backport workflow from Drupal 8 to Drupal 7, and there's still a *lot* of D7 fixing to do. I think I'd be in favor of postponing a merge from this sandbox into core until the D7 major + critical counts are a lot more under control.

Thanks,
-Derek

boombatower’s picture

It should work fine with existing pifr and testbots, but you will need to trigger it manually since the d.o integration doesn't support a sandbox that replaces core (although qa.d.o and testbots do). d.o only sends them as contrib modules.

Take one setup (I can do it if we want) and then we simply press the requeue button on qa.d.o.

rfay’s picture

Nope, there are at least two major problems

1. All testing is based on *releases*.
2. All testing assumes that *drupal* is the basis.

So both of those will need to be solved. IMO testing should be based on tags or other committish, but it's currently based on releases.

dww’s picture

Instead of bogging down this issue with the details, can the testbot team create the appropriate issues for these subtasks (if they don't already exist), perhaps standardize on an issue tag (if needed), link to those from here, and then we can safely leave this issue for discussion of the file restructuring itself?

Thanks!
-Derek

boombatower’s picture

@rfay: pift makes those assumptions (as I noted), pifr does not. You can override the core repository as Examiner and a few other companies using it do. Although after thinking about it I realize that you would need to alter a variable on the workers.

boombatower’s picture

To test a core fork using the current PIFR/PIFT requires among other things:
- alteration of a variable on the workers
- manual setup of the job via an XML-RPC call

Given that we only want to run this fork on a single issue, we would need to set up an alternate qa.d.o to run only this job to. Otherwise, if we changed the existing workers for qa.d.o all patches would use the forked repository.

This use case was incorporated in the design of ReviewDriven and is therefore supported out-of-the-box with ease. On behalf of ReviewDriven, we would like to offer use of the service for this issue. We will make available an account for two weeks to test this issue daily (or on demand). Those interested in this issue, please comment whether you would like to utilize this offer.

drifter’s picture

@boombatower: awesome, please do so. Thanks ReviewDriven!

yoroy’s picture

Is this still as per #74 about making the top level dir structure like this:

index.php
.htaccess
robots.txt
core
sites

?

To the extent that this is a usability issue: please go ahead. It was pretty scary figuring out where to put what when I started using Drupal. Having only two folders 'core' (sounds like a don't touch, great) and 'sites' (yeah, I want to make sites, cool!) would certainly help.

Oh, but where will readme.txt live now? Without reading all 220 comments here I'd expect readme.txt to live in root.

jstoller’s picture

+1 for including a README.txt file in the root directory, but only one such file. As I believe someone already suggested, it might be nice to move all the other text files currently in root to a new "docs" directory. The main README file could then refer users to specific docs as appropriate.

I'd also recommend adding a file at sites/README--about_sites.txt. This, more than anything else, would help newcomers to Drupal in getting their sites set up. This readme file could explain exactly what goes where in the sites directory, including for multi-site installations. It would duplicate much of the documentation in sites/default/default.settings.php, but that's a good thing. Newbies don't know to look in sites/default/default.settings.php for instructions. I sure didn't.

quicksketch’s picture

WOW, great work rfay! I didn't think I'd see this moving anywhere for quite some time. I'd really just about given up on it. I also would really love to see this committed sooner rather than later. The number of patches in the queue only goes up, not down. Though as I've heard, we've got a long ways to go towards maintaining the "no new features when there are more than 10 criticals or 100 majors". The number of blockers on this issue is ridiculous and the more compromises around testing people are willing to make the better. This has already been a herculean effort by the testing team to get this as far as it is currently (just passing the tests through patch form). As long as our patches are properly formatted with file moves instead of file delete/add, we're in good shape.

As for the README.txt file, please look at the actual sandbox: http://drupalcode.org/sandbox/quicksketch/1147512.git/tree/refs/heads/sa.... REAME.txt is still in the root, all other .txt files are in the "core" subdirectory.

Making the "docs" directory is a good idea but it's a followup. Same goes for moving the "profiles" directory. Please see #121 for a list of followup tasks.

quicksketch’s picture

StatusFileSize
new961.91 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch drupal.core_move_22336_223.patch.
[ View ]

I've merged all of @rfay's patches back into the 8.x branch in the sandbox (that's http://drupalcode.org/sandbox/quicksketch/1147512.git/tree/refs/heads/8.x). Let's continue using the 8.x branch for our work.

This patch is slowly getting bigger because every time we make compensating patches to fix conflicts it's another commit bundled into the patch. However with this reroll, all files are renamed properly and the patch size is a much more reasonable 900KB instead of 8MB.

Status:Needs review» Needs work

The last submitted patch, drupal.core_move_22336_223.patch, failed testing.

quicksketch’s picture

Status:Needs work» Needs review

For future reference, do NOT roll patches with git diff -C -M drupal/8.x 8.x. This does not handle file moves properly. Use this instead:

git format-patch -C -M origin/8.x --stdout > patch.patch
quicksketch’s picture

StatusFileSize
new481.74 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch drupal.core_move_22336_227.patch.
[ View ]

Okay, the problem with that approach is that it makes Git attempt to re-apply changes that have been fixed in later commits, thus causing conflicts when you try to apply them all at once. Here's the approach I used to fix this problem by making a new sandbox branch:

git checkout origin/8.x
git branch sandbox2_8.x
git checkout sandbox2_8.x
git am 1-core-move-files.patch # This is the first commit pulled manually from #224's patch.
git diff sandbox2_8.x sandbox/8.x > 2-core-update-files.patch
git apply 2-core-update-files.patch
git commit -a -m "Issue #22336 by quicksketch, scor, boombatower, and rfay. Move all core Drupal files under a core subdirectory."
git push sandbox2_8.x
git format-patch -C -M origin/8.x --stdout > patch.patch

It's not real pretty but I think this will work. If this patch passes, I wouldn't mind deleting the existing sandbox 8.x branch and replacing it with this new cleaner branch for ease of maintenance.

tstoeckler’s picture

Re #221, #222: As you can see in the patch README.txt is left in the root directory, all other text files move to core/. Moving those into a separate docs/ directory is for another issue, or a follow-up at least.

yoroy’s picture

Yes, thank you. It was not obvious for me to know where to look for what is being changed here. Please do continue :)

quicksketch’s picture

Status:Needs review» Reviewed & tested by the community

With all tests passing, my manual testing not showing any problems, and another review of the patch looking good also, I'm going to mark this RTBC.

Note that I'm not necessarily advocating this patch be immediately committed (although I'd personally be happy with that). We're still working on getting the Drupal core major/critical queues down lower, but I'll defer to maintainers to weigh in on the situation. (Though I will say that right now is an ideal time, with 7.4 just released and hovering right around the new official cap).

This patch does nothing beyond A: Moving most of the root files into a "core" directory and B: Updating files so that the new location works. There is also a side-effect change to the way folders are scanned for new themes/modules (see #124 for the explanation).

Please see #121 for a summary of what this issue does not cover.

catch’s picture

On commit timing, it would be nice to see this queue a bit shorter as well as any thresholds before we break every patch in it:

http://drupal.org/project/issues/search/drupal?status%5B%5D=14&version%5...

RobLoach’s picture

Hmmm......

/var/www/drupal/8 $ git apply drupal.core_move_22336_227.patch
error: patch failed: core/modules/simpletest/tests/file.test:427
error: core/modules/simpletest/tests/file.test: patch does not apply
catch’s picture

Status:Reviewed & tested by the community» Needs work
Issue tags:+DX (Developer Experience), +DrupalWTF

The last submitted patch, drupal.core_move_22336_227.patch, failed testing.

boombatower’s picture

Yea, the endless cycle of rerolling. Need a firm data point when this can get in or just get on with it, but no sense continually rerolling this with the _hope_ that it gets committed.

webchick’s picture

Well, one thing I definitely do know is that this can't get in before our thresholds are met. So if you want this in faster, start fixing some major/critical bugs/tasks. :)

Bojhan’s picture

I was pointed to this, with big surprise about @webchick's comment. What you are basically saying, any major change cannot happen unless we meet our thresholds. Even criticals, its unlikely we meet the standard anytime soon.

Sorry but enforcing this standard to prevent improvements to core, will halt innovation and most certainly demotivate major contributors from working on new stuff. Please reconsider or at least discuss this standard, because it leaves me in big worry about the future of Drupal 8 (which frankly hasn't moved much since Drupal 7 release).

webchick’s picture

That's exactly what I'm saying, yes. See http://drupal.org/node/1050616 for the full story. The goal is so that we never again end up with a release we can't ship for 18 months due to clean-up requirements.

I'm not exactly sure what you're concerned about though. We're only 3 issues off "unblocking" Drupal 8 (major bugs / tasks are > 100), and several of them are marked RTBC. I anticipate this problem being solved by roughly Tuesday.

Bojhan’s picture

Yes, we are close - which is very nice. But the bad precedent is here that we should focus on the critical/major issues and not on new issues that innovate. By saying, guys stop even thinking about getting this in before our thresholds are met - you are also saying stop thinking about it. I know that's not your goal at all, but I have seen this "stall" a number of larger issues.

Only time will tell if we will stay below the threshold long enough, to burst activity on these kind of major tasks. But I will shut up till Thursday and hope, this period of us being above the threshold doesn't occur too often. Because when it does, its very effective in stalling progress on issues that are about innovating/changing something big.

I think its at least a worthwhile consideration, if it does occur more often to have committers feedback or some kind of "can be committed when our thresholds are met" or even just some feedback.

I know its a hard policy to enforce, so I understand that being strict is important.

dww’s picture

@bojhan: Can we discuss the policy at #1050616: Figure out backport workflow from Drupal 8 to Drupal 7 and leave this issue for discussing the file reorganization? ;) Thanks.

quicksketch’s picture

Arg... well set out to reroll this today and after 30 minutes found it's not worth my time. Core has got not only a new conflict but a couple of new files. LMK when this is worth it to actually update. :P

webchick’s picture

Yeah, this issue is "special" because once it's committed, every. single. patch. thereafter gets immediately more difficult to backport. It also immediately breaks all existing patches in the queue, causing weeks or months of additional work doing patch re-rolls. So it's important to be strategic about the timing (assuming Dries is willing to commit it, which he hasn't said yet, afaik).

I'm going to try and bring this up with Dries this week.

quicksketch’s picture

every. single. patch. thereafter gets immediately more difficult to backport.

It's not going to be difficult to backport to D7, it's going to be ignoring 2 directories instead of 1. People applying patches regularly are using -p1 right now anyway to skip the a/b prefixes, just use -p2 and all D8 patches apply to D7 as well as they do today. Re-rolling existing patches means adding "core/" in front of existing patches.

I still maintain that the longer we wait, the MORE painful this becomes, not less.

webchick’s picture

YAY! Now that #1036132: Provide a mechanism for issue summaries is deployed, at 244 replies, this issue could really use one. :)

Could someone please edit the issue body above using the issue summary template? Dries said he can probably take a look at this issue tomorrow night, and this would save a bunch of time.

juan_g’s picture

Possibly scor's #157 (on backports to D7) suggests a reasonable solution for this timeline dilemma. After all, it's a "core/" string replacement.

mdupont’s picture

With Drupal 7.7 just released, and issue count below thresholds, isn't it a good time to commit this patch?

Bojhan’s picture

Status:Needs work» Needs review

Yes, it is a good time - would need a reroll, the summary is already done.

donSchoe’s picture

Status:Needs review» Needs work

I'm maintainging around 10 drupal sites, some of them are running for quite a while now and some of them had quite a few maintainers prior my time. All i can say after "cleaning up" some of them: this patch is a very good and very needed push forward for the maintainability and would make life easier for people who are not so familiar with drupal yet.

+1

lpalgarvio’s picture

wow, outstanding issue!

i agree with the issue summary:
Move all directories other than "sites" and "profiles" into a "core" directory.
Move all files other than .gitignore, .htaccess, index.php, README.txt, robots.txt, and web.config into the "core" directory also.
Adjust code and documentation throughout core to reflect the new locations.

open new issue for this one?:
Moving the "profiles" directory to "core/profiles"

core/profiles - drupal core profiles
sites/all/profiles - contrib sitewide profiles
sites/default/profiles - contrib default site profiles

aaron’s picture

Status:Needs work» Needs review
Issue tags:-DX (Developer Experience), -DrupalWTF

#227: drupal.core_move_22336_227.patch queued for re-testing.

Status:Needs review» Needs work
Issue tags:+DX (Developer Experience), +DrupalWTF

The last submitted patch, drupal.core_move_22336_227.patch, failed testing.

webchick’s picture

Assigned:Unassigned» Dries

Moving to Dries's queue for review.

juan_g’s picture

This was RTBC in #230, but the patch would need update just before committing, given that D8 is now a moving target.

Dries’s picture

I've looked at this and I'm comfortable with the patch. Given that it may break a lot of patches, I'd like to propose that I commit this on November 1st, 2011. Like that, (1) we don't break all patches in the issue queue just before we all go to DrupalCon London and (2) we give ourselves a bit more time to stabilize Drupal 7 with minimal effort.

EvanDonovan’s picture

So I guess that means this patch no longer "needs maintainer review"?

Bojhan’s picture

Status: To be committed

:')

moshe weitzman’s picture

Might make sense to commit this to a branch now and have folks work on larger patches against this branch.

quicksketch’s picture

I'd like to propose that I commit this on November 1st, 2011. Like that, (1) we don't break all patches in the issue queue just before we all go to DrupalCon London and (2) we give ourselves a bit more time to stabilize Drupal 7 with minimal effort.

Well isn't that just a poke in the eye.

I'd have to say I disagree with this proposal, as the number of patches in the queue will probably be at least double what it is now by that point, breaking many more patches than it would currently. If something as simple as a directory move can't get in until November, this puts a serious damper on my enthusiasm for attempting any other serious changes in Drupal core.

rfay’s picture

IMO #254 is a good proposal with a solid commitment, avoiding the need to
* Reroll the patch every 5 minutes
* Wonder if the patch will ever go in
* Delay a few months before we have to do a new D8 vs D7 patch dance (with different patches for each version)

Since all D7 patches will be different from D8 patches from Nov 1 on, it's going to introduce a nasty little process problem that we haven't had until now. We can handle it. And there are times to do something like this.

quicksketch’s picture

Status:Needs work» Postponed

Well might as well mark it what it is. Updated the issue summary.

quicksketch’s picture

Issue summary:View changes

Adding initial summary according to template.

jherencia’s picture

Sub.

jherencia’s picture

Ups, I think there is a bug with revision funcionality because I didn't make a revision, but I think quicksketch did.

rfay’s picture

@jherencia, you definitely created a new revision (thanks), and your information is there in the summary: http://drupal.org/node/22336/revisions

Yay for issue summaries!

dww’s picture

No, there's a bug. See #1217286: Posting a comment changes too many issue node revision properties. But please let's not discuss that further here. ;)

harryhirsch’s picture

very good thing

Shellingfox’s picture

+1

webchick’s picture

Assigned:Dries» Unassigned

Since Dries has chimed in on this, removing the assigned to value.

pillarsdotnet’s picture

Event posted: Move all core Drupal files into /core folder.

Also proposed:

This announcement has set a trend. The (unwritten as yet) policy appears to be that patches of this size (that affect everything in a file and would make many patches need rerolls) will be applied at pre-scheduled times. I propose that such schedules be announced on the event calendar such as I have done above.

moshe weitzman’s picture

According Stack Overflow, git handles merges even after there have been file renames. I'm not sure that we will really break pending patches. I wonder if it matters that we are patching with git apply versus git merge. If merge survives better, we might want to start using sandboxes more ... Not sure if this is best discussed elsewhere.

boombatower’s picture

Well if you use a branch for developing patches (and or create one for reroll) you can simply git pull --rebase and it deals with a heck of lot like renames.

cweagans’s picture

Coming here from #1109202: [meta] Fix coding standards violations across Drupal core. It might be good to commit the patch at #1109202-59: [meta] Fix coding standards violations across Drupal core immediately before committing this one.

(This one is relatively easy to reroll. The other one at #1109202: [meta] Fix coding standards violations across Drupal core would be a fairly significant amount of work to reroll, since Coder + Grammar parser would need to be updated and whatnot)

catch’s picture

I'd expect the other patch to be re-rollable just by find and replacing file names? i.e. s/includes\//core\/includes\//g (and the same for other directories). The issue with this patch is not that re-rolls will be hard so much, just that everything will need one.

If that's not the case for some reason we can try to co-ordinate it going in just before, but I'd want to makes sure it goes in before anyone tries to re-roll this one otherwise it messes up the scheduling of this - i.e. we need to make sure quicksketch is actually around on November 1st etc.

quicksketch’s picture

It might be good to commit the patch at #1109202-59: Drupal coding standards fixes immediately before committing this one.

(This one is relatively easy to reroll. The other one at #1109202: Drupal coding standards fixes would be a fairly significant amount of work to reroll, since Coder + Grammar parser would need to be updated and whatnot)

Let's please not commit any major changes "immediately" before committing this patch. Rerolling this patch *is* a serious pain. See comment #227 for the process you have to go through. :P

All the hubbub of "patches will break and will no longer be trivial" is blowing it out of proportion. All patches will still work fine if you just apply them to the "core" directory instead of the root, or if you do a simple find/replace on the existing patch paths. It's true patches will need a reroll, but I bet one person could reroll every single RTBC patch in less than 3 hours, by hand, with no scripts.

catch’s picture

OK just spoke to quicksketch in irc. So that this can happen smoothly on Nov 1st, I'm going to make an effort not to commit any patches at all on October 31st (or Nov 1st before this one).

I'm going to be away from next week until the 22nd October (not completely offline but not usual schedule), but will also try to keep the RTBC queue as empty as possible prior to October 31st - see how that goes in practice (I'm also trying to not commit any non-trivial patches quicker than around 5 days to give people time to review them who hadn't noticed them before they were RTBC, so bear in mind that empty means "empty of patches older than 5 days in the queue or assigned to Dries" rather than 0).

Crell’s picture

catch: Good to know that you have an RTBC "grace period". That makes sense, but it should be communicated more widely so people who need to know about it know about it.

RobLoach’s picture

Sounds like a good opportunity to start working off of forked branches rather than patch files :-) . quicksketch's sandbox for it might be a good place to keep this up to date.

pillarsdotnet’s picture

michaelfavia’s picture

While everything is being renamed has anyone given consideration to #7269: Add .php extension to PHP files? It would offer a little extra security, and dx in things like IDE's, greps, etc. I'd be happy to do it if there is significant buy in.

quicksketch’s picture

@michaelfavia Please see the issue summary at the top of the issue.

This issue has been going on for 6 years, please take into consideration the compromise that has already taken place. The scope of this issue is clear. These exact things have been agreed upon as what we're handling in this issue: [snip]

I also started a reroll of this issue and should be able to put it up for testbot to verify in the next day or so.

michaelfavia’s picture

@quicksketch: i did read the summary and didnt see #7269: Add .php extension to PHP files, itself 7 years old, mentioned.

I also didnt mean to add to the scope of the issue. To be honest as id love to see each go through.

I was just commenting that if one was happening it seemed like a good time to do both so you don't break all patches twice. ;)

Didn't mean to muddy the waters here though. If there is commitment/interest from catch, webchick or dries id be happy to devote some time to it to take place in the same time frame.

catch’s picture

Just to reiterate I'll stop committing any other patches some time on Saturday until this is in. That gives Sunday and Monday clear for re-rolling although I've been away and haven't properly caught up with Dries and webchick yet.

webchick’s picture

That's fine with me. And as far as I know, Dries is in India that week so probably no harm from that corner. :) I'll let him know, though.

catch’s picture

Status:Postponed» Needs work

We're not quite in pre-commit code freeze of this, but I'm moving this out of the postponed queue so anyone who's been under a rock since May 8th 2005 has a bit more chance to see it.

catch’s picture

Not sure whether Dries will get to these two in the next day or so, but otherwise I'm done with the RTBC queue this week (unless I managed to break 8.x in the past hour or something).

http://drupal.org/project/issues/search/drupal?status%5B%5D=14&version%5...

webchick’s picture

Ditto.

quicksketch’s picture

Reroll on Sunday is #1 priority for me. I'll post my patch asap.

quicksketch’s picture

Status:Needs work» Needs review
StatusFileSize
new659.75 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch drupal.core_move_22336_288.patch. This may be a -p0 (old style) patch, which is no longer supported by the testbots.
[ View ]

There were 5 or 6 new files added to Drupal core since the last reroll, which meant starting fresh again with 8.x, applying the first half of the last patch, moving the new files, make a new commit, then apply the second half of the last patch. Fortunately no code conflicts this time, so it wasn't too bad a reroll. Let's see what testbot thinks.

Status:Needs review» Needs work

The last submitted patch, drupal.core_move_22336_288.patch, failed testing.

quicksketch’s picture

Looks like that patch mismanaged the newer "modules/locale/locale-rtl.css" and "core/modules/node/tests/node_access_test.install" files. Working on another reroll.

quicksketch’s picture

Status:Needs work» Needs review
StatusFileSize
new494.74 KB
PASSED: [[SimpleTest]]: [MySQL] 33,759 pass(es).
[ View ]

I completely underestimated the amount of work it would take to get this going again. My last patch not only moved a few files wrong, it wasn't taking into account all the removed modules, Garland, and CSS files that have happened in the last 3 months. And of course there were new directories that were added in the Symfony patches that needed to be moved and new paths that needed updating.

I think I got them all, but I'd like to see test bot at least be able to apply the patch.

quicksketch’s picture

StatusFileSize
new494.25 KB
PASSED: [[SimpleTest]]: [MySQL] 33,762 pass(es).
[ View ]

Last patch I got lazy and just added another commit (for a total of 3) on top of the last changes. This patch rerolls the same changes but in 2 commits.

rfay’s picture

And congratulations!

sun’s picture

Status:Needs review» Needs work

Excellent work, all!

Looks good, except of:

+++ b/core/includes/common.inc
@@ -5133,16 +5134,17 @@ function drupal_cron_cleanup() {
function drupal_system_listing($mask, $directory, $key = 'name', $min_depth = 1) {
...
-  $profile = drupal_get_profile();
...
-  $searchdir = array($directory);
...
+  $searchdir = array('core/' . $directory);
...
+  $searchdir[] = 'profiles';  // We may be searching right in the profiles dir.
+  $profile = drupal_get_profile();
   if (file_exists("profiles/$profile/$directory")) {
     $searchdir[] = "profiles/$profile/$directory";
   }

The addition to search for $mask in 'profiles' itself looks like a sweeping and possibly unintended change to me.

If this is indeed intended, then we need 1) a better comment explaining the reason, and 2) a properly formatted comment (not on the same line).

quicksketch’s picture

The addition to search for $mask in 'profiles' itself looks like a sweeping and possibly unintended change to me.

A good catch. Looks like rfay added it in #202 with this comment:

This version resolves the xmlrpc tests and also allows drupal_system_listing() to peer into the profiles directory, which inexplicably is not in core/profiles. Is that OK? It sure seems like there should be a core/profiles directory and also a sites/all/profiles.

I think you're right though, this change is not within the scope of this issue and isn't necessary. I recapped why the profiles directory isn't in "core" directory (or elsewhere) in #121.

quicksketch’s picture

Status:Needs work» Needs review
StatusFileSize
new494.4 KB
FAILED: [[SimpleTest]]: [MySQL] 33,759 pass(es), 1 fail(s), and 0 exception(es).
[ View ]

Reroll with the offending line removed: +  $searchdir[] = 'profiles';  // We may be searching right in the profiles dir.. It's not needed for this patch and really should have its own issue. I also updated the comment on the next line there to clarify the "/modules" directory as "/core/modules".

Status:Needs review» Needs work

The last submitted patch, drupal.core_move_22336_297.patch, failed testing.

quicksketch’s picture

Status:Needs work» Needs review
StatusFileSize
new495.85 KB
PASSED: [[SimpleTest]]: [MySQL] 33,771 pass(es).
[ View ]

Well it looks as though it's inevitable that we'll hit the dreaded 300-comment page limit here. :(

After conversing with chx and catch in IRC, we reached the conclusion that the most logical thing to do here is to comment out the test. The test is an invalid use-case because profiles cannot currently exist in multiple locations, so drupal_get_filename() being used on profiles is unnecessary. There are already two other issues tracking problems with profiles:

#562042: Search for install profiles in sites/[all|site]/profiles folders, and move core profiles into /core/profiles
#1006714: drupal_get_path doesn't work for profiles

The second of these I've preemptively bumped to critical, due to our discussion of how we should handle broad system-wide changes at #1272266: Strategies for far-reaching core patches. Because I've commented out a test here, that issue gets a bump in priority until the test is restored. They're already working out many of the problems with drupal_get_filename when dealing with an inactive database or with profiles there, that discussion should continue but with a higher priority now.

This patch adds a third commit specifically commenting out the test. I think it makes sense to keep it separate from the other changes.

rfay’s picture

RE: #296 I apparently added that change, and judging from the comment in #202 was insecure about it. I'm fine with removing it as long as we're sure that drupal_system_listing() is behaving as we want it to be.

sun’s picture

Assigned:Unassigned» catch
Priority:Major» Critical
Status:Needs review» Reviewed & tested by the community

Thank you!

A couple of other patches in the queue are on hold for this one now. Thus, assigning to @catch and bumping to critical.

Let's get this in. :)

webchick’s picture

Weird, my comment earlier apparently got eaten.

Dries is planning to hold off on any more commits tonight, and is on his way to Singapore/India tomorrow. That, coupled with the fact that this might very well require some real-time coordination in IRC to get 100% correct confirms that yep, catch, this is all yours! :)

catch’s picture

Status:Reviewed & tested by the community» Fixed

OK, committed and pushed to 8.x (more than once since I forgot to use git am the first time and had to revert, but msonnabaum and chx fixed me up in irc).

webchick’s picture

webchick’s picture

scor’s picture

How to reroll old patches

In order to reroll patches which were created using the old file structure before the /core commit:
- make sure you're working on the latest version of the 8.x branch: git pull
- go into the core directory: cd core
- apply the patch: patch -p1 < old_patch_file.patch
- make sure it applied properly. See what to do if it doesn't below.
- reroll the patch into a new file: git diff > new_patch_file.patch

If the old patch does not apply, the patch command will tell you one of these messages:
- "can't find file to patch...": something is wrong with the file names path in the old patch file. The patch might have been rolled with the wrong command, or it might try to patch files which were not moved inside the new /core directory, such as profiles/, sites/, index.php, README.txt, robots.txt, or web.config. You can inspect the patch file with dreditor to see what files it affacts.
- "1 out of 3 hunks FAILED": the files to be patched have been found, but something went wrong when trying to make the changes. This often happens on old patches when the code surrounding where the changes are meant to happen has changed. Check the output of the patch command to see which hunk failed, so you can make the changes in the file manually.

(hopefully there will still be some patches left to reroll tonight for the "Boston Initiative" event).

webchick’s picture

Say, this was never announced at http://groups.drupal.org/core where we try and announce big major changes to core that affect lots of people.

Would someone in MAINTAINERS.txt involved in this issue please write up a short thing, preferably pointing to xjm's documents on how to re-roll patches for it?

pillarsdotnet’s picture

@webchick -- yes, it was: http://groups.drupal.org/node/179649

webchick’s picture

Yes, I know the "coming soon" was announced, but not "the aftermath" :)

catch’s picture

Assigned:catch» Unassigned

Pages