Formalize a structure for static assets in modules and themes

Problem/Motivation:

Since ~DrupalCon Nashville, there seems to be strong concensus around a reorganization of the source tree to better support composer and automatic updates. Most pertinent to this issue, the reorganization includes moving nearly everything out of the docroot: a "bootloader" script inside the docroot would select and include an "active" copy of the site's source tree that is located above/outside the docroot, allowing an automatic updater facility to operate on another "inactive" copy of the site's source tree and make that copy consistent with a new target version over possibly many cron or batch executions.

However, this poses a problem for non-php files that ship with modules and are expected to be served as static assets over http, especially images or other media that may not be declared in any existing .yml file or included on a page via existing hooks like hook_page_attachments().

To solve this problem, core needs to know which files that came with a module or theme must be made available to be served directly over http. With this knowledge it becomes possible to either copy/symlink them into the docroot as part of module installation, or serve them through php. This issue only seeks formalize and specifically define a module structure and any associated APIs for static assets. Once that's done, serving out those files from modules that have actually moved outside the docroot is best left to another issue.

It's important to note that the automatic updates initiative does not include updating contrib as an initial goal. However, it does include entire sites as a final goal, and to do that the answer to the serving static assets problem must be phased in now for contrib to begin adapting to it.

There's also probably an argument to be made for better DX if all a module's static files are in a predictable, standardized location within the module.

Proposed resolution:

This proposal captures the points raised in a slack conversation in #autoupdates between @pwolanin, @webflo, @mbaynton and @xjm on 7/30/18 with some additional specifics from @mbaynton.

• Just use special directories at the root of modules to inform core which files are to be treated as static, serveable assets. Any files these directories contain would be treated by core that way.
• The directories css and js would be regarded as containers of static files by default, because many modules already follow this convention.
• The directory assets would be regarded as a container of static files by default, as an easy, convention over configuration way for modules to ship images and other files.
• To streamline adaptation of existing modules, additional directories containing only static assets could be declared by the module in its info.yml file

Module authors could expect the following of static asset directories:

• The directories must be made completely free of php files.
• The directories may be relocated by core once the module is installed, so if for some reason
  the module wanted to make a static file public *and* also be able to read that file with php,
  it would need to locate the file using (A new API? A module:// stream wrapper?)
• Within the directories, core would arrange for the files to be served in response to http requests to modules/[module-name]/[directory-name]/[file-name]. Core would not rearrange or alter files or subdirectories relative to each other.

Remaining Tasks:

• Review & revise the proposed resolution.
• Implement API or stream wrapper for static asset I/O access.
• Develop an automated check to see if a module is likely to be not compliant, including verification that default and declared static asset dirs do not contain php, and scan for likely public files outside appropriate directories.

User interface changes:

None.

API changes:

New optional key in module info.yml files called assets_dirs containing an array of strings
listing directories within the module that should be treated as static, serveable asset files.

New API, stream wrapper, or both (TBD) to allow modules access to their own static asset directories after core potentially relocates them.

Data model changes:

None.