Rewrite 8.x-1.x

Commerce 2.x has been relying on composer_manager for the past few months, and in retrospective, it has been a very painful experience.
Our installation instructions (steps 3/4/5) tell the complete story:

1. Install the latest -dev version of Drupal 8.
2. Install the latest -dev versions of devel* and composer_manager.
3. Enable Commerce (but not any of the submodules!). This will add the Commerce libraries to the global composer.json
4. Go to your sites/default/files/composer directory and run composer install.
    This will download the required libraries into your sites/all/vendor directory.
5. Enable the Commerce submodules. Use the admin/modules page, Drush won't work currently**.

At DrupalCon I've spoken to DamZ and EclipseGC and we've tried to enumerate the problems we've noticed, and come up with solutions.

The problems

The first set of problems lies around the fact that the composer.json regeneration only happens on module enable:

1. composer_manager can't work as a part of the build process (running after drush make, for example)
2. Any code relying on a library must live in a submodule.
   The main module can't use a library interface as a part of any classes that must run during Drupal boostrap,
   For example, a library defines a CurrencyInterface, defining currency behavior. A drupal entity type class implements that interface.
   This entity type class must be in a submodule, so that the main module (commerce.module in our case) can ensure the regeneration of the composer.json file and the presence of the library before the submodule is installed. Otherwise Drupal will rebuild the entity types during bootstrap, and the missing library will crash the system.
3. Related modules can't be installed all at once (same reason as explained in #2).

The second set of problems lies around the placement of the global composer.json file and vendor/ directory.
Our site ends up with two different vendor directories, one in core/vendor and the second in sites/all/vendor.
Composer wasn't built to be used like this, a site/application must always have one vendor directory.
This triggers a set of (admittedly, solvable) Drupal bugs, where the autoloader isn't swapped early enough, causing Drush incompatibilities and other crashes.

The solutions

The composer.json generation is already neatly contained in its own service and related classes.
The solution to the first set of problems is to untie the service from ModuleHandler by crawling the module directories ourselves.
This would allow the process to be triggered without an active Drupal installation (via drush or by invoking a script manually).

Once we have the new composer.json file, we overwrite core's composer.json file. Thus, the module libraries end up in core/vendor.

The new flow becomes:

1. Get core & modules by running drush make.
2. Regenerate core's composer.json by running drush composer-manager regenerate (or whatever the command name ends up being).
3. Run composer install inside core/vendor.

Installing a new module now requires only going to core/vendor and running composer install again.

EclipseGC suggested implementing a Composer hook so that the composer.json regeneration happens automatically when "composer install" runs.
DamZ has looked into this, Composer has no such hook/event right now, but it can be added.
This would further simplify the process, the separate command wouldn't be needed and Composer Manager wouldn't need to care about module install at all.

Side effects

Libraries will be installed for uninstalled modules, but I'm prepared to live with that (by advising people to delete the modules they don't plan on using).

Each Drupal core update requires regenerating the global composer.json file and rerunning composer install.
This will get easier once Drupal core stops shipping with vendor/