Follow-up to #2829375: Composer dependencies might be lower cased (cause fatal error on composer require)

Problem/Motivation

For some reason, composer does not use the repositories defined in modules' composer.json files.
As the superfish library is not supported by packagist, we have to add it's repository in our root composer.json file to avoid the build to crash.

Proposed resolution

Remove the dependency and the repository from the composer.json file.
Add instructions in the README.txt file.

Comments

DuaelFr created an issue. See original summary.

duaelfr’s picture

Status: Active » Needs review
StatusFileSize
new2.24 KB
duaelfr’s picture

Priority: Normal » Major

Rising to Major as it is currently impossible to install superfish with composer.

duaelfr’s picture

geek-merlin’s picture

Title: Remove composer dependency and provide help in the readme instead » Add library to packagist
Status: Needs review » Needs work

Yes, subprojects can not define repositories in composer.

We should instead get the superfish-for-drupal library into packagist or drupal packagist.
This is super easy and i'd happily help the maintainer @mehrpadin if there are questions, just PM me.

geek-merlin’s picture

Note: we might need a note in the readme that a something like this is needed in the root composer.json:

    "extra": {
        "installer-paths": {
            "web/libraries/{$name}": ["type:drupal-library"]
        }
    }

(where web might be docroot in some projects)

Most projects have this, but not all.

duaelfr’s picture

I had a conversation with Mixologic that maintains the drupal packagist facade.
He is working on a workaround to make these things work. (see #2843461: Consider creating package data for repository references in modules)

In the meantime we really need a fix on this composer.json file. Publishing the library on packagist wouldn't be enough right now (have a look at the content of the composer.json file and you should understand why).

geek-merlin’s picture

> Publishing the library on packagist wouldn't be enough right now (have a look at the content of the composer.json file and you should understand why).

TBH, no. Can you elaborate?

duaelfr’s picture

The superfish module is trying to find its library in /libraries/superfish but it defines the library as "drupal/superfish-library" what would create a /libraries/superfish-library directory. That's why it tries to redefine the path via extra.installer-paths what, if it works here, could break all the libraries included via composer by downloading them in /libraries/superfish instead of their own directory.

geek-merlin’s picture

Hmm i thought that is the easy part of it.
* Decide on a good library path name, be it superfish, superfish-library or superfish-for-drupal (i'd suggest the latter, identical to the repo name and different from the original superfish lib - but hey, that's up to the maintainer to decide)
* Declare that name in the library's composer.json file as name
* Tell the module to find its library there (and optionally fallback to the original location for legacy sites)

Imho it's all about adapting to the composer ecosystem in the way it works instead of fighting against that principles (as long as we don't have that library packagist which may or may not happen).

duaelfr’s picture

Status: Needs work » Needs review

I agree, but we need a working version right now, even if it takes more time to decide how to do it properly (via libraries.drupal.org or Drupal facade). Right now, it's almost impossible to use this module from a composer-based project and these projects are becoming the new standard (even Drush plans to drop makefiles for composer). The #2 patch is a short term solution to allow site builders to use the module. As one of them, I'd like to get it commited asap ;)

mehrpadin’s picture

Hey there,

I've just applied the patch, will do the packagist too soon as possible.

https://packagist.org/packages/mehrpadin/superfish

Thank you both!

duaelfr’s picture

Thanks @mehrpadin!
Could we tag a new RC, maybe?

adamps’s picture

OK, I'm new to composer, so sorry I say something stupid. My understanding is that composer is a tool for automatic installation of dependencies. The users expects to run "composer require drupal/superfish", and that's it. Compose should automatically fetch dependencies on initial install, and automatically update them as needed.

The superfish README now has instructions "Add the proper repository to your composer.json file". So there is a manual step, it's not automatic. In the future there might be another manual step to alter the text that was added - e.g. change to version numbers.

I think it is possible to remove the need for manual steps. Here is my suggestion:

  • Register superfish library on packagist - done in #13.
  • Edit superfish module composer.json to add require of superfish library ("mehrpadin/superfish").
  • Edit superfish library composer.json to ensure library installs to the correct directory: add "type": "drupal-library" to the library composer.json (and require "composer/installers").
  • Remove README instruction for manual edit of composer.json.
adamps’s picture

Status: Needs review » Needs work
mehrpadin’s picture

Hi Adam,

You're right, it's just temporary I'll remove it soon, well today hopefully! tag created anyway.

PS: We are very close to 1.0 stable I believe so please do as much testing etc as you can, thanks.

adamps’s picture

Great, thanks. My D8 sites are still early in development, but I can certainly test the composer part when it's ready.

In case it helps you to have something to copy, I think I have just got my module RRSSB working seamlessly in composer - you must take the dev version.

abe’s picture

following this issue.
I submitted related question earlier in https://www.drupal.org/node/2857720#comment-11968844
I am new to composer. I have changed the composer.json file (generated from drupal-project) to contain:
----------
"web/libraries/{$name}": ["type:drupal-library",
"mehrpadin/superfish"],
----------
then I exec'ed
$ composer require mehrpadin/superfish
--> I got no errors, but
The superfish js/css libraries (version ^2.0 ) were installed in vendor/mehrpadin/superfish instead of web/libraries/superfish.
Thanks.

geek-merlin’s picture

@mehpadin: Can you do the 3 klicks? This would really ease things.

If you can use assistance, i'll be glad to help in chat or so.

Publishing to Packagist

Packagist is service used by Composer to manage libraries and dependencies.

Go to https://packagist.org/packages/submit
Paste link of this repository and click Check
After check phase is complete, click on Submit
That's it, the library is available on Packagist

mehrpadin’s picture

Hey everyone,

The library was added back then, but I think I'm gonna have to give up on this whole Composer thing for now and release the 1.0 stable anyway; just for the time being until Drupal finds a way to handle external libraries like the one we have here without any hassle.

The problem used to be this "composer/installers": "^1.0.24"; well, few months ago when I tried the composer.json I thought ok it's working but I must have been mistaken, I probably had the library installed already but thought the Composer installed it etc, when I tested it for real it did not work, I had some errors about the composer/installers version in the composer.json of the Drupal itself etc, I cannot recall much but I think it was something like it couldn't be bumped to the latest version, just in case you don't know you'd need v1.0.24 of the composer/installers in order to use directories other than /vendor, according to Drupal documentation for Composer:

Custom modules and themes paths requires composer/installers package v1.0.24 and up.

Back in the day the default version was 1.0.20 or something like that, so it wasn't working and I couldn't find a way.

When I had the v8.3.1 installed few days ago I realized they have updated the composer/installers to 1.0.24 that was good news until I tested again and although I didn't get that error message any more - well why would I - the "installer-paths" was always ignored and I was unable to put the library anywhere other than /vendor :\

Searching a little bit I came across this page from the Composer website,
https://getcomposer.org/doc/faqs/how-do-i-install-a-package-to-a-custom-...
This is misleading, having ["vendor/package"] there in that example made me think okey maybe it should be "drupal-library" instead of "type:drupal-library", so I tried that and it didn't work either.

Later I came across this new note on that same documentation page:

For instance, if you'd like to place the Dropzone package (which does not have a type of drupal-library) in the same directory as other Drupal libraries, you would first composer require oomphinc/composer-installers-extender, then add the following configuration to your composer.json file:

"extra": {
    "installer-paths": {
        "libraries/{$name}": [
            "type:drupal-library",
            "enyo/dropzone"
        ],
    }
}

Finally, you would composer require enyo/dropzone.

Guess what? that didn't work either, not even for Dropzone. I've also tried the drupal/webprofiler from the Devel module which had the composer-installers-extender as a requirement, and everything went to /vendor.

Searching even more I have found these:
https://www.drupal.org/node/2866988
https://www.drupal.org/node/2866109
https://www.drupal.org/node/2813897#comment-11707897
and also this:
https://www.drupal.org/node/667058

I'm not quite sure when this will be taken care of, but until then I'll give up on this, it appears there are other people giving up too:
https://www.drupal.org/node/2605130#comment-11868488

geek-merlin’s picture

So we just need the library-installer-path:

 "extra": {
        "installer-paths": {
            "web/libraries/{$name}": ["type:drupal-library"]
        }
    }

Even though this does not come from core undtil #2813897: Add composer installer-paths by default is decided, we can still add a README instruction to add it.
Many people have it anyway in their composer.json.

So great that we have the composer package, we only need to update README and composer.json

duaelfr’s picture

Status: Needs work » Needs review
StatusFileSize
new444 bytes

The latest dev version breaks all the things again.

  [RuntimeException]                                                                                                 
  Could not load package drupal/superfish in https://packages.drupal.org/8: [UnexpectedValueException] Could not pa  
  rse version constraint ^1.1.*: Invalid version string "^1.1.*"                                                     

  [UnexpectedValueException]                                                  
  Could not parse version constraint ^1.1.*: Invalid version string "^1.1.*"

The way the drupal packagist is done prevents all previous releases of superfish to be installed because of this wrong syntax. (It seems to only use the composer.json file of the latest dev version)
You are right, you should wait until there is a way to handle these dependencies properly.

geek-merlin’s picture

Title: Add library to packagist » Require library sanely
StatusFileSize
new3.64 KB

@mehrpadin: no need to do all the fight like #21 alone, just post here, that's how drupal works.
@DuelFr: Thanks for pointing that out and rolling a patch.

In fact putting the library on packagist and adding tha dependency is all there is to do.
Otoh we *must not* interfere with site setup by making additional requirements. This can even break a site.

That drupal core ist still discussing library handlin e.g. in #2813897: Add composer installer-paths by default is not a problem, as many (if not most) people use drupal-project which has the library installer path built in.

Patch flying in that
* builds on the previous patch
* removes all superfluous and possibly harmful composer.json content
* updates README

geek-merlin’s picture

Priority: Major » Critical

Ah, now i understand #23: An invalid version constraint prevents composer from installing the dev at all.
Setting critical thus.

mehrpadin’s picture

Hey there,

Thank you both, just applied the patch but I'm not much happy about how so such simple things are handled, D8 had to have all these covered, anyway! hopefully we can release the first stable version now?

geek-merlin’s picture

Status: Needs review » Needs work

Finally i came to test this. Not finally fixed yet.

@mehrpadin: We just need a tiny thing to be changed to work:
* Add "type: drupal-library" to the library json.

Then we should be able to close this!

Lowell’s picture

I'm pretty sure if you have "composer/installers": "^1.0.21" as a requirement and the "drupal-library" configuration in the composer.json, then composer will know to put the library in the /libraries folder. Might be worth a test run.

geek-merlin’s picture

Nope. I tested it. Without the #28 change, the library is recognized as "library", not "drupal-library", and goes into /vendor/...

Lowell’s picture

Yes, exactly. The library's composer.json needs the type: drupal-library configured so that the composer/installers knows what to do with it

Lowell’s picture

My thought is that with composer/installers as a requirement, there are no extra steps needed in the instructions for a composer based installation. It looks like it works with both the drupal-drupal and the drupal-composer based installs

Lowell’s picture

Well, actually I think composer/installers is already in the root composer.json for both methods, so only the type: drupal-library is really needed, just like you stated in #28

mehrpadin’s picture

Hey,

Thanks, just added that?

Lowell’s picture

Awesome, I just tested the latest dev branch and it does get the library as needed. Nice work.

geek-merlin’s picture

OK i see that's on 2.x.
Now we need to
* make a 2.1 release of the library
* require the in the module's composer.json
* test
* open champagne ;-)

mehrpadin’s picture

Great, thank you folks!

Axel, may I ask why we would need a v2.1?

geek-merlin’s picture

> Axel, may I ask why we would need a v2.1?

* the fix is in the library's 2.x dev version
* it is *not* in the 2.0 version that is required by the module

mehrpadin’s picture

Getting senile :\
v2.1 created & I've just also released the 1.0 stable for the module.

geek-merlin’s picture

Status: Needs work » Fixed

Tested a composer update today and it looks like everything works as expected.

Champagne!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.