Problem/Motivation

We have added versioned dependencies to Drupal 7. It relies on the info file having a version. If you use git clone then the version information is simply not there. Also, dev tarballs only have the branch information (7.x-3.x-dev) but not the latest tag (7.x-3.5). You cannot tell if Rules 7.x-2.x fulfills the requirement of Rules ( >= 2.3).

Proposed resolution

Before generating the version = line in the .info file, the packaging script will run git to find the latest tag, and use this to determine the version string. If we are exactly on a tag then we'll use that as the version. If not, then we add a .1-dev suffix. So dev tarballs will contain version = 7.x-3.5.1-dev instead of 7.x-3.x-dev, allowing us to fulfill dependencies like ( > 3.5 ).

To support site builders who use git clones of contributed projects in their sites, we will also add a mechanism to allow core to use a local version of git to determine the version string, as the packaging script does.

Remaining tasks

At this moment everything but check back in two hours ;)

User interface changes

Users will see very little change. They will see 7.x-2.x-dev less times and versioned dependencies will work.

API changes

We do not plan API changes. Unless something relies on

Original report by fago

I try to make use of a versioned dependency for modules relying on the entity API. However, Drupal says the version is incompatible with the version checked out directly via CVS/git, as there is no "version" entry in my .info file. According to [#542202] this is correct. It also says:

Because Drupal core uses a slightly different packaging process than contributed modules, the core modules have a version line predefined. This is an exception to the rule, not the model for contributed modules.

However, adding the version = VERSION line make it accept my CVS/Git version. So, is the documentation wrong and we should add version = VERSION or is this a bug?

Next, I was not able to define a versioned dependency compatible with a d.o dev snapshot containing the version string "version = "7.x-1.x-dev"" at all - I tried stuff like >= 1.x, >=1.x-dev, =7.x-1.x-dev, .. Nothing worked. So what should I declare to have compatibility with >=1.0-beta6 + but also the dev versions? E.g. I'd like (>=1.0-beta6, >=1.x-dev) to work.

Files: 
CommentFileSizeAuthor
#63 #1013302-db_update_impossible.png29.68 KBklonos
#64 1013302-db_update_impossible.png29.68 KBklonos
#44 Selection_010.png40.49 KBchx
#43 1013302_43.patch1.09 KBchx
PASSED: [[SimpleTest]]: [MySQL] 33,448 pass(es).
[ View ]
#42 1013302_42.patch965 byteschx
PASSED: [[SimpleTest]]: [MySQL] 33,450 pass(es).
[ View ]
#41 Selection_009.png20.16 KBchx
#38 1013302_pretty_regexp.patch974 byteschx
PASSED: [[SimpleTest]]: [MySQL] 33,446 pass(es).
[ View ]

Comments

mikey_p’s picture

You should probably be using git_deploy or cvs_deploy, since that information is added by the packaging script for modules on d.o. Those modules fill in the missing information from the git or CVS metadata if you're not using a downloaded package. The version = VERSION is just a method of keyword expansion for core modules, so that they all contain the default version.

fago’s picture

hm, I don't think it is very practicable for developers requiring cvs|git_deploy in order to be able to enable a module at all (due to the versioned dependencies).

fago’s picture

Version:7.x-dev» 8.x-dev
Priority:Normal» Major
Issue tags:+needs backport to D7
pillarsdotnet’s picture

A similar problem occurs when you add a new dependency. People upgrading from an old version will not know to add and enable the newly-required module, and neither update.php nor admin/reports/status provide helpful information as to why the newly-upgraded module crashes.

To resolve both problems I find myself writing elaborate hook_requirements() tests that react to missing dependencies by either enabling the required module, or disabling the dependent module and redirecting to the module selection page with an appropriate drupal_set_message().

This works, but it strikes me as particularly messy.

See for example htmlmail_requirements()

pillarsdotnet’s picture

And using Git Deployment is totally impractical until the following issue is resolved:

#1165694: Remove default nomask from file_scan_directory()

catch’s picture

Title:versioned dependencies and dev versions?» Versioned dependencies fail with git checkouts

Re-titling, this is a regression from D6, in that you used to only need cvs_deploy for update status. However I don't see it being fixed without moving git_deployment or at least some of it into core.

sun’s picture

Priority:Major» Normal

Additionally, cvs_deploy was a contrib module, gitdeploy is a contrib module. This practice was never recommended or officially supported by Drupal core. Thus, there's no way this issue can be major.

catch’s picture

Priority:Normal» Major

Sun, this is about not being able to install modules at all, D6 did not have versioned dependencies so it is a new bug. Update status is optional, installing modules is not.

fago’s picture

As I could really need versioned dependencies now to prevent an entity-API update mess, I had another look at this and figured out that putting manually a version like version = 7.x-1.x-dev in the repo fixes things. Once the packaging system adds a second version line, it looks like the second line wins.

Thus, it looks like that way finally versioned dependencies would work - event with Git checkouts. Any thoughts whether doing so might cause troubles somewhere?

chx’s picture

Can glip read tags? If yes (i think so -- it can read history for sure) then add glip to core and problem solved.

Edit: no time to properly tear glip apart but getTypeID has else if ($name == 'tag') return Git::OBJ_TAG; very promising.

Edit2: I think git_deploy has this already in _git_deploy_version_alter

fago’s picture

Title:Versioned dependencies fail with git checkouts» Versioned dependencies fail with dev versions

grml, I don't know why it seemed to work previously, but now it doesn't. Maybe because of different PHP versions? (5.2 vs 5.3)

unmet dependency:
> Entity API (>=7.x-1.0-beta9) (incompatible with version 7.x-1.x-dev)

That would mean users that want to upgrade entity api to dev version, would have to update all modules depending on it also to dev versions, where the dependency could be removed after rolling release? Not really an option.

That way, dependencies are unusable :(

chx’s picture

No, git is the unusable part because it does not have a hook firing post clone (guess what, bzr has). We can and will fix this with glip.

fago’s picture

That's the one problem, the other one is that #11 appears even with packaged dev-snapshot releases.

jbrown’s picture

subscribing

chx’s picture

packaged dev should die a fiery death.

catch’s picture

iirc there was a conscious decision to not support versioned dependencies for dev releases - in that it'd imply some kind of support for running dev releases.

But stopping git checkouts from being installed at all without a contrib module looks more of an oversight to me and we should move that part of git_deploy to core.

pillarsdotnet’s picture

fago’s picture

>iirc there was a conscious decision to not support versioned dependencies for dev releases - in that it'd imply some kind of support for running dev releases.

This is not about supporting dev releases, it is about being able to use them. If one cannot use them, #15 would be the more honest solution.

keichee’s picture

theres friction here right? if this is enabled or fixed, that friction to develop will disappear :)

some thoughts:
some hardcore developers use git to checkout modules for testing, its reasonable to support them in the core, not to wait until they finished a packaged release.

some motivation to those hard core devs/ module devs that porting low version to 8.x-dev, because they are appreciated and recognized by the core!

catch’s picture

Title:Versioned dependencies fail with dev versions» Versioned dependencies fail with dev versions and git clones
mrfelton’s picture

This issue is stopping me from being able to install the services module alongside the latest git checkout of ctools. #1219980: services Requires: Chaos tools (>=7.x-1.0-beta1), yet it doesn't work with the latest 7.x-1.x git branch

I echo the sentiments of others here, in that git checkouts need to be supported. Often in a drush make file you will need to lock down versions of modules to git checkouts from specific dates. You can't be restricted to using official releases.

keichee’s picture

thanks for the comment #22, the best way is to work with git versions if your a developer,
if your a troll. then its another story ;))

imho, dont be affected if your strucked :))

Robin van Emden’s picture

Encountered #22's as well, makes that Drush can not "enable" as well, see #1212746-12: Services Menu.

Damien Tournoud’s picture

Migrating git_deploy to core is not going to happen. The module depends on a fork of the Git PHP client (glip), and that's quite a very significant code base.

keichee’s picture

why does one could copy the git pull of drush?

dawehner’s picture

It is really important to support dependencies but on the other side it seems to be that it's not possible to use or anyhow detect this with git checkouts in drupal itself.

What about making this checking optional so people using git can disable it.
With disableing it, i mean make it not validate but still show the information on the modules page.

What do you think about this?

pillarsdotnet’s picture

rfay’s picture

The reality is that having even the major version in the checked-in info file is quite a lot of trouble, as it means that a 7.x and 8.x version of a module immediately diverge, even if there is no change to the code. So we end up maintaining multiple branches for no good reason.

The real answer here is probably to figure out some new way to inject version information (including major version information) into a module. Hmm. I don't know what it is. But I do hate our static versioning in the info file.

Dave Reid’s picture

This one just bit us in Media. We wanted to specify that Media requires File entity 2.x but users of the dev release will now get warnings.

mikey_p’s picture

I fail to see what using a dev release has to do with this issue. If you are using a dev release, you just need to adjust your dependencies to depend on the dev release or higher. I'm pretty sure there are tests for this and that specifying a dev should work.

Dave Reid’s picture

Because doing dependencies = file_entity (2.x) fails dev versions (7.x-2.x-dev) because 2.x-dev is actually 'less' than 2.x.

Dave Reid’s picture

I've even tried (2.x, =2.x-dev) and it also fails.

effulgentsia’s picture

@Dave Reid: I think (2.x, =2.x-dev) fails because when listing multiple dependecies, they are ANDed together, not ORed.

One way to fix this issue would be to introduce an OR syntax. For example: (>=2.0-beta1 or 2.x-dev). Beyond that though, while I think it's reasonable for 2.x-dev to always be considered less than 2.0-alpha1 (although in reality sometimes it's earlier and sometimes it's later, depending on when the person downloaded it, but that's the challenge in a version identifier whose contents changes), I don't understand at all why 2.x-dev fails a (2.x) check. If I want to force someone to a non-dev 2.x release, I can easily write (2.x, !=2.x-dev).

effulgentsia’s picture

By the way, for anyone interested, the way we currently implemented the Media related dependency checks is Media 2.0-unstable2 now depends on file_entity (>1.99) (i.e., all 2.x including 2.x-dev) and Media Gallery 1.0-beta7 now depends on media (<1.99) (i.e., all 1.x, including 1.x-dev, but not 2.x-dev or 2.0-*). Ugly, yes.

chx’s picture

Erm , you want (>= 2.x) or ( < 2.x) and you do NOT want to hijack this issue further with this.

chx’s picture

Issue summary:View changes

Issue summary by chx

chx’s picture

Issue summary:View changes

Expanded on the versions and dev tarballs

webchick’s picture

Issue summary:View changes

Re-wording proposed resolution to be a bit more clear.

webchick’s picture

Issue summary:View changes

Another sentence to clarify the problem being solved here.

webchick’s picture

chx and I spent about an hour discussing this issue in IRC tonight.

The version changes to the packaging script sound interesting, as long as they don't break Update Status, Drush, PIFR, and other tools that use and depend on this information. Being able to tell what version a dev release is between would be useful info, and not just for this issue.

Where I'm not as keen on is putting an exec() call to git inside Drupal core. I don't believe it's a good idea to couple our website software to the arbitrary version control tool that Drupal.org happens to be using at this time (imagine if we'd done this a year ago and coupled D7 to CVS, for example). That's why we created http://drupal.org/project/cvs_deploy and http://drupal.org/project/git_deploy in the first place, to de-couple these things, and support advanced use cases of people who want d.o clones of their projects.

I asked chx for information on why Git Deploy wasn't sufficient for this, and he said because it was "bloated" and that this would be an easy one-line-of-code way to find out the same information. He also said that we basically "beg" for people to clone their stuff from Git because of the "Version Control" tab on project pages and that this is a common thing that will come up, moreso than it did in CVS. The fact that 317 sites out of 550K run http://drupal.org/project/git_deploy I think blows this argument out of the water, however. (Interestingly, 1791 sites are running http://drupal.org/project/cvs_deploy still. But that's still 1791 of 550K.)

So basically, I'm not really convinced, and would prefer energy being dumped into fixing whatever the problems are with Git deploy instead. But I told chx to ask catch if this approach passed his sniff test. If it does, we need to make damned sure that it doesn't break on shared hosts and other places without a git client installed, but who may have Git clones thanks to local development environments.

webchick’s picture

Issue summary:View changes

Change from +1. -dev to -pl.

chx’s picture

Issue summary:View changes

We will use 7.x-3.5.0-dev ChX

chx’s picture

Status:Active» Needs review
StatusFileSize
new974 bytes
PASSED: [[SimpleTest]]: [MySQL] 33,446 pass(es).
[ View ]
catch’s picture

So I agree with webchick it's not nice to have git-specific code hard-coded in common.inc. However when more projects are using versioned dependencies I also think we'll see a lot more people run into this (in fact I think what's happening now is contrib developers just aren't using versioned dependencies because they keep running into this issue and users aren't running git-deploy). Before we had versioned dependencies the only feature you missed out on in core was update status, but not being able to install modules at all is a lot more severe.

However, having that logic directly in common.inc makes me wince. Could we move this to a pluggable class, and ship core with a git implementation of it? That way if we ever do switch version control again we could potentially just add a new implementation and change the default. It also makes that available to other projects (like potentially git_deploy) if they need it.

sdboyer’s picture

generally speaking, i'm +1 on the patch, though with some qualifications. i haven't tested it directly, but i've been over the regex and consulted with chx on the approach.

@chx, the only case we may have missed on the dev release thing is if there a) are no tags in the repo at all or b) there are no tags on the major branch version. in the first case, it seems like this should be fine - it'll just miss on the match and so do nothing to further manipulate the string. in the second case, i think it would still just pick the most recent tag that could be found, which would be on a previous major version branch. so i think these cases are fine.

note also that this approach, being decoupled from the data provided by update status, could also be a bit fouled up if someone pushed a tag but didn't actually tie a release to it yet. we'd get a count back to a tag that doesn't have any real release associated with it yet. in practice, though, i don't think there's actually any problem with that.

git_deploy's inappropriateness for this task have nothing to do with "bloat." it's worthwhile because *not* having git deploy but having some things, even temporarily, cloned down with git can grind your site to a halt, and without particularly good explanatory messages as to why. if the call fails, it fails - not a big deal, beyond the fact that the dependency checking will fail. unless, of course, i'm missing something, as i'm operating under the assumption that the patch here is really only run when we do version checking, not when we do normal update status inquiries. given that it's part of drupal_system_listing(), it seems it might touch both. if that's the case, i'm more inclined to side with @webchick as this really does get into the domain of stuff that git_deploy really ought to take care of.

chx’s picture

StatusFileSize
new20.16 KB

That patch does not work but if I move the code into system_system_info_alter it most certainly does.

chx’s picture

StatusFileSize
new965 bytes
PASSED: [[SimpleTest]]: [MySQL] 33,450 pass(es).
[ View ]

OK, now we see how pretty it is :) do we want to add a new module , enabled by default? I feel like that's an easy way to plug this in. This patch (unlike the one in #38) works. It is in system_system_info_alter but it could be in any other system_info_alter.

chx’s picture

StatusFileSize
new1.09 KB
PASSED: [[SimpleTest]]: [MySQL] 33,448 pass(es).
[ View ]

Cleanup, way more resilient. Changed format to 7.x-3.0-rc1.228-dev (it was 7.x-3.0-rc1-228-dev).

chx’s picture

StatusFileSize
new40.49 KB

Here's another screenshot, this time with the 7.x-1.0-beta1.47-dev format also showing that subdirs won't find the .git in the parent dir. Not a big deal IMO.

webchick’s picture

RE: #39, #37 outlines that we're talking here of 0.005% of active sites who are using clones/checkouts from drupal.org as opposed to tarballs. That number's probably off, but probably not by like a factor of 100. And even if it were, we're still talking < 1% of sites.

If those sites installed Git Deploy module, then this problem goes away, IMO. It's pretty easy to add a pointer to Git Deploy on the Version Control tab if we're concerned about people not finding it. We could even hook into Update Status info and display a big-ass warning on the modules page if we know there are modules which should have a version and do not, in an effort to highlight this problem more for site builders.

So are we sure that this why contrib developers aren't using versioned dependencies? Or is it because dev releases aren't supported ala #30-36 (which is a much, much more common scenario)? Or is it even because of some far less nefarious reason, like "They haven't read the documentation at http://drupal.org/node/542202 to realize they exist."

Adding additional metadata to the packaging script to -dev releases to try and pinpoint which tag/version they're between would solve the main problem here, and would solve it regardless if people are using git clones or tarballs.

chx’s picture

Sure we can do this in a contrib and point, I will talk to halstead about git deploy "lite". I have filed basically the same code against the packaging script under #1314180: Make dev release numbers useful. This would only solve it for dev tarballs, I fail to see what webchick means by "would solve it regardless if people are using git clones or tarballs.".

webchick’s picture

Sorry, I meant that the problem that bites more like 90% of sites is not knowing whether 7.x-2.x is > or < 7.x-2.3. And if we fixed this in the packaging script, then both the tarballs themselves, as well as Git Deploy, would have that information (I think?).

chx’s picture

No, git clone is not affected by the packaging script. I have committed this code into git_deploy 7.x-2.x and I am working with eliza411 (IRL, yay!) to get a handbook page and a link from the version control tab to git deploy and the handbook page up.

eliza411’s picture

When it's time to update docs and the text of the Version control tab, the unpublished handbook page is at http://drupal.org/node/1314752.

The proposed text for Project Git instructions is visible in context on git-dev.drupal.org ... I can make the changes to production when it's needed, but chx can, too.

Right after "Not working for you? See Troubleshooting Git clone." we thought the following sufficed:

Once it works, you need the Git deploy module. See Versioned dependencies and Git for an explanation.

kristiaanvandeneynde’s picture

Has there been any update on this issue?

I'm currently thinking of removing version requirements altogether in a module I'm working on, seeing as I otherwise cannot even enable it...

As catch wrote very nicely in #39:

So I agree with webchick it's not nice to have git-specific code hard-coded in common.inc. However when more projects are using versioned dependencies I also think we'll see a lot more people run into this (in fact I think what's happening now is contrib developers just aren't using versioned dependencies because they keep running into this issue and users aren't running git-deploy). Before we had versioned dependencies the only feature you missed out on in core was update status, but not being able to install modules at all is a lot more severe.

Bolded some bits for truth/emphasis.

chx’s picture

Yes there was massive progress, git deploy is updated once i can get hold of dww we will add the same code to the packing script without patching core this will be fixed.

webchick’s picture

#1314180: Make dev release numbers useful is now fixed. Yay! So can we close this one, or at least reduce the priority?

xjm’s picture

Well, if it works now. :) Let's find out!

chx’s picture

I will work eliza411 to get a new version of the "Version Control" out strongly urging using git deploy. Then we can close this.

xjm’s picture

I confirmed that I can now use a module with a versioned dependency both with a dev build and the latest HEAD from git. It seems to work without git deploy, because it understands the 7.11-dev version string that is there? Am I mistaken?

webchick’s picture

Priority:Major» Normal

Ok, well if all we're missing is some instructions on the version control tab to encourage the use of a module that people already know about, this doesn't get to block D8 features anymore. :) This should probably also be moved entirely to another queue, but I'm not sure which one so I'll leave it here for now.

eliza411’s picture

Status:Needs review» Fixed

I published http://drupal.org/node/1314752 and updated the version control instructions on drupal.org for maintainers and non-maintainers of full projects. I believe this should be closed now. If I'm mistaken, please reopen.

Status:Fixed» Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

clemens.tolboom’s picture

XREF to #1247476: drupal_parse_dependency fails on major version ops (>7.x) and (<=8.x) ... not sure whether some remarks made here fits to it.

mikeryan’s picture

Component:base system» install system
Status:Closed (fixed)» Active

There's still a pain point here - git_deploy doesn't work when doing db updates (see #1558158: wordpress_migrate.info "dependencies[] = migrate (>2.3)" not correctly detecting latest migrate.). In my situation, I have wordpress_migrate dependent on migrate (>2.3), and a git clone of migrate. This is just fine on the modules page, but after adding an update function to wordpress_migrate, update.php won't let me run it: "Migration from WordPress requires this module and version. Currently using Migrate version". The problem is that the system info data is fetched before doing the full bootstrap, so git_deploy's system_info_alter() hook doesn't get called.

Can we get to full bootstrap sooner? Or is it even worth it to accommodate this scenario?

chx’s picture

How is the system info data fetched before the full bootstrap? Does adding function git_deploy_boot() {} help?

mikeryan’s picture

This feels very familiar, I think I've seen this someplace before... Basically, an error causes hooks to get cached prematurely:

drupal_bootstrap(DRUPAL_BOOTSTRAP_SESSION)
drupal_session_initialize()
drupal_anonymous_user()
ip_address()
$ip_address = $_SERVER['REMOTE_ADDR']; // Note that REMOTE_ADDR is not present
_drupal_error_handler()
_drupal_error_handler_real()
_drupal_log_error()
watchdog()
module_implements()
drupal_alter()

The hooks get cached before we get to the drupal_alter('system_info', ...) and git_deploy never gets a peek at it.

No, implementing git_deploy_boot() doesn't get it in there early enough, must be before bootstrap_invoke_all()...

We can fix the narrow issue simply by checking the existence of $_SERVER['REMOTE_ADDR']. A more general solution would be... watchdog() forcing a full bootstrap before invoking its hooks? Otherwise, contrib modules don't get a shot at catching early watchdog messages...

#319844: Watchdog should work, even if the error is too early, before module.inc is loaded appears related, although in that case (which seems to have started with D6) it was erroring out completely.

klonos’s picture

StatusFileSize
new29.68 KB

I'm mostly using dev versions. The main reason being that they include latest patches. Here's the situation today where I simply cannot run db update:

db update impossible

klonos’s picture

StatusFileSize
new29.68 KB

...hmm

chx’s picture

Status:Active» Closed (works as designed)

If you use dev tarballs , they now contain something like this:

; Information added by drupal.org packaging script on 2013-06-27
version = "7.22+68-dev"

If you use git and are bothered by this please take the code from git deploy (or the patch in this issue) , put it into a script and use it as a post-checkout git hook script. The git deploy queue would be a good place for such a script.

klonos’s picture

I use dev versions and still got #63 in D7 dev. Perhaps #1405212-16: Requirements problem (Unresolved dependency, System Version >=7.12 required) is why but I cannot tell since I had to comment out all the offending dependencies[] = entries from the .info files of complaining modules in order to work around not being able to run update.php.

Wim Leers’s picture

Status:Closed (works as designed)» Active

This is indeed still a problem. Unbelievable :(

I also can't believe so few people are apparently using git_deploy. And probably most that do use drush updatedb.

#65 is not an actual solution, the git_deploy module should be able to inject its version information — see #62.

klonos’s picture

Finally! The whole "works as designed" status in this issue really started getting on my nerves.

klonos’s picture

Issue summary:View changes

.1-dev instead. It doesnt matter.

ciss’s picture

I'd just like to present our workaround to this problem, as it may be useful to others. We're currently using git checkouts for all projects (custom and contrib), created from make files, and thus run into dependency warnings all the time.
We've implemented a two-part workaround:

  1. A custom drush make-generate command, that pulls branch, tag and revision info from git and generates extended entries:
    ; Date: 2013-11-10
    ; Contains: views
    projects[views][type] = "module"
    projects[views][version] = "3.7"
    projects[views][real_version] = "3.7+14"
    projects[views][subdir] = "contrib"
    projects[views][download][type] = "git"
    projects[views][download][branch] = "7.x-3.x"
    projects[views][download][revision] = "80d9683"
    projects[views][download][url] = "gitlab:drupal-contrib/views"

    The "version" key is the last tagged revision before "revision" in the current branch. This is the information required by core for dependency checks. "real_version" is the last tag plus the number of commits made since. (On a side note: Naming dev releases in this style would solve a lot of problems, imo.)

  2. An implementation of hook_system_info_alter(), which reads make files to add version info. In cases where a module name doesn't match the project name it tries to determine the project name to retreive the version info. Unfortunately this rather costly operation would get triggered on every request, so we currently toggle it via a Drupal variable.

Of course this is somewhat a crutch, but it can also be a base for further improvements or decisions. If anyone's interested I can get permission to publish the code.

clemens.tolboom’s picture