Problem/Motivation

The conditions in joins in dynamic queries can be and are mostly SQL strings. The conditions can also be a Condition object, like they are in Select queries. The problem is that MongoDB does not support SQL strings.

// MongoDB does not support SQL string conditions.
$connection = \Drupal::database();
$query = $connection->select('node', 'n');
$query->join('inner', 'comment', 'c', '[n].[nid] = [c].[nid]');

Proposed resolution

Remove the use of SQL string conditions in joins in dynamic queries. With an exception for the join and relationship plugins of the views module. There is a followup issue for those changes #3420574: [PP-1] Make the conditions in joins in dynamic queries use Condition objects in the Views module.
Create a new method Drupal\Core\Database\Condition::compare() to compare two fields with each other.

// MongoDB supports conditions as Conditions objects
$connection = \Drupal::database();
$query = $connection->select('node', 'n');
$query->join('left', 'comment', 'c', $connection->joinCondition()->compare('n.nid', 'c.nid'));

Remaining tasks

TBD

User interface changes

TBD

API changes

A new method Drupal\Core\Database\Condition::compare() has been added. To compare two fields with each other.

Data model changes

None

Release notes snippet

TBD

Issue fork drupal-3398773

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

daffie created an issue. See original summary.

daffie’s picture

Status: Active » Needs review
StatusFileSize
new153.86 KB

The patch is for D10.3

needs-review-queue-bot’s picture

Status: Needs review » Needs work
StatusFileSize
new84 bytes

The Needs Review Queue Bot tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".

This does not mean that the patch needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

daffie’s picture

Version: 10.2.x-dev » 11.x-dev
Status: Needs work » Needs review
nitin shrivastava’s picture

@daffie #2 patch applied succesfully on drupal 11.x

daffie’s picture

Hide the patch files.

mondrake’s picture

Big work, @daffie!

I put some comments so far; still trying to grasp the entire concept. I see views is impacted heavily, will need someone knowing it better to review it.

mondrake’s picture

If we are deprecating passing a string as $conditions to Select::addJoin(), I think also ::join(), ::leftJoin() and ::innerJoin() should do the same.

daffie’s picture

If we are deprecating passing a string as $conditions to Select::addJoin(), I think also ::join(), ::leftJoin() and ::innerJoin() should do the same.

The methods ::join(), ::leftJoin() and ::innerJoin() are wrapper methods on top of Select::addJoin(). If you deprecate those on their own, you will get 2 deprecations. One from the wrapper method and one from Select::addJoin().

mondrake’s picture

Status: Needs review » Needs work
  1. How about adding a helper method like this (untested) to the Select class:
      public function joinCondition(array ...$joins): ConditionInterface {
        $condition = $this->connection->condition('AND');
        foreach ($joins as $join) {
          assert(count($join) === 2 || count($join) === 3);
          $condition->compare($join[0], $join[1], $join[2] ?? '=');
        }
        return $condition;
      }
    

    that would IMHO make DX better: instead of

    $query->leftJoin($this->dataTable, 'data', $this->database->condition('AND')->compare("revision.$this->idKey", "data.$this->idKey")->compare("revision.$this->langcodeKey", "data.$this->langcodeKey"));
    

    we would use

    $query->leftJoin($this->dataTable, 'data', $query->joinCondition(
      ["revision.$this->idKey", "data.$this->idKey"],
      ["revision.$this->langcodeKey", "data.$this->langcodeKey"]
    ));
    

    Also, the $operator argument of the compare() method can be made mandatory.

  2. Never mind the field vs column braindump. Still, I find the naming of compare too vague. How about compareFields?
  3. I do not see a problem in getting 2 deprecation messages for #10, and really think we need to deprecate all the related methods.
  4. I think we likely need to split this MR in one adding the feature and one doing the conversions, to make reviewing more palatable. But let's do that later.
daffie’s picture

Status: Needs work » Needs review

@mondrake: I like your idea and I have 2 variants I would like to propose.

The first one:

We create a helper method that would return the Condition object:

public function joinCondition(string $conjunction = 'AND') {
  return $this->connection->condition($conjunction);
}

The query before

$query->leftJoin($this->dataTable, 'data', $this->database->condition('AND')->compare("revision.$this->idKey", "data.$this->idKey")->compare("revision.$this->langcodeKey", "data.$this->langcodeKey"));

The query after

$query->leftJoin($this->dataTable, 'data', 
  $query->joinCondition()
    ->compare("revision.$this->idKey", "data.$this->idKey")
    ->compare("revision.$this->langcodeKey", "data.$this->langcodeKey")
);

The second one:

We allow the join condition is an array of arrays with each inner array having two values that must be equal. When it is not a simple join than the Condition object must be used.

The query before

$query->leftJoin($this->dataTable, 'data', $this->database->condition('AND')->compare("revision.$this->idKey", "data.$this->idKey")->compare("revision.$this->langcodeKey", "data.$this->langcodeKey"));

The query after

$query->leftJoin($this->dataTable, 'data', [ 
  ["revision.$this->idKey", "data.$this->idKey"],
  ["revision.$this->langcodeKey", "data.$this->langcodeKey"]
]);

For 11.2 I personally like compare() more, because it is closer to condition(). But I can live with compareFields().

For 11.3 If you think that is better, then lets do that.

@mondrake: Thank you for the reviewing and helping to get this issue to land.

mondrake’s picture

Hi @daffie

#12.1 - I like the first option! Keeps things open for uncommon join statements.

#12.2 - How about asking in Slack others' pov? So we don't go one way or another and then hit the wall later.

#12.3 - I do think that, but once again how about asking in Slack?

smustgrave’s picture

Is there a recommended way for testing this one?

smustgrave’s picture

Just following up if any recommended way of testing or if this needs any sign off?

poker10’s picture

Adding a note here, that in this Slack thread https://drupal.slack.com/archives/C1BMUQ9U6/p1699266961237289, there were concerns raised about adding so much deprecations (especially for contrib modules). Proposed approach was to either separate deprecations into a follow-up issue, or make the first step without deprecations - just the new functionality. It would also make the reviews here easier.

daffie’s picture

In the MR are the changes for the join and relationship plugins of the views module removed. I have created #3420574: [PP-1] Make the conditions in joins in dynamic queries use Condition objects in the Views module as a followup to make the changes to the views module.

daffie’s picture

The deprecation of the use of string conditions in joins in dynamic queries has been removed from the MR and a followup has been created for the deprecation (#3420576: [PP-2] Deprecate the use of string conditions in joins in dynamic queries).

daffie’s picture

Issue summary: View changes
daffie’s picture

Issue summary: View changes
smustgrave’s picture

So wasn't 100% sure how to best review this one

And of course gitlab is freezing up but one nitpicky thing I saw was

public function compare( could these new functions have a return typehint

As far as using the code I applied the patch for about an hour. Creating views with multiple relationships, created content and media, used layout builder and block layout. Just trying to make sure I touched as much as I could and nothing appeared to break.

I checked the CR and it reads well, the examples were extremely helpful so thank you for that!

For me I'm a +1 for giving this a shot but will leave in review for a bit longer for more eyes.

needs-review-queue-bot’s picture

Status: Needs review » Needs work
StatusFileSize
new90 bytes

The Needs Review Queue Bot tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".

This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

daffie’s picture

Status: Needs work » Needs review

@smustgrave: Thank for the review.

I have added the return type hint to the new method Condition::compare().

smustgrave’s picture

Status: Needs review » Reviewed & tested by the community

Thanks! Think I'm going to mark it so we can have it for 10.3 (not sure the cut off). But also see it's passing all database tests.

needs-review-queue-bot’s picture

Status: Reviewed & tested by the community » Needs work
StatusFileSize
new27.84 KB

The Needs Review Queue Bot tested this issue. It fails the Drupal core commit checks. Therefore, this issue status is now "Needs work".

This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

daffie’s picture

daffie’s picture

Status: Needs work » Reviewed & tested by the community

Rerolled for 11.x. Back to RTBC.

needs-review-queue-bot’s picture

Status: Reviewed & tested by the community » Needs work
StatusFileSize
new90 bytes

The Needs Review Queue Bot tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".

This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

daffie’s picture

Status: Needs work » Reviewed & tested by the community

Rebased and bacj to RTBC.

needs-review-queue-bot’s picture

Status: Reviewed & tested by the community » Needs work
StatusFileSize
new90 bytes

The Needs Review Queue Bot tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".

This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

daffie’s picture

Status: Needs work » Reviewed & tested by the community

Rebased the PR. Back to RTBC.

mondrake’s picture

@daffie couple comments inline

needs-review-queue-bot’s picture

Status: Reviewed & tested by the community » Needs work
StatusFileSize
new90 bytes

The Needs Review Queue Bot tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".

This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

daffie’s picture

Status: Needs work » Reviewed & tested by the community

Rebased and back to RTBC.

needs-review-queue-bot’s picture

Status: Reviewed & tested by the community » Needs work
StatusFileSize
new90 bytes

The Needs Review Queue Bot tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".

This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

daffie’s picture

Status: Needs work » Reviewed & tested by the community

Could I get a review from one of the committers, I have rebased this PR now for the seventh time.

needs-review-queue-bot’s picture

Status: Reviewed & tested by the community » Needs work
StatusFileSize
new26.18 KB

The Needs Review Queue Bot tested this issue. It fails the Drupal core commit checks. Therefore, this issue status is now "Needs work".

This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

dcam made their first commit to this issue’s fork.

dcam’s picture

Status: Needs work » Reviewed & tested by the community

Could I get a review from one of the committers, I have rebased this PR now for the seventh time.

I hear you. I rebased one for the 16th time in 3 months this morning.

Anyway, rebased again.

needs-review-queue-bot’s picture

Status: Reviewed & tested by the community » Needs work
StatusFileSize
new90 bytes

The Needs Review Queue Bot tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".

This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

dcam’s picture

Status: Needs work » Reviewed & tested by the community

Rebased. Restoring RTBC status.

benjifisher’s picture

@daffie:

The two snippets in the issue summary use $query->join('inner', ...) and $query->join('left', ...). I think that is a little odd, but it is all right since they are not explicitly described as equivalent.

But the first before/after example in the change record (CR) has the same examples. There, I think it is a problem: an inner JOIN is different from a left JOIN, isn't it? Or did you do that on purpose for some clever reason that I am missing?

I will not move the issue back to NW for that, but I will add an issue tag.

I have an even smaller suggestion about another example in the CR:

$condition = \Drupal::database()
  ->condition('AND')
  ->compare('d.entity_id', 'b.entity_id')
  ->compare('d.entity_type', 'b.entity_type')
  ->compare('d.field_name', 'b.field_name');

As I read it, the coding standards encourage breaking before each method if each method belongs to the same object. Here, condition() is a method on the Connection object, and the three compare() methods belong to the Condition object. So I would rewrite that snippet (saving one line) as

$condition = \Drupal::database()->condition('AND')
  ->compare('d.entity_id', 'b.entity_id')
  ->compare('d.entity_type', 'b.entity_type')
  ->compare('d.field_name', 'b.field_name');

alexpott made their first commit to this issue’s fork.

alexpott’s picture

Status: Reviewed & tested by the community » Needs work

Exciting work. I left a review on the MR.

joachim’s picture

This looks great. I just happened to stumble on the join methods allowing a condition object.

How do you do this:

> $connection->joinCondition()->compare('n.nid', 'c.nid'));

if you want to let the join() method provide the table alias automatically?

joachim’s picture

Another thing - the name 'compare' for the method makes me think of comparison operators like '<' and '>'.

Wouldn't something like 'equate' be more descriptive?

Version: 11.x-dev » main

Drupal core is now using the main branch as the primary development branch. New developments and disruptive changes should now be targeted to the main branch.

Read more in the announcement.