Problem/Motivation

We want collections to be filterable, that includes:

  1. Nested filters
  2. Filters with operators
  3. Filters with operators that apply to multiple values like IN, BETWEEN, …

Proposed resolution

Proposed filter structure:

filter[$index][$filter_type][$filter_property]=<value>

Available filter types: condition, group, exists.

Filter types can have properties like field, value, operator, etc.

The special property, group, can assign filters as children of a filter group. This is done by setting the parameter value to the index of the filter group.

To add a condition to a group, one would do something like the following:

filter[0][group][conjunction]=AND
filter[1][condition][group]=0
filter[2][condition][group]=0

The value filter type of the condition filter type may take an array of values. Thus, multiple values are achieved like so:

filter[$index][condition][value][]=<value0>
filter[$index][condition][value][]=<value1> // and so on.

Examples:

filter[$index][condition][field]=<public_field_name>
filter[$index][condition][value]=<value>
filter[$index][condition][operator]=<operator> // optional. Defaults to '='.
filter[$index][condition][group]=<group_index> // optional.

Condition groups would look like this:

filter[$index][group][conjunction]=<conjunction> // 'AND'|'OR'.

You can have a group within a group like so:

filter[$index_0][group][conjunction]=OR
filter[$index_1][group][conjunction]=AND
filter[$index_1][group][group]=$index_0

Exists would look like this:

filter[$index][exists][field]=<public_field_name>
filter[$index][exists][exists]=<boolean> // optional. 'TRUE'|'FALSE'. Defaults to 'TRUE'.
filter[$index][exists][group]=<group_index> // optional.

Shortcuts

  • If the operator is the equality operator, then the operator parameter can be omitted. Default: "="
  • It is not required to include the [condition] filter type as long as both a field and value are specified.
  • IS NULL and IS NOT NULL can use the exists filter type.
Examples:
filter[0][field]=<public_field_name>&filter[0][value]=<value>

expands to:

filter[0][condition][field]=<public_field_name>
filter[0][condition][value]=<value>
filter[0][condition][operator]=%3D

Remaining tasks

Parse the filters from the request, add them to some kind of request context object and apply them to the entity query object.

When it comes to conjunctions, I propose having the concept of groups, to remain consistent with the Views experience.

Map configured public field names to Drupal internal field names.

We still need to create issues to handle:

  1. Nested filters
  2. Conjunctions support

Comments

e0ipso created an issue. See original summary.

e0ipso’s picture

Issue summary: View changes
e0ipso’s picture

Status: Active » Needs review
StatusFileSize
new21.53 KB

More tests for the EntityResource class will be needed in a follow up issue.

e0ipso’s picture

Status: Needs review » Needs work
e0ipso’s picture

Status: Needs work » Needs review

Re-kicking tests.

e0ipso’s picture

StatusFileSize
new22.03 KB
new2.25 KB

Fixing test namespaces for testbot.

e0ipso’s picture

Issue summary: View changes
e0ipso’s picture

Status: Needs review » Fixed

Merged.

gabesullice’s picture

Assigned: e0ipso » gabesullice
Status: Fixed » Needs work

I'd like to propose the following syntax for filters:

filter[$index][$filter_type][$filter_property]=<value>

Available filter types: condition, group, exists.

Filter types can have properties like field, value, operator, etc.

The special property, group, can assign filters as children of a filter group. This is done by setting the parameter value to the index of the filter group. To add a condition to a group, one would do something like the following:

filter[0][group][conjunction]=AND
filter[1][condition][group]=0
filter[2][condition][group]=0

The value filter type of the condition filter type may take an array of values. Thus, multiple values are achieved like so:

filter[$index][condition][value][]=<value0>
filter[$index][condition][value][]=<value1> // and so on.

Examples:

filter[$index][condition][field]=<public_field_name>
filter[$index][condition][value]=<value>
filter[$index][condition][operator]=<operator> // optional. Defaults to ''.
filter[$index][condition][group]=<group_index> // optional.

Condition groups would look like this:

filter[$index][group][conjunction]=<conjunction> // 'AND'|'OR'.
filter[$index][group][group]=<group_index> // optional.

Exists would look like this:

filter[$index][exists][field]=<public_field_name>
filter[$index][exists][exists]=<boolean> // optional. 'TRUE'|'FALSE'. Defaults to 'TRUE'.
filter[$index][exists][group]=<group_index> // optional.
e0ipso’s picture

I like where this is going!

I have a couple of questions/requests.

1. Groups

Can we have:

filter[$index][group][name]=<group label> // optional.

Instead of:

filter[$index][group][group]=<group_index> // optional.

2. Exists

Do we need this if we can support the IS NOT NULL as an operator?

3. Shortcuts

I'd like to keep simplified versions for simple queries, since those will be the common case. Maybe:

filter[0][field]=<public_field_name>&filter[0][value]=<value>

expands to:

filter[0][condition][field]=<public_field_name>
filter[0][condition][value]=<value>
filter[0][condition][operator]=%3D
filter[0][condition][group]=asfdf
filter[asfdf][group][conjunction]=AND
gabesullice’s picture

@e0ipso:

1. Groups
How would you feel about filter[$index][group][index]=$index. There's no reason that the index can't be both alpha AND numeric. Machine generation might choose to use numeric indices, but it's entirely possible to have 'named' indices.

2. Exists
I'm fine leaving these out if you think it's confusing. Though, they're trivial to add and a nice shortcut.

3. Shortcuts
I think these look like useful features.

e0ipso’s picture

How would you feel about filter[$index][group][index]=$index.

Did you mean: filter[$index_0][group][index]=$index_1?

Or am I missing something.

gabesullice’s picture

@e0ipso: you are correct.

gabesullice’s picture

@e0ipso: reading your comment again now that I'm writing code, I think there was a miscommunication. [group][group] was not meant to be a label for the group. It was supposed to denote a parent-child relationship between groups. Like so:

filter[$index_0][group][conjunction]=OR

filter[$index_1][group][conjunction]=AND
filter[$index_1][group][group]=$index_0

filter[$index_2][field]=color
filter[$index_2][value]=red
filter[$index_2][group]=$index_0

filter[$index_3][field]=color
filter[$index_3][value]=blue
filter[$index_3][group]=$index_1

filter[$index_4][field]=color
filter[$index_4][value]=green
filter[$index_4][group]=$index_1

This would be the equivalent of the following pseudo-code:

($color == 'red' || ($color == 'blue' && $color == 'green'))

P.S. forgive the example, 'IN ARRAY' would be better, but I wanted to illustrate a nested group.

gabesullice’s picture

Status: Needs work » Needs review

The following patch makes use of the updated syntax, enables conjunctions, conjunctions within conjunctions, and eliminates the need for declaring that a field is multivalue.

gabesullice’s picture

gabesullice’s picture

Issue summary: View changes
e0ipso’s picture

This patch is amazing! I am so excited. It is also very big so the review is going to take quite some time.

First review pass:

  1. +++ b/jsonapi.services.yml
    @@ -48,3 +48,6 @@ services:
    +    arguments: [ '@entity.query' ]
    

    This service will disappear. We should be injecting the entity_type.manager and using getStorage() from there.

  2. +++ b/src/Query/ConditionOption.php
    @@ -0,0 +1,96 @@
    +   * Value of the field.
    

    Value of the condition for the given field.

  3. +++ b/src/Query/ConditionOption.php
    @@ -0,0 +1,96 @@
    +    $this->field    = $field;
    

    I believe that we should not have the extra spaces to align values, according to the Drupal Coding Standards.

  4. +++ b/src/Query/ExistsOption.php
    @@ -0,0 +1,79 @@
    +class ExistsOption implements QueryOptionInterface {
    

    There's some duplication with the ConditionOption class. Should we have a common parent (abstract) class to put the duplicated code there?

  5. +++ b/src/Query/GroupOption.php
    @@ -0,0 +1,140 @@
    +      if ($child_id) return $child_id;
    +      return ($groupOption->hasChild($target_id)) ? $condition->id() : NULL;
    

    We need the curly braces and new lines according to the DCS. Maybe:

    return $child_id ?: ($groupOption->hasChild($target_id)) ? $condition->id() : NULL;

  6. +++ b/src/Query/GroupOption.php
    @@ -0,0 +1,140 @@
    +    if ($this->id() == $target_id && $option instanceof GroupOption) {
    +      $this->childGroups[$option->id()] = $option;
    +      return TRUE;
    +    }
    +    elseif ($this->id() == $target_id) {
    +      $this->childOptions[$option->id()] = $option;
    +      return TRUE;
    +    }
    

    What about:

    if ($this->id() == $target_id) {
      $property = $option instanceof GroupOption ? 'childGroups' : 'childOptions';
      $this->{$property}[$option->id()] = $option;
      return TRUE;
    }
    </li>
    
    <li>
    &#10;+++ b/src/Query/GroupOption.php&#10;@@ -0,0 +1,140 @@&#10;+    elseif ($proper_child = array_reduce($this-&gt;childGroups, $find_proper_id, NULL)) {&#10;
    
    This array reduce will keep looping even after an ID is found.
    
    Maybe we need a helper protected method with an early return when the ID is found.
    </li>
    
    <li>
    &#10;+++ b/src/Query/GroupOption.php&#10;@@ -0,0 +1,140 @@&#10;+    case &#039;OR&#039;:&#10;...&#10;+    case &#039;AND&#039;:&#10;+    default:&#10;
    
    Indentation for the cases is off by 2 spaces.
    </li>
    
    <li>
    &#10;+++ b/src/Query/GroupOption.php&#10;@@ -0,0 +1,140 @@&#10;+    if (!isset($this-&gt;childOptions) || empty($this-&gt;childOptions)) return FALSE;&#10;
    
    Same comment about inline if statements.
    
    https://www.drupal.org/coding-standards#controlstruct says:
    
    <blockquote>Always use curly braces even in situations where they are technically optional. Having them increases readability and decreases the likelihood of logic errors being introduced when new lines are added. </blockquote>
    </li>
    
    <li>
    &#10;+++ b/src/Query/GroupOption.php&#10;@@ -0,0 +1,140 @@&#10;+    return array_reduce($this-&gt;groupOptions, function ($hasChild, $group) use ($id) {&#10;
    
    This keeps looping even after we know what the return value will be.
    </li>
    
    <li>
    &#10;+++ b/src/Query/QueryBuilder.php&#10;@@ -0,0 +1,212 @@&#10;+  public function addFilter() {&#10;
    
    I'm not sure what this is doing, but there is no docs or implementation code there.
    </li>
    
    <li>
    &#10;+++ b/src/Query/QueryBuilder.php&#10;@@ -0,0 +1,212 @@&#10;+    ¶&#10;
    
    Nit: Extra blank space.
    </li>
    
    <li>
    &#10;+++ b/src/Query/QueryBuilder.php&#10;@@ -0,0 +1,212 @@&#10;+    $applied_query = array_reduce($this-&gt;options, function ($query, $option) {&#10;+      $query = $option-&gt;apply($query);&#10;+      return $query;&#10;+    }, $query);&#10;
    
    Can we just do:
    <?php
    return $option->apply($query);
    
  7. +++ b/src/Query/QueryBuilder.php
    @@ -0,0 +1,212 @@
    +    case 'filter':
    

    Indendation for 'case'.

  8. +++ b/src/Query/QueryBuilder.php
    @@ -0,0 +1,212 @@
    +   * @param Drupal\jsonapi\Routing\Param\JsonApiParamInterface $param
    

    FQDN needs a leading backslash in the docblocks.

e0ipso’s picture

  1. +++ b/src/Query/ConditionOption.php
    @@ -0,0 +1,96 @@
    +  /**
    +   * Value of the field.
    +   *
    +   * @var string
    +   */
    +  protected $value;
    

    What if we have a multivalue operator? Should we make this an string|string[]?

  2. +++ b/src/Query/GroupOption.php
    @@ -0,0 +1,140 @@
    +class GroupOption implements QueryOptionInterface, QueryOptionTreeItemInterface {
    

    We'll need docblocks for classes as well.

  3. +++ b/src/Query/QueryBuilderInterface.php
    @@ -0,0 +1,26 @@
    +interface QueryBuilderInterface {
    

    Docblocks for class needed. (This happens in several other places, I won't report every one)

  4. +++ b/src/Resource/EntityResource.php
    @@ -87,6 +95,27 @@ class EntityResource implements EntityResourceInterface {
       /**
    +   * Gets a basic query for a collection.
    +   */
    +  protected function getCollectionQuery($entity_type_id, $params) {
    

    Needs docs for the parameters.

  5. +++ b/src/Routing/Param/Filter.php
    @@ -16,8 +16,6 @@ class Filter extends JsonApiParamBase {
    -  protected static $multivalue_operators = ['IN', 'NOT IN', 'BETWEEN'];
    

    Are we losing the ability to use multivalue operators?

e0ipso’s picture

I'm done with the review. We will definitely need some extensive functional tests for the filtering but in order to increase development velocity, I'd say that those can be a follow up.

I'm doing tests as part of #2726265: [BUGFIX] Add test coverage for EntityResource, Routes and ResourceManager, which will include testing for lists (without filters). Maybe you can base off that.

Will you create the follow up issue for the tests as part of the acceptance criteria for this one?

Finally, I always forget to add interdiffs but I will need it to review your updates to this patch.

Did I say how excited I am about this patch? :-D

e0ipso’s picture

Status: Needs review » Needs work
gabesullice’s picture

Status: Needs work » Needs review
StatusFileSize
new26.8 KB
new10.48 KB

@e0ipso: I appreciate all your excellent feedback!

#17
  1. I wasn't aware, thanks for the heads up.
  2. yep
  3. I've been in love with Golang for a while, hard to shake the format rules.
  4. Agreed, but I'm going to include that in a follow up issue so that we can get this committed to unblock RequestContext.
  5. Cleaned this up a bit.
  6. Good suggestion
  7. Yes, it will continue, but my typical pattern for this is to include a guard. if ($child_id) return $child_id; eliminates any real computation after a child has been found.
  8. Fixed.
  9. Sigh, fixed ;)
  10. See, #7
  11. Good catch, this was left over from an initial pass at what became configureFromParameter.
  12. Like nails on a chalkboard, no? Fixed.
  13. Yes, leftover from xdebug.
  14. Fixed.
  15. Fixed.
#18
  1. Updated docblock.
  2. Added
  3. Added
  4. Added
  5. No. Since we are using empty brackets after the [value] property, the parsed value immediately becomes an array so we don't need to declare them. Perhaps a follow up can be to add some checking and/or expansion.

  • e0ipso committed bce263e on 8.x-1.x authored by gabesullice
    Issue #2725725 by e0ipso, gabesullice: [FEATURE] Add support for...
e0ipso’s picture

Status: Needs review » Fixed

This was merged. Thanks!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.