Add XPath-style filtering ability in JSON data parser plugin

Lots of good comments in the patch. Thanks for your contributions. However, we need to be sure we maintain backward compatibility so having some tests that demonstrate that the current xpath approach works and that this new method also are supported would be nice. Setting back to NW for tests.

Yeah, that would work. There's nothing much special to these unit or kernel tests. Just normal unit test style stuff. If you haven't read it, there's some good docs over in https://phpunit.de/manual/4.5/en/writing-tests-for-phpunit.html about general testing and https://www.drupal.org/docs/8/phpunit

Thoughts on getting off the island that's being built here?

There's 2 very popular packages on packagist that implement JSONPath selectors which allows for much more advanced selectors. flow/jsonpath and mtdowling/jmespath.php

Seems like a better use of time to integrate one of these packages than to cobble together our own selection logic.

#2640514 has one example use case which deals with a product import.

I would much rather see us use an external library. From a quick survey of those two projects, flow/jsonpath seems the better fit for us. However, this also is a very likely break of BC. Shall we add a new json parser, built from scratch, that uses one of these? If we did that, I'd want to see some decent tests out of the gate.

Took an first pass at this. Just copied the existing json parser and modified the selection logic. Only part I'm not real happy about is

$field_data = (new JSONPathSelector($current))->find($selector);

if ($field_data->count() == 1) {
  $field_data = $field_data->first();
} else {
   $field_data = $field_data->data();
}

I might be missing something but I couldn't get a selector that returned just a string. Instead it was returning an array with one element as an example.
selected item:

[
  "label" => "all",
  "value" => "01-01-2017",
  "end_value" => "12-31-2017",
  "uuid" => "da9d4303-34db-402f-a7d7-a01b1b231d61"
]

field selector: label
result:

[
  0 => "all"
]

Example migration & data that's working for me:

id: seasons
label: JSON feed of Seasons
migration_group: Seasons
migration_tags:
  - demo data
source:
  plugin: url
  data_fetcher_plugin: file

  data_parser_plugin: jsonpath
  item_selector: $..seasons[*]

  fields:
    -
      name: label
      selector: label
    -
      name: start_value
      selector: value
    -
      name: end_value
      selector: end_value
    -
      name: uuid
      selector: uuid
  ids:
    uuid:
      type: string
process:
  name: label
  dates/value: start_value
  dates/end_value: end_value
destination:
  plugin: 'entity:custom_season'
migration_dependencies: {  }

{
    "years": [
        {
            "label": "2017",
            "uid": 1,
            "seasons": [
                {
                    "label": "all",
                    "value": "01-01-2017",
                    "end_value": "12-31-2017",
                    "uuid": "da9d4303-34db-402f-a7d7-a01b1b231d61"
                }
            ],
            "uuid": "0260f241-f528-4ade-9d8a-af8b2e1bed05"
        },
        {
            "label": "2018",
            "uid": 1,
            "seasons": [
                {
                    "label": "Winter",
                    "value": "01/01/2018",
                    "end_value": "03/31/2018",
                    "uuid": "2e169d04-d66a-4332-8a0c-3307f43f8c77"
                },
                {
                    "label": "Spring",
                    "value": "04/01/2018",
                    "end_value": "07/31/2018",
                    "uuid": "72d07dbf-92fa-4abe-9d12-60125aebd487"
                }
            ],
            "uuid": "041551e1-459c-4012-8510-fdfa3dd4b49b"
        }
    ]
}

Forgot to attach the patch :/

+1 for this feature, but I agree with #8 that bringing in a 3rd party library to do the heavy lifting for us would be the best approach. Being able to use JSONPath or proper Xpath expressions would be a great feature. E.g, I have a JSON feed I'm consuming, but I need to filter out some entries based on an attribute. Using Xpath like this works:

/data[attributes/parent_product/product_type = "Subscription"]

But isn't supported currently in the Json parser plugin.

I was about to open a new ticket, but I think this addresses my problem. You can see full troubleshooting here:

https://www.drupal.org/forum/support/module-development-and-code-questio...

Basically, I am trying to import 1 of 121 objects from an xpath selector. My migration works for all objects and creates 121 entities, but when I set the selector to the xpath of one of them, it errors out because the they are nested instead of in an array, like this:

{
"Abilene": {
"area": 589.4,
"condensed_name": "Abilene",
"conservation_capacity": 7900.0,
"conservation_pool_elevation": 2012.3,
"conservation_storage": 7757.0,
"dead_pool_elevation": 1968.8,
"elevation": 2012.07,
"full_name": "Lake Abilene",
"gauge_location": {
"coordinates": [
-99.88897705,
32.23457718
],
"type": "Point"
},
"percent_full": 98.2,
"short_name": "Abilene",
"tags": [
"climate_low_rolling_plains",
"monitored",
"municipal_abilene",
"region_brazos_g",
"water_supply",
"texas",
"major",
"basin_brazos"
],
"timestamp": "2019-03-13",
"volume": 7757.0,
"volume_under_conservation_pool_elevation": 7900.0
},
"Addicks": {
"area": null,
"condensed_name": "Addicks",
"conservation_capacity": null,
"conservation_pool_elevation": 67.5,
"conservation_storage": null,
"dead_pool_elevation": 67.5,
"elevation": 68.88,
"full_name": "Addicks Reservoir",
"gauge_location": {
"coordinates": [
-95.62355804,
29.79133987
],
"type": "Point"
}}

the parser identifies correct number of rows, but fails. if I select 'Addicks'.

Here is the full JSON feed:

https://waterdatafortexas.org/reservoirs/recent-conditions.json

Seems like this ticket addresses a similar problem, if not, I will be glad to open a different ticket.

I have applied with above patch to test it with the data below. There's an issue with the library itself when there's a "-" in the field selectors.The below migration template throws error when fetching the display-name from the json.
https://github.com/FlowCommunications/JSONPath/issues/16

Here's the migration template:

id: categories
label: Migrate categories.
status: true
migration_tags:
  - Sampel tag
migration_group: sample
source:
  plugin: url
  data_fetcher_plugin: http
  data_parser_plugin: jsonpath
  urls:
    - 'http://example.com/products.json'
  item_selector: '$.._child[*]'
  fields:
    -
      name: name
      selector: name
      label: 'Category Machine name'
    -
      name: display_name
      selector: display-name
      label: 'Category display name'
    -
      name: self_uri
      selector: self.uri
      label: 'Category Json Object document.'
  ids:
    self_uri:
      type: string
  constants:
    vocabulary: category
process:
  vid: constants/vocabulary
  name: display_name
  description: name
  parent:
    plugin: default_value
    default_value: 0
destination:
  plugin: entity:taxonomy_term
migration_dependencies: {}
dependencies:
  enforced:
    module:
      - example

Data.

{
  "self": {
    "type": "navigations.navigations",
    "uri": "/navigations/cf?zoom=element,element:child,element:child:child,element:child:child:child,element:child:child:child:child",
    "href": "https://example.com/cortex/navigations/cf?zoom=element,element:child,element:child:child,element:child:child:child,element:child:child:child:child"
  },
  "messages": [],
  "links": [
    {
      "rel": "element",
      "rev": "list",
      "type": "navigations.navigation",
      "uri": "/navigations/cf/mnxw24dvorsxe427mfxgix3umfrgyzluom=",
      "href": "https://example.com/cortex/navigations/cf/mnxw24dvorsxe427mfxgix3umfrgyzluom="
    },
    {
      "rel": "element",
      "rev": "list",
      "type": "navigations.navigation",
      "uri": "/navigations/cf/mntf643nmfzhi4din5xgk4y=",
      "href": "https://example.com/cortex/navigations/cf/mntf643nmfzhi4din5xgk4y="
    },
    {
      "rel": "element",
      "rev": "list",
      "type": "navigations.navigation",
      "uri": "/navigations/cf/mntf6y3bnvsxeylt=",
      "href": "https://example.com/cortex/navigations/cf/mntf6y3bnvsxeylt="
    },
    {
      "rel": "element",
      "rev": "list",
      "type": "navigations.navigation",
      "uri": "/navigations/cf/mntf653fmfzgcytmmvzq=",
      "href": "https://www.example.com/cortex/navigations/cf/mntf653fmfzgcytmmvzq="
    },
    {
      "rel": "element",
      "rev": "list",
      "type": "navigations.navigation",
      "uri": "/navigations/cf/mnzv643foj3gsy3fom=",
      "href": "https://www.example.com/cortex/navigations/cf/mnzv643foj3gsy3fom="
    },
    {
      "rel": "element",
      "rev": "list",
      "type": "navigations.navigation",
      "uri": "/navigations/cf/mfyha3djmnqw4y3fonsxe5tjmnsxg=",
      "href": "https://www.example.com/cortex/navigations/cf/mfyha3djmnqw4y3fonsxe5tjmnsxg="
    },
    {
      "rel": "element",
      "rev": "list",
      "type": "navigations.navigation",
      "uri": "/navigations/cf/ijsxg5ctmvwgyzlsom=",
      "href": "https://www.example.com/cortex/navigations/cf/ijsxg5ctmvwgyzlsom="
    }
  ],
  "_element": [
    {
      "self": {
        "type": "navigations.navigation",
        "uri": "/navigations/cf/mnzv643foj3gsy3fom=",
        "href": "https://www.example.com/cortex/navigations/cf/mnzv643foj3gsy3fom="
      },
      "messages": [],
      "links": [
        {
          "rel": "top",
          "type": "navigations.navigations",
          "uri": "/navigations/cf",
          "href": "https://www.example.com/cortex/navigations/cf"
        },
        {
          "rel": "child",
          "rev": "parent",
          "type": "navigations.navigation",
          "uri": "/navigations/cf/obzg6ztfonzws33omfwhizldna=",
          "href": "https://www.example.com/cortex/navigations/cf/obzg6ztfonzws33omfwhizldna="
        },
        {
          "rel": "items",
          "type": "searches.navigation-search-result",
          "uri": "/searches/navigations/cf/mnzv643foj3gsy3fom=/1",
          "href": "https://www.example.com/cortex/searches/navigations/cf/mnzv643foj3gsy3fom=/1"
        }
      ],
      "_child": [
        {
          "self": {
            "type": "navigations.navigation",
            "uri": "/navigations/cf/obzg6ztfonzws33omfwhizldna=",
            "href": "https://www.example.com/cortex/navigations/cf/obzg6ztfonzws33omfwhizldna="
          },
          "messages": [],
          "links": [
            {
              "rel": "parent",
              "rev": "child",
              "type": "navigations.navigation",
              "uri": "/navigations/cf/mnzv643foj3gsy3fom=",
              "href": "https://www.example.com/cortex/navigations/cf/mnzv643foj3gsy3fom="
            },
            {
              "rel": "top",
              "type": "navigations.navigations",
              "uri": "/navigations/cf",
              "href": "https://www.example.com/cortex/navigations/cf"
            },
            {
              "rel": "child",
              "rev": "parent",
              "type": "navigations.navigation",
              "uri": "/navigations/cf/mrqxiylsmvrw65tfoj4q=",
              "href": "https://www.example.com/cortex/navigations/cf/mrqxiylsmvrw65tfoj4q="
            },
            {
              "rel": "child",
              "rev": "parent",
              "type": "navigations.navigation",
              "uri": "/navigations/cf/ozuxe5ltm52wc4te=",
              "href": "https://www.example.com/cortex/navigations/cf/ozuxe5ltm52wc4te="
            },
            {
              "rel": "child",
              "rev": "parent",
              "type": "navigations.navigation",
              "uri": "/navigations/cf/mrqxiyluojqw443gmvza=",
              "href": "https://www.example.com/cortex/navigations/cf/mrqxiyluojqw443gmvza="
            },
            {
              "rel": "items",
              "type": "searches.navigation-search-result",
              "uri": "/searches/navigations/cf/obzg6ztfonzws33omfwhizldna=/1",
              "href": "https://www.example.com/cortex/searches/navigations/cf/obzg6ztfonzws33omfwhizldna=/1"
            }
          ],
          "_child": [
            {
              "self": {
                "type": "navigations.navigation",
                "uri": "/navigations/cf/mrqxiylsmvrw65tfoj4q=",
                "href": "https://www.example.com/cortex/navigations/cf/mrqxiylsmvrw65tfoj4q="
              },
              "messages": [],
              "links": [
                {
                  "rel": "parent",
                  "rev": "child",
                  "type": "navigations.navigation",
                  "uri": "/navigations/cf/obzg6ztfonzws33omfwhizldna=",
                  "href": "https://www.example.com/cortex/navigations/cf/obzg6ztfonzws33omfwhizldna="
                },
                {
                  "rel": "top",
                  "type": "navigations.navigations",
                  "uri": "/navigations/cf",
                  "href": "https://www.example.com/cortex/navigations/cf"
                },
                {
                  "rel": "items",
                  "type": "searches.navigation-search-result",
                  "uri": "/searches/navigations/cf/mrqxiylsmvrw65tfoj4q=/1",
                  "href": "https://www.example.com/cortex/searches/navigations/cf/mrqxiylsmvrw65tfoj4q=/1"
                }
              ],
              "details": [
                {
                  "display-name": "Category Description",
                  "display-value": "Data Recovery",
                  "name": "catDescription",
                  "value": "Data Recovery"
                },
                {
                  "display-name": "Name",
                  "display-value": "Data Recovery",
                  "name": "catName",
                  "value": "Data Recovery"
                }
              ],
              "display-name": "Data Recovery",
              "name": "datarecovery"
            },
            {
              "self": {
                "type": "navigations.navigation",
                "uri": "/navigations/cf/ozuxe5ltm52wc4te=",
                "href": "https://www.example.com/cortex/navigations/cf/ozuxe5ltm52wc4te="
              },
              "messages": [],
              "links": [
                {
                  "rel": "parent",
                  "rev": "child",
                  "type": "navigations.navigation",
                  "uri": "/navigations/cf/obzg6ztfonzws33omfwhizldna=",
                  "href": "https://www.example.com/cortex/navigations/cf/obzg6ztfonzws33omfwhizldna="
                },
                {
                  "rel": "top",
                  "type": "navigations.navigations",
                  "uri": "/navigations/cf",
                  "href": "https://www.example.com/cortex/navigations/cf"
                },
                {
                  "rel": "items",
                  "type": "searches.navigation-search-result",
                  "uri": "/searches/navigations/cf/ozuxe5ltm52wc4te=/1",
                  "href": "https://www.example.com/cortex/searches/navigations/cf/ozuxe5ltm52wc4te=/1"
                }
              ],
              "details": [
                {
                  "display-name": "Category Description",
                  "display-value": "Virus Guard",
                  "name": "catDescription",
                  "value": "Virus Guard"
                },
                {
                  "display-name": "Name",
                  "display-value": "Virus Guard",
                  "name": "catName",
                  "value": "Virus Guard"
                }
              ],
              "display-name": "Virus Guard",
              "name": "virusguard"
            },
            {
              "self": {
                "type": "navigations.navigation",
                "uri": "/navigations/cf/mrqxiyluojqw443gmvza=",
                "href": "https://www.example.com/cortex/navigations/cf/mrqxiyluojqw443gmvza="
              },
              "messages": [],
              "links": [
                {
                  "rel": "parent",
                  "rev": "child",
                  "type": "navigations.navigation",
                  "uri": "/navigations/cf/obzg6ztfonzws33omfwhizldna=",
                  "href": "https://www.example.com/cortex/navigations/cf/obzg6ztfonzws33omfwhizldna="
                },
                {
                  "rel": "top",
                  "type": "navigations.navigations",
                  "uri": "/navigations/cf",
                  "href": "https://www.example.com/cortex/navigations/cf"
                },
                {
                  "rel": "items",
                  "type": "searches.navigation-search-result",
                  "uri": "/searches/navigations/cf/mrqxiyluojqw443gmvza=/1",
                  "href": "https://www.example.com/cortex/searches/navigations/cf/mrqxiyluojqw443gmvza=/1"
                }
              ],
              "details": [
                {
                  "display-name": "Category Description",
                  "display-value": "Data Transfer",
                  "name": "catDescription",
                  "value": "Data Transfer"
                },
                {
                  "display-name": "Name",
                  "display-value": "Data Transfer",
                  "name": "catName",
                  "value": "Data Transfer"
                }
              ],
              "display-name": "Data Transfer",
              "name": "datatransfer"
            }
          ],
          "details": [
            {
              "display-name": "Category Description",
              "display-value": "Professional Tech",
              "name": "catDescription",
              "value": "Professional Tech"
            },
            {
              "display-name": "Name",
              "display-value": "Professional Tech",
              "name": "catName",
              "value": "Professional Tech"
            }
          ],
          "display-name": "Professional Tech",
          "name": "professionaltech"
        }
      ],
      "details": [
        {
          "display-name": "Category Description",
          "display-value": "Services",
          "name": "catDescription",
          "value": "Services"
        },
        {
          "display-name": "Name",
          "display-value": "Services",
          "name": "catName",
          "value": "Services"
        }
      ],
      "display-name": "Services",
      "name": "cs_services"
    },
    {
      "self": {
        "type": "navigations.navigation",
        "uri": "/navigations/cf/mfyha3djmnqw4y3fonsxe5tjmnsxg=",
        "href": "https://www.example.com/cortex/navigations/cf/mfyha3djmnqw4y3fonsxe5tjmnsxg="
      },
      "messages": [],
      "links": [
        {
          "rel": "top",
          "type": "navigations.navigations",
          "uri": "/navigations/cf",
          "href": "https://www.example.com/cortex/navigations/cf"
        },
        {
          "rel": "child",
          "rev": "parent",
          "type": "navigations.navigation",
          "uri": "/navigations/cf/obzg6ztfonzws33omfwhgzlsozuwgzlt=",
          "href": "https://www.example.com/cortex/navigations/cf/obzg6ztfonzws33omfwhgzlsozuwgzlt="
        },
        {
          "rel": "items",
          "type": "searches.navigation-search-result",
          "uri": "/searches/navigations/cf/mfyha3djmnqw4y3fonsxe5tjmnsxg=/1",
          "href": "https://www.example.com/cortex/searches/navigations/cf/mfyha3djmnqw4y3fonsxe5tjmnsxg=/1"
        }
      ],
      "_child": [
        {
          "self": {
            "type": "navigations.navigation",
            "uri": "/navigations/cf/obzg6ztfonzws33omfwhgzlsozuwgzlt=",
            "href": "https://www.example.com/cortex/navigations/cf/obzg6ztfonzws33omfwhgzlsozuwgzlt="
          },
          "messages": [],
          "links": [
            {
              "rel": "parent",
              "rev": "child",
              "type": "navigations.navigation",
              "uri": "/navigations/cf/mfyha3djmnqw4y3fonsxe5tjmnsxg=",
              "href": "https://www.example.com/cortex/navigations/cf/mfyha3djmnqw4y3fonsxe5tjmnsxg="
            },
            {
              "rel": "top",
              "type": "navigations.navigations",
              "uri": "/navigations/cf",
              "href": "https://www.example.com/cortex/navigations/cf"
            },
            {
              "rel": "child",
              "rev": "parent",
              "type": "navigations.navigation",
              "uri": "/navigations/cf/mrswy2lwmvzhs4dmovzws3ttorqwy3dboruw63q=",
              "href": "https://www.example.com/cortex/navigations/cf/mrswy2lwmvzhs4dmovzws3ttorqwy3dboruw63q="
            },
            {
              "rel": "child",
              "rev": "parent",
              "type": "navigations.navigation",
              "uri": "/navigations/cf/mrswy2lwmvzhs=",
              "href": "https://www.example.com/cortex/navigations/cf/mrswy2lwmvzhs="
            },
            {
              "rel": "items",
              "type": "searches.navigation-search-result",
              "uri": "/searches/navigations/cf/obzg6ztfonzws33omfwhgzlsozuwgzlt=/1",
              "href": "https://www.example.com/cortex/searches/navigations/cf/obzg6ztfonzws33omfwhgzlsozuwgzlt=/1"
            }
          ],
          "_child": [
            {
              "self": {
                "type": "navigations.navigation",
                "uri": "/navigations/cf/mrswy2lwmvzhs4dmovzws3ttorqwy3dboruw63q=",
                "href": "https://www.example.com/cortex/navigations/cf/mrswy2lwmvzhs4dmovzws3ttorqwy3dboruw63q="
              },
              "messages": [],
              "links": [
                {
                  "rel": "parent",
                  "rev": "child",
                  "type": "navigations.navigation",
                  "uri": "/navigations/cf/obzg6ztfonzws33omfwhgzlsozuwgzlt=",
                  "href": "https://www.example.com/cortex/navigations/cf/obzg6ztfonzws33omfwhgzlsozuwgzlt="
                },
                {
                  "rel": "top",
                  "type": "navigations.navigations",
                  "uri": "/navigations/cf",
                  "href": "https://www.example.com/cortex/navigations/cf"
                },
                {
                  "rel": "items",
                  "type": "searches.navigation-search-result",
                  "uri": "/searches/navigations/cf/mrswy2lwmvzhs4dmovzws3ttorqwy3dboruw63q=/1",
                  "href": "https://www.example.com/cortex/searches/navigations/cf/mrswy2lwmvzhs4dmovzws3ttorqwy3dboruw63q=/1"
                }
              ],
              "details": [
                {
                  "display-name": "Category Description",
                  "display-value": "Delivery + Installation",
                  "name": "catDescription",
                  "value": "Delivery + Installation"
                },
                {
                  "display-name": "Name",
                  "display-value": "Delivery + Installation",
                  "name": "catName",
                  "value": "Delivery + Installation"
                }
              ],
              "display-name": "Delivery + Installation",
              "name": "deliveryplusinstallation"
            },
            {
              "self": {
                "type": "navigations.navigation",
                "uri": "/navigations/cf/mrswy2lwmvzhs=",
                "href": "https://www.example.com/cortex/navigations/cf/mrswy2lwmvzhs="
              },
              "messages": [],
              "links": [
                {
                  "rel": "parent",
                  "rev": "child",
                  "type": "navigations.navigation",
                  "uri": "/navigations/cf/obzg6ztfonzws33omfwhgzlsozuwgzlt=",
                  "href": "https://www.example.com/cortex/navigations/cf/obzg6ztfonzws33omfwhgzlsozuwgzlt="
                },
                {
                  "rel": "top",
                  "type": "navigations.navigations",
                  "uri": "/navigations/cf",
                  "href": "https://www.example.com/cortex/navigations/cf"
                },
                {
                  "rel": "items",
                  "type": "searches.navigation-search-result",
                  "uri": "/searches/navigations/cf/mrswy2lwmvzhs=/1",
                  "href": "https://www.example.com/cortex/searches/navigations/cf/mrswy2lwmvzhs=/1"
                }
              ],
              "details": [
                {
                  "display-name": "Category Description",
                  "display-value": "Delivery",
                  "name": "catDescription",
                  "value": "Delivery"
                },
                {
                  "display-name": "Name",
                  "display-value": "Delivery",
                  "name": "catName",
                  "value": "Delivery"
                }
              ],
              "display-name": "Delivery",
              "name": "delivery"
            }
          ],
          "details": [
            {
              "display-name": "Category Description",
              "display-value": "Professional Services",
              "name": "catDescription",
              "value": "Professional Services"
            },
            {
              "display-name": "Name",
              "display-value": "Professional Services",
              "name": "catName",
              "value": "Professional Services"
            }
          ],
          "display-name": "Professional Services",
          "name": "professionalservices"
        }
      ],
      "details": [
        {
          "display-name": "Category Description",
          "display-value": "Applicance Services",
          "name": "catDescription",
          "value": "Applicance Services"
        },
        {
          "display-name": "Name",
          "display-value": "Applicance Services",
          "name": "catName",
          "value": "Applicance Services"
        }
      ],
      "display-name": "Applicance Services",
      "name": "applicanceservices"
    }
  ]
}

Re: #12
I made the following change to the selector and it worked fine as mentioned in the github issue. selector: '"display-name"'

I tested the JsonPath parser on some test data. When a field selector finds no matching field in the json source, the result is an empty array. In that case, the process plugins attached to this field are not executed. If I change fetchNextRow to return an empty string, the process plugins are executed.

protected function fetchNextRow() {

...

$field_data = (new JSONPathSelector($current))->find($selector);

if ($field_data->count() == 1) {
  $field_data = $field_data->first();
} else {
   $field_data = $field_data->data();
}
if ( is_array($field_data) && count($field_data) == 0 ) {
  $field_data = '';
}

Path in #8 works well for my simple use case.

It has an issue though if you try to use a SubProcess because the \Drupal\migrate\Plugin\migrate\process\SubProcess::transform() method tries to merge a JsonPath object with an array here $new_row = new Row($new_value + $source);
I have no idea on how to fix this for now.

I had to dig a bit around this SubProcess problem. It seems that JSONPath:first() can return JSONPath objects and also that JSONPath:data() converts JSON objects to PHP stdClass objects. The problem is that the SubProcess plugin uses a "+" operator to merge the source so it fails.

The attached patch solves this issue by recursively converting data to nested arrays.

Patch was not applying anymore

The is_scalar() check from #16 converts NULL values to an array which is causing database exceptions. Updated patch checks is_null() as well.

Looks like the patch needs a re-roll for v5.1. Also, looks like flow/jsonpath lib has a newer 0.5.0 release.

Updated patch to work with v5.1.

Do not forget https://www.drupal.org/project/migrate_source_jsonpath which uses https://github.com/Galbar/JsonPath-PHP under the hood, which is for my use case the better library.

In flow/jsonpath there is no possibility (at least I don't found any, see https://github.com/FlowCommunications/JSONPath/pull/57#issuecomment-6890...) to filter based on sub-properties, like

$.something[?(@.contentType.key=='my-value')].value

Galbar/JsonPath-PHP supports this and many more features.

flow/jsonpath has been deprecated in favor of softcreatr/jsonpath so this patch needs to be updated. I've planned to do it in a two weeks if no one does it before.

Bumped to 5.x then made a straight reroll to be able to provide an eventual interdiff after that.

It was easier than expected ;)

I replace the dependency, placed it in the require section and reverted the change on the author URL that is out of scope.

I have not tested at all, and I have no opinion on which library to use (#22).

The attached patch is just a reroll of the one in #25, which no longer applies. I also replace

        "softcreatr/jsonpath": "^0.5 || ^0.7"

with

        "softcreatr/jsonpath": "^0.7.4"

Same result as the 8.x-5.x branch: the test passes with Drupal 9, not with Drupal 8.9. I think it is the version of PHPUnit that matters.

Back to NR.

#26 wasn't applying with the latest dev version so I did a re-roll.

Updated #29 patch as it wasn't adding the new file in the correct directory for me.

NW for some tests of the source plugin.

This updated patch uses JSONPath->getData() instead of JSONPath->data() and ensures compatibility to softcreatr/jsonpath 0.8.0.

The branch is updated with the patch from #32 + a test case checking the plugin behavior when handling a Json file with null values and missing properties.
The jsonpath expression used in the test for null values is similar to something we used in a real migration.
The test for missing properties is lifted from the regular Json plugin test.
Not sure if the tests are sufficient.

This needs work as the tests are confirming comment #14 behavior.

Reroll #32 for 5.3.x and pin dependency to softcreatr/jsonpath 0.8.*

Hello,

I applied #35 patch but it throws below errors while running the migration so I've fixed it and added the patch.

Please review.

Thanks!

Errors which I am getting and added above patch to resolve it

PHP Fatal error:  Declaration of Drupal\migrate_plus\Plugin\migrate_plus\data_parser\JsonPath::openSourceUrl($url) must be compatible with Drupal\migrate_plus\DataParserPluginBase::openSourceUrl(string $url): bool in /app/docroot/modules/contrib/migrate_plus/src/Plugin/migrate_plus/data_parser/JsonPath.php on line 59


PHP Fatal error:  Declaration of Drupal\migrate_plus\Plugin\migrate_plus\data_parser\JsonPath::fetchNextRow() must be compatible with Drupal\migrate_plus\DataParserPluginBase::fetchNextRow(): void in /app/docroot/modules/contrib/migrate_plus/src/Plugin/migrate_plus/data_parser/JsonPath.php on line 71

Attached is an update of #38 patch and rerolled for the 6.x dev branch.

The main change is I added a test for the parser. I also added some type declarations that the 6.x regular JSON parser had.

An interdiff is included, but there were a lot of whitespace changes so I excluded those from it.

I am updating the version to 6.0.x. I also created a new MR based on that branch and the patch in #40.

I installed the MR version of migrate_plus using the instructions above on a vanilla D10 site with the minimal profile and created a test content type to use as the destination for the test migration.

I created a module with a simple migration using a sample JSON file source (attached) and was successful in using a selector to limit the items which were migrated. The JSON contains an attribute named "favoriteFruit"; the item_selector in the migration selects those with favoriteFruit=strawberry. My migration only imported the items with the matching selector with no errors.

My simple migration:

id: test_data
label: Migrate Test Data

source:
  plugin: url
  data_fetcher_plugin: file
  data_parser_plugin: jsonpath
  urls:
    - modules/custom/migration_test/data/people.json

  item_selector: '$.people[?(@.favoriteFruit=strawberry)]'

  fields:
    -
      name: guid
      label: 'GUID'
      selector: guid
    -
      name: name
      label: 'Person name'
      selector: name
    -
      name: age
      label: 'Age'
      selector: age
    -
      name: gender
      label: 'Gender'
      selector: gender
    -
      name: company
      label: 'Company'
      selector: company
    -
      name: about
      label: 'About'
      selector: about
    -
      name: favorite_fruit
      label: 'Favorite Fruit'
      selector: favoriteFruit

  ids:
    guid:
      type: string

process:
  type:
    plugin: default_value
    default_value: people
  title: name
  field_age: age
  field_gender: gender
  field_company: company
  body/value: about
  body/format:
    plugin: default_value
    default_value: plain_text
  field_favorite_fruit: favorite_fruit

destination:
  plugin: 'entity:node'

migration_dependencies: { }
dependencies:
  enforced:
    module:
      - migration_test

@pixlcat:

Thanks for testing this, and for adding JSONPath links to the issue summary.

Since we looked at this issue at DrupalCon earlier this month, I am adding the tag for that.

I reviewed the changes. It mostly looks good, but I would like a little code cleanup. I am setting the status to NW, and I will re-review promptly. Let's try to get this issue done!

One of my review comments will require a bit of research and/or experimentation.

@benjifisher I guess I didn't set this back to needs review. Doing that here.

Should this new class extend the current JSON class instead of DataParserPluginBase? I did some quick testing and it seems this new plugin doesn't support the pager enhancements as it doesn't implement getNextUrls()?

Added a patch file based upon #50

Nice idea here. Added some comments on the MR.

Looks like the CI pipeline failure "composer (previous minor)" can be disregarded here - #3565496: pipeline failure - composer (previous minor)

Tested with an existing JSONPath migration locally, dataparser goes away when softcreatr/jsonpath is not a dependency of the current project, is available and works when it is. Further testing welcome!

The only worry I have with CI test failures is the cspell on dataparser. If we can add that to the allow list, then this looks good to me. The default previous version of drupal core is about to change any day now so I'm not too worried about getting that green.