I am having trouble importing ProPublica's representatives information into my drupal 8 site. The JSON request I am hitting is a nested array, I am looking to grab the second level array items to import into my site. But when I try to just call that item_selector ID (members) I keep on getting a error message:

[error] Could not retrieve source count from member: Passed variable is not an array or object

I know the issue falls on the item_selector field in my migrate_plus.migration.member.yml file,

But i've tried everything and it still returns that same error message:
members
/members
members/
/results/members
/results/members/
results/members
results.0.members

nothing appears to work.

I have tested my migration yml file with doing a simple request on data that falls under results array and that does work. I attached below a sample size of the JSON data I'm trying to request as well as my migrate_plus.migration.member.yml file minus my x-api-key.
Any help or guidance would be greatly appreciated.

{
  "status": "OK",
  "copyright": " Copyright (c) 2019 Pro Publica Inc. All Rights Reserved.",
  "results": [
    {
      "congress": "116",
      "chamber": "House",
      "num_results": 442,
      "offset": 0,
      "members": [
        {
          "id": "A000374",
          "title": "Representative",
          "short_title": "Rep.",
          "api_uri": "https://api.propublica.org/congress/v1/members/A000374.json",
          "first_name": "Ralph",
          "middle_name": null,
          "last_name": "Abraham",
          "suffix": null,
          "date_of_birth": "1954-09-16",
          "gender": "M",
          "party": "R",
          "leadership_role": null,
          "twitter_account": "RepAbraham",
          "facebook_account": "CongressmanRalphAbraham",
          "youtube_account": null,
          "govtrack_id": "412630",
          "cspan_id": "76236",
          "votesmart_id": "155414",
          "icpsr_id": "21522",
          "crp_id": "N00036633",
          "google_entity_id": "/m/012dwd7_",
          "fec_candidate_id": "H4LA05221",
          "url": "https://abraham.house.gov",
          "rss_url": "https://abraham.house.gov/rss.xml",
          "contact_form": null,
          "in_office": true,
          "dw_nominate": 0.524,
          "ideal_point": null,
          "seniority": "6",
          "next_election": "2020",
          "total_votes": 482,
          "missed_votes": 197,
          "total_present": 0,
          "last_updated": "2019-07-17 15:00:22 -0400",
          "ocd_id": "ocd-division/country:us/state:la/cd:5",
          "office": "417 Cannon House Office Building",
          "phone": "202-225-8490",
          "fax": null,
          "state": "LA",
          "district": "5",
          "at_large": false,
          "geoid": "2205",
          "missed_votes_pct": 40.87,
          "votes_with_party_pct": 94.37
        },
        {
          "id": "A000370",
          "title": "Representative",
          "short_title": "Rep.",
          "api_uri": "https://api.propublica.org/congress/v1/members/A000370.json",
          "first_name": "Alma",
          "middle_name": null,
          "last_name": "Adams",
          "suffix": null,
          "date_of_birth": "1946-05-27",
          "gender": "F",
          "party": "D",
          "leadership_role": null,
          "twitter_account": "RepAdams",
          "facebook_account": "CongresswomanAdams",
          "youtube_account": null,
          "govtrack_id": "412607",
          "cspan_id": "76386",
          "votesmart_id": "5935",
          "icpsr_id": "21545",
          "crp_id": "N00035451",
          "google_entity_id": "/m/02b45d",
          "fec_candidate_id": "H4NC12100",
          "url": "https://adams.house.gov",
          "rss_url": "https://adams.house.gov/rss.xml",
          "contact_form": null,
          "in_office": true,
          "dw_nominate": -0.468,
          "ideal_point": null,
          "seniority": "8",
          "next_election": "2020",
          "total_votes": 482,
          "missed_votes": 13,
          "total_present": 0,
          "last_updated": "2019-07-17 15:00:24 -0400",
          "ocd_id": "ocd-division/country:us/state:nc/cd:12",
          "office": "2436 Rayburn House Office Building",
          "phone": "202-225-1510",
          "fax": null,
          "state": "NC",
          "district": "12",
          "at_large": false,
          "geoid": "3712",
          "missed_votes_pct": 2.7,
          "votes_with_party_pct": 98.93
        }
      ]
    }
  ]
}
Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

azussman created an issue. See original summary.

Postovan Dumitru’s picture

Stuck in a similar situation at the moment.

Any way of altering the JSON at the endpoint?

The only other way I fixed it in the past was to:
1. Create a Controller and request the data from there, alter it the way I need it and then return a
return new JsonResponse($data);
2. The second way would be to extend the url source plugin and play around with the data there.

I might go for the 2nd solution, just because I won't have to think about additional security for that Controller.

Edit: was just thinking about it, it should be the data parser plugin, not the url one.

azussman’s picture

Thanks for the follow up.

Sadly, I don't have control over the JSON endpoint, so I can't alter anything there. I will try your 2 suggested options. I've got to tweak some of the data results anyways so those suggestion will be perfect.

Postovan Dumitru’s picture

Priority: Normal » Minor
Status: Active » Closed (works as designed)

Closing down.

Don't think it was a bug to begin with.

@azussman, hope you figured out what you needed to do.

apaderno’s picture

Issue tags: -JSON API Request, -3rd party integration +3rd party

I am removing the tags used from few issues. I apologize for bumping this issue.

gapple’s picture

Version: 8.x-4.2 » 8.x-4.x-dev
Assigned: azussman » Unassigned
Priority: Minor » Normal
Status: Closed (works as designed) » Needs review
Issue tags: -
FileSize
748 bytes

There's a bug in the item_selector use for arrays, where it's checking the selector segments with !empty($selector).
For a selector like results/0/members it's evaluating if 0 is empty (which is true), making it impossible to get data from the array.

It instead should be evaluating whether the source data array has the provided index.

jhchnc’s picture

This issue needs bumping.

Thank you for the patch. Seems like it would work.

Valid JSON can have multiple parents, and in some cases we can't change JSON endpoint structure as it's used by multiple legacy applications, not just Drupal websites.

With more and more people using Migrate in lieu of Feeds, this patch should really be considered. This is a related ticket from 3 years ago, with the same, but less robust patch.

https://www.drupal.org/project/migrate_plus/issues/2883803

heddn’s picture

Version: 8.x-4.x-dev » 8.x-5.x-dev

Any chance for a test on this? Plus, it didn't pass all the tests:

1 coding standards message
	
src/Entity/Migration.php
line 83	Line exceeds 80 characters; contains 81 characters
jhchnc’s picture

Hi @heddn.

I would be willing to work with you on that if no one else chimes in. I'm pretty new to Drupal module testing but have done plenty of testing within Symfony and Laravel.

In any case, we appreciate attention to this issue!

jhchnc’s picture

FileSize
673 bytes

Comment #6 patch was producing warnings during drush migrate:status

[warning] in_array() expects parameter 2 to be array, null given metatag.module:817

I've retained the array check from patch #6 but have omitted the return of an empty array, per the original code.

Patch attached.

(I've never submitted a patch to drupal so go easy on me! ha.)

r00t_’s picture

Tried using the patch (#10) as I ran into the same issue. Didn't work for me though. I can still only get the first array. I can't access the Members nested array at all. Tried:

item_selector: results

fields:
members/id
/0/members/id
and any combination of things I could think of, etc.

Has anyone had any luck getting around this issue? I'm still new to Drupal so I don't think I could just code up a whole solution to this myself. Just trying to pull some json.

Matroskeen’s picture

Version: 8.x-5.x-dev » 6.0.x-dev
Status: Needs review » Needs work

I'll open an MR with a test coverage to prove it is already working.

  • Matroskeen committed 688e576 on 6.0.x
    Issue #3068584 by Matroskeen, jhchnc, gapple, azussman: item_selector...

  • Matroskeen committed 74ab05d on 8.x-5.x
    Issue #3068584 by Matroskeen, jhchnc, gapple, azussman: item_selector...
Matroskeen’s picture

Version: 6.0.x-dev » 8.x-5.x-dev
Status: Needs work » Fixed

Here we go! The test coverage proves that it's possible to retrieve inner elements.

However it also shows the limitation of existing JsonParser - it can't retrieve multiple elements because the path should be hard coded to a specific element. Luckily, there is the ability to use "depth" item_selector, which is also covered in the test.

More advanced selection would be possible when we land #3007709: Add XPath-style filtering ability in JSON data parser plugin.

I think we can consider this one as Fixed.
Thank you everyone!

gapple’s picture

It looks like this was partially addressed outside an issue some time ago with this commit: https://git.drupalcode.org/project/migrate_plus/-/commit/0a1f9752aa39f5c...
(and later modified by #3050058: Failure with Json parser when item_selector not present)

However, I'm not sure why $selector needs to be checked with empty() - the conditional should be just fine and much simpler with just array_key_exists

Matroskeen’s picture

Status: Fixed » Needs work

Let's do it then :)

I'll be happy to a commit a one-line patch if tests are happy.

Matroskeen’s picture

Status: Needs work » Needs review
FileSize
750 bytes

Let's see how it goes.

  • Matroskeen committed 4c328b8 on 6.0.x
    Issue #3068584 by Matroskeen, jhchnc, gapple, azussman: item_selector...

  • Matroskeen committed 7723b1b on 8.x-5.x
    Issue #3068584 by Matroskeen, jhchnc, gapple, azussman: item_selector...
Matroskeen’s picture

Status: Needs review » Fixed

It seems fine. Thanks!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.