By sandstrom on

Could anyone please help me on Drupal 8 migration from URL.

The source is a URL which is returning JSON data without any authentication & I chosen Drupal 'entity:node' as destination.

Please see below the migration script I've written, the script seems correct to me as per Drupal 8 migration standard, not sure if I am missing something.

migrate_plus.migration_group.icf_taleo.yml

id: icf_taleo

# A human-friendly label for the group.
label: ICF Taleo

# More information about the group.
description: Import taleo jobs from the json feed.

source_type: Authenticated HTTP as JSON

dependencies:
  enforced:
    module:
      - icf_taleo_importer

migrate_plus.migration.taleo_jobs.yml

langcode: en
status: true
dependencies:
  module:
    - migrate_plus
  enforced:
    module:
      - icf_taleo_importer
id: taleo_jobs
migration_tags: null
migration_group: icf_taleo
source:
  plugin: url
  data_fetcher_plugin: http
  data_parser_plugin: json
label: 'Import taleo jobs from the json feed.'
source:
  ids:
    Id:
      type: integer
      unsigned: true
  item_selector: Results/
  urls: 'http://some-source-url'
  fields:
    -
      name: id
      label: "Id"
      selector: Id
    -
      name: url
      label: "Url"
      selector: Url
    -
      name: created_date
      label: "Created Date"
      selector: CreatedDate
    -
      name: taleo_field
      label: "Field"
      selector: Field
    -
      name: location
      label: Location"
      selector: Location
    -
      name: taleo_title
      label: "Title"
      selector: Title
    -
      name: schedule
      label: "Schedule"
      selector: Schedule
    -
      name: level
      label: "Level"
      selector: Level

process:
  title: Title
  field_job_id: Id
  field_job_url: Url
  field_job_field: Field
  field_job_location: Location
  field_job_schedule: Schedule
  uid:
    plugin: default_value
    default_value: 1

destination:
  plugin: 'entity:node'
  default_bundle: taleo_jobs

migration_dependencies: null

The above 2 files I've created in my_custom_module/config/install/

I am using migration, migrate_plus, migrate_tools modules to achieve this. I installed migrate_plus & migrate_tools modules via composer.

Facing couple of issues.

  • When I run drush ms I see the following. There is no data showing to be processed.
Group: icf_taleo  Status  Total  Imported  Unprocessed  Last imported
  • When I go to /admin/structure/migrate I am able to see migration group I made, but clicking on 'List migrations' button (admin/structure/migrate/manage/icf_taleo/migrations) throwing the below error which is getting difficult to debug.
The website encountered an unexpected error. Please try again later.

Drupal\Component\Plugin\Exception\PluginNotFoundException: The "" plugin does not exist. in Drupal\Core\Plugin\DefaultPluginManager->doGetDefinition() (line 52 of core/lib/Drupal/Component/Plugin/Discovery/DiscoveryTrait.php).
Drupal\Core\Plugin\DefaultPluginManager->getDefinition(NULL) (Line: 53)
Drupal\migrate\Plugin\MigratePluginManager->createInstance(NULL, Array, Object) (Line: 335)
Drupal\migrate\Plugin\Migration->getSourcePlugin() (Line: 107)
Drupal\migrate_tools\Controller\MigrationListBuilder->buildRow(Object) (Line: 229)
Drupal\Core\Entity\EntityListBuilder->render() (Line: 22)
Drupal\Core\Entity\Controller\EntityListController->listing('migration')
call_user_func_array(Array, Array) (Line: 123)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->Drupal\Core\EventSubscriber\{closure}() (Line: 574)
Drupal\Core\Render\Renderer->executeInRenderContext(Object, Object) (Line: 124)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->wrapControllerExecutionInRenderContext(Array, Array) (Line: 97)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->Drupal\Core\EventSubscriber\{closure}()
call_user_func_array(Object, Array) (Line: 135)
Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object, 1) (Line: 57)
Symfony\Component\HttpKernel\HttpKernel->handle(Object, 1, 1) (Line: 57)
Drupal\Core\StackMiddleware\Session->handle(Object, 1, 1) (Line: 47)
Drupal\Core\StackMiddleware\KernelPreHandle->handle(Object, 1, 1) (Line: 99)
Drupal\page_cache\StackMiddleware\PageCache->pass(Object, 1, 1) (Line: 78)
Drupal\page_cache\StackMiddleware\PageCache->handle(Object, 1, 1) (Line: 47)
Drupal\Core\StackMiddleware\ReverseProxyMiddleware->handle(Object, 1, 1) (Line: 50)
Drupal\Core\StackMiddleware\NegotiationMiddleware->handle(Object, 1, 1) (Line: 23)
Stack\StackedHttpKernel->handle(Object, 1, 1) (Line: 656)
Drupal\Core\DrupalKernel->handle(Object) (Line: 19)

Not getting any clue which non-existent plugin it's looking for.

I installed drupal console & when I run drupal debug:plugin I see the following migration related plugins working in the application.

  migrate.destination            Drupal\migrate\Plugin\MigrateDestinationPluginManager                         

  migrate.id_map                 Drupal\migrate\Plugin\MigratePluginManager                                    

  migrate.process                Drupal\migrate\Plugin\MigratePluginManager                                    

  migrate.source                 Drupal\migrate\Plugin\MigrateSourcePluginManager                              

  migrate_plus.data_fetcher      Drupal\migrate_plus\DataFetcherPluginManager                                  

  migrate_plus.data_parser       Drupal\migrate_plus\DataParserPluginManager                                   

  migration                      Drupal\migrate\Plugin\MigrationPluginManager  

   

Seems all the required plugins are present. Checked throughly & googled a lot, I think the migration configurations are correct, not sure if am doing any mistake.

Please revert back if anyone having some idea on this, your help would really be appreciated. 

Thanks

Sandip

⋅ Categories: Drupal 8.x

Comments

sandstrom’s picture

sandstrom commented

I was able to fix the issue. The problem was with the migration configuration. I wrote 'source' 2 times by mistake.

Thanks

sandstrom’s picture

sandstrom commented

Hi,

I am facing a similar kind of issue. Trying to migrate data from a remote url which returns result in json format & the url doesn't have any authentication mechanism, but I have to be in office network or on VPN in order to get the result. I can see result when access the url from browser or postman plugin directly without any authentication. I am using migrate, migrate_plus & migrate_tools modules.

Here is the data structure of the result.

{
	"Results": [
		{
			"Id": "1800000474",
			"Url": "https://example.com/careersection/1",
			"CreatedDate": "2018-02-08T02:46:16.0000000Z",
			"Field": "Energy",
			"Location": "Fairfax, Virginia, United States",
			"Title": "Power Market Modeler and Analyst",
			"Schedule": "Full-time",
			"Level": "",
			"LocationTree": [
				"6560750684",
				"6360750684",
				"6160750684"
			],
			"FieldTree": [
				"11860750684"
			]
		},
		{
			"Id": "1800000473",
			"Url": "https://example.com/careersection/2",
			"CreatedDate": "2018-02-07T19:58:58.0000000Z",
			"Field": "Cyber Security",
			"Location": "Adelphi, Maryland, United States",
			"Title": "Network Security Analyst",
			"Schedule": "Full-time",
			"Level": "",
			"LocationTree": [
				"296560700958",
				"7160750684",
				"6160750684"
			],
			"FieldTree": [
				"60160700958"
			]
		}
        ]
}

Following is the migration configuration code I've written.

id: taleo_jobs
migration_tags: null
migration_group: icf_taleo
label: 'Import taleo jobs from the json feed.'
source:
  plugin: url
  data_fetcher_plugin: http
  data_parser_plugin: json
  urls: 'http://example.com/api/careerssearch/fetch?key=G3xCSJTVhmhKapfeorp8'
  item_selector: Results/
  ids:
    job_id:
      type: integer
  fields:
    -
      name: job_id
      label: 'Job Id'
      selector: /Id
    -
      name: job_url
      label: 'Job Url'
      selector: /Url
    -
      name: created_date
      label: 'Created Date'
      selector: /CreatedDate
    -
      name: job_field
      label: 'Job Field'
      selector: /Field
    -
      name: job_location
      label: 'Job Location'
      selector: /Location
    -
      name: job_title
      label: 'Job Title'
      selector: /Title
    -
      name: job_schedule
      label: 'Job Schedule'
      selector: /Schedule
    -
      name: job_level
      label: 'Job Level'
      selector: /Level
    -
      name: job_Location_tree
      label: 'Job Location Tree'
      selector: /LocationTree
    -
      name: job_field_tree
      label: 'Job Field Tree'
      selector: /FieldTree

process:
  type:
    plugin: default_value
    default_value: taleo_jobs
  title: job_title
  field_job_id: job_id
  field_job_url: job_url
  field_job_field: job_field
  field_job_location: job_location
  field_job_schedule: job_schedule
  uid:
    plugin: default_value
    default_value: 1

destination:
  plugin: 'entity:node'

migration_dependencies: {}

dependencies:
  enforced:
    module:
      - icf_taleo_importer

When I run drush ms, I see no data available to be processed. But in real there are more than 400 records available. 

 Group: icf_taleo  Status  Total  Imported  Unprocessed  Last imported       
 taleo_jobs        Idle    1      0         1

If I visit /admin/structure/migrate/manage/icf_taleo/migrations page, I see the below notices for every source field I defined. I suspect these notices are responsible for not showing any data to be processed from the remote json url.

Notice: Undefined index: /id in Drupal\migrate_plus\Plugin\migrate_plus\data_parser\Json->fetchNextRow() (line 106 of modules/contrib/migrate_plus/src/Plugin/migrate_plus/data_parser/Json.php).
Drupal\migrate_plus\Plugin\migrate_plus\data_parser\Json->fetchNextRow() (Line: 111)
Drupal\migrate_plus\DataParserPluginBase->next() (Line: 95)
Drupal\migrate_plus\DataParserPluginBase->rewind() (Line: 196)
Drupal\migrate_plus\DataParserPluginBase->count() (Line: 455)
Drupal\migrate\Plugin\migrate\source\SourcePluginBase->doCount() (Line: 414)
Drupal\migrate\Plugin\migrate\source\SourcePluginBase->count() (Line: 108)
Drupal\migrate_tools\Controller\MigrationListBuilder->buildRow(Object) (Line: 229)
Drupal\Core\Entity\EntityListBuilder->render() (Line: 22)
Drupal\Core\Entity\Controller\EntityListController->listing('migration')
call_user_func_array(Array, Array) (Line: 123)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->Drupal\Core\EventSubscriber\{closure}() (Line: 574)
Drupal\Core\Render\Renderer->executeInRenderContext(Object, Object) (Line: 124)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->wrapControllerExecutionInRenderContext(Array, Array) (Line: 97)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->Drupal\Core\EventSubscriber\{closure}()
call_user_func_array(Object, Array) (Line: 135)
Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object, 1) (Line: 57)
Symfony\Component\HttpKernel\HttpKernel->handle(Object, 1, 1) (Line: 57)
Drupal\Core\StackMiddleware\Session->handle(Object, 1, 1) (Line: 47)
Drupal\Core\StackMiddleware\KernelPreHandle->handle(Object, 1, 1) (Line: 99)
Drupal\page_cache\StackMiddleware\PageCache->pass(Object, 1, 1) (Line: 78)
Drupal\page_cache\StackMiddleware\PageCache->handle(Object, 1, 1) (Line: 47)
Drupal\Core\StackMiddleware\ReverseProxyMiddleware->handle(Object, 1, 1) (Line: 50)
Drupal\Core\StackMiddleware\NegotiationMiddleware->handle(Object, 1, 1) (Line: 23)
Stack\StackedHttpKernel->handle(Object, 1, 1) (Line: 656)
Drupal\Core\DrupalKernel->handle(Object) (Line: 19)

Notice: Undefined index: /Url in Drupal\migrate_plus\Plugin\migrate_plus\data_parser\Json->fetchNextRow() (line 106 of modules/contrib/migrate_plus/src/Plugin/migrate_plus/data_parser/Json.php).
Drupal\migrate_plus\Plugin\migrate_plus\data_parser\Json->fetchNextRow() (Line: 111)
Drupal\migrate_plus\DataParserPluginBase->next() (Line: 95)
Drupal\migrate_plus\DataParserPluginBase->rewind() (Line: 196)
Drupal\migrate_plus\DataParserPluginBase->count() (Line: 455)
Drupal\migrate\Plugin\migrate\source\SourcePluginBase->doCount() (Line: 414)
Drupal\migrate\Plugin\migrate\source\SourcePluginBase->count() (Line: 108)
Drupal\migrate_tools\Controller\MigrationListBuilder->buildRow(Object) (Line: 229)
Drupal\Core\Entity\EntityListBuilder->render() (Line: 22)
Drupal\Core\Entity\Controller\EntityListController->listing('migration')
call_user_func_array(Array, Array) (Line: 123)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->Drupal\Core\EventSubscriber\{closure}() (Line: 574)
Drupal\Core\Render\Renderer->executeInRenderContext(Object, Object) (Line: 124)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->wrapControllerExecutionInRenderContext(Array, Array) (Line: 97)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->Drupal\Core\EventSubscriber\{closure}()
call_user_func_array(Object, Array) (Line: 135)
Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object, 1) (Line: 57)
Symfony\Component\HttpKernel\HttpKernel->handle(Object, 1, 1) (Line: 57)
Drupal\Core\StackMiddleware\Session->handle(Object, 1, 1) (Line: 47)
Drupal\Core\StackMiddleware\KernelPreHandle->handle(Object, 1, 1) (Line: 99)
Drupal\page_cache\StackMiddleware\PageCache->pass(Object, 1, 1) (Line: 78)
Drupal\page_cache\StackMiddleware\PageCache->handle(Object, 1, 1) (Line: 47)
Drupal\Core\StackMiddleware\ReverseProxyMiddleware->handle(Object, 1, 1) (Line: 50)
Drupal\Core\StackMiddleware\NegotiationMiddleware->handle(Object, 1, 1) (Line: 23)
Stack\StackedHttpKernel->handle(Object, 1, 1) (Line: 656)
Drupal\Core\DrupalKernel->handle(Object) (Line: 19)

..............

Not sure if I am doing any mistake. Your help would be appreciated.

Thanks
Sandip

sandstrom’s picture

sandstrom commented

I was able to fix the issue & can be tracked in https://www.drupal.org/project/migrate_plus/issues/2940236