XML and JSON sources may have very different syntaxes, but are similar in terms of use cases - they are the most common formats used by web services (indeed, generally a web service may be requested either as XML or as JSON), they may provided paged feeds, they may follow a list/item pattern (read one endpoint to get a list of individual items to retrieve from another endpoint), etc. For general DX, and in particular to make it easy for migration developers to switch a migration between formats, we should try to make at least the YAML configuration for each source plugin as consistent as possible. Also, they should be functionally comparable - for example, right now the JSON plugin has some support for the list/item pattern, as well as the ability to provide custom headers (such as authentication tokens) on requests - both capabilities the XML plugin should share.
This is a meta issue for the JSON source plugin side - child issues will be specific changes to be made to the JSON source plugin to help bring it in line with the XML source plugin.
Comments
Comment #2
mikeryanI'm thinking that the XML and JSON source plugins should inherit from a common base class to address as many of these issues as possible: #2623012: Implement interfaces and base classes for URL-based sources.
Comment #3
mikeryanMaking progress - see https://www.drupal.org/node/2623012#comment-10634654 for thoughts on the new architecture. Note that the current state of this is in the 'refactor' branch of migrate_source_json - you'll all so need the 'refactor_source_plugins' branch from migrate_plus to use it.
Comment #4
karens commentedI tried out the refactor branch and it breaks many tests. In addition when I try to pass in custom headers they just get lost, nothing is passing them on to the client any more. And there used to be a way to pass in a custom class for the client and that's gone now. So this still needs work. I'll see if I can figure out some of the changes that are needed.
Comment #5
mikeryanYes, it's still very much a WIP (latest update at https://www.drupal.org/node/2623012#comment-10671346). The client class and custom header support should move into the generic support in migrate_plus, I haven't tackled that yet but they are essential for the project that's prompting my work here.
Comment #6
mikeryanPer https://www.drupal.org/node/2623012#comment-10693158, I've moved the JSON
reader/extractor/parser plugin into migrate_plus.