Problem/Motivation

We had our site set at 3x-dev which was, admittedly, not a great idea. When working on a feed after module updates, we noticed that, on the Mappings page, when selecting a source, there was no longer an option for "New Source" in the dropdown.

We are running 3x-dev on Drupal 9.3.4 with PHP 8. When I go to the "custom sources" tab, it tells me to add it through the drop-downs on the Mappings page.

Something happened that isn't being caught by updb.

Switching to Alpha 11 has fixed the issue. I just wanted you to know that it's possible this might be breaking existing feeds.

Steps to reproduce

  1. Use a parser that does not extend the ParserBase class. For example the parser "Parser with mapping form" from the module "feeds_test_plugin".
  2. On alpha11, confirm that the option to add a new custom source is available.
  3. Update to the latest dev and run updb/cr as normal.
  4. The add new source option for the mappings dropdowns will be gone.

Proposed resolution

The issue can possibly be fixed by moving code from ParserBase to MappingForm. More specifically code from ParserBase::mappingFormAlter() and ParserBase::getCustomSourceOptions(). It's possible the implementation in Feeds Extensible Parsers need to be updated as well after this code has been moved.

Remaining tasks

  1. Add a functional test that checks if custom sources can be added when using the parser "Parser with mapping form" from the module "feeds_test_plugin".
  2. Add a comment to "ParserWithMappingForm" that it is intended to not extend ParserBase.
  3. Move some code from ParserBase to MappingForm (see Proposed resolution).
  4. Update Feeds Extensible Parsers code. See #3209655: Integration with feeds custom source UI.

This was originally a comment on 2938505

Issue fork feeds-3263399

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

loopy1492 created an issue. See original summary.

megachriz’s picture

Status: Active » Postponed (maintainer needs more info)

Which parser do you use?

Can you share a configuration file from the alpha11 feed type that didn't update well?

loopy1492’s picture

Hey, Chriz. Just to clarify, we have been set at 3x-dev. On production, we were running off a build that happened on 12/23/21. This config export should represent whatever commit was at the HEAD of that branch at that time. I'm not totally sure if that's the same as the Alpha 11 build.

uuid: b61f01c5-2c45-4793-a62f-d809288e9ffc
langcode: en
status: true
dependencies:
  config:
    - core.base_field_override.node.our_agency_tracking.title
    - field.field.node.our_agency_tracking.field_at_agency_id
    - field.field.node.our_agency_tracking.field_at_anticipated_comp_date
    - field.field.node.our_agency_tracking.field_at_applicant_name
    - field.field.node.our_agency_tracking.field_at_app_award_amount
    - field.field.node.our_agency_tracking.field_at_app_number
    - field.field.node.our_agency_tracking.field_at_commencement_date
    - field.field.node.our_agency_tracking.field_at_contract_executed
    - field.field.node.our_agency_tracking.field_at_grant_money_dispersed
    - field.field.node.our_agency_tracking.field_at_latitude
    - field.field.node.our_agency_tracking.field_at_longitude
    - field.field.node.our_agency_tracking.field_at_press_ready_description
    - field.field.node.our_agency_tracking.field_at_program_name
    - field.field.node.our_agency_tracking.field_at_project_name
    - field.field.node.our_agency_tracking.field_at_region
    - field.field.node.our_agency_tracking.field_at_round_key
    - field.field.node.our_agency_tracking.field_at_round_number
    - field.field.node.our_agency_tracking.field_project_status_color_
    - node.type.our_agency_tracking
  module:
    - node
    - redc_feed
_core:
  default_config_hash: kHJ6Z5xxsvNAfzQkV5q7ekhmXMlNtWa4E5r7SweX7FI
id: agency_tracking
label: 'Agency Tracking'
description: ''
help: ''
import_period: 604800
fetcher: http
fetcher_configuration:
  auto_detect_feeds: false
  use_pubsubhubbub: false
  always_download: false
  fallback_hub: ''
  request_timeout: 30
parser: tracking
parser_configuration: {  }
processor: 'entity:node'
processor_configuration:
  langcode: en
  update_existing: 2
  update_non_existent: 'entity:unpublish_action:node'
  expire: -1
  owner_feed_author: true
  owner_id: 0
  authorize: true
  skip_hash_check: true
  values:
    type: our_agency_tracking
custom_sources: {  }
mappings:
  -
    target: field_at_grant_money_dispersed
    map:
      value: grant_money_dispersed_to_date
  -
    target: field_at_app_award_amount
    map:
      value: app_award_amount
  -
    target: field_at_round_number
    map:
      value: round_number
    unique: {  }
  -
    target: field_at_press_ready_description
    map:
      value: press_ready_description
  -
    target: field_at_latitude
    map:
      value: latitude
  -
    target: title
    map:
      value: applicant_program_id
    unique:
      value: '1'
  -
    target: field_at_anticipated_comp_date
    map:
      value: anticipated_completion_date
    settings:
      timezone: UTC
  -
    target: field_at_contract_executed
    map:
      value: contract_executed
    settings:
      timezone: UTC
  -
    target: field_at_commencement_date
    map:
      value: commencement_date
    settings:
      timezone: UTC
  -
    target: field_at_round_key
    map:
      value: round_key
  -
    target: field_at_app_number
    map:
      value: app_number
    unique: {  }
  -
    target: field_at_longitude
    map:
      value: longitude
  -
    target: field_at_agency_id
    map:
      target_id: agency_id
    settings:
      reference_by: name
      autocreate: '1'
  -
    target: field_at_applicant_name
    map:
      value: applicant_name
    unique: {  }
  -
    target: field_at_region
    map:
      value: region
  -
    target: field_at_project_name
    map:
      value: project_name
    unique: {  }
  -
    target: field_at_program_name
    map:
      value: program_name
    unique: {  }
  -
    target: field_project_status_color_
    map:
      value: project_status
    unique: {  }
megachriz’s picture

The build on December 23, 2021 is the same as alpha11. The first commit since the alpha11 release was on January 22, 2022.

I see the parser is called "tracking". Does that parser extends \Drupal\feeds\Feeds\Parser\ParserBase? Because I think that would probably solve the issue. But if it doesn't, then it's strange that the option "New source" is not available anymore, because that logic was already in the ParserBase class.

Maybe I'll check to see what happens if you use a parser that doesn't extend that class.

loopy1492’s picture

Hey, Chriz.

It does appear that this site is using a custom parser. That's quite vexing. I appreciate your help on this. Some new fields I'm adding weren't being inserted and now I know why.

As to the other issue, right now the parser is using the following:

```
class TrackingParser extends PluginBase implements ParserInterface {
```

megachriz’s picture

Status: Postponed (maintainer needs more info) » Active

Ah, I see that it isn't extending ParserBase. I'll see if I can do something about custom parser plugins that only implement the interface.

megachriz’s picture

Assigned: Unassigned » megachriz
Issue summary: View changes
Issue tags: +beta blocker, +Release blocker

I've been able to reproduce the issue with the parser "Parser with mapping form" from the module "feeds_test_plugin" (can be enabled when activating settings['extension_discovery_scan_tests'] = TRUE; in settings.php). Adding a custom source is available on alpha11 for this parser, but not on latest dev.

So I think this needs to be fixed before the next release. Marking it as a beta blocker as well.

megachriz’s picture

Issue summary: View changes

Added another task to the remaining tasks list.

megachriz’s picture

Issue summary: View changes
Status: Active » Needs work

I've written a test for this case. The fix hopefully comes somewhere in the next days.

megachriz’s picture

Issue summary: View changes
Status: Needs work » Needs review

@loopy1492
Can you check if with the code in the issue fork, you see the option to add a custom source again on the mapping form? You may still need to update the parser code a bit to comply with the change in ParserInterface.

loopy1492’s picture

Thanks for your work on this @MegaChriz. I'll add it to my to-do list.

loopy1492’s picture

Hey, @MegaChriz!

In my local environment, I have switched my code to reference the fork commit and run updb from the db export from production. The option "New Blank Source" now appears in the Source dropdown on the mappings page and I can add one without error. Success!

I did notice that updb doesn't create entries on the "custom sources" tab for the existing sources, though. Shouldn't it be? Note that it's possible that I threw a monkey in the works because, in production, I had created a couple while it was set to the 3x-dev branch. So it's posible that the update script looked at our feed type, saw that there were greater than zero custom sources in there, and decided not to run that part of the update. Not sure.

  • MegaChriz committed 93533ac on 8.x-3.x
    Issue #3263399 by MegaChriz, loopy1492: Fixed "New Source" option gone...
megachriz’s picture

Status: Needs review » Fixed

Thanks for reporting back! Merged the changes.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.