Problem/Motivation

Whever I try the upgrade, it hangs at 38%. Checking the error log, I see multiple entries with the message "Missing filter plugin: filter_null."

I know that the PHP plugin was enabled on the old site, so not sure if it's related to that. I did try enabling the contrib version of the D8 PHP filter, but php still showed up as a missing dependency, and there was no change in the import.

Proposed resolution

When replacing a filter with filter_null, be sure we are stripping out any settings that may fail a validation check.

Remaining tasks

User interface changes

API changes

Data model changes

Release notes snippet

In previous versions of Drupal 8, if a corresponding D8 filter was not found for a D6/D7 filter, it would get mapped to filter_null, but the settings would be retained. This "mixed-mapping" was a bug, which could later lead to configuration schema validation errors. The new behaviour is to instead drop the filter settings, with a message directing users to either re-configure the format by hand, or attempt the migration again with a D8 version of the dropped filter in question.

Issue fork drupal-2946889

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

NiklasBr created an issue. See original summary.

jcnventura’s picture

jcnventura’s picture

Status: Active » Closed (duplicate)
NiklasBr’s picture

Status: Closed (duplicate) » Active

#2618332 didn't fix the issue as described. Which is why I opened a new one.

heddn’s picture

Issue tags: -Migrate critical

Reviewed this in the migrate maintainers meeting. While this is important, it isn't migrate critical. Removing that tag.

NiklasBr’s picture

Question.
The original issue (#2618332) was/is tagged Migrate critical. This issue is a 1:1 copy of that issue because that issue did not resolve the reported bug, why is it no longer critical?

heddn’s picture

That issue was about how to handle null filters so it didn't skip importing any when it encountered a bad one or two. Additional logging was added as well. But it doesn't mean that null filters were all be fixed. There will still be the case were filters won't exist on the d8 site and it will fallback on the default null filter.

Rather than copy/pasting the IS from that previous issue, can you explain what you are seeing and what you were hoping to see?

NiklasBr’s picture

I am seing exactly the same error message as the issue summary says, everything as far as I can see is identical to the previous issue. I see no reason not to copy it. I see:

"Missing filter plugin: filter_null."

And then the migration fails. I was hoping to see no error messages.

SELECT DISTINCT node_revisions.format, filter_formats.*
FROM node_revisions
LEFT JOIN filter_formats
	ON filter_formats.format = node_revisions.format

0	NULL	NULL		NULL		NULL
5	5	Markdown	,2,3,8,4,5,9,7,	0
4	4	Full HTML	,3,4,5,10,13,	1
6	6	Plain text	,1,2,		1
SELECT * FROM filter_formats;

4	Full HTML	,3,4,5,10,13,	1
5	Markdown	,2,3,8,4,5,9,7,	0
6	Plain text	,1,2,		1
SELECT * FROM filters;

73	6	messaging	0	10
69	4	filter		2	0
68	4	filter		1	1
64	5	shortcode	0	-10
63	5	markdown	0	-9
62	5	filter		1	10
67	4	filter		3	10
61	5	filter		0	-7
66	4	codefilter	0	0
60	5	filter		3	-5
59	5	codefilter	0	-6
71	6	filter		0	-7
72	6	filter		1	10
70	6	filter		3	-5
65	5	filter		2	-8
heddn’s picture

Those are probably coming from shortcode, markdown or codefilter.

NiklasBr’s picture

Maybe. I don't know. How would the message "Missing filter plugin: filter_null." be related to those modules and how would we know and why does it cancel all migrations?

SELECT system.name, system.status
FROM system
WHERE system.name LIKE "%code%"
codefilter 			0
coder 				0
shortcode			0
shortcode_basic_tags		0
shortcode_embed_contents	0

We do not use them in Drupal 6 and do not plan on using them in Drupal 8.

heddn’s picture

It shouldn't cancel all migrations. I'm pretty sure the canceling is another issue unrelated to these messages in the logs.

NiklasBr’s picture

This is the only error message, then it stops. Maybe there is some other issue, but the log does not show any other error.

NiklasBr’s picture

Version: 8.4.x-dev » 8.6.x-dev

Issue still in D8.6.x-dev.

Version: 8.6.x-dev » 8.7.x-dev

Drupal 8.6.0-alpha1 will be released the week of July 16, 2018, which means new developments and disruptive changes should now be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

jcnventura’s picture

@NiklasBr, what exactly do you mean with 'migration fails'? Can you provide the output from this 'select * from migrate_message_d6_filter_format;'.

jcnventura’s picture

@heddn, if I can hijack this issue a bit, since it clearly seems to be the best one. My scenario is the following:

  • Drupal 7 site with the 'picture' module to handle responsive images. The picture module was eventually added to core and is now called 'responsive_image'. The D7 module defined an input filter named 'picture' that would transform images with data-picture-mapping='__id__' to a responsive format.
  • Migration works mostly fine, but because of the presence of the 'picture' filter I get the following errors during the migrate-upgrade:
    [error] Missing filter plugin: filter_null.
    And in migrate_message_d7_filter_format:
    Filter picture could not be mapped to an existing filter plugin; defaulting to filter_null.
  • After the migration finishes, I see no content on the site homepage, only titles. Editing the node shows the content.
  • Only after saving the format without any changes in /admin/config/content/formats/manage/full_html, does content show up.

The issue here is that since picture is now in core, it should be core's task to do a proper filter migration. I know that at one point this will be handled by #2822389: Allow responsive image style to be selected in Text Editor's image dialog (necessary for structured content), but until that point, there should be a 'safe' filter that would not remove all text, but simply show it as is. This would probably start by using a mapping between 'picture' and 'filter_none' (as opposed to 'filter_null').

aiphes’s picture

I get this Alert on dblog : Plugin de filtre manquant : filter_null. On node edit mode and on this kind of ting: quickedit/attachments?_wrapper_format=drupal_ajax

Then it break Coffee with message inside it "Could not load data, please refresh the page" (see https://www.drupal.org/project/coffee/issues/3000611).
Using Drush 8.1.6 and Drupal 8.6.1.UPgrading from D6 with drush.D8 PHP filter enabled.
Remove PHP filter from the incrimined filter doesn't change things, Coffee work randomly.
Other information, unable to edit a field settings in admin/structure/types/manage/XXX/form-display

In DB, migrate_message_d6_filter_format and migrate_message_upgrade_d6_filter_format tables:

Le filtre pathologic:0 n'a pas pu être associé à un plugin de filtre existant ; repli sur filter_nul...
  Le filtre php:0 n'a pas pu être associé à un plugin de filtre existant ; repli sur filter_null.
 Le filtre pathologic:0 n'a pas pu être associé à un plugin de filtre existant ; repli sur filter_nul...
 Le filtre php:0 n'a pas pu être associé à un plugin de filtre existant ; repli sur filter_null.
 Le filtre swftools:0 n'a pas pu être associé à un plugin de filtre existant ; repli sur filter_null.
siramsay’s picture

I was getting this same error "Missing filter plugin: filter_null."

#16
"Only after saving the format without any changes in /admin/config/content/formats/manage/full_html... ". worked form me.

+ removing/disabling some other filters too such as PHP filter.

Version: 8.7.x-dev » 8.8.x-dev

Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

drupalninja99’s picture

Is there a way to change the error from "error" to "warning"? Right now it stops the migration, I think this should be more of the "warning" category that won't interrupt migrations.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Wim Leers’s picture

Status: Active » Needs review
Issue tags: +migrate-d7-d8
FileSize
2.47 KB
171.36 KB

This is not yet perfect, but it does do what the issue title says: it improves the situation.

The reason that some filters can cause a migration to completely fail is that there may be config schema violation errors when a D6/D7 filter's "settings" are migrated when that filter does not exist on the destination D8 site. The D8 site will apply config schema validation, and the filter_null filter does not have any settings, therefore any settings being present causes it to fail completely.

So the only feasible solution is quite simple: drop those settings and warn the user explicitly (which should've already been the case, but alas), like so:

This still results in body fields (or any other field using formatted/processed text) that uses a text format with at least one filter_null to show up as empty. But at least now the migration is successful, and you can go and edit those Drupal 8 text formats.

Status: Needs review » Needs work

The last submitted patch, 22: 2946889-22.patch, failed testing. View results

Wim Leers’s picture

Status: Needs work » Needs review
FileSize
1.13 KB
3.55 KB
mikelutz’s picture

Title: Better handle replacement of missing filters with filter_null » Missing migration filters that are replaced with filter_null may have invalid settings applied
Priority: Major » Normal
Issue summary: View changes
Status: Needs review » Needs work
Issue tags: +Needs tests

This seems a sane and obviously correct issue and fix.

Setting to NW for tests.

Wim Leers’s picture

Status: Needs work » Needs review
Issue tags: -Needs tests
FileSize
827 bytes
4.32 KB

The last submitted patch, 26: 2946889-26-test_only_FAIL.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

quietone’s picture

Yes, nice fix. I like the more informative error message.
Filter @plugin_id could not be mapped to an existing filter plugin; defaulting to @fallback and dropping all settings. Either redo the migration with a viable successor, or modify the text format after the migration to remove this filter if it is no longer necessary.

I'd prefer if the comment didn't use the future perfect tense but that is preference. The comment is correct.

I found only one nit, which I fixed.

+++ b/core/modules/filter/tests/src/Kernel/Plugin/migrate/process/FilterSettingsTest.php
@@ -96,6 +96,13 @@ public function dataProvider() {
+      ]

Needs trailing comma.

Status: Needs review » Needs work

The last submitted patch, 28: 294688-28.patch, failed testing. View results

quietone’s picture

Sorry, bad patch. Will fix later today.

quietone’s picture

Status: Needs work » Needs review
FileSize
531 bytes
3.97 KB

Right, try this again.

Wim Leers’s picture

Given you're a Migrate maintainer, and you only fixed a nit, I think you're eligible to RTBC this 🤓

quietone’s picture

Status: Needs review » Reviewed & tested by the community

@Wim Leers, thanks. I was just waiting to make sure I hadn't make another mistake.

This is good to do. My review is in #28

Wim Leers’s picture

🥳

heddn’s picture

Status: Reviewed & tested by the community » Needs work

Sorry to kick this back on pedantics.

+++ b/core/modules/filter/src/Plugin/migrate/process/FilterID.php
@@ -77,7 +77,8 @@ public function transform($value, MigrateExecutableInterface $migrate_executable
+      $message = $this->t('Filter @plugin_id could not be mapped to an existing filter plugin; defaulting to @fallback and dropping all settings. Either redo the migration with a viable successor, or modify the text format after the migration to remove this filter if it is no longer necessary.', [

Nit (and yes I know that's how we did this before...) But we don't typically translate error messages. Can we use sprintf instead?

Instead of viable successor, could we say "format that exists on the destination site", or whatever we want them to do? I'm not sure we are explicit enough for folks to know what to do when we just say "viable successor". Alternatively, add a docs page and refer to it via a URL.

Wim Leers’s picture

#35++ 🤓😊

How about:
Filter @plugin_id could not be mapped to an existing filter plugin; defaulting to @fallback and dropping all settings. Either redo the migration with the module installed that provides an equivalent filter, or modify the text format after the migration to remove this filter if it is no longer necessary.

heddn’s picture

#36++ Let's go with that wording, unless someone speaks up.

Wim Leers’s picture

Issue tags: +Novice, +Needs reroll

#37 👍 🙂 — tagging to let somebody new to core update the patch!

quietone’s picture

Issue tags: +DrupalSouth 2019

Adding DrupalSouth tag.

jfussion’s picture

Assigned: Unassigned » jfussion

I'm working on this one. (DrupalSouth 2019)

jfussion’s picture

Assigned: jfussion » Unassigned
Status: Needs work » Needs review
FileSize
1.38 KB
4.14 KB

I have made changes from #36.

Thanks @jibran for helping me with creating the patch!

Status: Needs review » Needs work

The last submitted patch, 41: 2946889-41.patch, failed testing. View results

jfussion’s picture

Updated the patch from #41 to fix the failing test.

Thanks @dpi for the help.

jfussion’s picture

Status: Needs work » Needs review
quietone’s picture

Status: Needs review » Needs work

@jfussion, thanks for the patch. I noticed the size of the patch in #43 was much smaller than in #41. And after taking a look it seems that the interdiff is the patch and the patch is the interdiff.

jfussion’s picture

@quietone, thanks for the review. Here are the update interdiff and patch.

jfussion’s picture

Status: Needs work » Needs review
heddn’s picture

Status: Needs review » Needs work
  1. +++ b/core/modules/filter/src/Plugin/migrate/process/FilterID.php
    @@ -77,10 +77,8 @@ public function transform($value, MigrateExecutableInterface $migrate_executable
    +      $message = sprintf('Filter %s could not be mapped to an existing filter plugin; defaulting to %s and dropping all settings. Either redo the migration with the module installed that provides an equivalent filter, or modify the text format after the migration to remove this filter if it is no longer necessary.', $plugin_id, $fallback);
           $migrate_executable->saveMessage((string) $message, MigrationInterface::MESSAGE_WARNING);
    

    Nit: no need to cast to string.

  2. +++ b/core/modules/filter/tests/src/Kernel/Plugin/migrate/process/FilterIdTest.php
    @@ -66,7 +66,7 @@ public function testTransform($value, $expected_value, $invalid_id = NULL) {
    -          'Filter ' . $invalid_id . ' could not be mapped to an existing filter plugin; defaulting to filter_null.',
    +          'Filter ' . $invalid_id . ' could not be mapped to an existing filter plugin; defaulting to filter_null and dropping all settings. Either redo the migration with the module installed that provides an equivalent filter, or modify the text format after the migration to remove this filter if it is no longer necessary.',
    
    +++ b/core/modules/filter/tests/src/Kernel/Plugin/migrate/process/FilterSettingsTest.php
    @@ -96,6 +96,13 @@ public function dataProvider() {
    +      [
    +        [
    +          'foo' => 'bar',
    +        ],
    +        'filter_null',
    

    This could probably also use the same sprintf pattern as aboove. As I don't think we want to just hard code filter_null when we have the option to make this dynamic based on the data provider.

kostyashupenko’s picture

Issue tags: -Needs reroll
Wim Leers’s picture

Issue tags: +Needs reroll

Needs another reroll, for the additional nits described in #48.

NiklasBr’s picture

@NiklasBr, what exactly do you mean with 'migration fails'? Can you provide the output from this 'select * from migrate_message_d6_filter_format;'.

@jcnventura I gave up and tried now again with 8.8.0 available, but migrations still fails/choke/stop/crash on null filters. It stops.

jofitz’s picture

Status: Needs work » Needs review
Issue tags: -Needs reroll
FileSize
2.27 KB
4.25 KB

Addressed @heddn's code review comments from #48.

heddn’s picture

Status: Needs review » Reviewed & tested by the community

All feedback now addressed. Thanks for picking up the nits.

  • webchick committed 60b75a6 on 9.0.x
    Issue #2946889 by Wim Leers, jfussion, quietone, jofitz, heddn: Missing...

  • webchick committed 12517d0 on 8.9.x
    Issue #2946889 by Wim Leers, jfussion, quietone, jofitz, heddn: Missing...

  • webchick committed 06b5144 on 8.8.x
    Issue #2946889 by Wim Leers, jfussion, quietone, jofitz, heddn: Missing...
webchick’s picture

Issue summary: View changes
Status: Reviewed & tested by the community » Fixed
Issue tags: +8.8.2 release notes

Oooh, I love the error message improvements here very much!!

This bug fix represents a behaviour change from previous versions, so I'm tagging for release notes and making my best attempt at a release notes snippet.

@NiklasBr, could you try testing the 8.8.x-dev release (in about 12 hours) and see if it now resolves your issue? And if not, open a specific support request that we can try to dig in on?

Committed and pushed to 9.0.x; 8.9.x; 8.8.x. Thanks!

webchick’s picture

Version: 8.9.x-dev » 8.8.x-dev

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

donm’s picture

In case it helps others: Cleaning up log messages re "Missing filter plugin: filter_null." Context: Currently Drupal 8.8.1, with a database migrated out of D7 into D8 in whatever versions were current 2 years ago. In my case, I was getting 4 such log entries, and it seems that they were generated from 2 separate sources:

  1. PHP Filter: The D7 site had used a PHP snippet in a block. Which I had deleted in moving to D8. Per messages above in this thread, I tried to reinstall PHP Filter, which generated a message re "can't install because the particular filter already exists in current configuration". I used Easy Install module to purge that configuration. That left me with 2 log messages still being generated.
  2. Per #16 above, re-saving each text format (configuration/content authoring/text formats and editors): In my case, it was the "Display Suite code" text format that contained the message re missing filter null and that it would be removed once that format was saved. The previous log entries had no pointer to that particular page. But once I started doing the text format re-save, log entries were generated that provided a pointer to that page.
brooke_heaton’s picture

I'm getting this error on a Drupal 7 to Drupal 8 upgrade:

Is there a way to get around this issue? Can a default filter be set?

brooke_heaton’s picture

Still seeing this error on 8.9.16.

alexdmccabe made their first commit to this issue’s fork.

ressa’s picture

I saw "Missing filter plugin: filter_null" alerts in the log after a migration from Drupal 7 to Drupal 9, and saving all text formats seems to have solved it. I switched fallback_format from the old ('4') to new ('plain_text') with this:

# verify original value
$ drush config:get filter.settings fallback_format
'filter.settings:fallback_format': '4'

# set to new format
$ drush config:set filter.settings fallback_format 'plain_text'
Grevil’s picture

Small hint if anyone is still having this issue during a Drupal 7 to Drupal 9 Migration:

You can skip the empty filters through migration, by adding the "skip_on_empty" plugin in your "upgrade_d7_filter_format" migration yaml.
Either by directly inserting it in the process pipeline:

process:
  filters:
    -
      plugin: skip_on_empty
      method: row
    -
      plugin: sub_process
      source: filters
      [...]

Or through the usage of hook_migrations_plugins_alter():

  foreach ($migrations as $migrationId => &$migration) {
    if ($migrationId == 'upgrade_d7_filter_format') {
      array_unshift($migration['process']['filters'], [
        'plugin' => 'skip_on_empty',
        'method' => 'row',
      ]);
    }
  }