Problem / motivation

This is a spin off from https://www.drupal.org/node/2853872#comment-12224562. When testing #2853872: Migration for forum and article comments: duplicate comment types and incorrect comment_entity_statistics, I noticed that all forum threads were closed in D8.5.x

Every node has a setting which defines if the comments are open or closed. We are talking about this field in the UI:

Open = value 2
Closed = value 1

The node migrations, for example upgrade_d7_node_forum do not have any migration process for this setting. As a result, every node falls back to the default value of the comment type in question. See root cause analysis in comment #3.

Notes about related issues:
#2887043: D7 comment field instance migration: incorrect mapping for open/closed/hidden default value is not the root cause for this bug because that issue is talking about the default value and this issue is talking about individual nodes.

Proposed resolution

All node migrations (e.g. upgrade_d7_node_forum, upgrade_d7_node_article) should have a process for migrating this node setting. Source in D7 database is comment column in {node} table.

What makes this a bit tricky is that
a) D8 Forum has its own hard coded comment type comment_forum
b) Other D7-D8 migrations create separate comment types / fields with comment_node_{node_type} pattern.

Remaining tasks

Investigate further and identify root cause
Agree approach
Patch so that the comment types are properly handled
Add tests
Review
Commit
Dance all night

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

masipila created an issue. See original summary.

masipila’s picture

Title: All forum threads closed after migration from D7 » Comment "open / closed" stauts not migrated in node migrations
Issue summary: View changes
Priority: Normal » Major
Issue tags: +migrate-d7-d8
FileSize
18.37 KB

First quick glance to figure out the root cause.

We are talking about this D8 node field in UI:

When this setting is open for an individual node, the forum node has this when the node is checked with Devel.

  [comment_forum] => Array
                (
                    [x-default] => Array
                        (
                            [0] => Array
                                (
                                    [status] => 2
                                    [cid] => 0
                                    [last_comment_timestamp] => 1470864261
                                    [last_comment_name] => 
                                    [last_comment_uid] => 174
                                    [comment_count] => 0
                                )

                        )

                )

When the setting is closed, we the status is 1.

I looked at the migration upgrade_d7_node_forum. This migration process does not have any reference to the comment type field. As a result, the default value of the comment type is used for every node.

In plain English:

  • In Drupal 7 we have the option to define for each node separately if the comments are open or closed
  • Forum moderator can for example close "lock" a topic
  • The node migrations are not migrating this setting at all and as a result D8 falls back to the default value of the comment type.
  • I would consider this as Major. Module maintainers to evaluate and set the priority accordingly.
masipila’s picture

Issue summary: View changes

Minor updates to issue summary.

masipila’s picture

Title: Comment "open / closed" stauts not migrated in node migrations » Node "comment open / closed" status not migrated
masipila’s picture

Assigned: Unassigned » masipila

Reading the API, it looks to me that this mapping needs to be added to D7NodeDeriver::getDerivativeDefinitions.

Assinging to myself. I'll try to write the first draft of the patch in the next couple of days.

Cheers,
Markus

masipila’s picture

Assigned: masipila » Unassigned
Status: Active » Needs review
Issue tags: +Needs tests
FileSize
1.27 KB

Patch attached.

masipila’s picture

Issue summary: View changes

Apparently the patch didn't break any existing tests but I guess we should add tests to cover the additional mapping. I'll leave that for someone who is more experienced with automated tests.

Issue summary / remaining tasks updated accoringly.

Cheers,
Markus

heddn’s picture

Status: Needs review » Needs work
Issue tags: +Migrate Drupal

The patch in #7 doesn't necessarily account for installs using the minimal install profile, does it? At least for the article, it doesn't. I think we might be better off fixing the standard install profile and forum module to use the correct pattern. Back to NW for tests.

larowlan’s picture

+++ b/core/modules/node/src/Plugin/migrate/D7NodeDeriver.php
@@ -141,6 +141,22 @@ public function getDerivativeDefinitions($base_plugin_definition) {
+            $comment_type = 'comment_node_' . $node_type;

This isn't something the standard profile does - it must be specific to migrations.

Default behaviour in standard profile is 'comment' with forum having its own predictable comment type (via its /config/install/ folder).

Forum needs its own comment type, lots of code throughout the module assumes a specific field, and that is enforced via forum/config/install.

The pattern of using comment_{node type} is specific to migrate, see \Drupal\comment\Plugin\migrate\source\d7\CommentType::prepareRow where it allows for nuances between different node types in Drupal 7 - instead of finding unique combinations, it opts to create one for each node type.

masipila’s picture

Good morning,

The pattern of using pattern of using comment_node_{node type} is indeed coming from other migrations. Phenaproxima was trying to get rid of this back in June. See #2886839: D7 comment type migration should emulate Standard.

Summary:
1. D8 Standard installation profile uses comment comment type for Article.

2. D8 Forum must have its own comment type comment_forum. This is hard coded in Forum module just like it was mentioned also above in #10.

3. Phenaproxima was trying to get rid of the comment_node_{node_type} pattern in #2886839: D7 comment type migration should emulate Standard but that fell short. Conclusion was that we need to have separate comment types for each node types because the D7 comments might have additional fields in addition to the comment subject, body and author. These additional fields may be different for each node type in D7.

The patch in #7 assumes that 1-3 are valid. Do we have alternative approaches for this specific issue?

Cheers,
Markus

masipila’s picture

Hmm...

One thing we could consider would be as follows:

In D7NodeDeriver, we could check if the D8 node type has a comment field. If yes, then we add the mapping foo/0/status: comment where foo is the comment field we detected. If we find no comment fields, we don't add this mapping.

Worth thinking: it is possible that D8 site has multiple comment fields. Should we apply the mapping to all of them or just the first? If first, do we mean first what happens to come first from the D8 config check? Or first based on the weight in form settings? Or first in Display settings? If first in display settings, which view mode? Note that I'm not suggesting this heavy determination logic, I'm just making a point that if we can't assume comment_node_{node_type}, we need to think this through.

Markus

masipila’s picture

Issue summary: View changes

@heddn,

You proposed in https://www.drupal.org/node/2853872#comment-12242655 that we should let the other migrations create comment type comment_node_article . Now if the D8 site was installed with Standard installation profile, the Article has two comment types / comment fields.

Hence: the concern of multiple comment fields per content type I raised in #12 above is valid if we let Article to have two comment fields.

Proposed solution for this issue about node comment statuses:
1. In D7NodeDeriver we check from the D8 configuration if the content type has any comment fields. The result of this check can be 0-N.
2. If we find any comment fields in D8, we map foo/0/status: comment for each D8 comment field.

This should make your proposed approach in https://www.drupal.org/node/2853872#comment-12242655 feasible.

Issue summary updated with the proposed approach.

Cheers,
Markus

heddn’s picture

re #13: the issue with looking into the destination from the source deriver is that the source should have zero knowledge of the destination. None at all.

If having multiple comment fields is a problem, then we can update https://www.drupal.org/docs/8/upgrade/known-issues-when-upgrading-from-d... accordingly. But we already tell folks they should start their D8 site with minimal configuration and content. If they decide to use anything beside the minimal install profile, they'll have to do some work to bring themselves back to a minimal configuration state.

masipila’s picture

To verify that I have not misunderstood anything:

1. Regarding Articles
We have now concluded both in this issue and in #2853872: Migration for forum and article comments: duplicate comment types and incorrect comment_entity_statistics that we should not have special handling for Articles because of D8 Standard installation profile. If the site builder is using D8 Standard, he/she needs to manually remove the comment field comment from Article node type. This will be added to the known issues documenation heddn linked above.

2. Regarding the comment fields comment_node_{node_type}
These comment types / fields are created by the other D7-D8 migrations. I'm not quite sure if comments 9-10 by heddn and larowlan were arguing against mapping to these comment fields. That is what we need to achieve if we want to preserve the node level comment statuses.

3. About the new patch

  • This patch uses the same approach as patch #7 except it removes the special handling for Articles.
  • It also adds the optional dependency to comment field instance migration

Setting this to Needs Review to get feedback on the approach. This is still not complete as we need to add tests so once you provide feedback on the general approach, please set back to Needs work for at least tests.

Issue summary is also updated.

masipila’s picture

Status: Needs work » Needs review
dillix’s picture

Version: 8.5.x-dev » 8.4.x-dev
heddn’s picture

Status: Needs review » Needs work

#15. Just needs tests, back to NW for that.

jofitz’s picture

Here's a few tests.

(No need for an interdiff because all the changes are in the test_only patch)

The last submitted patch, 19: 2902766-19-test_only.patch, failed testing. View results

maxocub’s picture

@Jo Fitzgerald: Good start! But I think we should also test nodes with closed comments, and also forum nodes with closed and open comments.
Also did some clean up of the test file.

Status: Needs review » Needs work

The last submitted patch, 21: 2902766-21.patch, failed testing. View results

maxocub’s picture

Status: Needs work » Needs review
FileSize
18.13 KB
996 bytes

Fixed failing tests.

The last submitted patch, 21: 2902766-21.patch, failed testing. View results

masipila’s picture

Issue summary: View changes

Thanks for adding the test Jo Fitzgerald and maxocub!

I updated the issue summary to help reviewers. Remaining tasks at the moment are:
- Review by maintainers
- Commit
- Dance all night

Cheers,
Markus

heddn’s picture

Status: Needs review » Reviewed & tested by the community

Looks good now. There's full test coverage and we've picked up a few more edge cases.

Status: Reviewed & tested by the community » Needs work

The last submitted patch, 23: 2902766-23.patch, failed testing. View results

heddn’s picture

Issue tags: +Needs reroll
maxocub’s picture

Status: Needs work » Needs review
Issue tags: -Needs reroll
FileSize
18.17 KB

Here's the reroll.

heddn’s picture

Status: Needs review » Reviewed & tested by the community

Compared the diff stats, and we don't seem to have lost anything. Back to RTBC, assuming tests come back green.

  • catch committed b0a7be5 on 8.5.x
    Issue #2902766 by maxocub, masipila, Jo Fitzgerald: Node 'comment open...

  • catch committed 24804b5 on 8.4.x
    Issue #2902766 by maxocub, masipila, Jo Fitzgerald: Node 'comment open...
catch’s picture

Status: Reviewed & tested by the community » Fixed

Committed/pushed to 8.5.x and cherry-picked to 8.4.x. Thanks!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.