Support for Drupal 7 is ending on 5 January 2025—it’s time to migrate to Drupal 10! Learn about the many benefits of Drupal 10 and find migration tools in our resource center.
Problem/Motivation
We need to provide an upgrade path from 7.x for the core Statistics module.
Remaining Tasks
We need migrations, with tests, to cover:
- Any variables maintained by Statistics need to be moved into configuration
- The contents of the node_counter table may need to be migrated
Comment | File | Size | Author |
---|---|---|---|
#58 | interdiff-54-58.txt | 4.16 KB | jofitz |
#58 | interdiff-47-58.txt | 2.56 KB | jofitz |
#58 | 2500521-58.patch | 12.34 KB | jofitz |
#54 | interdiff-52-54.txt | 788 bytes | jofitz |
#54 | 2500521-54.patch | 13.59 KB | jofitz |
Comments
Comment #1
quietone CreditAttribution: quietone commentedJust making a start.
Comment #2
phenaproxima@quietone: There is already a migration in progress for the statistics settings (see the child issue), but we do need a migration for the node_counter table, if you're up for it.
Comment #3
quietone CreditAttribution: quietone commentedYea, I saw that after I posted and I was just leaving for the day. Not sure about node_counter table.
Comment #4
hussainwebI started working on this and realized it would be much easier if there was a service backing this. This is currently being handled in #1446932: Improve statistics performance by adding a swappable backend. I am postponing this issue on that and posting a patch I started with. Once that is in, we can continue working on the patch.
Comment #5
quietone CreditAttribution: quietone as a volunteer commentedComment #7
quietone CreditAttribution: quietone as a volunteer commentedNo longer postponed.
Comment #10
DamienMcKennaAdding the main D7-D8 issue as the parent issue.
Comment #11
DamienMcKennaLets nudge testbot to take a look at the patch (I'm sure it'll fail, but it's good to follow the standard process).
Comment #12
phenaproximaSelf-assigning for review.
Comment #13
phenaproximaThis is not ready for prime time. Sorry, y'all!
Two observations here. First, I'm not sure it will work if we just pass d7_node as the migration, seeing as how it has a deriver and encompasses many migrations, one per node type. We should be sure this will look in every d7_node derivative.
Second, if the migration plugin fails, it will return NULL. We should add an additional process plugin here that skips the row if nid is empty.
Again, we need to be sure this is respecting all d7_node derivatives.
Wat. We're not using this for anything.
No it's not! :)
I don't think we need any of this; we're not injecting any dependencies that I can see. We probably also don't need ContainerFactoryPluginInterface either, in that case.
Should be 'the ID of the node to which these statistics apply'.
So...we're not saving anything?
Also, there be no tests in them thar hills.
Comment #14
quietone CreditAttribution: quietone as a volunteer commentedThere doesn't seem to be a d6 migration for this. Can that be included here?
Comment #15
jofitz CreditAttribution: jofitz at ComputerMinds commentedLet's move this on a step.
Done: 3, 4, 5, 6, 7
To do: 1, 2
Still needs tests so no point in running past the testbot.
Comment #16
quietone CreditAttribution: quietone as a volunteer commentedOn the last migrate call we thought that d6 and d7 statistics were pretty much the same, so marking this for d6 as well.
Comment #17
Rewted CreditAttribution: Rewted commentedI originally noted the loss of statistics during my migration. Can anyone tell me how to copy over the views count directly from the old D6 Database to D8 manually?
Comment #18
jofitz CreditAttribution: jofitz at ComputerMinds commentedI'm still not sure how to address #13.1 & #13.2
Comment #20
mikeryanDon't have time to dig it up at the moment, but I seem to recall looking at a similar scenario and finding that the derivatives are automatically expanded... somewhere.
Comment #21
jofitz CreditAttribution: jofitz at ComputerMinds commentedMoved NodeCounterTest.php to the correct location.
While debugging through the code I can confirm that it does 'magically' expand the derivatives, but don't ask me why/how!
Comment #22
heddnNot necessarily critical if someone can dig up where the derivatives get expanded, but another way to prove this is by running a manual test. Tagging and marking Novice. To test this, attempt to migrate d6 of d7 statistics data into D8 and see if the numbers for nodes come over correctly.
Comment #23
heddn*Now* adding the novice tag.
Comment #24
dexter42 CreditAttribution: dexter42 commentedComment #25
dexter42 CreditAttribution: dexter42 commentedHi... one note - when there are translated nodes (e.g. English and German), there are two nodes in D6, but only one node in D8. I assume the statistics have to be summed up.
Comment #26
dexter42 CreditAttribution: dexter42 commentedBtw, in the destination plugin in import function should be following return statement:
return [$row->getDestinationProperty('nid')];
otherwise such row will be marked as failure.... As I have in the meantime this patch heavily customized to make workaround for my translations, I am unable to provide patch.
Comment #27
phenaproximaLooks damn near perfect to me. Just a few nits and it's RTBC once my and @dexter42's comments are addressed.
$old_destination_id_values should be [], not array().
I think I'd rather this were passed as a single associative array, where the database columns are the keys.
Should a source provider maybe be mentioned here?
Should say "Tests the migration of node counter data to Drupal 8."
This class name should be fully qualified, which means we can drop the use statement at the top of the file.
Should be rephrased as "Tests the migration of node counter data to Drupal 8."
This should be fully qualified, dropping the use statement at the top of the file.
I believe the coding standards require an empty line between these two annotations.
Comment #28
gaurav.kapoor CreditAttribution: gaurav.kapoor at OpenSense Labs commentedFixed 27.1,27.4,27.5,27.6,27.7 and 27.8.
Comment #29
phenaproximaThanks, @gaurav.kapoor! Looks good. I'm returning this to NW until the rest of the points are addressed, but it's definitely getting there.
#28.5 and #28.7 are not quite done yet -- the class name in the comment must be the fully qualified class name. So,
/** @var StatisticsViewsResult $statistics */
needs to be changed to:
/** @var \Drupal\statistics\StatisticsViewsResult $statistics */
Comment #30
quietone CreditAttribution: quietone as a volunteer commentedThanks all. Just some little bits left to do.
Fixed 27.2, 27.3, 27.5, 27.7 and #26. Also, I was getting schema errors when running the migration tests something about node.article.menu_ui. Adding 'menu_ui' to the list of modules fixed that. I think that is everything.
Comment #32
quietone CreditAttribution: quietone as a volunteer commentedAnd the same for D7.
This is failing because the new migrations have not been added to the big array of migrations in MigrateUpgradeForm. That array is being removed in #2569805: For Drupal migration, identify the source module. We could reroll here, but then we need to reroll there. No fun in that. So, I am going postpone this on that issue, which will solve the problem of having to maintain that large array of migrations.
Comment #34
RytoEX CreditAttribution: RytoEX as a volunteer commentedSince #2569805: For Drupal migration, identify the source module has been fixed, should this be shifted back to Needs Work for a reroll? I'm not familiar enough with the patch/issue to reroll this myself.
Comment #35
quietone CreditAttribution: quietone as a volunteer commented@RytoEX, yea, needs a reroll. But most of the files are new, so it will probably be the test fixture. Yes, just checked it is the test fixture and that can be tricky to reroll, it is a dump of the database prepared with the db-tools in core/scripts. However, it is just adding tables so might be straight forward. Don't know. Want to try?
Still, removing the novice tag because the test fixture is involved. And setting to NW
@RytoEX, thanks!
Comment #36
RytoEX CreditAttribution: RytoEX as a volunteer commentedIf it's just adding tables from a generated dump, I can try to take a look. Any hints on how to properly prepare and dump the database?
Comment #37
quietone CreditAttribution: quietone as a volunteer commentedThere are some notes in the handbook, Generating database fixtures for D8 Migrate tests that should help. If you find errors or think it needs improvement, please edit that page. Good luck.
Comment #38
quietone CreditAttribution: quietone as a volunteer commentedOh, I think it is safe to unassign phenaproxima, He assigned himself to review (#12) and did the review in #13.
Comment #39
RytoEX CreditAttribution: RytoEX as a volunteer commentedI forgot to ask. Should this be rerolled against 8.4.x or 8.5.x? This issue currently states 8.4.x, but I thought I'd check while I'm still tinkering.
I did finally get the database dumps working. Now I'm trying to reconcile the dumps with the patch from #30.
Comment #40
quietone CreditAttribution: quietone as a volunteer commentedLet's stay with 8.4.x.
Congratulations on getting the dump working. It is an extra part (i.e. hurdle) to working on migrate issues. I find that making the interdiff with a dump involved better results are obtained with 'git diff' instead of 'interdiff patchA patchB'.
Comment #41
RytoEX CreditAttribution: RytoEX as a volunteer commented@quietone
After staring at this for way too long, I think I might be overthinking it. I've tried rerolling the patch for 8.4.x, and the individual tests introduced in this patch seem to pass locally, so I'm uploading it. If it's wrong, I'm sure someone will catch it and say something or correct it. I figure it's probably better to try to move forward than for me to keep staring at it wondering if it's correct. If the tests are supposed to be rewritten to use the node_counter values introduced by #2872660: Migrate D6 node reference fields to D8 entity reference field definitions (commit 5380355) and #2669030: D6 Forum vocabulary is migrated to the wrong D8 field name (commit 31b67bb) instead of this patch inserting its own values (as it does here and previously did), say the word. If I've completely missed the mark, I hope to learn where I went wrong.
I couldn't figure out how to make a proper interdiff since the previous patch currently fails to apply on 8.4.x.
Also, I did make a few changes to the documentation page that you'd linked on generating database fixtures.
Comment #42
RytoEX CreditAttribution: RytoEX as a volunteer commentedI think the failures in #41 are partially due to #2569805: For Drupal migration, identify the source module, so I've made adjustments to deal with that. I've attached a new patch and an interdiff between the new patch and #41. This seems to fix at least a few of the test failures on my end.
Seeing "Source module found for d6_statistics_node_counter." when the test failed seemed a bit odd. I thought it would say "not found", but I may be unaware of some paradigm for test messages.
Again, feel free to wave me off if I've made a blunder.
Comment #43
RytoEX CreditAttribution: RytoEX as a volunteer commented#42 seems to have passed all tests. I noted some coding standards issues after the tests, and I have fixed those locally. To fully comply with #2569805: For Drupal migration, identify the source module and its change record (and prepare for #2908282: Throw exception for source plugins without a source_module property and its follow-ups), do we also need to include source_module and destination_module in the annotations for the NodeCounter source and destination classes? Again, assuming I haven't gone down the wrong path with the reroll in the first place.
Comment #44
RytoEX CreditAttribution: RytoEX as a volunteer commentedI've updated the patch to fix the Coding Standards issues and adds the source_module and destination_module annotations that I mentioned in #42. From what I can tell, the correct way to specify those for this case is in annotations instead of in a YAML file. I've attached interdiffs against #41 and #42.
Changing status to Needs Review and removing the Needs Reroll tag. I forgot to do these previously.
Comment #45
phenaproximaLooks very good. Really nice work -- I see nothing overly wrong here!
Core migrations must now live in the migrations directory, not migration_templates.
I think we could combine the d6_statistics_node_counter and d7_statistics_node_counter migrations into a single one. All we'd need to do is:
If this is insane, then we don't need to do it. Just suggesting it.
The 'migration' plugin is now 'migration_lookup'.
Does Statistics provide a service, or API function, we could use for this? Rather than directly writing to the database? If so, that'd be preferable.
Let's use assertSame() here instead.
Same here.
Comment #46
RytoEX CreditAttribution: RytoEX as a volunteer commentedThis new patch is now based against 8.5.x and addresses 45.1, 45.3, 45.5, and 45.6.
I'll let someone else weigh in on 45.2.
I didn't see anything in Statistics that could be used for 45.4 other than recordView, but that seems to use some expression logic for tracking node views, so I'm not sure how compatible it is with the needs here.
Comment #47
jofitz CreditAttribution: jofitz at ComputerMinds commentedAddressed 45.2.
I agree with @RytoEX about 45.4.
The 2 test failures in #46 (and retained in this patch too) are due to the changes suggested in 45.5 and 45.6: despite the doxygen, getTotalCount() etc return strings not integers hence why assertEquals() passed, but assertSame() fails. @phenaproxima I would be interested to hear your opinion on the best solution to this? The options include: simply making the expected values strings or editing the StatisticsViewsResult class so that it's values must be integers (e.g. typecasting in the constructor).
Once we iron out the finer points of 45.5 and 45.6, which should be simple enough, this will be ready for RTBC.
Comment #49
RytoEX CreditAttribution: RytoEX as a volunteer commentedThe interdiff in #47 omits the new file core/modules/statistics/migrations/statistics_node_counter.yml made to address 45.2, just in case anyone was only checking interdiffs.
I would also be curious to know more about the test messages that I mentioned in #42, but that may need to be left for a follow-up issue, if there is anything to address at all. My initial feeling about them is that the messages are incorrect, and that they should say "module not found" when the assertion fails rather than "module found".
Comment #50
phenaproximaCounts are integers by definition. It makes no sense for a count of any kind to be a string -- the inconsistency is usually due to PHP's loose typing rules, and the database drivers not bothering to cast return values, since they don't necessarily have any idea what type of data is contained in various columns.
So, if we're dealing with counts, we should always cast them to integers, and use assertSame().
Comment #51
phenaproximaMy understanding, which may be inaccurate, is that they can be either. They're only displayed if an assertion fails anyway. So if the message is positive, you'll know what failed; and, if it's negative, you'll know what failed. It's just a difference of phrasing -- your choice! :)
Comment #52
jofitz CreditAttribution: jofitz at ComputerMinds commentedI have cast the values to integer in the StatisticsViewsResult constructor.
Comment #54
jofitz CreditAttribution: jofitz at ComputerMinds commentedCorrect the test failure to expect an integer rather than a string (and remove the interdiff that I managed to include in the patch!)
Comment #55
heddnAll feedback addressed. Onward and upward.
Comment #56
xjmComment #57
phenaproximaReally looking fantastic. Only a couple more points, then this is completely ready to go.
Since this migration is generic for D6 and D7, it should probably just be "Node counter".
This is out of scope. I know we need to do it in order to get our tests to pass, but I think we can simply cast the value *in the tests*, rather than here. Let's revert this part of the patch, then open separate follow-up to make this fix in Statistics by itself.
Comment #58
jofitz CreditAttribution: jofitz at ComputerMinds commentedI did wonder if I was wandering down the wrong path!
Comment #59
phenaproximaNice one. Thanks everybody!
Comment #61
RytoEX CreditAttribution: RytoEX as a volunteer commentedPatch had failed testing for some reason (Aggregator?). Re-ran the tests and everything passed. Back to RTBC per #59.
Comment #63
catchCommitted eca600b and pushed to 8.5.x. Thanks!
Comment #64
ao2 CreditAttribution: ao2 as a volunteer commentedThanks, this works pretty well here too, after adjusting the migration lookup to my custom migrations.
As @dexter42 pointed out in #25, the
node_counter
table only knows about node ids and so if the D6 nodes have translations with the current patch the statistics of the translations are lost when migrating to D8.Not a big deal in my case as I have very few translations but worth mentioning IMHO.
Summing up the statistics of all the nodes in a translation set would be ideal, but it's probably more work than it's worth it.
Ciao,
Antonio
Comment #65
heddnRe #64: can we get a follow-up opened to address the i18n aspect there?
Comment #66
masipila CreditAttribution: masipila as a volunteer commentedDid this commit cover all the aspects that are cjrrently mentioned on the Known Issues page i.e. can we remove the section for Statistics on that handbook page? https://www.drupal.org/docs/8/upgrade/known-issues-when-upgrading-from-d...
Comment #67
masipila CreditAttribution: masipila as a volunteer commentedRe #64-65: I created the follow-up: #2930101: i18n / statistics - node counter not updated for translations