Problem/Motivation
SourcePluginBase supports a configuration option 'cache_key', the key to be used for caching source counts. This option is currently not documented.
Proposed resolution
Document the key alongside 'cache_counts'.
Remaining tasks
Do it.
User interface changes
None.
API changes
None.
Data model changes
None.
Original problem statement
see MigrationListBuilder.php
This part:
$row['total'] = $source_plugin->count();
When there is alot of data, the page will be unusable and timeout. Since I was having processing issues timing out with a large file (8,000+records), I split them out into smaller files < 1000 records each. So first off... there should be some sort of batch process involved with migrating large files but couldn't find anything in documentation around that so I assume that is functionality that doesn't exist yet. The root of this problem however is that whether i have 1 migration with 8000+ records or 8 migrations in a group of <1000 records each, that count function kills the loading of the page.
I realize stats are important and necessary but are there any ideas around somehow storing those in a table which would be set at the start or end of migration? Evaluating the total on multiple lists dynamically on this page is a huge issue.
Comment | File | Size | Author |
---|---|---|---|
#22 | 2973713-22.patch | 751 bytes | quietone |
#18 | 2973713-18.patch | 806 bytes | quietone |
| |||
#18 | 2973713-18.patch | 806 bytes | quietone |
| |||
#16 | interdiff-12-16.txt | 818 bytes | Ada Hernandez |
#16 | patch-2973713-16.patch | 731 bytes | Ada Hernandez |
Comments
Comment #2
apmsooner CreditAttribution: apmsooner commentedI should have mentioned, I'm using the migrate_plus module xml parser.
Comment #3
mikeryanIf you set
cache_counts: true
in your source plugin configuration then the count will only be retrieved once (well, or if you do a cache rebuild). Or, if you setskip_count: true
it won't do the counting at all. See https://api.drupal.org/api/drupal/core%21modules%21migrate%21src%21Plugi....Comment #4
apmsooner CreditAttribution: apmsooner commentedThanks. I'm able to get past the memory issues but cache_counts: true seems to have some issues with multiple migration lists within the group. It's as if all lists are using the same variable or something and the numbers are all out of whack and not making any sense at all. See attached screenshot and notice all the miscalculated numbers.
Comment #5
mikeryanAh - the cache key is derived from the source plugin ID. SQL source plugin IDs are (usually) unique, but your XML source plugins are going to pretty much all have an ID of "url". There is an explicit 'cache_key' config option you can use - we should document that.
The default cache key should be more unique - oh, and looks like I previously opened an issue for that: #2751829: Default source plugin cache key is insufficiently unique.
Comment #6
mikeryanComment #7
apmsooner CreditAttribution: apmsooner commentedThanks Mike, Setting the cache_key to the migration id worked out fine and now the numbers make some sense.
Comment #9
etecjdo CreditAttribution: etecjdo commentedHey there,
I am on the Drupal Europe and will try to fix this issue.
Regards.
Comment #10
gnuschichten CreditAttribution: gnuschichten at ETECTURE GmbH commentedI'm at drupal europe and I going to work on this issue.
Comment #11
tstoecklerAwesome @etecjdo and @gnuschichten, thanks! I am mentoring at Drupal Europe.
Comment #12
etecjdo CreditAttribution: etecjdo commentedHey there,
I have looked up the file and added a class documentation for the new cache-key.
Patch is attached.
Regards.
Comment #13
gnuschichten CreditAttribution: gnuschichten at ETECTURE GmbH commentedComment #14
gnuschichten CreditAttribution: gnuschichten at ETECTURE GmbH commentedWe are finished here and move to the next issue.
Comment #15
tstoecklerThanks @etecjdo and @gnuschichten!
I reviewed the patch and found one minor issue:
This should wrap at 80 characters, so cache_counts should be on the next line.
Maybe you can fix it?
Comment #16
Ada Hernandez CreditAttribution: Ada Hernandez at MTech, LLC commented##15 cache_counts on the next line
Comment #17
quietone CreditAttribution: quietone as a volunteer commented@etecjdo, Welcome to Drupal! Thank you for the patch. Good work.
The beginning 'This cache key' seems unnecessary here, we know it is the cache_key being described. Then I saw the same structure used in the description for the high water property and thought maybe this should stay as is. But then the high water does take a bit more to explain So, let's change this.
This is supposed to be a unique key as well, maybe that should be mentioned. I don't know maybe, "A uniquely named key used to cache the value of cache_counts"? Only a suggestion.
Comment #18
quietone CreditAttribution: quietone as a volunteer commentedBased on mikeryan's comment in #5 I came up with this documentation:
* - cache_key: (optional) Uniquely named key used to cache cache_counts.
Comment #21
quietone CreditAttribution: quietone as a volunteer commentedComment #22
quietone CreditAttribution: quietone as a volunteer commentedChanged to:
- cache_key: (optional) Uniquely named cache key used for cache_counts.
Comment #23
phenaproximaYup.
Comment #24
alexpottCrediting @apmsooner for creating the issue, @mikeryan and @tstoeckler for issue comments, and @gnuschichten and @tstoeckler for working together at DrupalEurope on this.
Committed and pushed e1832a2c2b to 8.7.x and 9efe560f76 to 8.6.x. Thanks!