Support for Drupal 7 is ending on 5 January 2025—it’s time to migrate to Drupal 10! Learn about the many benefits of Drupal 10 and find migration tools in our resource center.
I hastily cobbled this together. It is completely non-working.
Comment | File | Size | Author |
---|---|---|---|
#14 | 1890610-add-mongodb-source-14.patch | 5.51 KB | dstol |
#11 | migrate-mongodb-source-1890610-11.patch | 5.08 KB | havran |
#7 | migrate-mongodb-source-1890610-7.patch | 5.5 KB | TravisCarden |
#7 | interdiff.txt | 558 bytes | TravisCarden |
#1 | 1890610-add-mongodb-source.patch | 2.23 KB | dstol |
Comments
Comment #1
dstolComment #2
havran CreditAttribution: havran commentedThanks for work on this.
Comment #3
havran CreditAttribution: havran commentedI use for mongodb as source this class:
And here is migration class:
Comment #4
havran CreditAttribution: havran commentedI have working MongoDB source class. Here is code:
And here is migration class:
But this still need more work on performance (see comments on performRewind() function)
Comment #5
havran CreditAttribution: havran commentedToday progress.
- performance optimization - now mongodb query return only unmigrated documents
- some code cleanup
MigrateSource class:
Example migration class:
Comment #6
mikeryanPlease attach files rather than pasting all your code into the comment body.
Thank you.
Comment #7
TravisCarden CreditAttribution: TravisCarden commentedHere's a patch for @havran's code in #5. It works pretty well for me, but don't take that as a blanket affirmation of the details as I didn't give it a thorough review. The one comment I do have is that we can't assume that we have write access to the database to add an index, so I removed that code from the patch (see interdiff.txt below). I'm sure you could add it back conditioned on a test for permission, but I'm not sure you want to go modifying data sources without asking.
Comment #8
mikeryanJust a note - I don't have MongoDB data to test this with, or time to construct a test case. If someone not involved in developing the code can give it a good test and mark it rtbc I'll commit it.
Quick scan of the code looks fine to me, only quibble is to use @todo rather than ToDo.
Comment #9
dstolmikeryan, I'd be happy to share with you a mongo datasource and a sample migration that I've worked with.
Comment #10
mikeryan@dstol: I found the zips example data, and then noticed that's what @havran's sample code is using, so I'm good.
Line break needed (80 chars)
Add an option (defaulting to FALSE) for creating an index
@todo instead of ToDo, one c in necessary.
The place for index deletion would be postImport(), I think.
{@inheritdoc}
{@inheritdoc}
// If we have an existing idlist we use it.
If I follow correctly, the idea is to (in conjuction with the setMongoDocumentMigrationInfo and removeMongoDocumentMigrationInfo methods) write information to the source database to try to optimize the query? I'm... not a fan. For the first pass at this, to get into Migrate 2.6, forget about optimization and just make it work. Then, look at how MigrateSourceSQL optimizes highwater fields and see if a similar approach could be used here when there's a highwater mark.
Please return something human-readable - collection name, representation of query/sort components, ...
Comment #11
havran CreditAttribution: havran commentedI attached last version mongodb source plugin which i use in my project. I remove index related code and setMongoDocumentMigrationInfo/removeMongoDocumentMigrationInfo functions because i'm too not fan for this solution.
Part of performRewind() method is refactored because as row id can be used any field from mongodb collection (i use "old_id" field).
For performance optimatization we need still work - i try use highwater approach later in my migration (~650k articles with images, videos, comments and more).
Comment #13
mikeryanThe patch doesn't apply, see under "More git commands" at https://drupal.org/node/707484 how to add a new file in the patch.
Several lines have trailing whitespace which should be removed.
Why are the getMongoId and getSourceKeyNameAndType methods private? Is there a reason to prevent derived classes from overriding them?
Comment #14
dstolMoving this forward
Comment #15
mikeryanCommitted, thanks!
Could someone document this plugin under https://drupal.org/node/1006986?
Comment #17
Dane Powell CreditAttribution: Dane Powell commentedI documented this at https://drupal.org/node/2286035, so I think this issue is finished.