Closed (fixed)
Project:
Migrate
Version:
6.x-1.0
Component:
Documentation
Priority:
Normal
Category:
Support request
Assigned:
Unassigned
Reporter:
Created:
17 Mar 2010 at 14:06 UTC
Updated:
3 May 2010 at 19:00 UTC
I set up a data import from a cvs file. When I'm done, I want to delete the stuff from the import (the content set, the raw old data table, etc.)... so that I can do the same import again, with a new set of data.
how can I do this? (should I never remove the evidence of the import???)
I'm going to be doing a re-import with new data fairly frequently, like once a month.
Thanks!
Comments
Comment #1
mikeryanYour intent is simply to update the data from a new source file, right? Not to "remove evidence"? You just need to use the --update flag on drush migrate-import (or set the "Update previously-imported content" toggle when running interactively). Any source records with primary keys which were previously-imported will replace the data in the corresponding Drupal objects with the new data, and new records will be inserted.
Or am I misunderstanding what you're trying to do?
Comment #2
yesct commentedLets say grouplist1.csv is
1, Illinois
2, Michigan
3, Indiana
and I get the tw all set up with the view, and I do migrate, try it, change a few things, remigrate it, and then it is like I want it to be.
next month I want to import a new list of groups, grouplist2.csv:
4, Florida
5, New York
6, Texas
I want the import to proceed in the same way as previously. I saw three things to try:
a) rename grouplist2.csv to grouplist1.csv (so it just contains the 3 new things)
b) append grouplist2.csv to grouplist1.csv (so it contains all 6, and tell it to just work on the entries 4-6)
c) create a new view (and table) in tw, reconfigure grouplist2 in migrate to have the same settings as grouplist1 had.
a) I think I tried this, and it messed up some of the information, like it showed -3 entries processed, ...but
good things: did not need to re-edit in phpmyadmin the table structure (making the num an int, and setting the primary key), did not need to re-matchup the fields with the columns in migrate.
howto: make a copy of the previous csv file, overwrite the same filename with just the new data, go to table wizard and pick upload delimited files, browse to select it, then select import new file
bad things: if I pick clear, it clears (deletes) everything ever imported, not just the recent things in the current csv file/in the current view/table
b) entries 1-3 are still in the system, and if I forget to tell it to "work" on only 4-6, it will re-import 1-3, possibly changing things in those nodes that should not, now be changed.
c) is nice because it will not mess up what I did with grouplist1... but I cannot ask the people who will be admining the site to do the config of tw and migrate. also c) will leave grouplist1 and groulist2 etc in the screens for migrating... eventually we could have 30 iterations go by and those screens would be getting cluttered. Also, after grouplist1 (and 2, etc.) are imported, I wont need to go back and import them again, once the nodes are created, I dont want to clear/delete them
---- I guess I'm looking for a way to commit or freeze the import of grouplist1/entries 1-3. Delete those entries out of the table/view, and put in a new set of data the next month (or whenever I get the data on new things to import).
I'm new to using this, so I'm not quite sure what the workflow is that people normally use to import and then reimport new data.
I attached the screen shot of the migrate dashboard, I was looking for some button there that would commit or freeze an import (so it could not be cleared). Or a delete content set check mark (not to delete the nodes imported, just the content set listing on this page... which might have the effect of making it so people could not "clear" an import... but it would also have the effect, or needing to re-setup everything next time someone needed to do an other import.)
Comment #3
mikeryanJust do b - append your new rows to the MySQL table you've got set up in the view. You will now see 6 total, 3 imported (1-3) and 3 unimported (the new rows). Run the migration and it will just import the new rows - it will only reimport the first three if you explicitly tell it to (--update/Update previously-imported content).
Comment #4
yesct commentedhere is a first video of me trying out a)
http://cathytheys.blip.tv/file/3373803
I know you said to do b. I'll try that next, just wanted to put a link to a out in case anyone else was following this issue. :)
Comment #5
yesct commentedThe appending seems to be working out well! Thanks.
While testing this, I made a lot of new tables, tw views and migrate content sets.
I wanted to clear these out, so I thought: AH, uninstall the modules.
But when I uninstalled migrate, there are still these tables there:
# migrate_map_groupimport
# Structure migrate_map_grouplist2
# Structure migrate_map_groupsappend
# Structure migrate_map_groupsappend2
# Structure migrate_map_newgroups
# Structure migrate_msgs_groupimport
# Structure migrate_msgs_grouplist2
# Structure migrate_msgs_groupsappend
# Structure migrate_msgs_groupsappend2
# Structure migrate_msgs_newgroups
I'm going to drop them, are there any other tables I should drop?
Thanks
Comment #6
yesct commentedjust following up, in case anyone wants more info on what ended up working for me: a video of appending to do repeatedly batch imports
http://cathytheys.blip.tv/file/3403187/
in the last 4 videos of this series it starts with an overview, then showing the first batch and set up of tw and migrate, then finally appending to do batch 2 and 3, then something drupal people will probably ignore: showing how the order will interact with the group that was imported.
Comment #7
mikeryanre #5: That's it, migrate_map_* and migrate_msgs_*.
Comment #8
mikeryan