Is there a source class for using an existing entity type as the source?

I'm looking at a way to take all entities of one type, and break out certain values into a new, referenced entity.

A generic source class that works over all the entities of a given type seems like a useful base to have, so I'm wondering if it already exists before I code it :)

CommentFileSizeAuthor
#5 1883112.4.migrate.source-entity.wip_.patch8.76 KBjoachim

Comments

mikeryan’s picture

mikeryan
he/him
Primary language English
Location Pittsfield, MA, USA
commented
Category: support » feature

So you're looking to "migrate" data within a single Drupal install? No, there's no source class for that scenario.

mikeryan’s picture

mikeryan
he/him
Primary language English
Location Pittsfield, MA, USA
commented

But, of course, there is migrate_d2d...

joachim’s picture

joachim commented

Just had a look at that.

Unfortunately, I don't think that's what I'm after here.

That works by defining a migration, say, 'nodes', which then has as its destination the local DB node table, and as its source a MigrateSourceSQL which draws from the node table in another database.

What I am after here is using Migrate to do some internal processing on a database.

(Use case is that I want to break some data out from a field to a new entity to then reference.)

joachim’s picture

joachim commented

I've got a working source class for this, which I'll tidy up and roll as a patch soon.

A few problems though:

- parts of the migrate system expect the $row to be specifically a StdClass. If the entity being loaded as the source is form a module using EntityAPI, then it's a specific class.
- so far my use case has been to load entity A, and extract data from it to then create entity B. When the use case is processing entity A and re-saving it, I wonder whether #1693884: How to specify 'needs_update' for a row would be needed.

joachim’s picture

joachim commented
Status: Active » Needs work
StatusFileSize
new 1883112.4.migrate.source-entity.wip_.patch8.76 KB

Here is a patch of my work in progress.

An additional problem I found is that highwater mark testing appears to be too strict in some way I have not quite fathomed yet -- no rows were getting imported at all unless I commented that out.

Other than that, this works great. Obviously, I need to go digging some more into this highwater problem. The fix may turn out to be #1693884: How to specify 'needs_update' for a row.

My first use case for this was this sort of situation:

- input: nodes with a field 'foo' which has a value that is repeated over multiple nodes
- output: nodes with an entityref field to an entity which carries the 'foo' value.

This meant that while source and destination were different entities; the destination was new entities to be created by the migration.

My second use case, for the Image module upgrade, has the source and destination the same: the input is the node, the output is the same node to be updated. I'm less sure how to tackle this. I'm wondering a special 'null' destination, and then let the migration class do its own saving of the incoming node in prepare(), but I'd appreciate any thoughts on the best way to approach this.

bryrock’s picture

bryrock commented

Thanks for this patch. Going to try it out on some profile2 migrating.

bryrock’s picture

bryrock commented

Pretty nifty patch. Saves a lot of time in pulling together profile2 profiles for migration.

I've got something like this going...

   $query = new EntityFieldQuery();
   $query->entityCondition('entity_type', 'profile2')
    	      ->entityCondition('bundle', 'main')
	      ->propertyOrderBy('created', 'ASC');

    $this->source = new MigrateSourceEntity($query);

... and it returns the whole profile object. Only problem I'm having is figuring out how to direct the EntityFieldQuery to something other than my default (destination) database rather than my migration source.

joachim’s picture

joachim commented

> Only problem I'm having is figuring out how to direct the EntityFieldQuery to something other than my default (destination) database rather than my migration source.

I don't think that's possible with EFQ; certainly it's not part of what I was aiming for here. My use case was processing data from local entities into other local entities.

pifagor’s picture

pifagor
Location 🇺🇦 Rivne
commented
Issue summary: View changes
Status: Needs work » Closed (outdated)