7200 og_user_roles.inc update error

Duplicate? #1457964: AJAX 500 error on content migration

For me to debug you will have to attach your DB, same as described in #1457964: AJAX 500 error on content migration

I may have some more information on this issue...in my database I have the og table which looks like this:

SELECT gid, etid, entity_type FROM `og` LIMIT 0, 30

gid etid entity_type
2 1 node
3 2 node
18 9 node
27 190 node

SELECT * FROM `og_users_roles`

uid rid gid group_type
1 6 190 node
2 3 9 <---- fails here
2 3 26
2 6 1 node
2 6 190 node
20 6 1 node
30 3 8
30 3 9
43 3 0
43 3 19
43 3 25
43 3 26

The migration fails on the second row of og_users_roles here at line 56 of og_7200_user_roles.inc because og_migrate_get_group returns FALSE for gid=9:

    list($group_type, $gid) = og_migrate_get_group($row->gid);
    // Get the bundle from the entity.
    $group = entity_load_single($group_type, $gid);

I don't know the architecture too well to take it from here, I guessed at a potential solution which was just to check if og_migrate_get_group returned anything before proceeding with the entity_load_single, but that resulted in the migration hanging on the first record. Hope this helps.

If necessary I may be able to get you a database, but I cannot post it here. Let me know.

Just found this issue: http://drupal.org/node/1530148#pift-results-1530148 may be the same with a patch in progress.

Update: just tested above patch and issue still exists.

Please test this patch #1635366: Make og-migrate plugins less strict

Please let me know how I can get my DB to you without posting it here.

Send me an email with link to this issue, and a link to db dump.

edvanleeuwen
The DB you sent is very messy. I see phpbb stuff there, and there's drupal_users and users tables.

Please remove the unnecessary stuff so I can spend time on debugging not on setting up. Also please disable any module not related (e.g. HTML mail, etc')

@edvanleeuwen,
I'm sorry, but I'm loosing time just trying to set it up. I'm really trying to help you, but if the site isn't in English, and theme is completely corrupted so I can't see the batch -- I can't help you.

https://skitch.com/amitaibu/enmre/migrating-organic-groups-data-covs

Make sure you try yourself to import your DB to a clean installation. Make sure it's in English and a core theme is Enabled.

I am getting the same error when trying to update to 7.x-2.0-beta1, whether I'm updating from an old version of 7.x-1.x-dev or from 7.x-1.4. If I try instead updating to 7.x-2.x-dev, the update gets permanently stuck at "2 out of 4".

Investigating what's going on during the update, the source of my problem is a custom role that has been defined through the UI (i.e., via admin/config/group/roles, not through a module hook). The original og_role table has the custom role at rid=4, gid=0. The 7200_og_roles migration creates bundle-specific copies of all the DEFAULT roles, but does not create any bundle-specific copies of my custom role. This is because the og_roles_override function that creates the bundle-specific copies explicitly checks for whether gid=0, in which case it only operates on the roles provided by og_get_default_roles().

This then causes 7200_og_user_roles to fail when trying to migrate entries in the og_user_role table that have been assigned the original custom role. The migrate tries to change rid=4 to rid=''.

It seems likely that this is the same underlying problem as in the original post, given that the failed row initially has rid=6, i.e., a non-default role. It should also be possible to reproduce the problem in any environment by creating a custom global role and assigning that role to a member of any group.

I threw together a quick patch that causes og_roles_override() to create copies of the custom role. With the patch, the update runs to completion in both 7.x-2.0-beta1 and 7.x-2.x-dev. I've posted my 7.x-2.x-dev version of the patch.

Can you please add some docs and @param $force_all to better explain what it is supposed to do.

Done -- although in the process I decided to rename the param from $force_all to $in_og_migrate... seemed more appropriate.

For the sake of thoroughness, I also tracked down why I was originally getting stuck in an infinite loop in og_7200_users_roles when using 7.x-2.x-dev. So this patch contains everything from #15 plus one additional fix.

tagging

@Nephele, where have you been all this time?! ;)

+++ b/og.moduleundefined
@@ -2629,11 +2629,16 @@ function og_permissions_delete_by_module($modules = array()) {
+function og_roles_override($group_type, $bundle, $gid, $in_og_migrate=FALSE) {

@@ -2644,13 +2649,30 @@ function og_roles_override($group_type, $bundle, $gid) {
+  else {
+    // Copy ALL default roles
+    // This call to og_roles searches the database for all roles where bundle and type are blank
+    // Such entries only exist when a pre-2.0 version of og has not been fully migrated
+    // (This option is only intended to be used within og_migrate)

Lets move this logic directly unto og_7200_roles.inc instead of calling og_roles_override().

+++ b/plugins/og_migrate/7200/og_7200_user_roles.incundefined
@@ -35,6 +35,7 @@ function og_7200_user_roles(&$context) {
+  $initial_uid = $context['sandbox']['last'];

$intiial_uid => $last_uid

+++ b/plugins/og_migrate/7200/og_7200_user_roles.incundefined
@@ -66,6 +67,17 @@ function og_7200_user_roles(&$context) {
+        // $rid can come up empty for at least two reasons:
+        // (a) Because this row has already been converted in previous batch loop (uids overlap between loops)
+        //     In this case progress has already been incremented
+        // (b) Because the original row->rid can not be converted
+        //     In this case progress has probably not been incremented (and can't be 100% certain with available info)
+        // So: always increment if uid cannot have been previously done; also assume that (b) is true if $row->rid is found in $default_roles

This comment is a bit too complicated. Lets remove the (b) section.

+++ b/plugins/og_migrate/7200/og_7200_user_roles.incundefined
@@ -66,6 +67,17 @@ function og_7200_user_roles(&$context) {
+        if ($row->uid!=$initial_uid || array_key_exists($row->rid, $default_roles)) {

Missing space between !=. Also I (and Drupal core) prefer !empty($defaults_roles[$row->rid])

OK, take 4! Which has ended up looking almost completely different from patch #16, even though it accomplishes the same purpose.

I've moved the one chunk of code from og_roles_override into og_7200_roles. I'd originally avoided this approach because I was worried about unnecessary code duplication, but upon closer look I think there are only three lines of code that need to be copied from og_roles_override into og_7200_roles. In particular (unless I'm mistaken) the db query in og_roles_override that remaps og_users_roles doesn't need to be copied. So I agree that it is cleaner to just have all of the logic in og_7200_roles.

As for the infinite loops in og_7200_user_roles, I decided to take a completely different tack. The complex comments and other awkwardness were results of trying to work around the inaccuracy of the main query -- and the code still risked hitting infinite loops.

So in this go-around I instead changed the main query so each batch loop picks up exactly where the last one left off, no matter how large, complex, or error-filled the original database might be. Which allowed the deletion of all code/comments/logic about guessing at whether or not the progress counter needs to be incremented. Hopefully that makes more sense.

+++ b/plugins/og_migrate/7200/og_7200_user_roles.incundefined
@@ -35,12 +37,35 @@ function og_7200_user_roles(&$context) {
+  // Create a complex query condition to ensure that this loop picks up exactly where last one left off.
+  // Any simpler condition can potentially lead to infinite loops and/or loops that finish too early.
+  // The assembled condition is:
+  // (uid>last_uid) || (uid==last_uid && rid>last_rid) || (uid==last_uid && rid==last_rid && gid>last_gid)

Won't it be easier to change the above ->range(0... from 0 to the current progress?

Changing the range sounded like a great idea, but failed when I tested it.

The reason is that if rows 0-199 from batch#1 process successfully, then they no longer fall into the query criteria when batch#2 starts (group_type is no longer equal to ''). In other words, rows 200-399 from batch#1 have become rows 0-199 in the context of batch#2. On the other hand, any rows from batch#1 that don't get processed successfully (for example, because $gid is invalid), still have group_type=='', so batch#2 might want rows 10-209 or rows 30-229. The uid/rid/gid conditions are necessary to filter out the unprocessed rows.

So then I considered dropping the group_type=='' condition from the query. But even then, batch#2 doesn't necessary want rows 200-399. Now the problem is that any processed rows can end up being re-ordered (because their rid/gid values have been changed). So the original rows 0-199 might now be at 0-190/201-209.
Re-ordering shouldn't be a problem as long as the group_type=='' condition is included, because all processed rows are dropped from each new query.

Any other ideas?

Oops, I didn't mean to change the status.

Reposting #19 because the testbot seems to have forgotten about it and won't even let me requeue it.

#24: og_migrate_custom_roles-1626336-19.patch queued for re-testing.

I've fixed the comments to not go over 80 chars, and added a dot in the end of comments. Commited!

@Nephele,
Btw, do you have experience with Migrate module -- #1686394: Replace custom og-migrate with a proper integration with "Migrate" module :)