user import failes for salted password hashes [#171870]

Comment	File	Size	Author
#12	gallery_report3.html	1.46 MB	sgriffin
	gallery_report.html	19.12 KB	sgriffin

Comment #1

profix898 CreditAttribution: profix898 commented 30 August 2007 at 17:35

Thanks for reporting. This looks like a PHP version issue. stripos() is not available on PHP4. I will switch my dev environment to PHP4 and debug the stuff ... shouldnt be too difficult ...

Log in or register to post comments

Comment #2

sgriffin CreditAttribution: sgriffin commented 30 August 2007 at 17:49

ok. That error is probably not related to the user synch however.. I just threw that in there while I wrote this report up. Sorry.

Log in or register to post comments

Comment #3

profix898 CreditAttribution: profix898 commented 30 August 2007 at 17:52

I have just committed a bugfix for stripos().

The groups sync stuff is a more difficult. Could you please send me the content of your 'g2_externalidmap' table? It does not contain any useful information for other people, but if you like, you can also send it to me directly 'drupal /AT/ profix898 /DOT/ de'. Thanks.

Log in or register to post comments

Comment #4

sgriffin CreditAttribution: sgriffin commented 30 August 2007 at 18:06

Sent.
Thanks for your help. I truly appreciate it.

The import did throw a lot of errors, before about groups, but I removed all non vital groups from gallery2 and the error output stopped.
Leaving only the partial import message.

Log in or register to post comments

Comment #5

sgriffin CreditAttribution: sgriffin commented 30 August 2007 at 18:13

Ah, That is a very interesting table. I will try to clean that up and see what happens.

Log in or register to post comments

Comment #6

profix898 CreditAttribution: profix898 commented 30 August 2007 at 18:21

Did you have Gallery2 embedded into any other CMS (or PHP application) before? Can you please try what happens, if you truncate the whole table!?

Log in or register to post comments

Comment #7

profix898 CreditAttribution: profix898 commented 30 August 2007 at 18:22

Upps. Missed your post above :)

Log in or register to post comments

Comment #8

sgriffin CreditAttribution: sgriffin commented 30 August 2007 at 18:30

I cleared the table out, and it threw

*

A serious error has occured. This can happen due to an incorrect configuration or a bug in the gallery module. If you plan to submit a bug report to the issue queue at drupal.org consider to include the pre-generated report file.
* Error in function '_gallery_init()' (gallery_base.inc:102):
o Unable to initialize embedded Gallery. You need to configure your embedded Gallery.
Error (ERROR_MISSING_OBJECT) : 1 GalleryUser
+ in modules/core/classes/helpers/GalleryEntityHelper_simple.class at line 124 (gallerycoreapi::error)
+ in modules/core/classes/GalleryCoreApi.class at line 2290 (galleryentityhelper_simple::loadentitybyexternalid)
+ in modules/core/classes/GalleryEmbed.class at line 215 (gallerycoreapi::loadentitybyexternalid)
+ in modules/core/classes/GalleryEmbed.class at line 120 (galleryembed::checkactiveuser)
+ in xx/modules/gallery/gallery_base.inc at line 92 (galleryembed::init)
+ in xx/modules/gallery/gallery_user_admin.inc at line 38
+ in xx/modules/gallery/gallery.module at line 210
+ in ??? at line 0
+ in xx/includes/menu.inc at line 418
+ in xx/index.php at line 15

I will try to find your commit for strippos and/or re-import the table data.

Log in or register to post comments

Comment #9

profix898 CreditAttribution: profix898 commented 30 August 2007 at 18:36

Ah, that's because the Drupal superuser (uid=1) must be in this table. Otherwise sync doesnt work. You can 1. restore only this single entry or 2. go through the install wizard (admin/settings/gallery/install) again to get the entry back.

Log in or register to post comments

Comment #10

sgriffin CreditAttribution: sgriffin commented 30 August 2007 at 19:30

I re-imported the orginal gallery2 db from the old site.. but with the dev version of the module I get the stipos error from
admin/settings/gallery/install
Fatal error: Call to undefined function: stripos() in /home/xxx/modules/gallery/gallery_install.inc on line 883

Whats the fix?

Log in or register to post comments

Comment #11

profix898 CreditAttribution: profix898 commented 30 August 2007 at 20:12

It takes up to 8h until 5.x-2.x-dev is packaged again, but you can get the latest code from cvs directly (http://cvs.drupal.org/viewvc.py/drupal/contributions/modules/gallery/gal...).

I think you should try to ...
1. empty the externalidmap table of Galllery2
2. install the gallery module in Drupal (by going through the install wizard)
3. then try to import the users/groups again

Because you attached the debug trace with your original report, it is easily possible to see where the error occurred. The module complains that the G2 entity that is mapped to the Drupal role 4 (G2 group id is 3546) could not be loaded. The only question is why this happens. One reason could be an obsolete entry in this table. Thats why I asked you to flush the table.

Log in or register to post comments

Comment #12

sgriffin CreditAttribution: sgriffin commented 30 August 2007 at 21:23

File	Size
gallery_report3.html	1.46 MB

I installed your commit
I flushed the table.
I went to install.
Found a critical error in the user synch part
did a Reset and & Restart

It threw this error

* G2 groups for G2 user (uid: 3544):
Array
(
[4] => Everybody
[2] => Registered Users
[3] => Site Admins
)
* Drupal roles for Drupal user (uid: 1):
Array
(
[2] => 2
[1] => 1
)
* Drupal roles <> G2 groups map (for Drupal user):
Array
(
[1] => 4
[2] => 2
)
* Drupal roles <> G2 groups map (for G2 user):
Array
(
[27] => 3
[2] => 2
[1] => 4
)
* Remove user from these groups:
Array
(
[27] => 3
)
*

A serious error has occured. This can happen due to an incorrect configuration or a bug in the gallery module. If you plan to submit a bug report to the issue queue at drupal.org consider to include the pre-generated report file.
* Error in function '_gallery_groups_user()' (gallery_groups.inc:55):
o Error removing user from Gallery group (Gallery Group Id: 3)
Error (ERROR_STORAGE_FAILURE)
+ in modules/core/classes/GalleryStorage/GalleryStorageExtras.class at line 1106 (gallerycoreapi::error)
+ in modules/core/classes/GalleryStorage.class at line 517 (gallerystorageextras::removemapentry)
+ in modules/core/classes/GalleryCoreApi.class at line 2815 (mysqlstorage::removemapentry)
+ in modules/imageblock/classes/ImageBlockHelper.class at line 560 (gallerycoreapi::removemapentry)
+ in modules/core/classes/helpers/GalleryEventHelper_simple.class at line 117 (imageblockhelper::handleevent)
+ in modules/core/classes/GalleryCoreApi.class at line 2132 (galleryeventhelper_simple::postevent)
+ in modules/core/classes/helpers/GalleryUserGroupHelper_medium.class at line 92 (gallerycoreapi::postevent)
+ in modules/core/classes/GalleryCoreApi.class at line 1723 (galleryusergrouphelper_medium::removeuserfromgroup)
+ in /home/southern/public_html/soundzabound/modules/gallery/gallery_groups.inc at line 52 (gallerycoreapi::removeuserfromgroup)
+ in /home/southern/public_html/soundzabound/modules/gallery/gallery_user.inc at line 172
+ in /home/southern/public_html/soundzabound/modules/gallery/gallery_install.inc at line 648
+ in ??? at line 0
+ in /home/southern/public_html/soundzabound/includes/form.inc at line 428
+ in /home/southern/public_html/soundzabound/includes/form.inc at line 258
+ in /home/southern/public_html/soundzabound/includes/form.inc at line 80
+ in /home/southern/public_html/soundzabound/modules/gallery/gallery.module at line 171
+ in ??? at line 0
+ in /home/southern/public_html/soundzabound/includes/menu.inc at line 418
+ in /home/southern/public_html/soundzabound/index.php at line 15
*

A serious error has occured. This can happen due to an incorrect configuration or a bug in the gallery module. If you plan to submit a bug report to the issue queue at drupal.org consider to include the pre-generated report file.
* Error in function 'gallery_user_modify()' (gallery_user.inc:186):
o Error adding user to Gallery group (3)
Error (ERROR_STORAGE_FAILURE)
+ in modules/core/classes/GalleryStorage/GalleryStorageExtras.class at line 1106 (gallerycoreapi::error)
+ in modules/core/classes/GalleryStorage.class at line 517 (gallerystorageextras::removemapentry)
+ in modules/core/classes/GalleryCoreApi.class at line 2815 (mysqlstorage::removemapentry)
+ in modules/imageblock/classes/ImageBlockHelper.class at line 560 (gallerycoreapi::removemapentry)
+ in modules/core/classes/helpers/GalleryEventHelper_simple.class at line 117 (imageblockhelper::handleevent)
+ in modules/core/classes/GalleryCoreApi.class at line 2132 (galleryeventhelper_simple::postevent)
+ in modules/core/classes/helpers/GalleryUserGroupHelper_medium.class at line 62 (gallerycoreapi::postevent)
+ in modules/core/classes/GalleryCoreApi.class at line 1710 (galleryusergrouphelper_medium::addusertogroup)
+ in /home/southern/public_html/soundzabound/modules/gallery/gallery_user.inc at line 183 (gallerycoreapi::addusertogroup)
+ in /home/southern/public_html/soundzabound/modules/gallery/gallery_install.inc at line 648
+ in ??? at line 0
+ in /home/southern/public_html/soundzabound/includes/form.inc at line 428
+ in /home/southern/public_html/soundzabound/includes/form.inc at line 258
+ in /home/southern/public_html/soundzabound/includes/form.inc at line 80
+ in /home/southern/public_html/soundzabound/modules/gallery/gallery.module at line 171
+ in ??? at line 0
+ in /home/southern/public_html/soundzabound/includes/menu.inc at line 418
+ in /home/southern/public_html/soundzabound/index.php at line 15
* G2 groups for G2 user (uid: 3544):
Array
(
[4] => Everybody
[2] => Registered Users
[3] => Site Admins
)
* Drupal roles for Drupal user (uid: 1):
Array
(
[2] => 2
[1] => 1
)
* Drupal roles <> G2 groups map (for Drupal user):
Array
(
[1] => 4
[2] => 2
)
* Drupal roles <> G2 groups map (for G2 user):
Array
(
[27] => 3
[2] => 2
[1] => 4
)
* Remove user from these groups:
Array
(
[27] => 3
)
*

A serious error has occured. This can happen due to an incorrect configuration or a bug in the gallery module. If you plan to submit a bug report to the issue queue at drupal.org consider to include the pre-generated report file.
* Error in function '_gallery_groups_user()' (gallery_groups.inc:55):
o Error removing user from Gallery group (Gallery Group Id: 3)
Error (ERROR_STORAGE_FAILURE)
+ in modules/core/classes/GalleryStorage/GalleryStorageExtras.class at line 1106 (gallerycoreapi::error)
+ in modules/core/classes/GalleryStorage.class at line 517 (gallerystorageextras::removemapentry)
+ in modules/core/classes/GalleryCoreApi.class at line 2815 (mysqlstorage::removemapentry)
+ in modules/imageblock/classes/ImageBlockHelper.class at line 560 (gallerycoreapi::removemapentry)
+ in modules/core/classes/helpers/GalleryEventHelper_simple.class at line 117 (imageblockhelper::handleevent)
+ in modules/core/classes/GalleryCoreApi.class at line 2132 (galleryeventhelper_simple::postevent)
+ in modules/core/classes/helpers/GalleryUserGroupHelper_medium.class at line 92 (gallerycoreapi::postevent)
+ in modules/core/classes/GalleryCoreApi.class at line 1723 (galleryusergrouphelper_medium::removeuserfromgroup)
+ in /home/southern/public_html/soundzabound/modules/gallery/gallery_groups.inc at line 52 (gallerycoreapi::removeuserfromgroup)
+ in /home/southern/public_html/soundzabound/modules/gallery/gallery_user.inc at line 172
+ in /home/southern/public_html/soundzabound/modules/gallery/gallery_user_admin.inc at line 264
+ in /home/southern/public_html/soundzabound/modules/gallery/gallery_user_admin.inc at line 306
+ in /home/southern/public_html/soundzabound/modules/gallery/gallery.module at line 220
+ in ??? at line 0
+ in /home/southern/public_html/soundzabound/includes/menu.inc at line 418
+ in /home/southern/public_html/soundzabound/index.php at line 15
*

A serious error has occured. This can happen due to an incorrect configuration or a bug in the gallery module. If you plan to submit a bug report to the issue queue at drupal.org consider to include the pre-generated report file.
* Error in function 'gallery_user_modify()' (gallery_user.inc:186):
o Error adding user to Gallery group (3)
Error (ERROR_STORAGE_FAILURE)
+ in modules/core/classes/GalleryStorage/GalleryStorageExtras.class at line 1106 (gallerycoreapi::error)
+ in modules/core/classes/GalleryStorage.class at line 517 (gallerystorageextras::removemapentry)
+ in modules/core/classes/GalleryCoreApi.class at line 2815 (mysqlstorage::removemapentry)
+ in modules/imageblock/classes/ImageBlockHelper.class at line 560 (gallerycoreapi::removemapentry)
+ in modules/core/classes/helpers/GalleryEventHelper_simple.class at line 117 (imageblockhelper::handleevent)
+ in modules/core/classes/GalleryCoreApi.class at line 2132 (galleryeventhelper_simple::postevent)
+ in modules/core/classes/helpers/GalleryUserGroupHelper_medium.class at line 62 (gallerycoreapi::postevent)
+ in modules/core/classes/GalleryCoreApi.class at line 1710 (galleryusergrouphelper_medium::addusertogroup)
+ in /home/southern/public_html/soundzabound/modules/gallery/gallery_user.inc at line 183 (gallerycoreapi::addusertogroup)
+ in /home/southern/public_html/soundzabound/modules/gallery/gallery_user_admin.inc at line 264
+ in /home/southern/public_html/soundzabound/modules/gallery/gallery_user_admin.inc at line 306
+ in /home/southern/public_html/soundzabound/modules/gallery/gallery.module at line 220
+ in ??? at line 0
+ in /home/southern/public_html/soundzabound/includes/menu.inc at line 418
+ in /home/southern/public_html/soundzabound/index.php at line 15
* User synchronization (partially) failed.

Log in or register to post comments

Comment #13

profix898 CreditAttribution: profix898 commented 30 August 2007 at 21:32

Its time to sleep where I am (Germany) and it will take some time to debug this problem in depth. I know of at least 3 people, that were using the user/group import without issues. I really wonder why this fails for you. However I will work on this tomorrow. Stay tuned :)

Log in or register to post comments

Comment #14

sgriffin CreditAttribution: sgriffin commented 30 August 2007 at 21:58

Thank you, I appreciate all your efforts. Email me directly if you require direct access.

Log in or register to post comments

Comment #15

sgriffin CreditAttribution: sgriffin commented 31 August 2007 at 16:23

Is there a way to do the user import manually?
How does the gallery module authenticate against gallery hash?
Or how does the user record look when a user has been imported?

I will attempt a fresh install of 2.2.3 in order to answer some of these questions.

Log in or register to post comments

Comment #16

sgriffin CreditAttribution: sgriffin commented 31 August 2007 at 17:10

Weird.
I have learned that the password field in drupal is varchar(32) and in gallery2 the password is 38 characters..
When the synch is done.. both passwords are shortened to 32 and screwed up.

Log in or register to post comments

Comment #17

sgriffin CreditAttribution: sgriffin commented 31 August 2007 at 17:46

Changing the structure of the drupal pw to 38 or higher, fixes a bug in that you are still able to login to gallery2 with the username and password after the drupal synch.

However, logging into drupal still fails.

Conversely, if you set the password in drupal, logging into drupal and gallery2 works.

I don't understand how gallery2 does its hashing, it seems that a unique hash is made each time even it's the same pw.. some salt method I guess.

Does someone need a bounty to get this working?

Log in or register to post comments

Comment #18

profix898 CreditAttribution: profix898 commented 2 September 2007 at 22:31

Changing the structure of the drupal pw to 38 or higher, fixes a bug in that you are still able to login to gallery2 with the username and password after the drupal synch. However, logging into drupal still fails.

Looking through the code, you reminded me that the import code was incomplete. And you were right on the varchar(32) stuff. The password must be md5()ed for Drupal. I've committed a follow-up patch for the user-import code, that solves two problems:

It insert a dummy password (md5-hash of the G2 hash) into the Drupal db solving the issue with fieldsize of 32 vs. 38 characters
It implements hook_auth() to authenticate users against G2 (this was missing)

The whole import now works like this: the G2 user is imported into Drupal and the password is set to a dummy hash. Because of this any logon attempt of imported users will fail against the Drupal DB (thats what you experienced). However Drupal will also call hook_auth() trying to authenticate the user against external sources as well. And since gallery_auth() is now implemented, it will succeed to authenticate the user against G2 directly. When this happens, the clear password is md5()ed and inserted into the Drupal DB, so that next time the user logs in authentication will be successful in Drupal directly.

Conversely, if you set the password in drupal, logging into drupal and gallery2 works.

Yes, thats because G2 excepts both a md5-hash and a salted hash of the format $salt + md5($salt, $password).

How does the gallery module authenticate against gallery hash?

There is an utility function GalleryUtilities::isCorrectPassword($password, $hashedPassword), that calculates the salted password from the clear password and validates it against the provided hash version of the password. This exactly is done in hook_auth() now.

As it is impossible to calculate the md5-hash from the salted password hash, you cant import the users and disable the gallery.module afterwards. You will need to wait until all users have logged in at least once, so that the passwords are collected and stored correctly in the Drupal DB.

Is there a way to do the user import manually?

In principle: Yes. But it's not easy. You need to export all users (and details) from G2 and add them to the Drupal 'users' table (and 'profile' table if you need the fullnames). The second step is to build up the 'externalIdMap' table in G2 which contains the mapping of Drupal user ids to G2 user ids. If you need to preserve the groups/roles assignments, its getting even more difficult. All in all, I guess, it will be easier to understand why the module fails to import your users and fix the bug ;)

Thank you, I appreciate all your efforts. Email me directly if you require direct access.

I didnt have much time on the weekend to look at this, but I will walk through the code again on Monday. Did you make any manual modifications to the G2 tables? The debug dump you attached above, somehow doesnt look right. For example the Drupal superuser (uid=1) is mapped to the G2 user id 3544. It might be helpful to have a anonymized dump of your G2 userbase, but let me first look through the code again.

Does someone need a bounty to get this working?

I know some people out there are using gallery.module for their commercial sites and I take it very serious to contribute quality code. However I'm doing all the work on this module in my free time and it sometimes takes a few days to find bugs (esp. in new/unstable module versions like 5.x-2.x-dev). No need to push me ...

Log in or register to post comments

Comment #19

sgriffin CreditAttribution: sgriffin commented 3 September 2007 at 15:43

For my environment, I have a clean install of the galley 2 dev module from sept 2, and a fresh install of drupal 5.2 and an empty gallery 2.

I create a user in gallery 2 first.
When I do the import from gallery2, the password hash in the gallery2 db changes completely now. Whereas before it would only be truncated if the structure of drupal pw was not fixed.

In this case the login now fails for both.
For user settings, nothing is checked, and only import is checked on the adv synch tab.

Log in or register to post comments

Comment #20

sgriffin CreditAttribution: sgriffin commented 3 September 2007 at 21:56

Another thing I don't understand,.. I tried repairing/replacing the g2 hash with the previous hash.. and login fails.. I even replaced the db with the db that I used before I did the user synch and the login fails.
Thats interesting.

Log in or register to post comments

Comment #21

sgriffin CreditAttribution: sgriffin commented 4 September 2007 at 04:39

I believe I maybe running into a cache'ing issue I don't quite understand.
I blew out the hash completely, and was still able to authenticate against it with the password. I even cleared, the session table.
This would explain why I am not able to authenticate against a good hash, and why I can authenticate against garbage, and not be able to restore a previous db and authenticate against it.
A server reboot fixed the db restore problem, so thats a clue.
Also logging into gallery and changing the password there works.

Perhaps something with the QUERY CACHE;

Log in or register to post comments

Comment #22

profix898 CreditAttribution: profix898 commented 4 September 2007 at 12:27

Not sure about the QUERY CACHE stuff, I dont have it enabled on my dev-box. However I've found some bugs in the user import code introduced recently with a (considered) minor reorganization of the code. I will need to test it in more detail before I publish the patch or commit it to cvs. But help is on the way ... ;)

Log in or register to post comments

Comment #23

sgriffin CreditAttribution: sgriffin commented 4 September 2007 at 15:31

Thanks, I research query cache, and apparently it can not fetch stale data. Somehow, I guess I'm attaching to the old session.
I guess just messing around in phpmyadmin requires an extra step. I'm at a loss at the moment to understand how I can delete g2_hashedPassword for a user and still be able to authenticate.
when I deleted the record, It did stop it. but I could reinsert a record without a hash, and authenticate.

This is all besides the point hopefully.

Log in or register to post comments

Comment #24

profix898 CreditAttribution: profix898 commented 5 September 2007 at 16:18

Title:	User synchronization (partially) failed.	» user import failes for salted password hashes
Status:	Active	» Fixed

I've just committed a patch to fix user import with salted G2 hashes. In my dev environment import works now using both md5 and salted hash methods in G2. As explained above, whenever a user with a salted hash gets imported, a dummy password is inserted into the Drupal DB (the password hash in the G2 DB remains unchanged). If the user tries to login, he/she is authenticated against the G2 DB directly, the md5 hash is calculated from the password and inserted in the Drupal DB, so that next time he/she can be authenticated in Drupal internally.

Regarding 'query cache' stuff: I just received a private message from a user complaining about G2 fetching invalid data as well. I have created a new issue for this at http://drupal.org/node/173300.
In the meantime I suggest that you setup a G2/Drupal installation on a different host (e.g. your desktop with caching disabled), perform the user import and move the resulting {users} table to your working site.

Please let me know if import of salted password hashes still fails for you ...

Log in or register to post comments

Comment #25

sgriffin CreditAttribution: sgriffin commented 5 September 2007 at 18:50

Oh my god, I think it worked.
On the test site, import worked, logging into drupal was successful and logging back into gallery2 was successful.

It will take me some time to get back to you with a live data test.
Plus figuring out whats wrong with my user db or group map.

Thank you so much!

Am I to understand I can edit the external map table so that
g_externalId = the drupal role
g_entityId = the gallery role
So that the roles are mapped correctly?

Log in or register to post comments

Comment #26

profix898 CreditAttribution: profix898 commented 5 September 2007 at 18:58

Oh my god, I think it worked. On the test site, import worked, logging into drupal was successful and logging back into gallery2 was successful. ... Thank you so much!

Glad to read that :) You're welcome.

Am I to understand I can edit the external map table ... So that the roles are mapped correctly?

Yes. Thats exactly how this table works. Make sure to set 'g_entityType' to 'GalleryGroup' for groups mapping and to 'GalleryUser' for user mapping. Otherwise manually editing this table shouldnt break anything on your site.