Migration from Drupal 7 to Drupal 9

Not yet, but it should be trivial to do so, at least for the TFA module settings. Ideally, the migration would also handle the user data information, but it is probably OK to ask all the users to re-login after the update anyway.

Interesting, @jcnventura -- so there is no defined path or even set of guidelines for a migration of TFA for users at this point. I was also interested in the thread above. On a project, that I am working on we have about ~2,500 active users on D7 going to a D9 site, and it would be ideal to not have to force our users to set it up again. We only have a limited support team running TFA help sessions with users.

I think the conceptual part that I am struggling with is what would be actually migrated or ported over in the user migration to make this work, so that users do not have to set up their TFA codes again. If we could set up the general module settings, but what is actually attached to the user entity to make the TFA code work that we could avoid having them redo their TFA codes on their handheld devices, etc? Please feel free also to correct anything that I am saying as well :).

Equivocation: I am only aware of the core concepts of the TFA module and the general implementation, but I would be curious if folks might be able to sketch this out concretely because this seems like a common occurrence (?).

My problem is that I only run Drupal 9 (and 10) at the moment, so migrations from D7 are something I no longer do myself, but I'm willing to add it to the module if someone else is willing to do it. From one of the modules that I was maintaining more often during the D7 -> D8 transition (5 years ago), there is this example on how to migrate the settings: https://git.drupalcode.org/project/adsense/-/blob/8.x-1.x/migrations/d7_...

As to how to migrate the user_data, to be honest that should usually be handled by Drupal core, as that information is managed by core. You can always test a D7->D10 migration to see what happens to the contents of the users_data table after the migration.

At some point, and since this module is used in Drupal.org, it is very likely that when d.o wants to move to Drupal 10, someone will really take an interest in writing that migration. If you can't wait so long, then you might need to start work on it yourself.

Hi @JackG102 I am also trying to migrate TFA data from Drupal 7 to Drupal 9.

Were you able to migrate the data of TFA? Thanks.

I've pushed an initial prototype of this to the issue-fork.

I have not yet run a full end-to-end test on this yet as my d7 and d8 labs are not currently setup with the same cryptography.

This code at the moment is mostly provided to a base to jump-start discussion. Ideally sites should have been off D7 before Aug 1 deadline when D.O. and the DST cut support going forward, however without having a migration in place we certainly have contributed to sites missing that transfer date.

We do need some form of migration as none of this is stored in the D7 equvilient of user_data and without migrating the token data this isn't just "users need to log in again" this is "users need to setup their tokens again" which would be a security failure.

Key items as this sample exists now:

• No config is imported, TFA must be manually configured in D8+. Import of config may be doable I just have not looked into it since the real necessity as noted in #3 is that we get the token data.
• It is required that the same encryption is used on the source D7 install and the destination D8+ install as this makes no attempt to rotate encryption settings. We might need to check with the Encrypt module on if they plan to create a process plugin for encryption rotation during migrations.
• No attempt to import the trusted browsers has yet been made. This also may be doable however since the impact of not doing so is just a UX requirement that a user is prompted to provide a token it may be a much lower priority.

Concerns:
The tfa_basic module doesn't appear to list the TOTP plugin or the Trusted Browser plugin as a configured plugin in its users data field so when we are currently not importing that they are enabled. This may require a consult with the tfa_basic maintainer to determine how to detect and support.

I too have this requirement and I have worked on the logic for the data migration from D7 to D10. But I am having some issue with encryption. I would like to have some input on the error I am getting.

I have made a contributed module for TFA Migration I would like to someone other than me to test it and provide input. Currently the module has functionality to migrate TFA Seed and TFA Users Settings.

My Workflow is as follows :

Get the TFA Seed from D7 and decrypt it using the private key from D7 and Encrypt it with the Key from D10.

I am able to create the functionality to process this. I am getting the correct seed size in D10 but seems decryption and encryption logic need correction as I am getting below error on D10 once form is submitted after OTP is enter in the site.

The website encountered an unexpected error. Please try again later.

RangeException: Base32::doDecode() only expects characters in the correct base32 alphabet in ParagonIE\ConstantTime\Base32::doDecode() (line 421 of /var/www/html/vendor/paragonie/constant_time_encoding/src/Base32.php).
ParagonIE\ConstantTime\Base32::decodeUpper('') (Line: 81)
ParagonIE\ConstantTime\Encoding::base32DecodeUpper('') (Line: 266)
Drupal\tfa\Plugin\TfaValidation\TfaTotpValidation->validate('405072') (Line: 222)
Drupal\tfa\Plugin\TfaValidation\TfaTotpValidation->validateForm(Array, Object) (Line: 273)
Drupal\tfa\Form\EntryForm->validateForm(Array, Object)
call_user_func_array(Array, Array) (Line: 82)
Drupal\Core\Form\FormValidator->executeValidateHandlers(Array, Object) (Line: 275)
Drupal\Core\Form\FormValidator->doValidateForm(Array, Object, 'tfa_entry_form') (Line: 118)
Drupal\Core\Form\FormValidator->validateForm('tfa_entry_form', Array, Object) (Line: 593)
Drupal\Core\Form\FormBuilder->processForm('tfa_entry_form', Array, Object) (Line: 325)
Drupal\Core\Form\FormBuilder->buildForm(Object, Object) (Line: 73)
Drupal\Core\Controller\FormController->getContentResult(Object, Object)
call_user_func_array(Array, Array) (Line: 123)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->Drupal\Core\EventSubscriber\{closure}() (Line: 592)
Drupal\Core\Render\Renderer->executeInRenderContext(Object, Object) (Line: 121)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->wrapControllerExecutionInRenderContext(Array, Array) (Line: 97)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->Drupal\Core\EventSubscriber\{closure}() (Line: 182)
Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object, 1) (Line: 76)
Symfony\Component\HttpKernel\HttpKernel->handle(Object, 1, 1) (Line: 44)
Drupal\redirect_after_login\RedirectMiddleware->handle(Object, 1, 1) (Line: 58)
Drupal\Core\StackMiddleware\Session->handle(Object, 1, 1) (Line: 48)
Drupal\Core\StackMiddleware\KernelPreHandle->handle(Object, 1, 1) (Line: 106)
Drupal\page_cache\StackMiddleware\PageCache->pass(Object, 1, 1) (Line: 85)
Drupal\page_cache\StackMiddleware\PageCache->handle(Object, 1, 1) (Line: 48)
Drupal\Core\StackMiddleware\ReverseProxyMiddleware->handle(Object, 1, 1) (Line: 51)
Drupal\Core\StackMiddleware\NegotiationMiddleware->handle(Object, 1, 1) (Line: 51)
Drupal\Core\StackMiddleware\StackedHttpKernel->handle(Object, 1, 1) (Line: 704)
Drupal\Core\DrupalKernel->handle(Object) (Line: 19)

Any advice will be helpful. My logic for decryption and encryption is available here

@cmlara I have looked into your code in the Fork, I believe it wont work as the seed size is different in D7 and D10.

In the code you are just copying seed data from D7 to D10 without processing.

I think my approach is correct.

D7 Seed 64 Chars long 

gIhLBSTBzzDOi8D0LceftNtyFHyZVTttmB2WKzlIvrIXS2KnJcAGaTYe3gxzX33r

D10 268 Chars long  

ZGVmNTAyMDA2YmU2MTljZTFjNjA1YmI5MmQwZjEyM2ZkNmU0M2RhODE0ZGM1NWE0NTQyYTdiNWJjNWE0ZDNhMjAwMTFhNzYxMmU2NGE5MzMxYWU0MWIzNzhlMDg0YTcwMWJiNmQ4MzY0ZWI5MjBiZDE3ZjA5NzZiMmRhNDNlNzNkMzAxYzUzODU3NTFkNDY3ZTE0MDkxOWVjMjk3ZjY0MDE4YTY4NDMyMDIwNmZmYmQyMTAyY2RhOWJjZDQwNTNiN2RmMTZiNzM=

Coming to accepted codes I think we can not utlize migrated data as in Drupal 7 Accepted codes are stored using this logic

    $hash = hash('sha1', drupal_get_hash_salt() . $code);

As the Accepted code is stored in the form of SHA1 + salt hash . It is highly impossible to reverse the hash and get the original OTP . So It is not possible to verify against the codes used in D7 in D10. I am not sure how to proceed here.

This issue partially stalled out on waiting for tfa_basic to fix [# 3386346], though with recent security fixes we may need to evaluate if that is still a blocker.

Coming to accepted codes I think we can not utlize migrated data as in Drupal 7 Accepted codes are stored using this logic

That does indeed pose an issue, as we now use sha256.
We could still import the previous used codes and it would require a custom plugin that can review older tokens as well as current.

I will note for the TFA included plugins we are working to move away from accepted codes in #3403462: Remove usage of alreadyAcceptedCode()/storeAccepedCode() in the TOTP,HOTP, Recovery Code plugins.. For the tfa_basic/totp plugin migrations could do similar as we plan to do by assuming the last accepted token is (roughly) time()/30 + Max Window Count

In the code you are just copying seed data from D7 to D10 without processing.

It was written (but never tested) under the assumption that one will use the same encryption on the source and destination. This means that the data should be returned by decryption the same in D7 and D10.

Seed length isn't necessarily a good indicator of functioning correctly, its dependent upon both the secret length, and encryption method (including decryption method key length)

Regarding your error in tfa_migrate:

RangeException: Base32::doDecode() only expects characters in the correct base32 alphabet in ParagonIE\ConstantTime\Base32::doDecode() (line 421 of /var/www/html/vendor/paragonie/constant_time_encoding/src/Base32.php).
ParagonIE\ConstantTime\Bas e32::decodeUpper('') (Line: 81)
ParagonIE\ConstantTime\Encoding::base32DecodeUpper('') (Line: 266)
Drupal\tfa\Plugin\TfaValidation\TfaTotpValidation->validate('931540') (Line: 222)

This appears an empty value is being provided to the base32 decoder, getSeed() will return false if the seed is empty (the Real AES Encrypt plugin has been known to return an empty string rather than throw an exception in its early releases, I can't recall if/when this was fixed.) This also could be a fault in conversion with the data not stored correctly.

Coming to accepted codes I think we can not utlize migrated data as in Drupal 7 Accepted codes are stored using this logic

That does indeed pose an issue, as we now use sha256.
We could still import the previous used codes and it would require a custom plugin that can review older tokens as well as current.

For that we need to store D7 Site Hash in the D10 Site. Got any suggestions on where we can store it ? Normally we would store it in migration related configurations as we will be normally uninstalling migration related modules. But we can't do it here as we need that value later. Should we add a new configuration in TFA module ?

During Migration we can prefix old TFA Accepted codes with legacy word.

Got any suggestions on where we can store it ?

Ideally the 7.x->8.x conversion would of been directly compatible in TfaPluginBase, however we are long past being able to correct that. At this point I'm not sure we can make this TFA responsibility.

it would likely be the plugin provider responsibility to store the old site hashes and to lookup if any are needed. TFA wouldn't need to know that older codes are being queried as that would happen inside the plugins.

During Migration we can prefix old TFA Accepted codes with legacy word.

#3404162: storeAcceptedCode()/alreadyAcceptedCode should check token id. would be relevant, at the moment the code storage has no support for naming the source of a used validation code. That is assuming we even keep storing validated codes in 2.x. Once we no longer use it internally there is room to question if we should leave it on the base API or not.

I think we can use the below logic to validate the Legacy Accepted Code.(untested)

  /**
   * Check whether the code has been used in Legacy System.
   *
   * @param string $code
   *   Entered OTP Code.
   * @param string $hash_salt
   *   Legacy Site Hash.
   * @param int $uid
   *   UID of the User.
   *
   * @return bool
   *   Returns whether the code already used or not.
   */
  protected function alreadyAcceptedCodeInLegacySystem(string $code, string $hash_salt, int $uid): bool {

    $fields = [
      'uid',
      'module',
    ];

    $hash = hash('sha1', $hash_salt . $code);

    $legacy_code = 'legacy_tfa_accepted_code_' . $hash;

    $query = $this->connection->select('users_data', 'ud')
      ->fields('ud', $fields)
      ->condition('name', $legacy_code)
      ->condition('uid', $uid)
      ->condition('module', 'tfa')
      ->execute();

    $result = $query->fetchAssoc();

    if (!empty($result)) {
      return TRUE;
    }

    return FALSE;

  }

We can Just plug-in in the above code in the validate function and add OR condition to check legacy and current system accepted codes.

That would (minus the breaches of API by directly querying the user_data table) look roughly correct for what a plugin would need to do to lookup historical codes.

I think I have fixed the issue #3420305: Fix Decoding error after migration can any one test it and confirm ?

I believe this was solved by the TFA Migration module