Problem/Motivation

The Brightcove module needs API access tokens, which expire every 5 minutes. Thereby it is using the KV expireable to invalidate these access tokens.
In a normal HTTP request, this is cool, but during a migration we ran into an issue, when the migration actually executes for longer than 5 minutes.
Along the long-running migration process, we REQUEST_TIME used inside the KV expireable never changes, even the real world time moves on.

In short, REQUEST_TIME does not equal to the \Drupal::time()->getCurrentTime().

Proposed resolution

Use the actual time in \Drupal\Core\KeyValueStore\DatabaseStorageExpirable. \Drupal\user\SharedTempStore and \Drupal\user\PrivateTempStore by leveraging the datetime.time service and its ::getCurrentTime method.

Remaining tasks

User interface changes

API changes

Data model changes

CommentFileSizeAuthor
#49 2855457-49.patch23.39 KBjungle
#48 2855457-48.patch0 bytesjungle
#48 interdiff-46-48.txt2.83 KBjungle
#46 interdiff-44-46.txt4.87 KBjungle
#46 2855457-46.patch22.11 KBjungle
#44 2855457-44.patch22.14 KBjungle
#44 interdiff-42-44.txt15.34 KBjungle
#42 interdiff_41_42.txt10.57 KBmukesh88
#42 2855457-42.patch21.52 KBmukesh88
#41 interdiff_39-41.txt6.98 KBsahil.goyal
#41 2855457-41.patch20.28 KBsahil.goyal
#39 interdiff_38-39.txt2.26 KBRishabh Vishwakarma
#39 2855457-39.patch18.12 KBRishabh Vishwakarma
#38 2855457-38.patch18.82 KBkkalashnikov
#30 interdiff-28-30.txt8.98 KBjungle
#30 2855457-30.patch31.32 KBjungle
#28 interdiff-27-28.txt9.73 KBthalles
#28 2855457-28.patch30.97 KBthalles
#27 2855457-27.patch29.88 KBmoltam
#22 interdiff-20-22.txt2.95 KBmpdonadio
#22 2855457-22.patch28.75 KBmpdonadio
#20 2855457-20.patch27.12 KBmpdonadio
#8 interdiff-2855457-4-8.txt5.06 KBshadcn
#8 use_the_exact_current-2855457-8.patch20.7 KBshadcn
#4 use_the_exact_current-2855457-4.patch16.67 KBshadcn
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

dawehner created an issue. See original summary.

dawehner’s picture

Issue summary: View changes
dawehner’s picture

Issue tags: +Novice, +php-novice
shadcn’s picture

Status: Active » Needs review
FileSize
16.67 KB

OK let's see what fails.

mpdonadio’s picture

Looks good.

  1. +++ b/core/lib/Drupal/Core/KeyValueStore/DatabaseStorageExpirable.php
    @@ -25,9 +33,11 @@ class DatabaseStorageExpirable extends DatabaseStorage implements KeyValueStoreE
    +   * @param \Drupal\Component\Datetime\TimeInterface $time
    

    Nit, needs to have a comment on the line below it, something like "The injected Time service."

  2. +++ b/core/lib/Drupal/Core/KeyValueStore/DatabaseStorageExpirable.php
    @@ -25,9 +33,11 @@ class DatabaseStorageExpirable extends DatabaseStorage implements KeyValueStoreE
    -  public function __construct($collection, SerializationInterface $serializer, Connection $connection, $table = 'key_value_expire') {
    +  public function __construct($collection, SerializationInterface $serializer, Connection $connection, $table = 'key_value_expire', TimeInterface $time = NULL) {
         parent::__construct($collection, $serializer, $connection, $table);
    +    $this->time = $time ?: \Drupal::service('datetime.time');
    

    This is the only place in the patch where the time service is optional. The patch should be consistent; either do this on all of the constructors or non of them. Of the options, I think it should be mandatory. If it is mandatory, then the patch needs an empty post_update hook to make sure the container gets rebuilt (prob in system.module?).

  3. +++ b/core/modules/user/src/PrivateTempStore.php
    @@ -81,12 +89,13 @@ class PrivateTempStore {
    -  public function __construct(KeyValueStoreExpirableInterface $storage, LockBackendInterface $lock_backend, AccountProxyInterface $current_user, RequestStack $request_stack, $expire = 604800) {
    +  public function __construct(KeyValueStoreExpirableInterface $storage, LockBackendInterface $lock_backend, AccountProxyInterface $current_user, RequestStack $request_stack, $expire = 604800, TimeInterface $time) {
    

    Looks like missing a @param?

shadcn’s picture

Assigned: Unassigned » shadcn

Thanks @mpdonadio. I'll update the patch.

dawehner’s picture

IMHO ideally we would have some test coverage here. For example you could use an additional sleep() call here.

shadcn’s picture

OK let's test this one.

  1. Added the comment.
  2. Made $time mandatory. Updated a few other places.
  3. Added missing @param.
dawehner’s picture

I'm confused, I could not find a real test in your patch.

  1. +++ b/core/modules/system/system.post_update.php
    @@ -65,3 +65,10 @@ function system_post_update_add_region_to_entity_displays() {
    +/**
    + * Rebuild the container after injecting time service in tempstore.
    + */
    +function system_post_update_inject_time_service_in_tempstore() {
    +  // Empty post-update hook.
    +}
    

    This is not needed as we will rebuild the container anyway when moving to 8.4.

  2. +++ b/core/modules/user/tests/src/Unit/PrivateTempStoreTest.php
    @@ -79,12 +86,14 @@ protected function setUp() {
    +    $this->time = $this->getMock('Drupal\Component\Datetime\TimeInterface');
    
    +++ b/core/modules/user/tests/src/Unit/SharedTempStoreTest.php
    @@ -74,12 +81,14 @@ protected function setUp() {
    +    $this->time = $this->getMock('Drupal\Component\Datetime\TimeInterface');
    

    Note: you can use TimeInterface::class

mpdonadio’s picture

Issue tags: +Needs tests

@dawener, so this is a bug-report but we are only going to add it to 8.4.x and not cherry pick to 8.3.x? That is what I mentioned the post update hook. If that is the case, we should probably change this to a Task.

shadcn’s picture

What kind of tests do we need here?

Do \Drupal\KernelTests\Core\KeyValueStore\DatabaseStorageExpirableTest, \Drupal\Tests\user\Unit\SharedTempStoreTest and \Drupal\Tests\user\Unit\PrivateTempStoreTest cover these?

dawehner’s picture

@dawener, so this is a bug-report but we are only going to add it to 8.4.x and not cherry pick to 8.3.x? That is what I mentioned the post update hook. If that is the case, we should probably change this to a Task.

Personally I'd be totally fine with that.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.0-alpha1 will be released the week of July 31, 2017, which means new developments and disruptive changes should now be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.0-alpha1 will be released the week of January 17, 2018, which means new developments and disruptive changes should now be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

borisson_’s picture

Status: Needs review » Needs work

#11, it looks like all of them are testing this, I think this can be a new test-method in the kernel test.

Setting to needs work to add the tests.

@dawehner, if you have a better idea where the test should go, I would be helpful to provide more direction.

dawehner’s picture

For me at least \Drupal\KernelTests\Core\KeyValueStore\DatabaseStorageExpirableTest sounds like a perfectly appropriate place.

Version: 8.6.x-dev » 8.7.x-dev

Drupal 8.6.0-alpha1 will be released the week of July 16, 2018, which means new developments and disruptive changes should now be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

nginex’s picture

Issue tags: +LutskGCW19
mpdonadio’s picture

Issue tags: +Needs reroll

This is a messy reroll b/c the array syntax change and #2935617: Move User module's temp stores to core.

mpdonadio’s picture

Assigned: shadcn » Unassigned
Status: Needs work » Needs review
Issue tags: -Needs reroll
FileSize
27.12 KB

Here is a reroll, more of the re-do variety.

Status: Needs review » Needs work

The last submitted patch, 20: 2855457-20.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

mpdonadio’s picture

Status: Needs work » Needs review
FileSize
28.75 KB
2.95 KB

Status: Needs review » Needs work

The last submitted patch, 22: 2855457-22.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

Version: 8.7.x-dev » 8.8.x-dev

Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

mrweiner’s picture

Status: Needs work » Needs review

Patch from #22 seems to still apply cleanly on 8.8.x-dev

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

moltam’s picture

Issue tags: -
FileSize
29.88 KB

I updated the patch and made it compatible with version 8.8.x.

Also fixed the issues that I've found.

thalles’s picture

I fix some issues finded by Drupal phpcs in this patch.

FILE: /home/thallesvf/Desktop/lando/d8x/core/lib/Drupal/Core/KeyValueStore/DatabaseStorageExpirable.php
-------------------------------------------------------------------------------------------------------
FOUND 1 ERROR AND 1 WARNING AFFECTING 2 LINES
-------------------------------------------------------------------------------------------------------
  38 | ERROR   | Arguments with default values must be at the end of the argument list
 120 | WARNING | Possible useless method overriding detected
-------------------------------------------------------------------------------------------------------

Status: Needs review » Needs work

The last submitted patch, 28: 2855457-28.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

jungle’s picture

Status: Needs work » Needs review
FileSize
31.32 KB
8.98 KB

Fix Drupal\Tests\ComposerIntegrationTest and 9 coding standards messages

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

smustgrave’s picture

This issue is being reviewed by the kind folks in Slack, #needs-review-queue-initiative. We are working to keep the size of Needs Review queue [2700+ issues] to around 400 (1 month or less), following Review a patch or merge request as a guide.

Tagging for framework managers thoughts as this feels like something that could be disruptive for other sites.
Tagging for IS update for remaining tasks as this will have to be rescoped for D10
Tagging for change record update on if this moves forward it will need one for sure.

kkalashnikov’s picture

Patch for Drupal version 10.1.x

Rishabh Vishwakarma’s picture

Status: Needs work » Needs review
FileSize
18.12 KB
2.26 KB

Resolved PHPCS errors from #38

smustgrave’s picture

Status: Needs review » Needs work

Still needs issue summary update and change record

sahil.goyal’s picture

Trying to resolve CCF corresponding to #39, upload a patch and interdiff with it.

mukesh88’s picture

Fixed the custom commands on #41
upload a patch and interdiff with it.

jungle’s picture

Assigned: Unassigned » jungle

I am on this. CR added. Constructor property promotion will be used.

jungle’s picture

Assigned: jungle » Unassigned
Status: Needs work » Needs review
FileSize
15.34 KB
22.14 KB
jungle’s picture

Added "in short, REQUEST_TIME does not equal to \Drupal::time()->getCurrentTime()." to the IS.

jungle’s picture

Status: Needs review » Needs work

The last submitted patch, 46: 2855457-46.patch, failed testing. View results

jungle’s picture

FileSize
23.39 KB

Oops. 2855457-48.patch is empty.

smustgrave’s picture

Status: Needs review » Needs work

Removed credit for #38, #39, #41, and #42 as it's expected the patch to be tested locally before uploading.

The last patch doesn't pass commit checks, could you make sure to run ./core/scripts/dev/commit-code-check.sh before uploading a patch to make sure there are no issues with code formatting. see https://www.drupal.org/docs/develop/development-tools/running-core-devel...

Reviewing #49
Changes look good but think we will need a simple kernel test (maybe Unit) for checking the deprecation of each of the constructors when a null time parameter is passed.

Going to post to the slack channel also for the framework feedback.

longwave’s picture

+++ b/core/lib/Drupal/Core/KeyValueStore/DatabaseStorageExpirable.php
@@ -24,9 +25,15 @@ class DatabaseStorageExpirable extends DatabaseStorage implements KeyValueStoreE
-  public function __construct($collection, SerializationInterface $serializer, Connection $connection, $table = 'key_value_expire') {
+  public function __construct($collection, SerializationInterface $serializer, Connection $connection, $table = 'key_value_expire', protected ?TimeInterface $time = NULL) {

The problem here is that we will have to remove the default for $table in Drupal 11 as you can't have mandatory arguments following optional ones. Is it worth doing the argument dance here so $time comes before $table?

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.