Problem/Motivation

There are times when the batch processing functionality needs to be swapped out.

Proposed resolution

Create a class to process batches and add it as a service.

The current functionality is not to be changed at this stage. The default batch processor service class will call the existing functions. Further issues will arise when migrating the functionality to the service class at a later stage. See #2875151.
Issues are already created:

User interface changes

None.

API changes

A service will be introduced to process batches.

Data model changes

None.

Issue fork drupal-2959723

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

John Cook created an issue. See original summary.

john cook’s picture

Status: Active » Needs review
StatusFileSize
new8.95 KB

Initial interface and service class.

The service can be acquired with \Drupal::service('batch.processor'). This means that a batch can be queued using the following (8.6 and higher):

\Drupal::service('batch.processor')->queue($batch_builder->toArray());
borisson_’s picture

Status: Needs review » Needs work

Nitpicks + 2 actual questions ahead:

  1. +++ b/core/lib/Drupal/Core/Batch/BatchProcessor.php
    @@ -0,0 +1,130 @@
    + * @package Drupal\Core\Batch
    

    There are only 2 classes in core where @package shows up, we don't really have a cs rule about this I think, but let's remove this.

  2. +++ b/core/lib/Drupal/Core/Batch/BatchProcessor.php
    @@ -0,0 +1,130 @@
    +class BatchProcessor implements BatchProcessorInterface {
    ...
    +   * Creates a new BatchQueueController.
    

    Naming mismatch.

  3. +++ b/core/lib/Drupal/Core/Batch/BatchProcessor.php
    @@ -0,0 +1,130 @@
    +  public function &getCurrentBatch() {
    
    +++ b/core/lib/Drupal/Core/Batch/BatchProcessorInterface.php
    @@ -0,0 +1,136 @@
    +  /**
    +   * Retrieves the current batch.
    +   */
    +  public function getCurrentBatch();
    

    Do we document that this is by reference or should we not do that?

  4. +++ b/core/lib/Drupal/Core/Batch/BatchProcessorInterface.php
    @@ -0,0 +1,136 @@
    +   * @internal This is an internal function and will be removed. Use
    +   *   _batch_current_set() instead.
    

    So basically this is @deprecated? I don't see why we're adding a deprecated method that doesn't have any uses.

  5. +++ b/core/lib/Drupal/Core/Batch/BatchProcessorInterface.php
    @@ -0,0 +1,136 @@
    +   * @internal This is an internal function and will be removed. Use
    +   *   _batch_next_set() instead.
    

    ^

  6. +++ b/core/lib/Drupal/Core/Batch/BatchProcessorInterface.php
    @@ -0,0 +1,136 @@
    +   * @return \Symfony\Component\HttpFoundation\RedirectResponse|false
    +   *   A redirect response to the completed page or NULL to stay at the current
    +   *   URL.
    

    Is it false or null that gets returned? There's a mismatched between the description and the parameter documentation.

john cook’s picture

Status: Needs work » Needs review
StatusFileSize
new8.53 KB
new2.7 KB

Thanks for the review @borisson_.

The first patch was copy-pasted from an experiment I had done, and was uploaded so I didn't lose it.

I've fixed the problems in comment #3.

borisson_’s picture

Status: Needs review » Reviewed & tested by the community

All the nits I had to pick in #3 were fixed in #4. I would love to see usages of this as well, but I think we can do that in a follow-up.

alexpott’s picture

Status: Reviewed & tested by the community » Needs work

There are times when the functionality to for the processing of batches is required to be swapped out.

This needs more detail.

Also we don't add unused code to core - this would just confuse people and is a recipe for adding stuff that does not work - because we're not using it.

john cook’s picture

Status: Needs work » Needs review
StatusFileSize
new52.44 KB
new58.93 KB

I've gone through core and changed the functional code to use the service. Uploading to make sure all the tests still pass.

Status: Needs review » Needs work

The last submitted patch, 7: 2959723-7.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

Version: 8.6.x-dev » 8.7.x-dev

Drupal 8.6.0-alpha1 will be released the week of July 16, 2018, which means new developments and disruptive changes should now be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

john cook’s picture

StatusFileSize
new59.17 KB

Re-roll of patch from #7.

john cook’s picture

Status: Needs work » Needs review

Status: Needs review » Needs work

The last submitted patch, 10: 2959723-10.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

voleger’s picture

Status: Needs work » Needs review
StatusFileSize
new44.46 KB
new94.52 KB

Rerolled
Added required DI, moved functions body into the appropriate methods.
Still, need to move some functions into the service.

Status: Needs review » Needs work

The last submitted patch, 13: 2959723-13.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

john cook’s picture

StatusFileSize
new94.23 KB

Another re-roll

john cook’s picture

StatusFileSize
new94.44 KB
new808 bytes

I found an issue where the queue name and class weren't being derived while the batch was being processed. The new changes fix this.

john cook’s picture

Status: Needs work » Needs review

Status: Needs review » Needs work

The last submitted patch, 16: 2959723-16.patch, failed testing. View results

Version: 8.7.x-dev » 8.8.x-dev

Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

john cook’s picture

Version: 8.8.x-dev » 8.7.x-dev
Status: Needs work » Needs review
StatusFileSize
new2.28 KB

So #16 was a red herring. The problem was because a variable needed to be passed by reference.

#16 has been reverted and a new fix has been implemented. Because of this, the interdiff goes from 15 to 19.

john cook’s picture

StatusFileSize
new93.86 KB

Forgot the patch.

Status: Needs review » Needs work

The last submitted patch, 21: 2959723-19.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

john cook’s picture

Status: Needs work » Needs review
StatusFileSize
new93.86 KB
new756 bytes

Fixed a typing error.

Status: Needs review » Needs work

The last submitted patch, 23: 2959723-23.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

mikelutz’s picture

mikelutz’s picture

StatusFileSize
new94.1 KB

reroll

vacho’s picture

voleger’s picture

Version: 8.7.x-dev » 8.8.x-dev
StatusFileSize
new94.26 KB

Just reroll patch against 8.8.x

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

ridhimaabrol24’s picture

Status: Needs work » Needs review
StatusFileSize
new89.92 KB

Rerolling patch for 9.1.x

ridhimaabrol24’s picture

StatusFileSize
new89.92 KB
new668 bytes

Fixing PHP lint error

Status: Needs review » Needs work

The last submitted patch, 32: 2959723-32.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

ridhimaabrol24’s picture

Assigned: Unassigned » ridhimaabrol24
ridhimaabrol24’s picture

Assigned: ridhimaabrol24 » Unassigned
Status: Needs work » Needs review
StatusFileSize
new89.92 KB
new831 bytes

Fixing failed test cases.

Status: Needs review » Needs work

The last submitted patch, 35: 2959723-35.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

voleger’s picture

Status: Needs work » Needs review

Rerolled #35

voleger’s picture

Status: Needs review » Needs work

Those errors show that the batch.processor handler has to rely on the database connection during runtime, and the database connection has not to be passed using DI. This service has to be instantiated before establishing a database connection.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

voleger’s picture

Rebased over 9.4.x
The test result shows that we are pretty close to finishing the definition of the batch processor. The failing test shows that the conversion still has an issue with porting _batch_append_set() function to set proper execution of batches added during queue execution. I'm currently trying to address that.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

voleger’s picture

Version: 9.5.x-dev » 10.1.x-dev

Rebasing the brranch

Bhanu951 made their first commit to this issue’s fork.

bhanu951’s picture

There are still references to batch_set() in comments.

grep -R "batch_set("
./core/includes/batch.inc:  // batch_set() or hook_batch_alter().
./core/includes/form.inc: * batch_set($batch);
./core/includes/form.inc: * the code calling batch_set() to sanitize them first with a function like
./core/includes/form.inc:function batch_set($batch_definition) {
./core/lib/Drupal/Core/Form/form.api.php: * Callback for batch_set().
./core/lib/Drupal/Core/Form/form.api.php: *   array passed to batch_set().
./core/lib/Drupal/Core/Form/form.api.php: * Callback for batch_set().
./core/lib/Drupal/Core/Form/form.api.php: *   The associative array of batch information. See batch_set() for details on
./core/lib/Drupal/Core/Extension/module.api.php: *       definition suitable for batch_set() or an array of batch definitions
./core/lib/Drupal/Core/Extension/module.api.php: *       suitable for consecutive batch_set() calls. The installer will then
./core/lib/Drupal/Core/Batch/BatchBuilder.php: * batch_set($batch_builder->toArray());
./core/modules/user/user.api.php: * by using batch_set().
voleger’s picture

Assigned: Unassigned » voleger

Let's make the test green again

voleger’s picture

Fixed autowiring test
Added deprecation of _batch_needs_update()
#47 still needs work

voleger’s picture

Assigned: voleger » Unassigned
bhanu951’s picture

Status: Needs work » Needs review
voleger’s picture

Assigned: Unassigned » voleger
Status: Needs review » Needs work
voleger’s picture

Assigned: voleger » Unassigned
bhanu951’s picture

StatusFileSize
new155.4 KB

adding Patch for testing since GITLAB is having Issues.

bhanu951’s picture

Status: Needs work » Needs review
voleger’s picture

Assigned: Unassigned » voleger

Please, reset your local branch before applying additional changes next time. I'll make clean up duplications and redundant commits in the forked branch.

_utsavsharma’s picture

StatusFileSize
new762 bytes
new155.38 KB

Fixed CCF for #54.

Status: Needs review » Needs work

The last submitted patch, 57: 2959723-57.patch, failed testing. View results

bhanu951’s picture


Remaining self deprecation notices (946)

  946x: Calling Drupal\Core\Form\FormSubmitter::__construct without the $batch_processor argument is deprecated in drupal:10.1.0 and it will be required in drupal:11.0.0. See https://www.drupal.org/node/3229844
    946x in InstallUninstallTest::testInstallUninstall from Drupal\Tests\system\Functional\Module
Remaining self deprecation notices (94)

  63x: Calling Drupal\Core\Form\FormSubmitter::__construct without the $batch_processor argument is deprecated in drupal:10.1.0 and it will be required in drupal:11.0.0. See https://www.drupal.org/node/3229844
    53x in UpdatePathTestBaseFilledTest::testUpdatedSite from Drupal\Tests\system\Functional\UpdateSystem
    5x in UpdatePathTestBaseFilledTest::testPathAliasProcessing from Drupal\Tests\system\Functional\UpdateSystem
    2x in UpdatePathTestBaseFilledTest::testDatabaseLoaded from Drupal\Tests\system\Functional\UpdateSystem
    1x in UpdatePathTestBaseFilledTest::testUpdateHookN from Drupal\Tests\system\Functional\UpdateSystem
    1x in UpdatePathTestBaseFilledTest::testModuleListChange from Drupal\Tests\system\Functional\UpdateSystem
    ...

  31x: Calling Drupal\Core\Batch\BatchStorage::__construct without the $time argument is deprecated in drupal:10.1.0 and it will be required in drupal:11.0.0. See https://www.drupal.org/node/2959723
    6x in UpdatePathTestBaseFilledTest::testModuleListChange from Drupal\Tests\system\Functional\UpdateSystem
    5x in UpdatePathTestBaseFilledTest::testUpdatedSite from Drupal\Tests\system\Functional\UpdateSystem
    5x in UpdatePathTestBaseFilledTest::testDatabaseLoaded from Drupal\Tests\system\Functional\UpdateSystem
    5x in UpdatePathTestBaseFilledTest::testUpdateHookN from Drupal\Tests\system\Functional\UpdateSystem
    5x in UpdatePathTestBaseFilledTest::testPathAliasProcessing from Drupal\Tests\system\Functional\UpdateSystem

Not sure how to handle deprecation notices here, anyone got any pointers? Thanks.

voleger’s picture

It is pointless to require dependency on the batch processor as the form submitter is a batch processor dependency too. So it has to be an optional dependency: a batch processor for the form submitter or a form submitter for the batch processor.

voleger’s picture

Assigned: voleger » Unassigned

Unassigning. Cleaned up the MR history, get rid of duplicated commits and leaked commits from previous major version branches.
@Bhanu951 please reset your local branch to get rid of messed history from the forked branch.
`git reset --hard drupal-2959723/2959723-create-an-interface`
Then re-apply your changes as I see you worked on the branch during I was assigned to the issue.

bhanu951’s picture

@voleger Thanks for the clean-up. Seems my latest changes are already present.

After latest changes there are still 6 errors in tests.

As testbot status is not being displayed. Refer below link for tests errors.

https://www.drupal.org/pift-ci-job/2569990

https://www.drupal.org/pift-ci-job/2570005

bhanu951’s picture

bhanu951’s picture

Seems we are having test failures related to batch functionality.

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

voleger changed the visibility of the branch 2959723-create-an-interface-10-1-x to hidden.

voleger’s picture

The last issue left is related to the session opening problem. It only happens during the tests run.

RuntimeException: Failed to start the session: already started by PHP. in Symfony\Component\HttpFoundation\Session\Storage\NativeSessionStorage->start() (line 112 of vendor/symfony/http-foundation/Session/Storage/NativeSessionStorage.php).
Drupal\Core\Session\SessionManager->startNow() (Line: 94)
Drupal\Core\Session\SessionManager->start() (Line: 59)
Symfony\Component\HttpFoundation\Session\Session->start() (Line: 115)
Drupal\Core\Batch\BatchStorage->create() (Line: 107)
Drupal\Core\ProxyClass\Batch\BatchStorage->create() (Line: 322)
Drupal\Core\Batch\BatchProcessor->process() (Line: 62)
Drupal\Core\Form\FormSubmitter->doSubmitForm() (Line: 619)
Drupal\Core\Form\FormBuilder->processForm() (Line: 351)
Drupal\Core\Form\FormBuilder->buildForm() (Line: 73)
Drupal\Core\Controller\FormController->getContentResult()
call_user_func_array() (Line: 123)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->{closure:Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber::wrapControllerExecutionInRenderContext():121}() (Line: 633)
Drupal\Core\Render\Renderer::{closure:Drupal\Core\Render\Renderer::executeInRenderContext():633}()
Fiber->resume() (Line: 647)
Drupal\Core\Render\Renderer->executeInRenderContext() (Line: 121)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->wrapControllerExecutionInRenderContext() (Line: 97)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->{closure:Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber::onController():96}() (Line: 183)
Symfony\Component\HttpKernel\HttpKernel->handleRaw() (Line: 76)
Symfony\Component\HttpKernel\HttpKernel->handle() (Line: 36)
Drupal\Core\Test\StackMiddleware\TestWaitTerminateMiddleware->handle() (Line: 53)
Drupal\Core\StackMiddleware\Session->handle() (Line: 48)
Drupal\Core\StackMiddleware\KernelPreHandle->handle() (Line: 28)
Drupal\Core\StackMiddleware\ContentLength->handle() (Line: 118)
Drupal\page_cache\StackMiddleware\PageCache->pass() (Line: 92)
Drupal\page_cache\StackMiddleware\PageCache->handle() (Line: 48)
Drupal\Core\StackMiddleware\ReverseProxyMiddleware->handle() (Line: 51)
Drupal\Core\StackMiddleware\NegotiationMiddleware->handle() (Line: 53)
Drupal\Core\StackMiddleware\AjaxPageState->handle() (Line: 54)
Drupal\Core\StackMiddleware\StackedHttpKernel->handle() (Line: 716)
Drupal\Core\DrupalKernel->handle() (Line: 19)

Backtrace points to the Session `start()` method called in Drupal\Core\Batch\BatchStorage->create(). Any thoughts on why the session may be started outside the session service?

nicxvan made their first commit to this issue’s fork.

nicxvan’s picture

Self applied two suggestions to fix the new attributes for ignoring deprecations.

Also this will conflict with the issue to use the callable resolver for batches which is RTBC.

I think we can begin hammering out some of what is going on here though so we can hopefully make batch better!

voleger’s picture

Status: Needs work » Needs review

Opened new !13522 MR. In the v2 branch, I split the changes as much as possible to make it as easy as possible to rebase and update from upstream. So I introduced the interface and class as an OOP wrapper around the procedural API, then did conversion function by function in a separate commit. Then, I applied deprecation replacements, and finally, a couple of changes were required to make the changes pass the test.

One change that requires attention is BatchStorage. It produces a weird issue with session handling. Once the API moved into a service, the session manager in the batch storage service was unable to handle starting a session as before, as it was handled in the kernel first, and it looks like the session manager instance is different from the one that was used in the kernel, so it failed to pass the internal properties check.
Because of that, I replaced the session manager with a request proxy, which allows bringing the session instance from the request. The purpose of starting the session is to initiate CSRF checks, so this replacement in the dependency should not change the initial purpose of starting the session there.

In the deprecation policy, there is no case of dependency replacements (only removal and addition). Asking for a review of dependency replacement.

Another change is related to the views batch user action plugin. Change related to https://www.drupal.org/node/3542837 CR, to allow autowiring of new dependency.

voleger’s picture

Title: Create an interface and initial class for the batch processor service » [PP-1] Create an interface and initial class for the batch processor service
Assigned: Unassigned » voleger
Status: Needs review » Postponed

Reviews are welcome. But this issue will not reach 11.3.0, so it has to wait until 11.3.0 is released.
Before that, I hope #3539919: Convert batch callbacks to CallableResolver will be merged in, and I'll be able to rebase the v2 branch to move the introduced changes into the batch processor implementation.

Affected methods that need to be updated once issue #3539919: Convert batch callbacks to CallableResolver is merged are:
- \Drupal\Core\Batch\BatchProcessor::processQueue() [_process_queue()]
- \Drupal\Core\Batch\BatchProcessor::nextSet() [_batch_next_set()]
- \Drupal\Core\Batch\BatchProcessor::finishedProcessing() [_batch_finished()]
- \Drupal\Core\Batch\BatchProcessor::process() [batch_process()]

voleger’s picture

voleger’s picture

Title: [PP-1] Create an interface and initial class for the batch processor service » Create an interface and initial class for the batch processor service
Status: Postponed » Needs review

Back to needs review again after merging #3539919: Convert batch callbacks to CallableResolver

voleger’s picture

Assigned: voleger » Unassigned

Addressed review comments

voleger’s picture

Updated CR

voleger’s picture

smustgrave’s picture

This is a pretty large MR, any room for breaking up?

voleger’s picture

Yes, since the v2 recreation attempt, I see that we can introduce the interface first and use the service as a wrapper around most procedural functions. Once it is done, we can deprecate procedural functions one by one in the follow-up issues. Is it ok to do it like that? I'll prepare an alternative PR then.

Version: 11.x-dev » main

Drupal core is now using the main branch as the primary development branch. New developments and disruptive changes should now be targeted to the main branch.

Read more in the announcement.

godotislate’s picture

Yes, since the v2 recreation attempt, I see that we can introduce the interface first and use the service as a wrapper around most procedural functions. Once it is done, we can deprecate procedural functions one by one in the follow-up issues.

I agree with this plan. I think it will make it easier to get it in.

Thanks for the work so far!

godotislate’s picture

Status: Needs review » Needs work

voleger’s picture

Status: Needs work » Needs review

Here is the MR !14593 for review.

Do we need to create follow-ups now, or once the interface is merged?

nicxvan’s picture

We can create follow ups now, just postpone them on this issue.

voleger changed the visibility of the branch 2959723-create-an-interface-v2 to hidden.

voleger’s picture

Merge request !14593 already contains replacements in the tests.
Created the first follow-up #3570981: [PP-1] Convert procedural internal batch processor functions to OOP issue.
Public API functions will have one issue per function.

nicxvan’s picture

One comment on the MR i think it's the way we're doing things now.

Somehow I missed the big pivot.

At first I questioned the wrapper methods, but it let's us get a pretty minimal change in.

I'm curious to see if core committers will agree.

Three questions to consider here:

1 should we convert one real call or do we think tests are enough.

2 Is there a way to do this without injecting the app root? Feel a shame to do that since it seems temporary.

3 do we need an interface? I see that question getting asked pretty often including the new field purge process we're working on.

I think this is real close.

voleger’s picture

Re #90.
1: We create follow-ups for that purpose. Tests show that the procedural API continues to work as expected even when wrapped in a processor class. This will make it easier to test replacements in the follow-up.
2: The service is internal. We can remove the @internal annotation once the processor changes are complete and the follow-up issues are resolved, without BC layers.
3: I guess @john cook may have more input on that topic. I think the initial intent was to do the interface to allow decoration in the contrib space.

nicxvan’s picture

Thank you!
1. Ok, I can agree with that, no change requested.
2. While it's internal, this is a pretty important service so we still would go through BC, but the way this is done, I think just one parameter is fine, no change requested.
3. I think we should remove the interface

I don't see anywhere else in core we do something like this, I don't want to derail this further, but I wonder if an approach where we just do one or two methods to reduce scope without doing this shuffle.

I think the only actionable thing to do here is just remove the interface, move the comments to the methods directly and update where it's injected to use the actual BatchProcessor, we did the same thing for the FieldPurger.

nicxvan’s picture

Sorry, why is that related?

voleger’s picture

Title: Create an interface and initial class for the batch processor service » Create an initial class for the batch processor service

Addressed #92 and ready for review

voleger’s picture

Issue summary: View changes
voleger’s picture

Rebased

nicxvan’s picture

Status: Needs review » Reviewed & tested by the community

Still not 100% this will be accepted, but I asked in slack and no one objected. We do have the plan and all of the follow ups are created.

This is far easier to review than the initial MR that converted everything in one go.

The method names either match the procedural equivalent or are clear improvements.
I updated the CR to remove the Interface references since we don't have one now.

I think this is ready!

quietone made their first commit to this issue’s fork.

quietone’s picture

Status: Reviewed & tested by the community » Needs review

I read the comments and made a few suggestions. I then accepted the suggestions as they don't change meaning and some were formatting. There is also one question about docs that should be answered. I am setting back to NR for that one question. Plus, someone should check the suggestions I made.

Also, the doc block for constructors isn't required so there is the option to remove those in the new classes in this MR.

quietone’s picture

And I updated credit.

needs-review-queue-bot’s picture

Status: Needs review » Needs work
StatusFileSize
new91 bytes

The Needs Review Queue Bot tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".

This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

voleger’s picture

Status: Needs work » Needs review

Updated docs, and rerolled one test after the node module functions deprecation replacement.

needs-review-queue-bot’s picture

Status: Needs review » Needs work
StatusFileSize
new1.52 KB

The Needs Review Queue Bot tested this issue. It fails the Drupal core commit checks. Therefore, this issue status is now "Needs work".

This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

voleger’s picture

Status: Needs work » Needs review
nicxvan’s picture

Status: Needs review » Reviewed & tested by the community

I reviewed the changes in 103 and the rebase in 105 and this looks good again!

quietone’s picture

Status: Reviewed & tested by the community » Needs work
Issue tags: +Needs change record updates

I reviewed the change record and it refers to a new interface, which is not in the MR. It also refers to functions as being deprecated when they have not been deprecated. I am setting this to needs work for an update to the change record.

For the change record, it should be concise and state the change. It should not start with a question. It can start with a simple sentence something like" A new batch processor, Drupal\Core\Batch\BatchProcessor has been added". In this case, examples of how to use it are also suitable. And the list of issues should be limited to just one issue, the one with the change being described.

voleger’s picture

Status: Needs work » Needs review

RE #107:
- The mention of the interface in CR has been removed.
- The functions will be deprecated - see the child issues of the meta issue due to a request to break down the required changes to fully introduce the class. You can review MR !14593 for the full set of replacements, which will be introduced across all 5 child issues.
- The list of related issues has been reduced to only one meta issue.

nicxvan’s picture

Ok thank you for pointing out in slack where the CR went.

I think this highlights the trickiness of this approach.

I think this very much deserves a CR, but we aren't deprecating anything.

We no longer have an interface so we could do this method by method.

I'm still a little worried about what I mentioned in 98, but I think this is an interesting approach too.
Incidentally I had cleaned up the interface changes in the CR in 98, it looks like it was accidentally.

I'd propose this to move this forward:

I'll create a minimal CR introducing the new service on this issue and we get this in front of the committers. If they decide this isn't the way forward, since we have just a class now that isn't an interface we can choose just a method or two to convert and do that issue by issue.

I'll leave the CR on the meta for reference.

nicxvan’s picture

Status: Needs review » Reviewed & tested by the community
Issue tags: -Needs change record updates

Ok the CR is now here again and it's just a very simple mapping of the procedural to methods.

Let's get this back in front of the committers for a decision if this direction is the way to go.

If not I'd propose we do batch_set, get, and process here.

nicxvan’s picture

larowlan’s picture

Status: Reviewed & tested by the community » Needs work

Left some comments on the MR and on the meta - would be good to see a big picture plan of the end state if possible

daffie made their first commit to this issue’s fork.

daffie’s picture

Status: Needs work » Needs review

All remarks of @larowlan have been addressed.
Back to NR.

nicxvan’s picture

Status: Needs review » Needs work

We intentionally removed the interface in 92, I'm not sure why we would add that back here.

More and more services are being added directly workout an interface, I think that mostly addresses @larowlan's concern about the underscore functions, even if they are public, without an interface they are internal.

We could exclude them until we convert them.

I think we should revert that part at least.

daffie’s picture

Status: Needs work » Needs review

Removed the BatchProcessorInterface as requested.
Back to NR.

nicxvan’s picture

Thanks!

berdir’s picture

Status: Needs review » Needs work

Did a first review.

This is challenging. I understand we tried a few different things and it's a very complex thing that's also used early in the installer and some functions have a lot of calls, but this doesn't yet seem to result in a workable API, especially around those protected methods. We can make up things that in practice aren't going to work because some of these calls are external, plus challenges around BC. And having everything on one service is going to make that huge.

Naming seems to be mixed up. batch_set() became queue(), which is something completely different from getQueue() on the same service.

I think we'll more likely make meaningful progress by focusing on the public few methods here and not worry yet about most of those protected ones just yet. I'm also wondering if we should split it up into at least two services? My initial thought was one service that manages the batch state, stuff like batch set/get and the queue, and then one to process it, downside is that many cases would then need to interact with two services. And dependencies are tricky. batch_process() calls into _batch_process(), but that one calls back into batch_get(), so we need to be careful to not have circular dependencies.

So my current thought is that the public API is one service, maybe a generic and boring BatchManager, that covers what most cases need, adding/checking current batch and starting to process it. That would wrap two internal services , something like a BatchRegistry/BatchState (since we already have storage) that contains the actual storage for batch_get() and the queue stuff, and the BatchProcessor that would also interact with the first.

And then in this issue, we'd focus on BatchManager public API only?

nicxvan’s picture

Yes, I think you're right.

I'm willing to pick this up, the truth is we need to give this the locale treatment and build the correct api, which beyond your recommendations likely includes a value object for passing info around.

I'll pick this up once things settle around 12.

berdir’s picture

A value object would be nice but also scary complicated, there's a reason we only did the builder for now. I think it would be neat to add or even require a builder object for addBatch() or what we want to call it, but changing from the array to object is going to be extremely challenging, the data structure is by reference and pretty complex, there's custom form stuff and queue stuff mixed in and what not.

One thought I have is that we'd make the two inner wrapped services internal and BatchManager would not expose the internal data structure but offer convenience methods for common things that people do with batch_get() for example. Looking through the calls, I see multiple times in core and contrib this:

      $batch =& batch_get();
      // Mark this batch as non-progressive to bypass the progress bar and
      // redirect.
      $batch['progressive'] = FALSE;
      batch_process();

So we could have a setProgressive(bool $progressive).

And other places just check if there's a batch, we could have a needsProcessing() method or so:

// Execute a batch if required. A batch is only used when remote files
    // are checked.
    if (batch_get()) {
      return batch_process('admin/reports/translations');
    }

with those two, the majority of calls to batch_get() can be simplified and don't need to know about the data structure anymore. Then the impact will be much smaller once we're refactor the then internal functions and data structure.

daffie’s picture

Assigned: Unassigned » daffie

I am working on this.

daffie’s picture

Status: Needs work » Needs review

Disclosure: I have used AI in the making of the MR.

daffie’s picture

Assigned: daffie » Unassigned
needs-review-queue-bot’s picture

Status: Needs review » Needs work
StatusFileSize
new811 bytes

The Needs Review Queue Bot tested this issue. It fails the Drupal core commit checks. Therefore, this issue status is now "Needs work".

This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

ghost of drupal past’s picture

My dayjob currently includes making sure batch works when Drupal is run in an app server. This is a lot of fun, read this comment explaining how broken this is. Ahem.

So I feel I need to comment on this issue: I do not understand the very direction of this. If I were doing this, I would move all the code in batch.inc unchanged into a class and have the existing functions call this service. Nothing else for a first step. Very easy, painless, zero fuss. Yes, keep even the (rather accursed) statics.

Then it's possible to fan out the followups:

  1. move batch init functionality in form.inc into a service in a similar issue
  2. swap calls to batch functionality to the new services in one or more issues as patch size warrants
  3. carefully swapping out statics but read the comment in batch_get()
  4. make batch queue a proper service instead of a classname
  5. work out the BC policy for batch_get(). This will hurt.
  6. Once that's done move the internal state to an object
  7. potentially other modernizing steps in the service as needed.
nicxvan’s picture

Assigned: Unassigned » nicxvan

Let me take a crack at this, I've been thinking we needed to go that route for a while.

What I'll do is just convert the batch.inc in a naive BatchProcessor class as a starting point, if it looks decent I'll update the CR.

nicxvan’s picture

Assigned: nicxvan » Unassigned

The drupal_static change wasn't pushed yet, gonna wait for that.

berdir’s picture

I was wondering about that too, but I think there's also value in having a plan/understand of how the complete API and interaction between the functions will look like. It's not as simple as batch.inc in one service and form.inc in another, for example because of cross dependencies, batch_get() and others in form.inc are also used by batch.inc, and vice versa: batch_process() calls _batch_process().

It was my idea, so obviously, but I think the new structure with the three services looks better to me and if we agree on that overall grouping, we _could_ now revert the reverted process to something more usual and start converting the inner services bottom up and properly deprecate them and make this a meta instead of a starting point with follow-ups. Pretty sure we're going to hit a bunch of roadblocks either way.

ghost of drupal past’s picture

I do not think the number of services matter really, all of batch could be thrown into a single service -- I just didn't like the current MR introducing new services calling old code, that looks backwards to me.

nicxvan’s picture

I'm working on this again. I'm going to start with just the batch.inc conversion and then we call look at moving things around.

nicxvan’s picture

Ok I took a first pass in 17276. I made as much of it protected as I can, some things were called outside of batch so I had to make them public, there was one test I removed that I'm not sure was the most valuable test.

currentSet needs better docs and page return.

CR and IS needs updating if this direction looks good.

I think this is worth taking a first pass look at.

voleger’s picture

Previous changes were more like trying to keep the existing Batch API architecture.

                 ┌────────────────────┐
                 │ Existing contrib   │
                 │ modules            │
                 └─────────┬──────────┘
                           │
                    old procedural API
                           │
                           ▼
                 ┌────────────────────┐
                 │ Compatibility      │
                 │ facade              │
                 └─────────┬──────────┘
                           │
                           ▼
                 ┌────────────────────┐
                 │ BatchManager       │
                 │ BatchExecutor      │
                 │ BatchStorage       │
                 └─────────┬──────────┘
                           │
              ┌────────────┼────────────┐
              ▼            ▼            ▼
           HTTP          CLI         Queue
          executor     executor     executor

we could use the service as the foundation for a proper service-oriented architecture with a BatchManager, pluggable storage/execution, and a clear separation between batch definition and execution. This would let us migrate the existing functions incrementally while preserving backward compatibility, rather than creating another abstraction that ultimately has to be redesigned later. The existing follow-up issues already point toward this direction, so perhaps we could align them around this broader architecture and make the service the stable foundation for the next stage of Batch API evolution.

voleger’s picture

@nicxvan, check MR https://git.drupalcode.org/project/drupal/-/merge_requests/13522, as I previously resolved that issue with early calls to the container before the DB connection was available.

voleger’s picture

Also, regarding session handling in `core/lib/Drupal/Core/Batch/BatchStorage.php`, we discussed that 10 months ago we should avoid starting the session once it has already started.