Needs work
Project:
Drupal core
Version:
main
Component:
batch system
Priority:
Normal
Category:
Task
Assigned:
Unassigned
Issue tags:
Reporter:
Created:
9 Apr 2018 at 23:08 UTC
Updated:
28 Sep 2026 at 16:01 UTC
Jump to comment: Most recent, Most recent file
Comments
Comment #2
john cook commentedInitial interface and service class.
The service can be acquired with
\Drupal::service('batch.processor'). This means that a batch can be queued using the following (8.6 and higher):Comment #3
borisson_Nitpicks + 2 actual questions ahead:
There are only 2 classes in core where
@packageshows up, we don't really have a cs rule about this I think, but let's remove this.Naming mismatch.
Do we document that this is by reference or should we not do that?
So basically this is
@deprecated? I don't see why we're adding a deprecated method that doesn't have any uses.^
Is it false or null that gets returned? There's a mismatched between the description and the parameter documentation.
Comment #4
john cook commentedThanks for the review @borisson_.
The first patch was copy-pasted from an experiment I had done, and was uploaded so I didn't lose it.
I've fixed the problems in comment #3.
Comment #5
borisson_All the nits I had to pick in #3 were fixed in #4. I would love to see usages of this as well, but I think we can do that in a follow-up.
Comment #6
alexpottThis needs more detail.
Also we don't add unused code to core - this would just confuse people and is a recipe for adding stuff that does not work - because we're not using it.
Comment #7
john cook commentedI've gone through core and changed the functional code to use the service. Uploading to make sure all the tests still pass.
Comment #10
john cook commentedRe-roll of patch from #7.
Comment #11
john cook commentedComment #13
volegerRerolled
Added required DI, moved functions body into the appropriate methods.
Still, need to move some functions into the service.
Comment #15
john cook commentedAnother re-roll
Comment #16
john cook commentedI found an issue where the queue name and class weren't being derived while the batch was being processed. The new changes fix this.
Comment #17
john cook commentedComment #20
john cook commentedSo #16 was a red herring. The problem was because a variable needed to be passed by reference.
#16 has been reverted and a new fix has been implemented. Because of this, the interdiff goes from 15 to 19.
Comment #21
john cook commentedForgot the patch.
Comment #23
john cook commentedFixed a typing error.
Comment #25
mikelutzComment #26
mikelutzreroll
Comment #27
vacho commentedPatch #27 reroll
Comment #28
volegerJust reroll patch against 8.8.x
Comment #31
ridhimaabrol24 commentedRerolling patch for 9.1.x
Comment #32
ridhimaabrol24 commentedFixing PHP lint error
Comment #34
ridhimaabrol24 commentedComment #35
ridhimaabrol24 commentedFixing failed test cases.
Comment #40
volegerRerolled #35
Comment #41
volegerThose errors show that the
batch.processorhandler has to rely on the database connection during runtime, and the database connection has not to be passed using DI. This service has to be instantiated before establishing a database connection.Comment #43
volegerRebased over 9.4.x
The test result shows that we are pretty close to finishing the definition of the batch processor. The failing test shows that the conversion still has an issue with porting _batch_append_set() function to set proper execution of batches added during queue execution. I'm currently trying to address that.
Comment #45
volegerRebasing the brranch
Comment #47
bhanu951 commentedThere are still references to
batch_set()in comments.Comment #48
volegerLet's make the test green again
Comment #49
volegerFixed autowiring test
Added deprecation of _batch_needs_update()
#47 still needs work
Comment #50
volegerComment #51
bhanu951 commentedComment #52
volegerComment #53
volegerComment #54
bhanu951 commentedadding Patch for testing since GITLAB is having Issues.
Comment #55
bhanu951 commentedComment #56
volegerPlease, reset your local branch before applying additional changes next time. I'll make clean up duplications and redundant commits in the forked branch.
Comment #57
_utsavsharma commentedFixed CCF for #54.
Comment #59
bhanu951 commentedNot sure how to handle deprecation notices here, anyone got any pointers? Thanks.
Comment #60
volegerIt is pointless to require dependency on the batch processor as the form submitter is a batch processor dependency too. So it has to be an optional dependency: a batch processor for the form submitter or a form submitter for the batch processor.
Comment #61
volegerUnassigning. Cleaned up the MR history, get rid of duplicated commits and leaked commits from previous major version branches.
@Bhanu951 please reset your local branch to get rid of messed history from the forked branch.
`git reset --hard drupal-2959723/2959723-create-an-interface`
Then re-apply your changes as I see you worked on the branch during I was assigned to the issue.
Comment #62
bhanu951 commented@voleger Thanks for the clean-up.
Seems my latest changes are already present.
After latest changes there are still 6 errors in tests.As testbot status is not being displayed. Refer below link for tests errors.
https://www.drupal.org/pift-ci-job/2569990https://www.drupal.org/pift-ci-job/2570005
Comment #63
bhanu951 commentedComment #64
bhanu951 commentedSeems we are having test failures related to batch functionality.
Comment #67
volegerThe last issue left is related to the session opening problem. It only happens during the tests run.
Backtrace points to the Session `start()` method called in Drupal\Core\Batch\BatchStorage->create(). Any thoughts on why the session may be started outside the session service?
Comment #69
nicxvan commentedSelf applied two suggestions to fix the new attributes for ignoring deprecations.
Also this will conflict with the issue to use the callable resolver for batches which is RTBC.
I think we can begin hammering out some of what is going on here though so we can hopefully make batch better!
Comment #72
volegerOpened new !13522 MR. In the v2 branch, I split the changes as much as possible to make it as easy as possible to rebase and update from upstream. So I introduced the interface and class as an OOP wrapper around the procedural API, then did conversion function by function in a separate commit. Then, I applied deprecation replacements, and finally, a couple of changes were required to make the changes pass the test.
One change that requires attention is BatchStorage. It produces a weird issue with session handling. Once the API moved into a service, the session manager in the batch storage service was unable to handle starting a session as before, as it was handled in the kernel first, and it looks like the session manager instance is different from the one that was used in the kernel, so it failed to pass the internal properties check.
Because of that, I replaced the session manager with a request proxy, which allows bringing the session instance from the request. The purpose of starting the session is to initiate CSRF checks, so this replacement in the dependency should not change the initial purpose of starting the session there.
In the deprecation policy, there is no case of dependency replacements (only removal and addition). Asking for a review of dependency replacement.
Another change is related to the views batch user action plugin. Change related to https://www.drupal.org/node/3542837 CR, to allow autowiring of new dependency.
Comment #73
volegerReviews are welcome. But this issue will not reach 11.3.0, so it has to wait until 11.3.0 is released.
Before that, I hope #3539919: Convert batch callbacks to CallableResolver will be merged in, and I'll be able to rebase the v2 branch to move the introduced changes into the batch processor implementation.
Affected methods that need to be updated once issue #3539919: Convert batch callbacks to CallableResolver is merged are:
- \Drupal\Core\Batch\BatchProcessor::processQueue() [_process_queue()]
- \Drupal\Core\Batch\BatchProcessor::nextSet() [_batch_next_set()]
- \Drupal\Core\Batch\BatchProcessor::finishedProcessing() [_batch_finished()]
- \Drupal\Core\Batch\BatchProcessor::process() [batch_process()]
Comment #74
volegerAttaching related issue #3539919: Convert batch callbacks to CallableResolver
Comment #75
volegerBack to needs review again after merging #3539919: Convert batch callbacks to CallableResolver
Comment #76
volegerAddressed review comments
Comment #77
volegerUpdated CR
Comment #78
volegerRebase after #3536726: Use CallableResolver for form callbacks got in
Comment #79
smustgrave commentedThis is a pretty large MR, any room for breaking up?
Comment #80
volegerYes, since the v2 recreation attempt, I see that we can introduce the interface first and use the service as a wrapper around most procedural functions. Once it is done, we can deprecate procedural functions one by one in the follow-up issues. Is it ok to do it like that? I'll prepare an alternative PR then.
Comment #82
godotislateI agree with this plan. I think it will make it easier to get it in.
Thanks for the work so far!
Comment #83
godotislateComment #85
volegerHere is the MR !14593 for review.
Do we need to create follow-ups now, or once the interface is merged?
Comment #86
nicxvan commentedWe can create follow ups now, just postpone them on this issue.
Comment #88
volegerMerge request !14593 already contains replacements in the tests.
Created the first follow-up #3570981: [PP-1] Convert procedural internal batch processor functions to OOP issue.
Public API functions will have one issue per function.
Comment #89
volegerPublic procedural API conversion/replacements:
Comment #90
nicxvan commentedOne comment on the MR i think it's the way we're doing things now.
Somehow I missed the big pivot.
At first I questioned the wrapper methods, but it let's us get a pretty minimal change in.
I'm curious to see if core committers will agree.
Three questions to consider here:
1 should we convert one real call or do we think tests are enough.
2 Is there a way to do this without injecting the app root? Feel a shame to do that since it seems temporary.
3 do we need an interface? I see that question getting asked pretty often including the new field purge process we're working on.
I think this is real close.
Comment #91
volegerRe #90.
1: We create follow-ups for that purpose. Tests show that the procedural API continues to work as expected even when wrapped in a processor class. This will make it easier to test replacements in the follow-up.
2: The service is internal. We can remove the @internal annotation once the processor changes are complete and the follow-up issues are resolved, without BC layers.
3: I guess @john cook may have more input on that topic. I think the initial intent was to do the interface to allow decoration in the contrib space.
Comment #92
nicxvan commentedThank you!
1. Ok, I can agree with that, no change requested.
2. While it's internal, this is a pretty important service so we still would go through BC, but the way this is done, I think just one parameter is fine, no change requested.
3. I think we should remove the interface
I don't see anywhere else in core we do something like this, I don't want to derail this further, but I wonder if an approach where we just do one or two methods to reduce scope without doing this shuffle.
I think the only actionable thing to do here is just remove the interface, move the comments to the methods directly and update where it's injected to use the actual BatchProcessor, we did the same thing for the FieldPurger.
Comment #93
andypostrelated may affect current MR #3560672: [policy, no patch] Decide if and where to adopt the #[NoDiscard] attribute
Comment #94
nicxvan commentedSorry, why is that related?
Comment #95
volegerAddressed #92 and ready for review
Comment #96
volegerComment #97
volegerRebased
Comment #98
nicxvan commentedStill not 100% this will be accepted, but I asked in slack and no one objected. We do have the plan and all of the follow ups are created.
This is far easier to review than the initial MR that converted everything in one go.
The method names either match the procedural equivalent or are clear improvements.
I updated the CR to remove the Interface references since we don't have one now.
I think this is ready!
Comment #100
quietone commentedI read the comments and made a few suggestions. I then accepted the suggestions as they don't change meaning and some were formatting. There is also one question about docs that should be answered. I am setting back to NR for that one question. Plus, someone should check the suggestions I made.
Also, the doc block for constructors isn't required so there is the option to remove those in the new classes in this MR.
Comment #101
quietone commentedAnd I updated credit.
Comment #102
needs-review-queue-bot commentedThe Needs Review Queue Bot tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".
This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.
Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.
Comment #103
volegerUpdated docs, and rerolled one test after the node module functions deprecation replacement.
Comment #104
needs-review-queue-bot commentedThe Needs Review Queue Bot tested this issue. It fails the Drupal core commit checks. Therefore, this issue status is now "Needs work".
This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.
Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.
Comment #105
volegerComment #106
nicxvan commentedI reviewed the changes in 103 and the rebase in 105 and this looks good again!
Comment #107
quietone commentedI reviewed the change record and it refers to a new interface, which is not in the MR. It also refers to functions as being deprecated when they have not been deprecated. I am setting this to needs work for an update to the change record.
For the change record, it should be concise and state the change. It should not start with a question. It can start with a simple sentence something like" A new batch processor,
Drupal\Core\Batch\BatchProcessorhas been added". In this case, examples of how to use it are also suitable. And the list of issues should be limited to just one issue, the one with the change being described.Comment #108
volegerRE #107:
- The mention of the interface in CR has been removed.
- The functions will be deprecated - see the child issues of the meta issue due to a request to break down the required changes to fully introduce the class. You can review MR !14593 for the full set of replacements, which will be introduced across all 5 child issues.
- The list of related issues has been reduced to only one meta issue.
Comment #109
nicxvan commentedOk thank you for pointing out in slack where the CR went.
I think this highlights the trickiness of this approach.
I think this very much deserves a CR, but we aren't deprecating anything.
We no longer have an interface so we could do this method by method.
I'm still a little worried about what I mentioned in 98, but I think this is an interesting approach too.
Incidentally I had cleaned up the interface changes in the CR in 98, it looks like it was accidentally.
I'd propose this to move this forward:
I'll create a minimal CR introducing the new service on this issue and we get this in front of the committers. If they decide this isn't the way forward, since we have just a class now that isn't an interface we can choose just a method or two to convert and do that issue by issue.
I'll leave the CR on the meta for reference.
Comment #110
nicxvan commentedOk the CR is now here again and it's just a very simple mapping of the procedural to methods.
Let's get this back in front of the committers for a decision if this direction is the way to go.
If not I'd propose we do batch_set, get, and process here.
Comment #111
nicxvan commentedComment #112
larowlanLeft some comments on the MR and on the meta - would be good to see a big picture plan of the end state if possible
Comment #114
daffie commentedAll remarks of @larowlan have been addressed.
Back to NR.
Comment #115
nicxvan commentedWe intentionally removed the interface in 92, I'm not sure why we would add that back here.
More and more services are being added directly workout an interface, I think that mostly addresses @larowlan's concern about the underscore functions, even if they are public, without an interface they are internal.
We could exclude them until we convert them.
I think we should revert that part at least.
Comment #116
daffie commentedRemoved the BatchProcessorInterface as requested.
Back to NR.
Comment #117
nicxvan commentedThanks!
Comment #118
berdirDid a first review.
This is challenging. I understand we tried a few different things and it's a very complex thing that's also used early in the installer and some functions have a lot of calls, but this doesn't yet seem to result in a workable API, especially around those protected methods. We can make up things that in practice aren't going to work because some of these calls are external, plus challenges around BC. And having everything on one service is going to make that huge.
Naming seems to be mixed up. batch_set() became queue(), which is something completely different from getQueue() on the same service.
I think we'll more likely make meaningful progress by focusing on the public few methods here and not worry yet about most of those protected ones just yet. I'm also wondering if we should split it up into at least two services? My initial thought was one service that manages the batch state, stuff like batch set/get and the queue, and then one to process it, downside is that many cases would then need to interact with two services. And dependencies are tricky. batch_process() calls into _batch_process(), but that one calls back into batch_get(), so we need to be careful to not have circular dependencies.
So my current thought is that the public API is one service, maybe a generic and boring BatchManager, that covers what most cases need, adding/checking current batch and starting to process it. That would wrap two internal services , something like a BatchRegistry/BatchState (since we already have storage) that contains the actual storage for batch_get() and the queue stuff, and the BatchProcessor that would also interact with the first.
And then in this issue, we'd focus on BatchManager public API only?
Comment #119
nicxvan commentedYes, I think you're right.
I'm willing to pick this up, the truth is we need to give this the locale treatment and build the correct api, which beyond your recommendations likely includes a value object for passing info around.
I'll pick this up once things settle around 12.
Comment #120
berdirA value object would be nice but also scary complicated, there's a reason we only did the builder for now. I think it would be neat to add or even require a builder object for addBatch() or what we want to call it, but changing from the array to object is going to be extremely challenging, the data structure is by reference and pretty complex, there's custom form stuff and queue stuff mixed in and what not.
One thought I have is that we'd make the two inner wrapped services internal and BatchManager would not expose the internal data structure but offer convenience methods for common things that people do with batch_get() for example. Looking through the calls, I see multiple times in core and contrib this:
So we could have a setProgressive(bool $progressive).
And other places just check if there's a batch, we could have a needsProcessing() method or so:
with those two, the majority of calls to batch_get() can be simplified and don't need to know about the data structure anymore. Then the impact will be much smaller once we're refactor the then internal functions and data structure.
Comment #121
daffie commentedI am working on this.
Comment #122
daffie commentedDisclosure: I have used AI in the making of the MR.
Comment #123
daffie commentedComment #124
needs-review-queue-bot commentedThe Needs Review Queue Bot tested this issue. It fails the Drupal core commit checks. Therefore, this issue status is now "Needs work".
This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.
Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.
Comment #125
ghost of drupal pastMy dayjob currently includes making sure batch works when Drupal is run in an app server. This is a lot of fun, read this comment explaining how broken this is. Ahem.
So I feel I need to comment on this issue: I do not understand the very direction of this. If I were doing this, I would move all the code in batch.inc unchanged into a class and have the existing functions call this service. Nothing else for a first step. Very easy, painless, zero fuss. Yes, keep even the (rather accursed) statics.
Then it's possible to fan out the followups:
Comment #126
nicxvan commentedLet me take a crack at this, I've been thinking we needed to go that route for a while.
What I'll do is just convert the batch.inc in a naive BatchProcessor class as a starting point, if it looks decent I'll update the CR.
Comment #127
nicxvan commentedThe drupal_static change wasn't pushed yet, gonna wait for that.
Comment #128
berdirI was wondering about that too, but I think there's also value in having a plan/understand of how the complete API and interaction between the functions will look like. It's not as simple as batch.inc in one service and form.inc in another, for example because of cross dependencies, batch_get() and others in form.inc are also used by batch.inc, and vice versa: batch_process() calls _batch_process().
It was my idea, so obviously, but I think the new structure with the three services looks better to me and if we agree on that overall grouping, we _could_ now revert the reverted process to something more usual and start converting the inner services bottom up and properly deprecate them and make this a meta instead of a starting point with follow-ups. Pretty sure we're going to hit a bunch of roadblocks either way.
Comment #129
ghost of drupal pastI do not think the number of services matter really, all of batch could be thrown into a single service -- I just didn't like the current MR introducing new services calling old code, that looks backwards to me.
Comment #130
nicxvan commentedI'm working on this again. I'm going to start with just the batch.inc conversion and then we call look at moving things around.
Comment #132
nicxvan commentedOk I took a first pass in 17276. I made as much of it protected as I can, some things were called outside of batch so I had to make them public, there was one test I removed that I'm not sure was the most valuable test.
currentSet needs better docs and page return.
CR and IS needs updating if this direction looks good.
I think this is worth taking a first pass look at.
Comment #133
volegerPrevious changes were more like trying to keep the existing Batch API architecture.
we could use the service as the foundation for a proper service-oriented architecture with a BatchManager, pluggable storage/execution, and a clear separation between batch definition and execution. This would let us migrate the existing functions incrementally while preserving backward compatibility, rather than creating another abstraction that ultimately has to be redesigned later. The existing follow-up issues already point toward this direction, so perhaps we could align them around this broader architecture and make the service the stable foundation for the next stage of Batch API evolution.
Comment #134
voleger@nicxvan, check MR https://git.drupalcode.org/project/drupal/-/merge_requests/13522, as I previously resolved that issue with early calls to the container before the DB connection was available.
Comment #135
volegerAlso, regarding session handling in `core/lib/Drupal/Core/Batch/BatchStorage.php`, we discussed that 10 months ago we should avoid starting the session once it has already started.