When trying to view a page with a lot of ajax buttons (or anything else resulting in bigpipe no_js placeholders) the page fails to load.
The error that occurs is:
preg_split(): Compilation failed: regular expression is too large at offset 102392 in Drupal\big_pipe\Render\BigPipe->sendNoJsPlaceholders() (line 396 of core/modules/big_pipe/src/Render/BigPipe.php).
The offending line:
$fragments = preg_split('/' . implode('|', $preg_placeholder_strings) . '/', $html, NULL, PREG_SPLIT_NO_EMPTY | PREG_SPLIT_DELIM_CAPTURE)
This combines all the placeholder keys into one big regex. Each key is around 87 bytes so depending on your PHP version and settings this can fail with only a few hundred placeholders.
Possible "solutions":
- use a different way of splitting the html into fragments
- limit the amount of placeholders before splitting
- check if $fragments equals FALSE and make a nice error.
| Comment | File | Size | Author |
|---|---|---|---|
| #29 | 2981584-20.patch | 5.25 KB | wim leers |
| #24 | interdiff-20-24.txt | 2.75 KB | ndobromirov |
| #24 | 2981584-24.patch | 3.56 KB | ndobromirov |
| #20 | 2981584-20.patch | 5.25 KB | wim leers |
| #20 | interdiff.txt | 3.92 KB | wim leers |
Comments
Comment #2
cilefen commentedComment #3
wim leersWow, that means we're generating a 100KB regexp, based on the placeholder strings! 😳 You also say "large amounts of placeholders" in the issue title. How large?
Would you mind doing
and report back?
Without steps to reproduce, I can't help.
Comment #4
JvE commentedIn our case it's a view rendering all the posts and people you have flagged including the flag (that uses ajax to unflag) and bigpipe crashes around 900 flags.
As a workaround we've added paging to the view.
The regex looks like this, but then with even more of the same:
I tried looking into the existing test coverage of big_pipe but could find no easy way to add a testcase.
Comment #5
wim leersThanks, that makes this much more understandable.
Each of those placeholders is legitimate AFAICT. On the one hand I wonder if I should make BigPipe
\Drupal\big_pipe\Render\Placeholder\BigPipeStrategy::processPlaceholders()do nothing in case of e.g. >100 placeholders, and ask of modules like Flag which can result in many small placeholders to instead implement it via JavaScript. On the other hand, this is an entirely valid and legitimate use of BigPipe, and it's totally reasonable to expect BigPipe to handle hundreds of placeholders.IOW: this is a question of algorithmic scalability. The current algorithmic implementation does not scale this far. Perhaps a strategy like
\Drupal\Core\EventSubscriber\ActiveLinkResponseFilter::setLinkActiveClass()'s could work. Needs investigation.Comment #6
wim leersThe simpler solution for now would be to optimize how the Flag module generates placeholders — does each of those placeholders really need to be unique? I suspect all of them are being replaced with a unique URL. But perhaps only the query string is unique, and everything else remains the same? Could we in that case omit the query string from the placeholder?
Moving to the Flag issue queue to get feedback from their maintainer on this proposal. Dear maintainer: I'm willing to help make the changes in the Flag module to make this happen — just looking for feedback first! :)
Comment #7
joachim commentedI don't know much about how BigPipe works, and I'm pretty hazy on render array caching too.
I can tell you that each flag link will have a unique URL, of the form: /flag/ACTION/FLAG_ID/ENTITY_ID
So if for example, you have a view listing nodes, and you show a flag on each node, you'll get links like this:
- flag/flag/my_flag/1
- flag/flag/my_flag/2
- flag/unflag/my_flag/3 (supposing node 3 is currently flagged)
- etc
If you show links for more than one flag, then you'll get:
- flag/flag/my_flag/1
- flag/flag/my_other_flag/1
- flag/flag/my_flag/2
- flag/flag/my_other_flag/2
The other thing to consider is that our lazy builder, FlagLinkBuilder, doesn't just return the URL. It returns a render array that includes wrapping classes for the flag link. That's because:
1. the link can be themed, with the template receiving variables such as the flag, the current user, the flaggable entity
2. flag links can use AJAX, and show a hover message on the link when clicked, so the HTML for the flag link needs wrapper classes for this to work
So each of the flag links output by FlagLinkBuilder could be a completely unique piece of HTML.
Comment #8
berdirMost importantly, the link is the result of checking whether a user already flagged that entity or not. *That* is the main reason we use a placeholder, everything else is just the result of that.
Comment #9
JvE commentedI've added a rather straightforward patch, but am having serious difficulty in finding the correct place to implement a test.
Comment #10
JvE commentedCreated new version which also avoids notices.
And added a new unit test specifically for this case.
Comment #11
JvE commentedComment #12
JvE commentedAfter all these years I'm still struggling with Drupal's patch based workflow and ticket/issue UI.
edit: And now I realize it's trying to patch big_pipe inside Flag :(
Comment #13
socketwench commentedMoving this to Drupal core instead of Flag.
Comment #14
socketwench commentedComment #15
JvE commentedComment #16
JvE commentedComment #19
wim leers#7: Thanks for the feedback! And sorry for dropping the ball here :( This coincided with https://wimleers.com/blog/state-of-jsonapi-2018-07, after which it was a whirlwind until a week ago, when JSON:API landed in Drupal 8.7 :)
#12: thanks for the patch! 🙏 VERY well done! 👏 I just fixed nits in your test, and am uploading a separate patch that only includes the test — this should fail.
Comment #20
wim leersReview of the actual code changes:
Ideally we'd use a PHP-provided constant, but … PHP doesn't provide that, even though it's known in the PHP source code: https://github.com/php/php-src/search?q=MAX_PATTERN_SIZE&type=Code
So … yeah, let's stick to this heuristic, since we know it works on PHP 5.5 too.
Comment #22
ndobromirov commentedWhy not consider different placeholder format?
With a start and end...
From
big_pipe_nojs_placeholder_attribute_safe:y4JIEH3ptT3rvRFhbx0i52qoMHYFpvef7R9NIbEUSh8To
big_pipe_nojs_placeholder_attribute_safe\:y4JIEH3ptT3rvRFhbx0i52qoMHYFpvef7R9NIbEUSh8:big_pipe_nojs_placeholder_endAt that point the regex will be the same for any amount of placeholders in there.
big_pipe_nojs_placeholder_attribute_safe\:.+?\:big_pipe_nojs_placeholder_endOr even keeping the current placeholders with this:
big_pipe_nojs_placeholder_attribute_safe\:[a-zA-Z_\-0-9]{43}... Whatever the number of characters we are expecting.
I am not getting why we need the white-list splitting, when we know the format we look for.
Comment #23
wim leers@ndobromirov Ohhhh, very interesting idea! Would you like to turn that into a patch? :) The test coverage can remain the same!
Comment #24
ndobromirov commentedImplemented proposal from #22. Let's see what the test bot will say...
Comment #26
ndobromirov commentedAfter reviewing all the cases listed here
BigPipePlaceholderTestCases.The format will not work all the time :(, mainly because this is not the only format of placeholders that could be happening.
But still this can be used as a special case when all placeholders are in this format or to replace all placeholders that comply with this format with a single reg-ex. This should result in reduction of the regex length in much more cases.
For cases like this one:
They can be rewritten in the regex to drop the common prefix, essentially doubling the fast case capacity (or even more).
This way we should get way more often into the fast regex splitting scenario than in the slow fallback.
Comment #27
wim leersLet me guess: your proposal works fine for the attribute value placeholder case but not the HTML placeholder case?
Comment #28
ndobromirov commentedYes :).
The last option will work for all possible ones, but will speed-up only the attributes case and will still need the fallback from #20. I think its very safe to commit 20 and any further improvements can live in a follow-up if anyone hits the limit :).
It was just an idea, without knowing any of the details of BigPipe. I assumed there was a single format based on the output from #4.
Moving back to RTBC as per #20. No time to dive into this :(.
Comment #29
wim leersAgreed!
And thank you for attempting to come up with an even better solution! 🙏
Re-uploading #20 to make it clear that that is the patch thats' RTBC.
Comment #31
catchCommitted 0f73e39 and pushed to 8.8.x. Thanks!
We're in a commit freeze for 8.7.x at the moment, so moving to to-be-ported there.
Comment #32
wim leersYay!
Comment #33
alexpottDid the cherry-pick for 8.7.x
Comment #35
wim leersLovely, thanks! That means this shipped with Drupal 8.7.1! 🥳
Comment #36
berdir> Lovely, thanks! That means this shipped with Drupal 8.7.1! 🥳
Security releases do not contain bugfixes, see https://www.drupal.org/project/drupal/releases/8.7.1:
No other fixes are included.
Comment #37
wim leersOops! I didn't notice 8.7.1 was a security release — getting back into it after a vacation, I guess it shows 😅
Comment #38
wim leersIt shipped today, with https://www.drupal.org/project/drupal/releases/8.7.2!