When trying to view a page with a lot of ajax buttons (or anything else resulting in bigpipe no_js placeholders) the page fails to load.

The error that occurs is:
preg_split(): Compilation failed: regular expression is too large at offset 102392 in Drupal\big_pipe\Render\BigPipe->sendNoJsPlaceholders() (line 396 of core/modules/big_pipe/src/Render/BigPipe.php).

The offending line:

$fragments = preg_split('/' . implode('|', $preg_placeholder_strings) . '/', $html, NULL, PREG_SPLIT_NO_EMPTY | PREG_SPLIT_DELIM_CAPTURE)

This combines all the placeholder keys into one big regex. Each key is around 87 bytes so depending on your PHP version and settings this can fail with only a few hundred placeholders.

Possible "solutions":
- use a different way of splitting the html into fragments
- limit the amount of placeholders before splitting
- check if $fragments equals FALSE and make a nice error.

Comments

JvE created an issue. See original summary.

cilefen’s picture

Priority: Normal » Major
wim leers’s picture

Category: Bug report » Support request
Priority: Major » Normal
Status: Active » Postponed (maintainer needs more info)
Issue tags: +Needs steps to reproduce, +Needs tests

offset 102392

Wow, that means we're generating a 100KB regexp, based on the placeholder strings! 😳 You also say "large amounts of placeholders" in the issue title. How large?

Would you mind doing

var_dump(implode('|', $preg_placeholder_strings))

and report back?

Without steps to reproduce, I can't help.

JvE’s picture

In our case it's a view rendering all the posts and people you have flagged including the flag (that uses ajax to unflag) and bigpipe crashes around 900 flags.
As a workaround we've added paging to the view.

The regex looks like this, but then with even more of the same:

(big_pipe_nojs_placeholder_attribute_safe\:y4JIEH3ptT3rvRFhbx0i52qoMHYFpvef7R9NIbEUSh8)|(big_pipe_nojs_placeholder_attribute_safe\:S1elZEr0MQ8yiTv\-afiZAHJsrLxEx2uDccAuvC_oBhQ)|(big_pipe_nojs_placeholder_attribute_safe\:Co69fKYlQSHIlSH5Jy5Szdg2J5ZD2pdPVFh\-4rt8wEQ)|(big_pipe_nojs_placeholder_attribute_safe\:drlfivd2ainYksfd5\-RQCairL6q6CiMNfh6cxuzgSdw)|(big_pipe_nojs_placeholder_attribute_safe\:YWOzoqIp7uW\-8UR1ixJmPSRJkta3IhOt0cq_Tp5wIrg)|(big_pipe_nojs_placeholder_attribute_safe\:vTNz5abQMfYbYYHQ_LRxaXZb4qFj_V8z8IK_4vEvjM0)|(big_pipe_nojs_placeholder_attribute_safe\:qNvXzKY_nZu1JGJO1FSzXvUpskCY7cSBEQj5XKeosyA)|(big_pipe_nojs_placeholder_attribute_safe\:aJAuoXWQ3MvLLN0fXFFJPgOvjZd7lN2oPSGasgpBlvc)|(big_pipe_nojs_placeholder_attribute_safe\:WKX1MUUi2jtBnUaReQfO_s1gm1l0OJBkQddX_IZphiA)|(big_pipe_nojs_placeholder_attribute_safe\:ZWhrW3XqRLUi1KYbHra_FNSuh8ThnzczDgGi6gy5Vo8)|(big_pipe_nojs_placeholder_attribute_safe\:yO_EsjdlYqw8oHcDUHFbgMCkyIUmIL0\-gUNENqs24a4)|(big_pipe_nojs_placeholder_attribute_safe\:E\-ymE3E2D3joMThPwrfenTBsn5Dx0ItlTAuzACrQ7G0)|(big_pipe_nojs_placeholder_attribute_safe\:oKyqYCve40wVMxOAsvVXCEXB2cJMs2KHegkz\-\-RIXts)|(big_pipe_nojs_placeholder_attribute_safe\:bmKPZy0ZKMG6bqQEAfX1t9rVMobUf36FM0FolzTqUmQ)|(big_pipe_nojs_placeholder_attribute_safe\:sTJwn_So7c9CxvAyevOAUJMZnf8bc2Z3zVxjsTP8\-3c)|(big_pipe_nojs_placeholder_attribute_safe\:YuTEvBFTrNY\-By0cIB7njU2CUgVOB3jUbA2W7LXhXjo)|(big_pipe_nojs_placeholder_attribute_safe\:2KtXRuIS7ZQJ8lAU9s4\-WxRtImZIg8eMR_OxpWq5F9E)|(big_pipe_nojs_placeholder_attribute_safe\:cBHIRWhLHDEITnQVmI2DnEvYOAlByasowPw6zcLzDk4)|(big_pipe_nojs_placeholder_attribute_safe\:eSN0W9\-zKA87wot_er08R\-ZFFTcYDcEsiX4kxn1jz2k)|(big_pipe_nojs_placeholder_attribute_safe\:oV_VepBGfzN1OomAELb8iJT8l9kOTYpqm0YjCcE0OIQ)|(big_pipe_nojs_placeholder_attribute_safe\:4AVYIiS4JBcTWARF1p0gIu2cEJLgAbzqJB4\-h6SYBEg)|(big_pipe_nojs_placeholder_attribute_safe\:QZWTS0evCsi3z\-qBWLAMKiltk50CohWaufg9\-huaMKM)|(big_pipe_nojs_placeholder_attribute_safe\:dPaL6YAhF_0jqVNoG4\-eT8ZO33xaRjMYd1RKL13iTD8)|(big_pipe_nojs_placeholder_attribute_safe\:FFTECWNi4jHvGeIMopqf5x5R7\-edVNsdJqi1Q\-Pa_EM)|(big_pipe_nojs_placeholder_attribute_safe\:XdagZkGUNlOmzsBV72tcfnnAPnmzpk\-XDFClng_LBak)|(big_pipe_nojs_placeholder_attribute_safe\:KA8Tn\-XZtbZOW6kXEML31RqgO7DP7QfqF661t2JJ\-ww)|(big_pipe_nojs_placeholder_attribute_safe\:P2gEAVR8qztYvFVy7NL6ZB8hhl5QM\-yN2nrhSEZxZyM)|(big_pipe_nojs_placeholder_attribute_safe\:elBI9PYngQ9L8_GQP3efaV8\-_8sX7cjeN6aRlRKTL1U)|(big_pipe_nojs_placeholder_attribute_safe\:Dm_xCrbq_D3yAHyxkD6fPClTPpooHJrZAkNDJtQC00Q)|(big_pipe_nojs_placeholder_attribute_safe\:52Udu8NjXI4fip_SNTejXFO31u8DR1ECTBH2vvJD4uM)|(big_pipe_nojs_placeholder_attribute_safe\:Zz8bPH6_sff8ws3Mu0OWK1hYeY8gYysHl5jRXvjzi2s)|(big_pipe_nojs_placeholder_attribute_safe\:b88VrSEjxx5JFcrvkM9MWnBYVdbCDsvQehT28LgWllM)|(big_pipe_nojs_placeholder_attribute_safe\:dGh6Ap_oo3ui3jNb1VlmT3oGTycxl\-3Nt6u5tIwUrk4)|(big_pipe_nojs_placeholder_attribute_safe\:ayWjWiRrmSp2kRk2Svr7axM8WxVQTWJLtri1t2SMDZA)|(big_pipe_nojs_placeholder_attribute_safe\:EWVGG2OXA7j1kh7OT1VT1PlZSiistWXOGOtN2rWCF74)|(big_pipe_nojs_placeholder_attribute_safe\:fGl4LBkjxpIPGq9mVskB\-pRB8vz0iS9URM3S5pbnyhA)|(big_pipe_nojs_placeholder_attribute_safe\:FTD1G4omSklnyWEv2k157E8S2JCfM\-9pD007ZRXaoa4)|(big_pipe_nojs_placeholder_attribute_safe\:FdEOrDCGbkKLuzYhikPxuk6IsEmCRu7TUaOZtyVoJ_Q)|(big_pipe_nojs_placeholder_attribute_safe\:OQxvqnxGWdcWrdWCanNQK7P58K4rjIMtGke34\-93Buo)|(big_pipe_nojs_placeholder_attribute_safe\:wfmor\-\-yF3qAeiLNAlYoqQjRBtDh4OO1eJL6cehn_Lk)|(big_pipe_nojs_placeholder_attribute_safe\:tpj\-\-l60vd6kRWYbDSyNTEgBw4GWXfjwcOKYoFal2VE)|(big_pipe_nojs_placeholder_attribute_safe\:aYPpbXfdGuwO\-smW7VYIJnlUMdEhmuccyrZVSjHvTpo)|(big_pipe_nojs_placeholder_attribute_safe\:mQuPVrvkURahxjPIrt35T3658AsC2NNNr3M7Afwkbr0)|(big_pipe_nojs_placeholder_attribute_safe\:xFjJIgWuQksOdmWK7iCuwPptUMNkI5YtfxrGmdr3UgI)|(big_pipe_nojs_placeholder_attribute_safe\:hDfyi5bU8QhXuEjCyCDUPWgwYfSQbElFt\-cfbNiNjlo)|(big_pipe_nojs_placeholder_attribute_safe\:2_2tYTpnLsvKMl4ZS8YbxJG_OGb6h_uuors6TI0jHXI)|(big_pipe_nojs_placeholder_attribute_safe\:uyVV9JXbSmGK_TWFwMImB7g5uLsTA_PKa8Aj7Emz1tM)|(big_pipe_nojs_placeholder_attribute_safe\:Slunjb9PL1gc4B_V95Tj_OqJ_LA5JgFe29D0bMKDfj4)|(big_pipe_nojs_placeholder_attribute_safe\:N2t202OXx7DRraL\-71PuEq\-r2pHGC6fQ5RLKWSEgnqo)|(big_pipe_nojs_placeholder_attribute_safe\:pijcTvrrjxZYFypKmvmhyL4\-\-kB8HqBIXZlJVu0UmDg)
...

I tried looking into the existing test coverage of big_pipe but could find no easy way to add a testcase.

wim leers’s picture

Title: BigPipe breaks on large amounts of placeholders » BigPipe breaks on large amounts of placeholders (e.g. Flag module on view displaying ~1000 entities)
Status: Postponed (maintainer needs more info) » Active
Issue tags: -Needs steps to reproduce +scalability

Thanks, that makes this much more understandable.

Each of those placeholders is legitimate AFAICT. On the one hand I wonder if I should make BigPipe \Drupal\big_pipe\Render\Placeholder\BigPipeStrategy::processPlaceholders() do nothing in case of e.g. >100 placeholders, and ask of modules like Flag which can result in many small placeholders to instead implement it via JavaScript. On the other hand, this is an entirely valid and legitimate use of BigPipe, and it's totally reasonable to expect BigPipe to handle hundreds of placeholders.

IOW: this is a question of algorithmic scalability. The current algorithmic implementation does not scale this far. Perhaps a strategy like \Drupal\Core\EventSubscriber\ActiveLinkResponseFilter::setLinkActiveClass()'s could work. Needs investigation.

wim leers’s picture

Project: Drupal core » Flag
Version: 8.6.x-dev » 8.x-4.x-dev
Component: big_pipe.module » Flag core

The simpler solution for now would be to optimize how the Flag module generates placeholders — does each of those placeholders really need to be unique? I suspect all of them are being replaced with a unique URL. But perhaps only the query string is unique, and everything else remains the same? Could we in that case omit the query string from the placeholder?

Moving to the Flag issue queue to get feedback from their maintainer on this proposal. Dear maintainer: I'm willing to help make the changes in the Flag module to make this happen — just looking for feedback first! :)

joachim’s picture

I don't know much about how BigPipe works, and I'm pretty hazy on render array caching too.

I can tell you that each flag link will have a unique URL, of the form: /flag/ACTION/FLAG_ID/ENTITY_ID

So if for example, you have a view listing nodes, and you show a flag on each node, you'll get links like this:

- flag/flag/my_flag/1
- flag/flag/my_flag/2
- flag/unflag/my_flag/3 (supposing node 3 is currently flagged)
- etc

If you show links for more than one flag, then you'll get:

- flag/flag/my_flag/1
- flag/flag/my_other_flag/1
- flag/flag/my_flag/2
- flag/flag/my_other_flag/2

The other thing to consider is that our lazy builder, FlagLinkBuilder, doesn't just return the URL. It returns a render array that includes wrapping classes for the flag link. That's because:

1. the link can be themed, with the template receiving variables such as the flag, the current user, the flaggable entity
2. flag links can use AJAX, and show a hover message on the link when clicked, so the HTML for the flag link needs wrapper classes for this to work

So each of the flag links output by FlagLinkBuilder could be a completely unique piece of HTML.

berdir’s picture

Most importantly, the link is the result of checking whether a user already flagged that entity or not. *That* is the main reason we use a placeholder, everything else is just the result of that.

JvE’s picture

StatusFileSize
new3.12 KB

I've added a rather straightforward patch, but am having serious difficulty in finding the correct place to implement a test.

JvE’s picture

StatusFileSize
new5.38 KB

Created new version which also avoids notices.
And added a new unit test specifically for this case.

JvE’s picture

JvE’s picture

StatusFileSize
new5.72 KB

After all these years I'm still struggling with Drupal's patch based workflow and ticket/issue UI.

edit: And now I realize it's trying to patch big_pipe inside Flag :(

socketwench’s picture

Project: Flag » Drupal core
Version: 8.x-4.x-dev » 9.x-dev
Component: Flag core » ajax system

Moving this to Drupal core instead of Flag.

socketwench’s picture

Version: 9.x-dev » 8.6.x-dev
Category: Support request » Bug report
JvE’s picture

Component: ajax system » big_pipe.module
JvE’s picture

Status: Active » Needs review

Version: 8.6.x-dev » 8.7.x-dev

Drupal 8.6.0-alpha1 will be released the week of July 16, 2018, which means new developments and disruptive changes should now be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.7.x-dev » 8.8.x-dev

Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

wim leers’s picture

Issue tags: -Needs tests +Performance
StatusFileSize
new2.86 KB
new5.69 KB
new2.31 KB

#7: Thanks for the feedback! And sorry for dropping the ball here :( This coincided with https://wimleers.com/blog/state-of-jsonapi-2018-07, after which it was a whirlwind until a week ago, when JSON:API landed in Drupal 8.7 :)

#12: thanks for the patch! 🙏 VERY well done! 👏 I just fixed nits in your test, and am uploading a separate patch that only includes the test — this should fail.

wim leers’s picture

Status: Needs review » Reviewed & tested by the community
StatusFileSize
new3.92 KB
new5.25 KB

Review of the actual code changes:

  1. Bunch of nits, fixed them all.
  2. +++ b/core/modules/big_pipe/src/Render/BigPipe.php
    @@ -754,4 +750,48 @@ protected function getPlaceholderOrder($html, $placeholders) {
    +    if (strlen($pattern) < 31000) {
    +      // Only small (<31K) patterns can be handled by preg_split.
    

    Ideally we'd use a PHP-provided constant, but … PHP doesn't provide that, even though it's known in the PHP source code: https://github.com/php/php-src/search?q=MAX_PATTERN_SIZE&type=Code

    So … yeah, let's stick to this heuristic, since we know it works on PHP 5.5 too.

The last submitted patch, 19: 2981584-19-tests_only_FAIL.patch, failed testing. View results

ndobromirov’s picture

Why not consider different placeholder format?
With a start and end...

From
big_pipe_nojs_placeholder_attribute_safe:y4JIEH3ptT3rvRFhbx0i52qoMHYFpvef7R9NIbEUSh8

To
big_pipe_nojs_placeholder_attribute_safe\:y4JIEH3ptT3rvRFhbx0i52qoMHYFpvef7R9NIbEUSh8:big_pipe_nojs_placeholder_end

At that point the regex will be the same for any amount of placeholders in there.
big_pipe_nojs_placeholder_attribute_safe\:.+?\:big_pipe_nojs_placeholder_end

Or even keeping the current placeholders with this:
big_pipe_nojs_placeholder_attribute_safe\:[a-zA-Z_\-0-9]{43}
... Whatever the number of characters we are expecting.

I am not getting why we need the white-list splitting, when we know the format we look for.

wim leers’s picture

Status: Reviewed & tested by the community » Needs review

@ndobromirov Ohhhh, very interesting idea! Would you like to turn that into a patch? :) The test coverage can remain the same!

ndobromirov’s picture

StatusFileSize
new3.56 KB
new2.75 KB

Implemented proposal from #22. Let's see what the test bot will say...

Status: Needs review » Needs work

The last submitted patch, 24: 2981584-24.patch, failed testing. View results

ndobromirov’s picture

After reviewing all the cases listed here BigPipePlaceholderTestCases.
The format will not work all the time :(, mainly because this is not the only format of placeholders that could be happening.

But still this can be used as a special case when all placeholders are in this format or to replace all placeholders that comply with this format with a single reg-ex. This should result in reduction of the regex length in much more cases.

For cases like this one:

big_pipe_nojs_placeholder_attribute_safe:form_action_cc611e1d
big_pipe_nojs_placeholder_attribute_safe:&lt;hello
big_pipe_nojs_placeholder_attribute_safe:1111111111111111111111111111111111111111
big_pipe_nojs_placeholder_attribute_safe:2222222222222222222222222222222222222222

They can be rewritten in the regex to drop the common prefix, essentially doubling the fast case capacity (or even more).


# Reuse the common prefix where possible only.
big_pipe_nojs_placeholder_attribute_safe:(form_action_cc611e1d|&lt;hello|1111111111111111111111111111111111111111|2222222222222222222222222222222222222222)|

# Reuse common prefix + the common pattern...
big_pipe_nojs_placeholder_attribute_safe:(form_action_cc611e1d|&lt;hello|[a-zA-Z_\-0-9]{43})|(others...)

This way we should get way more often into the fast regex splitting scenario than in the slow fallback.

wim leers’s picture

Let me guess: your proposal works fine for the attribute value placeholder case but not the HTML placeholder case?

ndobromirov’s picture

Status: Needs work » Reviewed & tested by the community

Yes :).

The last option will work for all possible ones, but will speed-up only the attributes case and will still need the fallback from #20. I think its very safe to commit 20 and any further improvements can live in a follow-up if anyone hits the limit :).

It was just an idea, without knowing any of the details of BigPipe. I assumed there was a single format based on the output from #4.

Moving back to RTBC as per #20. No time to dive into this :(.

wim leers’s picture

StatusFileSize
new5.25 KB

Agreed!

And thank you for attempting to come up with an even better solution! 🙏

Re-uploading #20 to make it clear that that is the patch thats' RTBC.

  • catch committed 0f73e39 on 8.8.x
    Issue #2981584 by Wim Leers, JvE, ndobromirov: BigPipe breaks on large...
catch’s picture

Version: 8.8.x-dev » 8.7.x-dev
Status: Reviewed & tested by the community » Patch (to be ported)

Committed 0f73e39 and pushed to 8.8.x. Thanks!

We're in a commit freeze for 8.7.x at the moment, so moving to to-be-ported there.

wim leers’s picture

Yay!

alexpott’s picture

Status: Patch (to be ported) » Fixed

Did the cherry-pick for 8.7.x

  • alexpott committed 4c1adcd on 8.7.x authored by catch
    Issue #2981584 by Wim Leers, JvE, ndobromirov: BigPipe breaks on large...
wim leers’s picture

Lovely, thanks! That means this shipped with Drupal 8.7.1! 🥳

berdir’s picture

> Lovely, thanks! That means this shipped with Drupal 8.7.1! 🥳

Security releases do not contain bugfixes, see https://www.drupal.org/project/drupal/releases/8.7.1:

No other fixes are included.

wim leers’s picture

Oops! I didn't notice 8.7.1 was a security release — getting back into it after a vacation, I guess it shows 😅

wim leers’s picture

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.