Core Web UI interfaces can go over php's max_input_vars

This problem (or one very like it) drove me mad for a whole day. Eventually I tracked the problem down, using Firebug and lots of Google. The series of "is not a function" messages turned out to be something of a red herring. In actual fact, the problem was caused by a truncated Ajax request. The fact that the Ajax data was being truncated was causing Drupal to try to load an incomplete page; hence apparently missing files.

There could be several reasons for an incomplete Ajax request. In MY PARTICULAR CASE, fixing these settings in my php.ini solved the problem:

max_input_nesting_level = 1000
max_input_vars = 100000

This is because the form that the Ajax was being called from was huge, with lots of field collections within field groups. So I think it hit a limit either with its nesting or with the number of variables it could handle. Setting these to larger values allowed all the Ajax data to get through.

Various other settings that could possibly cause this problem include settings like post_max_size, publish_max_size (but not in my case).

EDIT: important note - initially I set the above variables to 0 assuming that this would remove the limit as is the normal convention. This was not in fact the case and caused ALL forms on ALL sites on my server to fail on submission.

I should also note, for anyone using Suhosin, you'll also want to set the following variables in your php.ini:

suhosin.post.max_vars = 100000
suhosin.request.max_vars = 100000
suhosin.post.max_value_length = 1000000
suhosin.request.max_value_length = 1000000

Obviously your exact values don't have to be the same as mine. The point is, unless you do something about it, by default Suhosin is going to chop off your form submissions if they're too long, and not tell you about it. This is a problem I've seen with other very large forms, e.g. the User Permissions form on a site with a very large number of modules.

This seems like more of a Drupal core bug. Should those values be included in our default settings.php file? With many more modules opting to make use of ajax, it would be nice to accomodate them out of the box. Does this need documentation somewhere too?

Related issues:
#1517092: admin/structure/menu/manage/MENU_NAME cannot be submitted on PHP 5.3.9+ for large menus (>~1000 items)
#1870014: Admin Interface Fail: PHP max_input_vars & suhosin post variable limit with a node with lots of fields (50+).
#329056: Unable to save changes to menu

There's lots more in contrib regarding this issue.

Tagging.

Edit:
V--- not sure if you intended to remove the tag, when you bumped priority. Would be useful to go through contrib and find similar issues.

Need steps to replicate. It sounds like it's this particular setup that causes an issue. If it's a php limit it's probably because something is going overboard with the fields/settings/whatever.

I wouldn't say it's a core issue, but if it can be reproduced, why not.

Actually, is that related/duplicate of #1517092: admin/structure/menu/manage/MENU_NAME cannot be submitted on PHP 5.3.9+ for large menus (>~1000 items)?

@_nod: its easily reproducable. Please check out my.explanation into why this is happening here.
#1870014: Admin Interface Fail: PHP max_input_vars & suhosin post variable limit with a node with lots of fields (50+).

Yes, you are most likely going to need to go a little overboard with something before you run in to this problem.

If you created a page with 1001 unique HTML id tags, then attempt to perform any Ajax request and your limits were set to 1000, my bet is query would get included twice and cause problems in the request.

In chrome inspector check the post fields to /system/ajax . Once the number of those goes over 1000 you'll start running into ajax issues.

While this is a configuration issue, if possible we should try and reduce the Ajax post fields if possible. It would help with speed as well. Otherwise we need to document the problem and make people aware what's going on. Gonna be pretty embarrassing for drupal to have to tell people to bump up their max post cars to over 1000 because drupal can't figure out a more elegant solution to this problem. For me I've had this issue for months and its very hard to debug.

Sorry, didn't mean to remove the issue tag!

Thanks for the great documentation so far. Is there a way to reproduce this bug using only the latest Drupal core? That may help clarify where the problem exists.

Well, in D8 the ids are in 1 variable only. It's a space separated string of ids so that shouldn't happen because of it (and I think there is a D7 patch waiting in the issue).

So I guess it's something to add to this: #956186: Allow AJAX to use GET requests.

@nod_ There's no mention of suhosin in issue #956186, so it's something that should get brought up there. The suhosin log is usually added to /var/log/syslog, which is probably why most people don't notice these problems. I would assume they happen more frequently, but are not getting reported because this is not a log a PHP developer usually checks.

For D7 a long path to a javascript file could potentially also trigger something like post.max_name_length as well. In my install for instance, I believe these should of been getting triggered as well...but it seems to get passed them despite vars like
"ajax_page_state[js][sites/all/modules/contrib/context/plugins/context_reaction_block.js] : 1" being 88 chars long.

If you search a default suhosin.ini for 'post.max_' you'll find some other variables which we need to keep in mind.

;suhosin.post.max_array_depth = 100
;suhosin.post.max_array_index_length = 64
;suhosin.post.max_name_length = 64
;suhosin.post.max_totalname_length = 256
;suhosin.post.max_value_length = 1000000
;suhosin.post.max_vars = 1000
;suhosin.post.disallow_nul = on

@nod_ do you think it would be wise to write a test for core, which would test the values in $options against suhosin defaults in `ajax.beforeSend` in misc/ajax.js

Would be a good test, or at very least a warning to see if an ajax request would ever overflow suhosin / php defaults.

Yeah that sounds like a plan. I'm happy to make the ajax call fail and throw an error if it goes over the limit.

@nod_, we'll this is technically a PHP configuration issue, so a hard fail would cause more harm than good. Except if you were detecting the settings and they were in fact going to mess up the request.

But some kind of notice needs to happen so this doesn't go under the radar.

I've seen a Drupal 6 permissions screen go over max_input_vars as well, in the UI it just failed silently (i.e. any vars over the limit just get ignored but the form still submits).

We could potentially do something in PHP to detect if the number of vars == max_input_vars then throw a warning although it's a shame that PHP itself doesn't throw an exception or similar.

Configuration issue though so downgrading from major.

Additional data

I believe I just ran into this problem on D7 using the Menu Editor module #1032270: Menu not changed, no error message displayed (max_input_vars). I tried to make some changes to my main menu and the form appeared to submit as expected, but none of the changes were saved. I realize this is technically a configuration issue, but what concerns me is that there is no error, or visual indication of why it isn't working. If I hadn't stumbled upon some existing issues, I never would have figured this out.

We could potentially do something in PHP to detect if the number of vars == max_input_vars then throw a warning although it's a shame that PHP itself doesn't throw an exception or similar.

I think this would help, yes. The counting could happen after all hook_form_alter() stuff has been applied.
Then maybe show a message and write to watchdog? Or remember the form ids and url of big forms, and show this on the requirements page?
In my experience, the watchdog easily gets easily clogged up with PHP strict notices, or file permission stuff. So for things where a straightforward solution is available (change your PHP configuration), a report on the status/requirements page seems more useful.

The following line in misc.js will cause a huge request array:
options.data['ajax_html_ids[]'].push(this.id);

It can be reproduced by creating a multiple field on a node with 1000 instances of the field (or a number of more complex fields like ckeditor textarea).

This is a real issue, especially if you run a site with huge forms, which is sometimes a necessity. In a perfect world 'ajax_html_ids' should be killed with fire, along with the dev that came up with such a horrible solution.

But since stuff has grown dependant on this it might actually make more sense to bypass the max_vars settings simply by letting drupal pass the post data from the input stream instead, something like:

$_POST = parse_str(file_get_contents('php://input'));

in the bootstrap would bypass max_vars for posts entirely.

@EJTH developers come up with horrible solutions all the time. Especially when they're working for free on a volunteer open source project I don't think that deserves the death penalty.

Killing of drupal_html_id() though is happening in #1305882: drupal_html_id() considered harmful; remove ajax_html_ids to use GET (not POST) AJAX requests.

Adding #1203766: With large number of permissions /admin/people/permissions becomes unusable

Here's a hasty patch (without tests) that provides feedback for the core permissions form.

It's perhaps possible to generalize this in FormBuilder::doBuildForm; after any #after_builds have run, one could in principal determine how many form elements would be generated. It might not always be accurate, especially after contrib gets involved, and it won't be beneficial in the vast majority of cases. I think it's better to address this on a per-form basis. Maybe add a helper function on FormBase to keep things DRY.

Bumping this to critical, it's a long standing bug, but it's silent data loss with no workaround.

I'm posting a patch here for D7 based on work we've done to solve this issue in Backdrop CMS. It will need to be reworked for Drupal 8+, but I'm hoping it might still be helpful.

Attached patch converts @Jenlamptom's d7 backdrop patch to 9.2.x.

I first tried to move this to form validation, before I realized that it needs to happen here, before form validation, so that we check the input variables and display a warning /BEFORE/ the form is submitted.

I simplified some of the messaging, so that we can further discuss it messaging here. I think it would be nice to link to a d.o. page about why/how to increase this, the backdrop patch had that. Should we add logging?

I'm not sure if adding StringTranslationTrait, MessengerTrait to FormBuilder has any drawbacks...

This still needs a test :(

Updated patch, fixes PHP fatals using SubformState (errors noticed during manual block placement tests)

These errors highlight even more so, that this still needs a test...

Updated patch fixes missing trait in test.

Updated patch adds the new methods to FormStateDecoratorBase, although I'm not sure what this is used for.

Patch from #35 still applies to 7.80.

Sadly the return from TestBot Build Successful doesn't mean all went well, in fact, there are more errors then could be shown at once.

If you click through you can see them, like for the last test..

Back to Needs work.

Attached patch is a minor tweak, to limit the hook_requirements() check to runtime, because most systems use different settings for commands line and runtime, and this check is really meaningless for the command line (which doesn't have form variables).

This patch contains a proof of concept for another approach: allow forms to be marked as "large", if a post for a large form contains more input vars than max_input_vars, parse the raw payload of the request to retrieve all input vars.

What do you think?

#45 did seem to stop the max_input_vars error from appearing when editing the Management menu.

I'm not sure about #45 as we're silently bypassing a security feature added by site admins.

The backdrop approach of showing a warning seems much better

The long term fix here would be to redesign the forms so they don't go over this limit, but that was already being discussed before 2012 when this issue was opened and it doesn't look likely to be resolved imminently.

Given how intractable this is, and also how silent the failure is when it's a problem, I think it would be a good idea to add warnings in a couple of different places:

1. In hook_requirements() when max_input_vars is quite low. We could either multiply roles * permissions and warn if it's more than 80% of max_input_vars, or set some arbitrary amount. This could link to a change record.

2. Similar checks on the user permissions and/or menu links forms themselves, not sure what the form-specific wording should look like, it might want to link to the status report maybe?

This would help to resolve the data loss issue, we can then have a follow-up to remove the warnings when we've resolved the form design issues themselves. For example with the user permissions system, this could be a hierarchical permissions system, or AJAX-ifying the permissions form to write to temp store + a confirm changes button instead of submitting everything at once.

#1 and #2 could be split to their own issues and maybe convert this to a meta/plan.

Tagging for issue summary update, a bit pre-occupied at the moment but might be able to tackle that +spinning the issues out later if no-one beats me to it.

Updated the issue summary with new plan

One idea that works only with JS enabled is when submitting the form, adding disabled to all unchecked checkboxes, that will exclude them from the POSTed values. Not sure how well it'd work when unchecking a permission…

I think #53 is worth looking into, we've already had this bug for years with or without js and making it work js-only would be a big improvement over not at all.

I'm not sure we can help more on the frontend side, browsers only submit checked checkboxes already. So on my 4000 checkbox pages, there are only the 143 checked checkboxes sent in the POST request.

browsers may have gotten smarter about all this since then.

Aren't there "shadow" hidden inputs as well? In order to get rid of those you'd need to e.g. round-trip a hash of the current state to detect if another user saved permissions in the meantime. Then the front-end could submit only the checkbox transitions. That would require JS as well.

While we're talking JS, if we send the max_post_vars as a setting, we can stop the submission if we would exceed it.

Or turn it into a progressive enhancement exercise, where the no-JS form can only look at one role at a time, and only once you enable JS can you edit all roles at once.

Hmmm, another terrible idea: I wonder if you could implement the transition idea without JS using radio buttons. If previously checked, make the "checked" input have an empty value, and the "unchecked" input have a value attr that means "remove this permission from the role". You could even use CSS to get close to the current interface.

I'm guessing the radio value always gets submitted, though, even with a blank value.

And there are likely to be a11y concerns.

Reviving this as I just ran into the issue once again (and think it happened to clients > 30 times in the last years).

Maybe a good first shot would be to ensure a viable minimum for max_input_vars for Drupal installations when installing Drupal and in the status report as warning?

Just having a mid-size Drupal project, for example the admin menu administration page (/admin/structure/menu/manage/admin) or the permissions administration typically grows > 1000 input elements very fast.
So before going to be much into details here, adding a hook_requirements() check for max_input_vars being at least 10000, otherwise showing a warning (not an error) to raise attention would be super helpful I think.

While discussing this for a long time, I think many people are hurt by having a too low default hosting provider value (typically '1000'). What do you think? Everything else as follow-up perhaps?

@anybody while it only improves the permissions page, I think #3464530: Improve performance of the user.permissions.js script running in /admin/people/permissions. may have improved things there - do you have a site you could confirm that on?

Hi @catch and thanks for the fast reply! Maybe there's a misunderstanding. My comment is about forms with many inputs not being saved at all, getting people into trouble, not only on the permissions page, but also for example on the menu administration and other pages with many form elements.

It's not about performance, but a risky (mis)configuration in php.ini for Drupal.

PS: Especially for non-technical users and admins that don't know this nasty issue yet, this thing can cost hours and you curse Drupal until you find out that it is because of the php.ini configuration and not Drupal's own fault. But Drupal should help by checking the config for this typical problem!

@anybody that issue at least theoretically also reduced the number of checkboxes that the permissions page submits, so it would be good to see what if any effect it has on a real site affected by this issue.

Agreed the permissions page isn't the only problem here though.

(Ignore) Sorry, wrong issue.

@geek-merlin: Are you sure this landed in the right issue?

It can be database-related, yes, but what we see here can also happen without any database interaction, what ever is done with the form values. (Of course in most cases they will be stored in a database).

This issue is about forms with many inputs, so that the sent request exceeds max_input_vars and values are dropped without a warning.

I'm not sure if this belongs into database system. Guess you maybe worked in several related issues? (That clearly exist)

The main issue is at the http/php level

@tedfordgif we have done all we can at the html level. no more cruft to remove. the shadow inputs are not submitted

@nod_ thanks! Could you maybe give a short comment on my suggestions in #60? That could be a quick win to at least introduce a warning if the limit is low?

Additionally, we could show a warning if a form submitted exactly matches max_post_vars and allow opt-out via settings.php? With a high max_post_vars value this warning would then become improbable.

Once we have a plan, I think it would make sense to split this into sub-issues for the different points to implement. But it first needs a decision which way to follow.

Thanks!

hook_requirement make sense. i would prevent the form from being submitted with a warning if we know it will go over

@nod_ thanks! I don't think we can detect the form is going over, because we simply won't receive values for the values over the limit, but we should elaborate on that!

Let's add a first subtask then to implement a hook_requirements check for a max_post_vars limit < 10 000 to show on installation and in the status report. I'll ask our team.

Here we should proceed checking how to detect a form submit that exceeds the limit.

we can get the max input var value and check with JS at submit for example. It's a pretty dangerous situation so special casing makes sense

When max_input_vars is exceeded an E_WARNING is triggered according to https://www.php.net/manual/en/info.configuration.php#ini.max-input-vars - could we special-case this in the error handler and then do something with it?

Idea from #71 would be great. Using an arbitrary big number doesn't ensure issue is not going to happen.

Other option is adding the check in the Form API layer. During form rendering the number of variables needed can be known, and raise a warning to the watchdog. #71 idea is easier to implement, I guess.

Other option is adding the check in the Form API layer. During form rendering the number of variables needed can be known, and raise a warning to the watchdog. #71 idea is easier to implement, I guess.

Indeed, I think that would be better than JS and server-side. On the other hand, that could also get complicated with compound elements, but definitely solvable.

I hope I'll find the time in the next days to check #71 that would be best! Adding a warning to the status report for very low limit still makes sense to me anyway.

I've been looking into the suggestion by @catch in #71. Here's a patch (no MR because I'm not sure we want to go this direction).

Have not reviewed

Fixes should be in MRs and will also need test coverage

Appears to have pipeline issues.

Also appears to be missing test coverage, tagging for such.

I don't think this can be tested unless we're able to add it to .htaccess within a test, otherwise it would depend on the environment.

If we detect the E_WARNING on submission, it needs to stop the form from processing immediately to prevent data loss.

If a form does get processed with more than max_input_vars values, that can sometimes cause things to be deleted - for menus it causes the menu items to be removed, and for permissions it causes those permissions to be un-granted, and removed from the config files (not always database as discussed above).

> I don't think this can be tested unless we're able to add it to .htaccess within a test, otherwise it would depend on the environment.

For tests, we could use a test module with 10,008 checkboxes where the checkboxes are created by a loop.

For tests, we could use a test module with 10,008 checkboxes where the checkboxes are created by a loop.

We can't guarantee that the test environment has max_input_vars set to 10,000.

We can't guarantee that the test environment has max_input_vars set to 10,000.

Any reason not to read the max_input_vars value of the environment and just create more checkboxes than whatever the value is?

I agree with @azinck, I believe we can create a test.

Yeah max_input_vars +1 is a great idea.

This is ready for review again, complete with a test :)

This can't be easily tested with the current approach. Perhaps the testbot CI can handle things, but running the test locally, (5000 max input var is my php's default) and the test never runs to completion. I tried to set the value lower via an ini_set in the test's setup method. But that still left the test form still using 5000 from max_input_vars.

Instead I manually tested things with the test by artificially setting my max_input_vars to 5. That let things run to completion in a very happy way. Moving this back to NW because we either need to be OK with manual testing (and get rid of the tests) or figure out a way to runtime set the max_input_vars to a lower value.

As I expected, I tracked down over in https://www.php.net/manual/en/ini.core.php#ini.sect.file-uploads that we can't set that value at run-time.

If it works on gitlab, could we check max_input_vars first, and if it's over a certain number, skip the test? That would mean the test doesn't time out locally and give people a hint how they can run it locally if it's likely to fail.

It runs on the testbot, but at a huge cost, 36 minutes: https://git.drupalcode.org/issue/drupal-1565704/-/pipelines/397114/test_...

It runs on the testbot, but at a huge cost, 36 minutes

OMG definitely not, let's remove the test from the MR.

Could the max_input_vars error get hidden by a subsequent error?

That seems possible, but I don't think we know exactly when it will get triggered, so we would need to hard-code support for it in core's error handler probably - maybe that would be OK though.

I resolved the conflicts and made the test manual.

The Needs Review Queue Bot tested this issue. It fails the Drupal core commit checks. Therefore, this issue status is now "Needs work".

This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

We're still using the patch from #45 in a few production sites. The patch doesn't apply anymore against 11.3, this is a reroll.

For us only showing a message or globally increasing the max_input_vars not a solution. We have a few sites with large menu's, which couldn't be saved with the current approach from MR10822.