Currently $form_state['storage'] is intended to be set in a #submit callback. So when creating a multistep form you can put your stuff into $form_state during the form build, then populate the storage during #submit.

This works fine, but when you start to add some #ahah stuff to the form, #cache will be set and the form will be cached when it's loaded the first time. As an affect the #submit callback is called
* without an initialized $form_state, as the form got cached through #cache
* without a form storage, as it wasn't set yet
So there is no way to make use of the $form_state apart from setting $form_state['storage'] during the form build. However currently this isn't intended and doesn't work well, because when you do it
* the form is immediately rebuilt even when it's loaded the very first time
* $form is written two times into the cache, once due to the storage and once due to #cache

To solve this I'd propose support setting the form storage during form build and fix the two mentioned issues. If there is an agreement that this is the way to solve it, I'd do a patch.

Note: This issue applies to 6.x too.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

fago’s picture

Title: form storage doesn't work well on cached forms » fix various problems when using form storage
FileSize
1.18 KB

Furthermore there is another related issue: When the storage is already set and there is a validation error, the form is rebuilt nevertheless! So when one uses a multistep form and saves stuff into the storage with the submit callback, then in this case the submit callbacks are skipped due to the validation error and the form is rebuilt without an updated storage, so any changes are lost.

So I've done a patch which fixes these problems:
* It makes sure the form storage lets the form rebuild only when the form has been submitted and there are no validation errors.
* It also saves the form storage to the cache table when #cache is activate by #ahah properties or something else.

These simple fixes now make it possible to
* put your thingie to edit into the form storage during the initial form build
* to update your thingie after each step out of a #submit callback
- which I'd consider as typical usage.

Furthermore now it's no problem to use the form storage together with #cache.

Note that I've tested the changes with a complex case - the rules admin UI for d6. It's a multistep form using storage which might have some #ahah properties in there - with this patch the form runs fine!

It applies to d7 and d6.

fago’s picture

I think it would make sense to write a test for a multistep form using storage and #cache in d7 - however I'm not sure how to do it. Is there a way to add a testcase-module containing such a form just for testing?

Damien Tournoud’s picture

I just created a book page about our mock modules. Please see http://drupal.org/node/302577

fago’s picture

Status: Active » Needs review
FileSize
5.91 KB

ah thanks!

ok, so I've written some tests using a mock module. The module provides a multistep form using storage and optionally #cache. Then the tests ensure that
* the storage properly stores the values
* the form builders aren't called too often
* the storage keeps working, when #cache is activated
* the form values are kept when an validation error occurs.

attached is a patch containing *only* the tests. Make sure to have the patch from #1 included or some tests will fail.

Robin Monks’s picture

Status: Needs review » Reviewed & tested by the community

Test patch and issue patch apply file, and the FormsAPI simpletest runs correctly (88 passes, 0 fails, 0 exceptions).

Robin

moshe weitzman’s picture

Status: Reviewed & tested by the community » Needs review

@Robin - please slow down. Significant FAPI changes need review from Eaton or chx.

chx’s picture

The second hunk is a bugfix -- you can have storage populated from a previous step and then a validate error will get the form rebuilt still. Whether we want the builder to be able to populate storage I am less sure. I am open to what Eaton says :)

fago’s picture

thanks chx.

>Whether we want the builder to be able to populate storage I am less sure. I am open to what Eaton says :)
I'd call this a bug making $form_state unusable, as it requires one to use $form for storing values instead when caching by #cache should work too.

Anyway when putting stuff into the storage during build is not supported, the first hunk makes no difference either. It's unnecessary to pass NULL to form_set_cache when there is no storage.

dropcube’s picture

I had posted a similar patch in other issue #291939: Multistep forms should not be rebuilt if there are validation errors. because I did not found this one.

Making sure the form storage lets the form rebuild only when there are not validation errors fixes the bugs reported and work Ok form me.

Anyway when putting stuff into the storage during build is not supported

Could you explain why this is not supported. I have some use case where is required to put stuff into the storage during the form builder function. Is there an alternative way to do this?

fago’s picture

>Anyway when putting stuff into the storage during build is not supported, the first hunk makes no difference either. It's unnecessary to pass NULL to form_set_cache when there is no storage.

Anyway, I think putting stuff into storage during build *should* be supported. Currently it's not, but nobody knows. It's not documented and as it's useful people do - yes, I have seen several cases where people do so. Then some days ago I held a copy of the latest pro drupal development book in my hands - even the multistep form example in there sets $the storage in $form_state during the form build!

@dropcube: What happens when you populate the storage during form build currently?
* the form is immediately rebuilt even when it's loaded the very first time. So the builder runs 2x when the form is shown the first time.
* When #cache gets set, $form is written two times into the cache, once due to the storage and once due to #cache.

To summarize this simple patch
1. It fixes validation for multistep forms (prevent the rebuild!)
2. It fixes issues when populating form storage during form build

Furthermore thanks to the fix of point 2, it makes it possible to use the form storage in a clean way even when #cache gets activated. See post #1.

Imo the patch from #1 is an important patch which should get into d7 and d6! For d7 there is also a test (#4).

Wim Leers’s picture

Subscribing.

sun’s picture

Status: Needs review » Needs work

Can we add some inline comments explaining the how's and why's of the enhanced condition?

fago’s picture

Status: Needs work » Needs review
FileSize
1.9 KB

ok, I've updated the patch.

Furthermore I've improved it a bit: When $form_state['rebuild'] is explicitly set, now it's too checked whether the form has been submitted and validation errors occured. It doesn't make sense to allow people skipping this check, so I've improved the patch to check it for both cases. As an affect you can set $form_state['rebuild'] in your form builder now too.

Updated and improved patch from #1 attached, applies to d6 & d7. The test for d7 (#4) still works fine once the patch is applied.

dropcube’s picture

The patch looks good for me. I tested and it prevents that forms that contains validation errors get rebuild during a multi-step process. The Form API docs should be updated to clearly explains the workflow.

Status: Needs review » Needs work

The last submitted patch failed testing.

lilou’s picture

Status: Needs work » Needs review
sun’s picture

Re-test.

rb7’s picture

Subscribing.

bilgehan’s picture

Subscribing.

Damien Tournoud’s picture

Status: Needs review » Needs work

Those two fixes looks reasonable at first sight. The test in #4 should be rolled in the patch (so that it can be tested by the test bot). There is some code style convention issues there (please see the Test writers guidelines for details). Plus I would like to be sure that the test cover both hunks (ie. it should break when any of them is not applied).

doq’s picture

Subscribing.

fago’s picture

Status: Needs work » Needs review
FileSize
7.19 KB

so, next try.

* I've updated the patch, as the form api received some improvements in the meanwhile.
* I took a look at the test writer guidelines (thanks damien) and overhauled the test. The test already covers both hunks.
* Attached is the updated patch for d7, containing the both, the changes and test.

Dries’s picture

Some feedback:

- We use dashes instead of underscores for menu/url paths. form_test/form_storage should be form-test/form-storage.

- The test case looks OK but took me some time figuring out. Some additional documentation would have helped. I wonder if the test could be simplified.

- In testFormCached() why don't we pass array('query' => 'cache=1') to both calls to drupalPost()? It seems like that could make for a better test.

Let's get this patch in!

fago’s picture

FileSize
7.95 KB

thanks.

The exiting menu items of the already existing form-API test use the path "form_test" - so I stayed with form_test for consistency, but fixed it to use

form_test/form-storage

. So probably form_test should be fixed there to be form-test for all those menu items, but that's another issue..

>In testFormCached() why don't we pass array('query' => 'cache=1') to both calls to drupalPost()? It seems like that could make for a better test.

It doesn't matter the second time, as the form is already cached and won't be constructed again. But yes, it's more logical to add it to the second call too, so I've done so.

>Some additional documentation would have helped. I wonder if the test could be simplified.

Unfortunately I can't think of a way simplifying the test. Multistep forms with form storage aren't that simple ;) However I added some more comments to the form and the tests. I also improved the language to talk about "form constructions" not "form builds", as form builds could be easily confused with the build of form_builder(), which is happening for cached forms too and isn't counted.

fago’s picture

FileSize
7.98 KB

I forgot to rename the counting "builds" variable too, fixed that.

Dries’s picture

Status: Needs review » Fixed

Code looks good, fixes a real problem, comes with tests and documentation. Excellent work. Committed to CVS HEAD. Thanks fago.

fago’s picture

Version: 7.x-dev » 6.x-dev
Status: Fixed » Needs review

yeah, thanks!

So let's backport this to 6.x? The patch from #13 still applies fine to d6 and it's the same fix as got committed in d7.

Alan D.’s picture

Thanks fago, the patch from #13 is working nicely on my local Drupal 6.12 install.

Is this still on the cards for a back port?

John Morahan’s picture

Status: Needs review » Reviewed & tested by the community

#13 still applies and still works for me on 6.x-dev.

Gábor Hojtsy’s picture

Status: Reviewed & tested by the community » Fixed

Committed to Drupal 6 too, thanks.

dww’s picture

Not sure if it's because project_issue's comment handling is doing something unholy and relying on the bug that this fixed, but deploying this change on d.o broke issue comment previews. See #579900: Previewing comments is broken by FAPI change in Drupal core 6.14. I don't have time to dig much deeper (I've got a massive deadline this weekend), other than narrowing it down to this API change as the source of the problem. I haven't even read this issue yet or studied the patch. But, if anyone familiar with this change is willing/able to take a look at issue comment previews, that'd be great... Thanks!

mattyoung’s picture

subscribe

chx’s picture

Title: fix various problems when using form storage » button broken due to fix various problems when using form storage
Status: Fixed » Active

!empty($form_state['submitted']) && !form_get_errors() <= why is form_state['submitted'] there? the !form_get_errors yeah but the submitted check broke buttons as dww shows.

dww’s picture

To clarify, by "buttons" we mean FAPI form elements of #type 'button', like core's "Preview" button on comment forms...

sun’s picture

Version: 6.x-dev » 7.x-dev
Priority: Normal » Critical
Status: Active » Needs review
FileSize
812 bytes

yeah, looks wrong.

However, comment previews are working in HEAD, so this doesn't seem to be covered by tests.

dww’s picture

comment previews are working in D6, too. Core comment previews do *not* require rebuilding the form (and in fact, you break the preview if you rebuild the form). what's broken are forms that use #type 'button' to trigger something that requires the form to rebuild. But right, that might not be tested in D7, either...

sun’s picture

Like that? Not sure.

sun’s picture

#type = button.

dww’s picture

Status: Needs review » Needs work

In terms of the new test in #38 -- no, I think we really want to test both #type == 'submit' and 'button', not one or the other. So long as both exist in FAPI (and API bug IMHO, but whatever), we should test both properly for this case.

dww’s picture

p.s. I know you love to clean up every file you touch, but all the whitespace and code style fixes make this patch harder to read. ;) Can't we do that stuff in separate issues? *shrug* Either way, thanks for fixing them. ;)

grendzy’s picture

This change also broke multistep forms in Drupal 6.14. If '#button' is used, $form_state['values'] is empty.

boombatower’s picture

Frank Steiner’s picture

This is really serious for 6.14! E.g. the biblio module stopped working after upgrading from 6.13 to 6.14. The patch in http://drupal.org/node/591696 makes it working again, so I can get around this, but I guess there should be an official patch for 6.14.

Gábor Hojtsy’s picture

Tagging

bjcool’s picture

+

mrfelton’s picture

Which patch should I apply to 6.14 to fix this?

Frank Steiner’s picture

The one from #43 works for our 6.14.

mikeytown2’s picture

Does this fix the bug where button D was pressed but the function for button A was called instead? For 6.14

vesapalmu’s picture

This multistep form bug also effects a custom module that we have for a customer site. Looking for ways of working around it currently.

sun’s picture

Assigned: Unassigned » sun
Status: Needs work » Needs review
FileSize
5.92 KB

Status: Needs review » Needs work

The last submitted patch failed testing.

sun’s picture

Status: Needs work » Needs review
FileSize
6.47 KB

Fixed the tests.

fago’s picture

Status: Needs review » Needs work

ouch, um, my fault :( I'm sry for that.

@patch:
The fix looks good. However the number of form constructions should be 1 less for the version with cache activated. I think you are testing it both times without having #cache activated as you are missing the query parameter for the GET call.

sun’s picture

Status: Needs work » Needs review
+++ modules/simpletest/tests/form_test.module	17 Nov 2009 18:17:55 -0000
@@ -297,8 +297,10 @@ function form_storage_test_form($form, &
     $form_state['storage'] += array('step' => 1);
   }
+  // Output how often this form has been constructed.
+  drupal_set_message('Form constructions: ' . $_SESSION['constructions']);
 
-  // Count how often the form is constructed
+  // Count how often the form is constructed.
   $_SESSION['constructions']++;

uhm - are you sure?

It counts the number of constructions in a session variable - not the number of submitted steps or cached rebuilds or whatever.

This review is powered by Dreditor.

fago’s picture

Status: Needs review » Needs work

Right, it counts how often the form constructor is called. But this depends whether caching is on or not.

Workflow without $form_state['cache']:
Get page -> Form construction #1
Post step1 ->
* Form is constructed (#2) and the form submitted.
* Form is rebuilt -> Construction (#3) + Form gets cached.
Post step2 ->
* Form is loaded from cache and the form submitted.
* Form is rebuilt -> Construction (#4)

Workflow with $form_state['cache'] (e.g. due to #ajax):
Get page -> Form construction #1 + Form gets cached.
Post step1 ->
* Form is loaded from cache and the form submitted.
* Form is rebuilt -> Construction (#2) + Form gets cached.
Post step2 ->
* Form is loaded from cache and the form submitted.
* Form is rebuilt -> Construction (#3)

Thus the we should have one construction less with caching. As said I think you are missing array('query' => array('cache' => 1)) for your initial GET request when trying to test with cache. Thus it's not enabled. I'd fix it myself however this testcase doesn't work on my local test environment - somehow the session is lost on every page load when running the test :(

Status: Needs work » Needs review

rfay requested that failed test be re-tested.

rfay’s picture

Status: Needs review » Needs work

The patch in #52 will have to be re-rolled due to #634440: Remove auto-rebuilding magic for $form_state['storage'] landing. I was about to do it, but since this is the most sensitive area of all our current undertakings, I thought I'd leave it to the principals on this issue.

sun’s picture

Issue tags: +D7 Form API challenge

.

fago’s picture

also we can't just remove the check for 'submitted' as else the form might rebuild even when it's loaded initially. I think checking "$form_state['process_input']" instead should do it as we only want to rebuild when the input has been processed beforehand.

joep.hendrix’s picture

The previous step button will not work any longer on multistep forms (D6.14).
Please rollback or point me to a an example how to build a multistep form with previous step buttons.

Cheers!

rfay’s picture

@JoepH: This is a D7 issue, and nothing has been committed yet. I'm not familiar with a 6.14 regression on this, but you may find one. But this isn't the place for the discussion.

When you do pursue this, you'll have to give a very specific example of your problem or nobody will be able to help you out.

Alan D.’s picture

@rfay Just a reminder, this is also a D6 issue to as of #30 when the changes were committed into the D6 branch.

This change is preventing a couple of our sites from being upgrading to 6.14, although the change itself is required by two of our other D6 sites.

rfay’s picture

I apologize for the misdirection and error. I am so used to things going into 7 and then working their way into 6.

aaron’s picture

joep.hendrix’s picture

Would it be a good idea to create a new issue for D6?
Please read #61.

Thanks

grendzy’s picture

JoepH: The D6 fix will be in this issue, indicated by the issue tag drupal-6.15-blocker.

dww’s picture

@rfay re: #61: Uhh, please read this issue, starting at comment #31. This is absolutely a D6 regression in 6.14, and we definitely need to fix it. In fact, the only reason this issue has version "7.x-dev" is the policy of fixing in HEAD first and then backporting.

Bilmar’s picture

subscribing

fago’s picture

FileSize
11.24 KB

So let's fix it in d7 now! I've setup a working dev-environment in a virtual machine, so I gave this a try.

@#42: The form shouldn't rebuild if there is a validation error, this was broken before this patch landed initially.

@proposed fix:
The fix was broken as 1) it triggered form rebuilds even when the form was shown the first time, 2) it tried to invoke a #submit handler for a button of type 'button', however this is the only distinction of 'button' buttons to 'submit' buttons - they don't submit the form. 1) was the cause for the wrong construction counts as I noted in #55.

So to fix I changed it to check for "$form_state['process_input']" instead 'submitted', that way the form doesn't build twice for the first page load, but also rebuilds for buttons of type button.

I also overhauled the test case to test this, however this wasn't so easy as in d7 it's not so easy to trigger form rebuilding for buttons of type button because
* just setting the storage doesn't trigger rebuilding any more
* enabling $form_state['rebuild'] during the constructor doesn't work right yet once caching is turned on, see #648170: Form constructors cannot enable form caching or form rebuilding
* submit callbacks aren't invoked for buttons of type button.
So I used a validation handler to set $form_state['rebuild']. For that I added a "Reset" button to the test, which just rebuilds the form with the previous values from storage.
However if rebuild isn't set the form would just re-load - so the same form would be presented again. This might be useful just to run the validation handlers. So I tested this case by implementing a simple "Reload" button.

Thus the test covers now
* rebuilding through a submit button
* rebuilding with a button button
* not rebuilding with a button button

fago’s picture

Status: Needs work » Needs review
sun’s picture

Status: Needs review » Reviewed & tested by the community
+++ modules/simpletest/tests/form.test
@@ -589,13 +621,10 @@ class FormsFormStorageTestCase extends DrupalWebTestCase {
     // At this point, the form storage should contain updated values, but we do
     // not see them, because the form has not been rebuilt yet due to the
-    // validation error. Post again with an arbitrary 'title' (which is only
-    // updated in form storage in step 1) and verify that the rebuilt form
-    // contains the values of the updated form storage.
-    $edit = array('title' => 'foo', 'value' => '');
-    $this->drupalPost(NULL, $edit, 'Save');
-    $this->assertFieldByName('title', 'title_changed', t('The altered form storage value was updated in cache and taken over.'));
-    $this->assertText('Title: title_changed', t('The form storage has stored the values.'));
+    // validation error. Post again and verify that the rebuilt form contains
+    // the values of the updated form storage.
+    $this->drupalPost(NULL, array('title' => 'foo', 'value' => 'bar'), 'Save');
+    $this->assertText("The thing has been changed.", 'The altered form storage value was updated in cache and taken over.');

I really hope that this is 100% the same result, 'cause I need this to function properly in D7. It looks like it is, so I'm going to trust you. ;)

This review is powered by Dreditor.

fago’s picture

Yep, it's the same.

I had to adapt it as I changed the test-form to not 'reload' the form once it's finished, because else we would get another "form construction message" which might generate a false positive for the other tests. Also the title output reflects now the actual form value and not the storage, anyway a dedicated output ("The thing has been changed.") is easier to grasp.

Gábor Hojtsy’s picture

Given this turned out to be more complex then to be possible for D6.15, it did not actually block that release. I'd very welcome a D6 fix for this soon, right after D7.

Dries’s picture

Version: 7.x-dev » 6.x-dev
Priority: Critical » Normal
Status: Reviewed & tested by the community » Fixed

Committed to CVS HEAD. Thanks! Changing version to D6.

Gábor Hojtsy’s picture

Priority: Normal » Critical
Status: Fixed » Patch (to be ported)

It is actually a critical issue for Drupal 6 as well (requires followup to original patch), not at all fixed there as far as I understand.

jbrown’s picture

Subscribing

ssm2017 Binder’s picture

subscribing

bjcool’s picture

When can we have a fix for 6.x? Since now months we have to life with a patched core...

alex_b’s picture

Also OpenID Provider relied on the pre 6.14 behavior. What's interesting is that a coincidentally set $_POST global concealed the bug. When $_POST was accidentally set (from a different form submission), the new form was eventually rebuilt and $form_state - on which the OpenID Provider form in question relied - was cached.

Detailed writeup here: #621956-7: Redirecting fails when not logged in on the provider site

Haven't had the time to look at #69.

fago’s picture

Status: Patch (to be ported) » Needs review
FileSize
1.66 KB

Here is a backport of the d7 fix (#69). As for d7, this re-enables form rebuilds for buttons of #type 'button'.

sun’s picture

Status: Needs review » Reviewed & tested by the community

Looks good - and looks like an API change at first sight, but we'd have to duplicate a very long + awkward condition otherwise.

fago’s picture

Yep, it sets the 'process_input' flag known from d7, so we can avoid this long + awkward condition. However a new flag won't break anything, so it should be fine that way.

joep.hendrix’s picture

#80 still breaks a multi-step form that has been working fine on 6.14 6.13.
I will investigate further.

edit:
on 6.14 the previous step functionality is not working when required fields are not filled in.

for 6.13 we created an ugly workaround by clearing the messages in the session. From 6.14 this not enough anymore, we will also need to set the $form_state['submitted'] = true and clear the messages.
see http://drupal.org/node/472566

off topic: is there a clean way of implementing previous step functionality that bypasses the internal required fields validation?

profix898’s picture

subscribing

fago’s picture

@#83: Well clearing the messages shouldn't be enough, when it was, then because you triggered the "form rebuilds even when there are validation errors" bug fixed by this issue.

Summit’s picture

Subscribing,
greetings, Martijn

sun’s picture

Version: 6.x-dev » 6.15
Issue tags: +drupal-6.16-blocker

When comment previews are configured to be required, no one is currently able to submit a comment in D6. This patch also fixes that bug.

pembeci’s picture

Subscribing.

Gábor Hojtsy’s picture

Status: Reviewed & tested by the community » Fixed

Thanks, committed to Drupal 6.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

Damien Tournoud’s picture

Version: 6.15 » 7.x-dev
Priority: Critical » Normal
Status: Closed (fixed) » Needs work

Reopening. There is a discrepancy between the code and the intention expressed in the comment:

  // If $form_state['rebuild'] has been set and input has been processed, we
  // know that we're in a multi-part process of some sort and the form's
  // workflow is not complete. We need to construct a fresh copy of the form,
  // passing in the latest $form_state in addition to any other variables passed
  // into drupal_get_form().
  if ($form_state['rebuild'] && $form_state['process_input'] && !form_get_errors()) {
    $form = drupal_rebuild_form($form_id, $form_state);
  }

The intent here is to rebuild only if the form has been processed and passed validation. But !form_get_errors() means something set a form error. Especially, it means that you cannot set a non-rebuild-preventing error in a submit handler, just before rebuilding the form. I suggest we store the information "we executed form_execute_handlers('submit', $form, $form_state);" somewhere in the form state, and use that here.

sun’s picture

it means that you cannot set a non-rebuild-preventing error in a submit handler

Can you elaborate on that? It doesn't make sense to me - form errors need to be set during form validation, not in submit handlers. We are only allowed to rebuild after successful submit. Any validation errors need to be handled already.

Damien Tournoud’s picture

What if I want to display an error message, without preventing the form from being submitted/rebuilt?

fago’s picture

>What if I want to display an error message, without preventing the form from being submitted/rebuilt?
FAPI currently doesn't support that. But you could use a #pre_render callback to set the error - that should work.

sun’s picture

As of now, we don't have a concept for pseudo-error, warnings, and recommendation messages. That's a very dusty area, as we've learned from #limit_validation_errors and all the related trouble with entities built off from in-edit data in $form_state. I'm not sure what the use-case at hand is, but you can always invoke drupal_set_message('Message', 'error') manually -- in the end, for a pseudo error message, it would be the same as calling form_set_error(), but just without preventing the form from successfully validating and submitting.

Damien Tournoud’s picture

Version: 7.x-dev » 6.x-dev
Status: Needs work » Closed (fixed)