Prepopulated values are nuked if hidden

Before I can fix this I need to find out how to tell the difference between hidden and non-hidden fields post-submission.

I did a quick implementation of hook_webform_submission_presave with just some watchdogs in, and can confirm that the fields that are being nuked are not in the &$submission object. The issue affects at least one core field (job title) as well as custom fields (in our case a multiselect checkbox of school subjects).

If I fill in the form so that job title is visible, and I put in a value, I see my test value in $submission->data[75][0] (75 being the webform component id of the job title field).

If I then repeat the form and follow a path such that the job title field is not shown, the key 75 does not appear in $submission->data. However, the job title has been erased from the contact.

Dear Coleman,

We've discovered that if a custom field is not in the submission object that it is loaded and then nuked for currently existing contacts in saveCustomData. The comment specifically says "Only save if this is not blank or data already exists and record is known".

foreach ($custom as $k => $v) {
  // Only save if this is not blank or data already exists and record is known
  if ($v !== '' || ($suf > 0 && $known)) {
    $params[$k . '_' . $suf] = $v;
  }
}

However, we've also shown that if a field is hidden by conditional statements then it isn't in the submission object.

It seems in the instance of the job title that it is being overwritten by the fillDataFromSubmission. However the current employer is not being. Again we found a comment:

// Only known contacts are allowed to empty a field
if (($val !== '' && $val !== NULL) || !empty($this->existing_contacts[$c])) {
  $this->data[$ent][$c][$table][$n][$name] = $val;
}

We found that this fillDataFromSubmission was introducing the differences shown in the attached meld screenshot.

We are stuck now due to not being sure how the conditional hiding and showing works, and how this ties into what comprises the submission object.

not sure if this helps at all - but here is an exported webform with civi and conditionals - if used with cid in url then submitting will cause data in the conditional 'hidden' fields to be deleted - but i suspect Coleman is way ahead of needing this.

I've been doing further testing by filling in a contact with absolutely every field filled and box ticked in Civi itself, then running through the webform we're having this problem with, selecting the 'Don't want to answer' option which hides all the other custom fields on that webform.

I found that the only custom fields nuked were two multi-select checkbox sets. No custom text or radiobutton fields were cleared. (Plus the job title, but that may be a separate bug.)

Around line 457 of wf_crm_webform_base.inc, where it says:

if ($v !== '' || ($suf > 0 && $known)) {

If I add some code there to also exclude a string that consists solely of separator characters (ASCII 0x01), our couple of multiselect fields no longer get nuked.

As I found below that loop, unless I do the above, that a print_r of params looked like:

Array ( [entityID] => 131763 [custom_41_0] =>  [custom_45_0] =>  [custom_21_0] => 8123 [custom_33_0] => Don’t_want_to_answer )

and hidden after the [custom_41_0] => and [custom_45_0] => were a couple of separator characters.

And tracing further back, I find in fillDataFromSubmission, the line

$val = $sp . implode($sp, $val) . $sp;

If I replace it with

$val = empty($val) ? '' : $sp . implode($sp, $val) . $sp;

then again the custom fields don't get nuked- because now $val is actually an empty string, not a couple of separators, so it doesn't get past the $val !== '' at the end of fillDataFromSubmission and get added to the $this->data structure.

Ah nice, I like that last patch.
So does that completely solve your issue or is there more left to do?

Progress :)

It looks to me like the problem is entirely solved by this tweak to wf_crm_webform_postprocess::fillDataFromSubmission:

--- a/includes/wf_crm_webform_postprocess.inc
+++ b/includes/wf_crm_webform_postprocess.inc
@@ -1673,7 +1673,7 @@ class wf_crm_webform_postprocess extends wf_crm_webform_base {
     $sp = CRM_Core_DAO::VALUE_SEPARATOR;
     foreach ($this->enabled as $field_key => $fid) {
       $val = $this->submissionValue($fid);
-      if ($val !== FALSE) {
+      if ($val !== NULL && $val !== array(NULL)) {
         list( , $c, $ent, $n, $table, $name) = explode('_', $field_key, 6);
         // Fieldsets and existing contact fields are not strictly CiviCRM fields, so ignore
         if ($name === 'existing' || $name === 'fieldset') {

I think it also removes the necessity of your above suggested patches to the implode code. Can you test and let me know?

I tested this patch against both core and custom fields, and it worked on all of them, ensuring they would be ignored if hidden by a webform conditional rule. But I'd like to hear how your testing goes and see if it solves your problems.

Is your email/phone issue also related to conditional fields? Can you give me steps to reproduce that one?

Job title and custom multiselects seem to work for us with that fix, as do name/email/phone.

That issue was- if you have one webform that collects name/email/phone, and passes cid and checksum to a second webform which adds further information to the contact including custom fields, after that is all done the name/email/phone got deleted. We haven't actually observed it recently, as our multi-stage webforms include and hide the name/email/phone.

Ok I think we can call this issue fixed :)
If you are able to reproduce the other, please do file an issue for it.

I just noticed that too. Filed as #2389991: Custom data blank values cannot be saved.

Re-opening as our testing with a wide range of field types shows that Birth Date - and possibly all date type fields but not tested this - still get nuked.

I can give you access to this form if it helps

Kaylie tested on test form with fields of:
gender
birth date
street
country
phone
email
free text
multi select
radio
yes/no
integer

Had no issues with the data being deleted EXCEPT FOR BIRTH DATE FIELD.

@petednz I couldn't reproduce that with the following test I just set up:

- Create a webform with first name, last name, email & birth date
- Fill in anonymously as Joe Tester, providing birthdate
- Add a conditional where if last name is "Tester" the birth date field is hidden
- Fill it in anonymously as Joe Tester again (so that the default strict rule matches on the same email) - birth date field was hidden
- Verified contact record was updated, but birth date field was not affected.