One of the website I am working on has a very big webform. After enabling draft between pagination I was getting data too long for column `property` in webform_submission_data table. Based on suggestion on one reported issue, I increased column size from 128 to 255 and this problem looks like fixed.

Webform has total 11 pages. After reaching page 5 and going forward to page fix, it shows nginx error. After checking nginx log I got "recv() failed connection reset by peer" error. Tried restarting php-fpm as per some posts but still same.

Does this module support such big forms? Form has around 100 questions paginated in 11 pages.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

bpadaria created an issue. See original summary.

jrockowitz’s picture

Looks like a likert or composite sub-element's key is greater than 128 characters. We need to figure out where in the UI you were able to do this.

jrockowitz’s picture

Status: Active » Needs review
FileSize
6.82 KB

I think the issue is that Likert questions support 255 characters and the property column is limited to 128 characters.

@bpadaria Can you please confirm that your webform has a Likert element with a question whose value is more than 128 characters?

The attached patch limits a likert element's question option value to 128 characters.

Status: Needs review » Needs work

The last submitted patch, 3: 2937156-3.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

jrockowitz’s picture

Status: Needs work » Needs review
FileSize
6.88 KB

  • jrockowitz committed c1fe2c0 on 8.x-5.x
    Issue #2937156 by jrockowitz: error with very big webform submission
    
jrockowitz’s picture

I committed the patch. Please download the latest dev release to review.

bpadaria’s picture

Thanks jrockowitz for quick response.

Looks like patch resolves string too Long error.
But pagination error after getting through page 5 through "next page >>" is still there.

We have 11 pages for multistep form.
Drupal log doesn't give much information except draft saved for ?page=4.
Nginx log says "[error] 32032#32032: *27241 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: xxx.xxx.xxx.xx, server: xxxxx, request: "POST /webform/myform/test?page=6 HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.clientcms.sock:", host: "xxxxxx", referrer: "webform/uhn_research_quality_self_clone/test?page=5"

We have installed APC cache. So opcode caching is still not installed on server.
Will installing opcode caching help for his issue?

Thanks.

jrockowitz’s picture

Status: Needs review » Fixed

The NGINX issue is not related to the Webform. I would not be surprised if you installed your form on http://simplytest.me and it worked without any errors.

I have seen NGINX/PHP FPM have problems with older version of PHP but generally these types of error are related to memory usage.

I strongly recommend using memcache or redis, which will significantly reduce your memory usage and increase site performance.

Also see...

https://www.google.com/search?q=recv%28%29+failed+connection+reset+by+peer
https://www.google.com/search?q=Drupal+recv%28%29+failed+connection+rese...

I am marking this as fixed since the webform related error has been fixed.

bpadaria’s picture

Thanks jrockowitz.

Tried it over simplytest.me and still same.
That might be an issue with form elements.

Think I have to debug it in detail.
Will keep posting my investigation :)

flowerts’s picture

Hi @jrockowitz
I'm getting similar issue trying to get 45 paged webform to be submitted.
Tried the patch, but couldn't get it to work. Is it possible the issue to be with the number of the pages?

jrockowitz’s picture

Status: Fixed » Needs work

I think the issue is related to the amount memory being used by a large webform.

The patch does not fix the NGINX issue.

I am going to generate and attach a massive multiple page test webform that we can use for testing.

jrockowitz’s picture

Status: Needs work » Needs review

Attached is a gigantic multistep form with 300 elements. It works fine via Apache.

Can you please test this form and if it throws any errors?

If it works it possible that either a certain element or the number of elements on a page is causing issues.

jrockowitz’s picture

Here is the form.

flowerts’s picture

FileSize
274.35 KB

Hi @jrockowitz,
I've tested your example, btw the elements are not visible when I tested the submission, and indeed it works.
My example (a real one) attached, fails every time. I do suppose that some of the text fields are longer than 128 (or even 255). Not sure how we can inspect them (except manually). We have created it as a yml file that I have verified
I've made few tests by deleting some of the texts and removing sections, but still the result is the same.

jrockowitz’s picture

I think we need to increase the 'property' column length from 128 to 255 since 255 has no performance impact.

https://stackoverflow.com/questions/1262174/mysql-why-use-varchar20-inst...

jrockowitz’s picture

@flowerts Your form definitely has likert question values that are greater than 128 characters.

The attached patch does increase the character limit for likert question values to 255 characters. I am still not sure I want to make this change because I think people should generally store simpler machine names for likert question values and not the full question.

Status: Needs review » Needs work

The last submitted patch, 17: 2937156-17.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

jrockowitz’s picture

The errors indicate that we can't change the 'property' length from 128 to 255 characters because it will break the 'webform_submission_data' DB index.

@flowerts You need to go thru all your #questions and shorted the values.

For example the below #questions. The second question is 155 characters.

'#questions':
        'assess levels of access to adequate quantity and quality of food': 'Assess levels of access to adequate quantity and quality of food'
        ' Provide food preparation utilisation instructions (including instructions on how to modify food for easier consumption by persons with difficulties eating)': ' Provide food preparation utilisation instructions (including instructions on how to modify food for easier consumption by persons with difficulties eating)'

...should be changed to something like...

'#questions':
        'access_food': 'Assess levels of access to adequate quantity and quality of food'
        ' food_preparation': ' Provide food preparation utilisation instructions (including instructions on how to modify food for easier consumption by persons with difficulties eating)'

BTW, using shorter question keys/values will make your CSV export much cleaner.

flowerts’s picture

Hi @jrockowitz
First thanks :)
I do agree that making the likert question values greater than 128 characters is in general not the correct way.
I think the best approach will be to revise the form.

Even though as it will be much faster I'll test the patch and will change the 'property' to 255.

jrockowitz’s picture

@flowerts Don't use the patch it will mess up your database. The below errors logged by the broken test should not be ignored.

Syntax error or access violation: 1071 Specified key was too long; max key length is 767 bytes: CREATE TABLE {webform_submission_data}...

jrockowitz’s picture

flowerts’s picture

Hi @jrockowitz

Saw your comment a bit late ;).
Tried to play around with the db and confirmed that it won't work this way.

So, after clean install and new db (without any changes) and rewriting all likert questions keys by limiting them to max 128 (110 to be sure) webform submission works great.

Thanks again.

jrockowitz’s picture

Status: Needs work » Fixed

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

bpadaria’s picture

Thank you both of you for investigation.

Rewriting likert question keys resolved my issue as well.