Problem/Motivation

Some third-party integrations support the processing of data via attachments to an email inbox. For example, SAP ERMS has built-in parsing of XML email attachments in the below format.

<?xml version="1.0" encoding="UTF-8"?>
<asx:abap xmlns:asx="http://www.sap.com/abapxml" version="1.0">
   <asx:values>
      <VERSION>1.0</VERSION>
      <SENDER>emailaddress</SENDER>
      <WEBFORM_ID>ContactUsForm</WEBFORM_ID>
      <SOURCE>
            <NAME>Lastname</NAME>
            <VALUE>Lastname Value</VALUE>
         </o2PARAVALU>
         <o2PARAVALU>
            <NAME>Firstname</NAME>
            <VALUE>Firstname value</VALUE>
         </o2PARAVALU>
         <o2PARAVALU>
            <NAME>Emailaddress</NAME>
            <VALUE>Emailaddress value</VALUE>
         </o2PARAVALU>
      </SOURCE>
   </asx:values>
</asx:abap>

Proposed resolution

Add an 'Attachment' element which allows site builders to dynamically create files which can be attached to emails and downloaded via the UI.

Remaining tasks

  • Refactor WebformManagedFileBase
  • Create WebformAttachment form element
  • Create WebformAttachment webform element
  • Add a route for generating and returning a dynamic file. /webform/{webform}/submissions/{webform_submission}/attachment/{element_key}
  • Update EmailWebformHandler to handle the WebformAttachment element
  • Write tests
  • Create change record

Notes

  • Downloading of generated files is only available to submissions stored in the database.
  • Webform and element access controls will be applied to download route.
  • Submission purging should be used for temp submission storage and file generation.

WebformAttachment element properties

  • #name (string)
  • #content (long text)
  • #twig (boolean)

Display formats

  • Link
  • URL
  • Name

User interface changes

New WebformAttachment element

API changes

EmailWebformHandler needs to be updated.

Data model changes

N/A

Comments

jrockowitz created an issue. See original summary.

jrockowitz’s picture

Issue summary: View changes
jrockowitz’s picture

Issue summary: View changes
jrockowitz’s picture

Issue summary: View changes

  • jrockowitz committed 3d4df87 on 3013498-attachement
    Issue #3013498: Allow custom files to be attached to emails and...
  • jrockowitz committed 43a7e5b on 3013498-attachement
    Issue #3013498: Allow custom files to be attached to emails and...
  • jrockowitz committed 7723167 on 3013498-attachement
    Issue #3013498: Allow custom files to be attached to emails and...

  • jrockowitz committed 68f4adb on 3013498-attachement
    Issue #3013498: Allow custom files to be attached to emails and...
  • jrockowitz committed d0ebfef on 3013498-attachement
    Issue #3013498: Allow custom files to be attached to emails and...

  • jrockowitz committed 42578ea on 3013498-attachement
    Issue #3013498: Allow custom files to be attached to emails and...
  • jrockowitz committed ff55319 on 3013498-attachement
    Issue #3013498: Allow custom files to be attached to emails and...

  • jrockowitz committed 7f630c1 on 3013498-attachement
    Issue #3013498: Allow custom files to be attached to emails and...

  • jrockowitz committed 2d273af on 3013498-attachement
    Issue #3013498: Allow custom files to be attached to emails and...
  • jrockowitz committed 43b0a89 on 3013498-attachement
    Issue #3013498: Allow custom files to be attached to emails and...
  • jrockowitz committed 9daa38f on 3013498-attachement
    Issue #3013498: Allow custom files to be attached to emails and...
jrockowitz’s picture

Status: Active » Needs review
StatusFileSize
new78.72 KB

The attached patch is really not ready for review because it has no test coverage. I just want to see if it is going to cause any regressions.

Status: Needs review » Needs work

The last submitted patch, 10: 3013498-10.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

  • jrockowitz committed 382d7fc on 3013498-attachement
    Issue #3013498: Allow custom files to be attached to emails and...
jrockowitz’s picture

Status: Needs work » Needs review
StatusFileSize
new82.36 KB

  • jrockowitz committed 030cf9f on 3013498-attachement
    Issue #3013498: Allow custom files to be attached to emails and...

  • jrockowitz committed 80da062 on 3013498-attachement
    Issue #3013498: Allow custom files to be attached to emails and...

  • jrockowitz committed 8b87b62 on 3013498-attachement
    Issue #3013498: Allow custom files to be attached to emails and...
jrockowitz’s picture

StatusFileSize
new103.81 KB
jrockowitz’s picture

  • jrockowitz committed 5a92883 on 3013498-attachement
    Issue #3013498: Allow custom files to be attached to emails and...

  • jrockowitz committed 46818a4 on 3013498-attachement
    Issue #3013498: Allow custom files to be attached to emails and...
  • jrockowitz committed 9345f78 on 3013498-attachement
    Issue #3013498: Allow custom files to be attached to emails and...
jrockowitz’s picture

StatusFileSize
new99.48 KB

This patch refactors the code to remove the unneeded WebformAttachmentManager.

jrockowitz’s picture

StatusFileSize
new6.16 KB

The attached webform is a working demo of the SAP ERMS XML email attachments.

jrockowitz’s picture

To make it easier for themers to alter the generated attachments we should add some Twig template wrappers.

The Twig templates would be just for Twig and Token attachments.

webform-attachment-token.html.twig
webform-attachment-twig.html.twig

These templates would also support some basic template suggestions.

webform-attachment-twig--WEBFORM_ID.html.twig
webform-attachment-twig--WEBFORM_ID--ELEMENT_KEY.html.twig

  • jrockowitz committed 5de4a34 on 3013498-attachement
    Issue #3013498 by jrockowitz: Allow custom files to be attached to...
jrockowitz’s picture

StatusFileSize
new102.18 KB
new7.02 KB

Considering how powerful and flexible Twig is, I don't think we need to add template wrappers. Attached is the email example for which now include the below Twig.

<?xml version="1.0" encoding="UTF-8"?>
<asx:abap xmlns:asx="http://www.sap.com/abapxml" version="1.0">
   <asx:values>
      <VERSION>1.0</VERSION>
      <SENDER>{{ webform_id }}</SENDER>
      <WEBFORM_ID>{{ data.email }}</WEBFORM_ID>
      <SOURCE>
        {% for key, value in data %}
         <o2PARAVALU>
            <NAME>{{ (elements_flattened[key]['#xml_name']) ? elements_flattened[key]['#xml_name'] : key }}</NAME>
            <VALUE>{{ value }}</VALUE>
         </o2PARAVALU>
        {% endfor %}      
      </SOURCE>
   </asx:values>
</asx:abap>
jrockowitz’s picture

StatusFileSize
new109.95 KB

The attached patch allows the file name to be sanitized.

  • jrockowitz committed 4e387f3 on 3013498-attachement
    Issue #3013498 by jrockowitz: Allow custom files to be attached to...

  • jrockowitz committed 65443b3 on 3013498-attachement
    Issue #3013498 by jrockowitz: Allow custom files to be attached to...

  • jrockowitz committed cb380f2 on 3013498-attachement
    Issue #3013498 by jrockowitz: Allow custom files to be attached to...
jrockowitz’s picture

StatusFileSize
new111.53 KB

Status: Needs review » Needs work

The last submitted patch, 30: 3013498-30.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

ksavoie’s picture

This is getting really close the file naming looks to be working.
However I am occasionally getting the following error on submission. Not exactly sure the conditions that cause it.

The website encountered an unexpected error. Please try again later.</br></br><em class="placeholder">Drupal\Core\Entity\EntityStorageException</em>: Parameter &quot;filename&quot; for route &quot;entity.webform.user.submission.attachment&quot; must match &quot;[^/]++&quot; (&quot;20190110020130_Rental Property Owner/Manager_Authorization to  Allow Management Company to act on behalf of owner.xml&quot; given) to generate a corresponding URL. in <em class="placeholder">Drupal\Core\Entity\Sql\SqlContentEntityStorage-&gt;save()</em> (line <em class="placeholder">783</em> of <em class="placeholder">core/lib/Drupal/Core/Entity/Sql/SqlContentEntityStorage.php</em>). <pre class="backtrace">Drupal\Core\Routing\UrlGenerator-&gt;getInternalPathFromRoute(&#039;entity.webform.user.submission.attachment&#039;, Object, Array, Array) (Line: 293)
Drupal\Core\Routing\UrlGenerator-&gt;generateFromRoute(&#039;entity.webform.user.submission.attachment&#039;, Array, Array, 1) (Line: 105)
Drupal\Core\Render\MetadataBubblingUrlGenerator-&gt;generateFromRoute(&#039;entity.webform.user.submission.attachment&#039;, Array, Array, 1) (Line: 753)
Drupal\Core\Url-&gt;toString(1) (Line: 167)
Drupal\Core\Utility\LinkGenerator-&gt;generate(&#039;20190110020130_Rental Property Owner/Manager_Authorization to  Allow Management Company to act on behalf of owner.xml&#039;, Object) (Line: 94)
Drupal\Core\Render\Element\Link::preRenderLink(Array)
call_user_func(Array, Array) (Line: 378)
Drupal\Core\Render\Renderer-&gt;doRender(Array, ) (Line: 195)
Drupal\Core\Render\Renderer-&gt;render(Array) (Line: 490)
Drupal\Core\Template\TwigExtension-&gt;escapeFilter(Object, Array, &#039;html&#039;, NULL, 1) (Line: 51)
__TwigTemplate_29b41d9fed80b44f5f2a03f764defe940d5d4da6172995af1d26ead3a6607a06-&gt;doDisplay(Array, Array) (Line: 432)
Twig_Template-&gt;displayWithErrorHandling(Array, Array) (Line: 403)
Twig_Template-&gt;display(Array) (Line: 411)
Twig_Template-&gt;render(Array) (Line: 64)
twig_render_template(&#039;modules/webform/templates/webform-element-base-html.html.twig&#039;, Array) (Line: 384)
Drupal\Core\Theme\ThemeManager-&gt;render(&#039;webform_element_base_html&#039;, Array) (Line: 437)
Drupal\Core\Render\Renderer-&gt;doRender(Array) (Line: 450)
Drupal\Core\Render\Renderer-&gt;doRender(Array, 1) (Line: 195)
Drupal\Core\Render\Renderer-&gt;render(Array, 1) (Line: 151)
Drupal\Core\Render\Renderer-&gt;Drupal\Core\Render\{closure}() (Line: 582)
Drupal\Core\Render\Renderer-&gt;executeInRenderContext(Object, Object) (Line: 152)
Drupal\Core\Render\Renderer-&gt;renderPlain(Array) (Line: 842)
_webform_token_get_submission_values(Array, Object) (Line: 445)
webform_tokens(&#039;webform_submission&#039;, Array, Array, Array, Object)
call_user_func_array(&#039;webform_tokens&#039;, Array) (Line: 403)
Drupal\Core\Extension\ModuleHandler-&gt;invokeAll(&#039;tokens&#039;, Array) (Line: 304)
Drupal\Core\Utility\Token-&gt;generate(&#039;webform_submission&#039;, Array, Array, Array, Object) (Line: 196)
Drupal\Core\Utility\Token-&gt;replace(&#039;Submitted on [webform_submission:created]
Submitted by: [webform_submission:user]
Submitted values are:
[webform_submission:values]
&#039;, Array, Array, Object) (Line: 138)
Drupal\webform\WebformTokenManager-&gt;replace(&#039;Submitted on [webform_submission:created]
Submitted by: [webform_submission:user]
Submitted values are:
[webform_submission:values]
&#039;, Object, Array, Array, Object) (Line: 190)
Drupal\webform\WebformTokenManager-&gt;replaceNoRenderContext(&#039;Submitted on [webform_submission:created]
Submitted by: [webform_submission:user]
Submitted values are:
[webform_submission:values]
&#039;, Object, Array, Array) (Line: 928)
Drupal\webform\Plugin\WebformHandler\EmailWebformHandler-&gt;getMessage(Object) (Line: 856)
Drupal\webform\Plugin\WebformHandler\EmailWebformHandler-&gt;postSave(Object, , NULL) (Line: 2224)
Drupal\webform\Entity\Webform-&gt;invokeHandlers(&#039;postSave&#039;, Object, , NULL) (Line: 1097)
Drupal\webform\WebformSubmissionStorage-&gt;invokeWebformHandlers(&#039;postSave&#039;, Object, ) (Line: 1010)
Drupal\webform\WebformSubmissionStorage-&gt;doPostSave(Object, ) (Line: 432)
Drupal\Core\Entity\EntityStorageBase-&gt;save(Object) (Line: 774)
Drupal\Core\Entity\Sql\SqlContentEntityStorage-&gt;save(Object) (Line: 390)
Drupal\Core\Entity\Entity-&gt;save() (Line: 757)
Drupal\webform\Entity\WebformSubmission-&gt;save() (Line: 1637)
Drupal\webform\WebformSubmissionForm-&gt;save(Array, Object)
call_user_func_array(Array, Array) (Line: 111)
Drupal\Core\Form\FormSubmitter-&gt;executeSubmitHandlers(Array, Object) (Line: 51)
Drupal\Core\Form\FormSubmitter-&gt;doSubmitForm(Array, Object) (Line: 589)
Drupal\Core\Form\FormBuilder-&gt;processForm(&#039;webform_submission_contact_us_add_form&#039;, Array, Object) (Line: 318)
Drupal\Core\Form\FormBuilder-&gt;buildForm(&#039;webform_submission_contact_us_add_form&#039;, Object) (Line: 48)
Drupal\Core\Entity\EntityFormBuilder-&gt;getForm(Object, &#039;add&#039;) (Line: 1046)
Drupal\webform\Entity\Webform-&gt;getSubmissionForm() (Line: 84)
Drupal\webform\Controller\WebformEntityController-&gt;addForm(Object, Object)
call_user_func_array(Array, Array) (Line: 123)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber-&gt;Drupal\Core\EventSubscriber\{closure}() (Line: 582)
Drupal\Core\Render\Renderer-&gt;executeInRenderContext(Object, Object) (Line: 124)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber-&gt;wrapControllerExecutionInRenderContext(Array, Array) (Line: 97)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber-&gt;Drupal\Core\EventSubscriber\{closure}() (Line: 151)
Symfony\Component\HttpKernel\HttpKernel-&gt;handleRaw(Object, 1) (Line: 68)
Symfony\Component\HttpKernel\HttpKernel-&gt;handle(Object, 1, 1) (Line: 57)
Drupal\Core\StackMiddleware\Session-&gt;handle(Object, 1, 1) (Line: 47)
Drupal\Core\StackMiddleware\KernelPreHandle-&gt;handle(Object, 1, 1) (Line: 99)
Drupal\page_cache\StackMiddleware\PageCache-&gt;pass(Object, 1, 1) (Line: 78)
Drupal\page_cache\StackMiddleware\PageCache-&gt;handle(Object, 1, 1) (Line: 47)
Drupal\Core\StackMiddleware\ReverseProxyMiddleware-&gt;handle(Object, 1, 1) (Line: 52)
Drupal\Core\StackMiddleware\NegotiationMiddleware-&gt;handle(Object, 1, 1) (Line: 23)
Stack\StackedHttpKernel-&gt;handle(Object, 1, 1) (Line: 669)
Drupal\Core\DrupalKernel-&gt;handle(Object) (Line: 19)
</pre>

Also is there any way to enhance the phone field validation. Currently is allows anything as a phone number.

jrockowitz’s picture

To prevent the error, you need to check the 'Sanitize file name" setting for the attachment element.

Also is there any way to enhance the phone field validation. Currently is allows anything as a phone number.

.

Please post general support questions to https://drupal.stackexchange.com/questions/tagged/webforms

ksavoie’s picture

The sanitation flag lets the form submit but it strips the date I have configured at the beginning.

I discovered the issue with submitting is with the drop downs containing '/'.

Additionally, if you submit a form that had a file name that would have erred if not for the sanitation flag. Then you disable the flag, the 'result' page errors until you re-enable the flag.

jrockowitz’s picture

Okay, I will figure out how to prevent the fatal error. I think we might want to turn on sanitize on by default.

ksavoie’s picture

That's what I was thinking.
Is it a requirement as part of sanitation the lower casing of all the characters?

jrockowitz’s picture

Status: Needs work » Needs review
StatusFileSize
new110.18 KB

The attached patch fixes the / in file name issue .

The sanitize code is from http://stackoverflow.com/questions/2021624/string-sanitizer-for-filename and Drupal's standard code practice which is to lower case file names.

Status: Needs review » Needs work

The last submitted patch, 37: 3013498-37.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

jrockowitz’s picture

Status: Needs work » Needs review
StatusFileSize
new110.2 KB

  • jrockowitz committed 8ca0433 on 3013498-attachement
    Issue #3013498 by jrockowitz: Allow custom files to be attached to...

Status: Needs review » Needs work

The last submitted patch, 39: 3013498-39.patch, failed testing. View results

ksavoie’s picture

Is patch 39 OK to test?

jrockowitz’s picture

Yes please try the patch from 39

ksavoie’s picture

The sanitation error looks to be mitigated, however sanitizing still strips the date.

jrockowitz’s picture

What exactly is being removed from the date?

One solution would be to add support for a ':sanitize' token prefix and you can decide which tokens are sanitized.

ksavoie’s picture

The literal values of the two fields that append the file name
"Rental Property Owner/Manager"
"Authorization to Allow Management Company to act on behalf of owner"

File Name configuration:

[webform_submission:completed:custom:Ymdhis]_[webform_submission:values:category:raw]_[webform_submission:values:sub_moving:raw][webform_submission:values:sub_budget:raw][webform_submission:values:sub_billing_question:raw][webform_submission:values:sub_refund:raw][webform_submission:values:sub_rental_prop_owner_manager:raw][webform_submission:values:sub_web_administration:raw][webform_submission:values:sub_payments:raw][webform_submission:values:sub_energy_assistance:raw][webform_submission:values:sub_letter_received:raw]_webform.xml

Filename result with sanitize not checked

20190111112153_Rental Property OwnerManager_Authorization to  Allow Management Company to act on behalf of owner_webform.xml

Filename result with sanitize checked

manager_authorization-to-allow-management-company-to-act-on-behalf-of-owner_webform.xml
jrockowitz’s picture

Status: Needs work » Needs review
StatusFileSize
new110.19 KB

The attached patch fixes the slash issue with sanitizing enabled.

  • jrockowitz committed 4a7f5c9 on 3013498-attachement
    Issue #3013498 by jrockowitz: Allow custom files to be attached to...
jrockowitz’s picture

StatusFileSize
new107.33 KB
ksavoie’s picture

Email Subject field length needs to be increased. Similar issue as with file name. Once I start adding in potential token values, the field truncates.

This is what I'm trying to currently use as a email subject.
Website_ContactUsForm_[webform_submission:completed:custom:Ymdhis]_[webform_submission:values:category:raw]-[webform_submission:values:sub_budget_bill:raw][webform_submission:values:sub_bill_amount:raw][webform_submission:values:sub_refund:raw][webform_submission:values:sub_payment:raw][webform_submission:values:sub_web_admin:raw][webform_submission:values:sub_miscellaneous:raw]

ksavoie’s picture

Category: Feature request » Bug report

Clicking edit on a field set as required doesn't perpetuate the required status into the edit box. Will disable if saved without re-selecting.

  • jrockowitz committed fa39688 on 3013498-attachement
    Issue #3013498: Allow custom files to be attached to emails and...
jrockowitz’s picture

StatusFileSize
new107.96 KB

The attached patch fixes #50 but I am not sure where #51 is happening. Can you please include a screenshot?

ksavoie’s picture

StatusFileSize
new37.22 KB

In this case (see attached image), if I click edit on this select (that is currently set as required), the edit panel doesn't have 'required' selected. If I click save without making any change, it will no longer be required.

jrockowitz’s picture

You need to click the 'Save' button to update the 'Required' states via the form builder.

ksavoie’s picture

Doesn't make a difference. If I save in main screen or save in edit panel, as soon as I edit the field it is always deselected.

jrockowitz’s picture

StatusFileSize
new276.7 KB

Here is a screenshot of the 'required' and 'Save elements', I mentioned in #55.

ksavoie’s picture

Yes, this is exactly what has been done. The state is saved, I can go away from that form and reopen it and the required state is selected, however as soon as I click edit, the edit panel does not have required selected at the bottom under 'Form Validation'.

jrockowitz’s picture

You are right the Required checkbox is no longer working for the element edit form.

What is strange is all versions of Webform are broken. This might be a Drupal core issue.

jrockowitz’s picture

Okay, the regression is #3024568: Conditional field requirement, customize requrement text and only the latest dev release of the webform module is affected.

jrockowitz’s picture

I committed the updated patch from #3024568: Conditional field requirement, customize requrement text. Please download the latest dev release to review.

BTW, make sure to clear your browser's cache.

ksavoie’s picture

Both fixes appear to be working, however custom required text no longer displays. The required fields will focus but no text. (remember this is a case where two fields are conditionally required, one or the other).

jrockowitz’s picture

@ksavoie Please make sure to clear your browser cache. The custom required text will only appear if an element is conditionally required.

  • jrockowitz committed 29a5fd2 on 8.x-5.x
    Issue #3013498 by jrockowitz, ksavoie: Allow custom files to be attached...
jrockowitz’s picture

Status: Needs review » Fixed

I committed the patch. Please download the latest dev release to review. If there are any new issues please create a new ticket.

ksavoie’s picture

What is the format of the token to write the attachment into the body of a custom message format.
Tried
[webform_submission:values:sap_contact_us_results_xml]

But it just wrote it verbatim into the body.

jrockowitz’s picture

The attachment token only renders the link to the attachment's file. Please open a new ticket and we can probably add support for a [webform_submission:values:sap_contact_us_results_xml:content] token.

ksavoie’s picture

A link would have been fine.

Should [webform_submission:values:sap_contact_us_results_xml] have written the link?

jrockowitz’s picture

It depends, please create a new ticket and we can figure out what is happening with attachment element token.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

idebr’s picture

I was about to file a new issue where the 'Resend' option generated an error for a custom file that is being attached with a custom EmailWebformHandler when I stumbled upon this issue. After applying the patch in #53 the form is working again beautifully!

Many thanks @jrockowitz for your great work!