Problem/Motivation

From #2822298: Provide a show / hide headers link / button in the mail display we've noticed that

Using the multiple-attachments.eml example, it turns out the Message-ID line is wrapped into two lines still. We need to investigate where the newline is coming from.

This is the followup where to investigate and discuss about it.

Proposed resolution

- investigate
- discuss
- fix it

Remaining tasks

User interface changes

API changes

Data model changes

CommentFileSizeAuthor
#25 interdiff-2822786-23-25.txt2.41 KBtoncic
#25 multiple-attachments_message-id-28222786-25.patch8.93 KBtoncic
#23 interdiff-2822786-21-23.txt4.73 KBtoncic
#23 multiple-attachments_message-id-28222786-23.patch9.16 KBtoncic
#21 interdiff-2822786-19-21.txt4.81 KBtoncic
#21 multiple-attachments_message-id-28222786-21.patch6.94 KBtoncic
#19 interdiff-2822786-17-19.txt2.12 KBtoncic
#19 multiple-attachments_message-id-28222786-19.patch4.08 KBtoncic
#17 interdiff-2822786-15-17.txt1.54 KBtoncic
#17 multiple-attachments_message-id-28222786-17.patch1.97 KBtoncic
#15 multiple-attachments_message-id-28222786-15.patch2.83 KBtoncic
#13 interdiff-2822786-10-13.txt3.91 KBtoncic
#13 multiple-attachemts_message-id-282278-13.patch7.28 KBtoncic
#10 inmail_2822786_header_10.patch3.58 KBmiro_dietiker
#7 interdiff-2822786-4-7.txt2.3 KBtoncic
#7 multiple-attachemts_message-id-2822786-7.patch1.91 KBtoncic
#4 message_id_after.png138.64 KBtoncic
#4 multiple-attachemts_message-id-2822786-4.patch718 bytestoncic
#4 message_id_before.png138.25 KBtoncic
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

tduong created an issue. See original summary.

ModernMantra’s picture

What breaks to new line is Header::toString(). Debugged and verified and there is regex expression which breaks string after 78 characters.

$field_string = !$encoded ? preg_replace('/(.{0,78})(\h|$)/', "\\1\n\\2", $field_string) : $field_string;

miro_dietiker’s picture

Funny stuff...

Yeah this is because we are using toString() to go into the SMTP domain...
Not sure what is the clean solution. I guess there should be a different method getHeaderString or so that does this wrapping and keep toString() unencoded... However you will need to change all explicit and implicit string consumers to that method.

toncic’s picture

Issue summary: View changes
Status: Active » Needs review
FileSize
138.25 KB
718 bytes
138.64 KB

Change toString() method just to skip Message-ID for regex and it is working for me.

toncic’s picture

Assigned: Unassigned » toncic
miro_dietiker’s picture

Status: Needs review » Needs work

No, this approach is the wrong direction.

All headers need to behave the same way. This wrapping would happen with every other line too if it is too long.
So it's about making the method toString NOT wrap at all and let everyone who expects that wrapping call the new method that contains the old wrapping. I tried to explain if this is fine in #3 and in order to decide it, we need it implemented to see the complexity / shift.

toncic’s picture

Status: Needs work » Needs review
FileSize
1.91 KB
2.3 KB

I added new function toStringNotWrap() and use that now.
I think maybe we can add new checkbox in setting, something like write header into multi lines.

mbovan’s picture

Status: Needs review » Needs work
Issue tags: +Needs tests
  1. +++ b/inmail.module
    @@ -462,6 +462,7 @@ function inmail_preprocess_inmail_message(&$variables) {
    +  $variables['all_headers'] = $message->getHeader()->toString();
    
    +++ b/templates/inmail-message.html.twig
    @@ -75,7 +75,7 @@
    -      <pre>{{ message.header.toString() }}</pre>
    +      <pre>{{ all_headers }}</pre>
    

    It seems the same as before.

    In case we are creating new variables, we should document/add them in theme info and the twig template doc block.

  2. +++ b/src/MIME/MimeHeader.php
    @@ -199,6 +199,23 @@ class MimeHeader {
    +  public function toStringNotWrap() {
    

    Considering #3, not sure if this new method should go into MessageDecomposition service... But, as toString() also does encoding, I guess we can have it on MimeHeader.

    Also, having optional $wrapped parameter in toString() could be an option to consider in this case.

Tests are missing as well.

miro_dietiker’s picture

You might underestimate the complexity and architectural decisions we need to think about here.

Instances of MimeMessageInterface / MimeEntityInterface contains a parsed message and calling toString() should reconstruct what was parsed WITHOUT changes. Currently toString simply force encodes header, which is also needed because we decode on parsing. Thus we do not really know if parsing and later converting back to string will result in the same. As soon as the header contains special characters that are inline encoded, things get worse. Originally, they could be unencoded 8bit while our conversion might apply inline encoding.

So the first thing here is, writing / extending unit tests that check the cycle. Parsing examples with special encodings.
Then later convert toString and check if the very same thing result again.
This will require analysis of the implementation in \Drupal\inmail\MIME\MimeHeader::toString and its \Drupal\inmail\MIME\MimeHeader::mimeHeaderEncode

The mess already starts at \Drupal\inmail\MIME\MimeParser::parseHeaderFields where we decode with the lines:

      $decoded_body = str_replace("\n", '', Unicode::mimeHeaderDecode(trim($body)));
      $header->addField(trim($name), $decoded_body, FALSE);

So it means lots of cases we encode here are not guaranteed to be returned the same way later.

In contrast, if you look at the body, we have two methods:
getBody() that keeps the body unchanged and getDecodedBody() that applies decoding.
The body is not modified when parsing.

So it seems that if we want to be able to show the original header from parsing, we need to additionally store the undecoded header when parsing and keep it with \Drupal\inmail\MIME\MimeHeader::addField
I think it should be a separate method addParsedField() with an extra argument.

We now need to decide if toString defaults to encoding and if we implement an additional getRaw() that returns the parsed values (plus the added ones without Raw data through regular encoding)... or if we make toString to default returning the unchanged data, offering an alternative call (param, method) to return the encoded value.

IMHO, we should default to return unchanged things as much as possible. and toString should only trigger encodings, ... if we do not have original raw information.

As a result, toStringNotWrap() is not needed. :-)

miro_dietiker’s picture

Status: Needs work » Needs review
FileSize
3.58 KB

Attached some implementation that follows my principles outlined.

So we need examples like this overlong MessageId in \Drupal\Tests\inmail\Unit\MIME\MimeHeaderTest::provideHeaders()
We might need to move this data to a different dataProvider and switch testParse to that.

I guess there aren't too many other changes needed here except deep test coverage.

Status: Needs review » Needs work

The last submitted patch, 10: inmail_2822786_header_10.patch, failed testing.

The last submitted patch, 10: inmail_2822786_header_10.patch, failed testing.

toncic’s picture

Trying to fix test fails but unfortunately not successful.

Status: Needs review » Needs work

The last submitted patch, 13: multiple-attachemts_message-id-282278-13.patch, failed testing.

toncic’s picture

Status: Needs work » Needs review
FileSize
2.83 KB

After discussion with both @miro_dietiker and @Berdit, we deal to switch the way for solving this issue. Now the goal is to have $raw in MimeHeader class and to have getRaw() as a way to access to raw.

miro_dietiker’s picture

Status: Needs review » Needs work
  1. +++ b/src/MIME/MimeHeader.php
    @@ -32,18 +32,29 @@ class MimeHeader {
    +   * The row field.
    

    raw

  2. +++ b/src/MIME/MimeHeader.php
    @@ -182,19 +193,26 @@ class MimeHeader {
    +    if (isset($this->raw)) {
    +      $header[] = $this->raw;
    +    }
    +    else {
    

    Revert. It's unused.

toncic’s picture

Implemented comm #16.

Status: Needs review » Needs work

The last submitted patch, 17: multiple-attachments_message-id-28222786-17.patch, failed testing.

toncic’s picture

Fixing test failing.

miro_dietiker’s picture

Status: Needs review » Needs work
+++ b/src/MIME/MimeHeader.php
@@ -185,16 +196,10 @@ class MimeHeader {
-      $field_string = !$encoded ? preg_replace('/(.{0,78})(\h|$)/', "\\1\n\\2", $field_string) : $field_string;

We don't want to remove this.
The new method getRaw() is not used at all now.
You want to write a unit test for it.
And then the twig "All headers" output needs to be switched to this method.
And there we should once assert an overlengthy row from the example in display.

toncic’s picture

Added test for getRaw() and switch to use getRaw() in twig.

Status: Needs review » Needs work

The last submitted patch, 21: multiple-attachments_message-id-28222786-21.patch, failed testing.

toncic’s picture

Fixing test failing.

ModernMantra’s picture

Small syntax stuff that can be optionally fixed.

  1. +++ b/src/MIME/MimeHeader.php
    @@ -32,18 +32,29 @@ class MimeHeader {
    +   * @param $raw
    

    Missing type.

  2. +++ b/src/MIME/MimeHeader.php
    @@ -32,18 +32,29 @@ class MimeHeader {
    +
    

    IMHO, that it is not necessary.

  3. +++ b/tests/src/Kernel/ModeratorForwardTest.php
    @@ -170,9 +170,9 @@ class ModeratorForwardTest extends KernelTestBase {
    +    $expected_headers =  $forward_header . wordwrap($received_header, 78, "\n ") . $expected_headers;
    

    Unrelated change. Seems that you hit two times space causing that it appears in patch.

  4. +++ b/tests/src/Unit/MIME/MimeHeaderTest.php
    @@ -165,4 +173,17 @@ class MimeHeaderTest extends UnitTestCase {
    +      ]],
    

    Wrong indentation.

toncic’s picture

  • miro_dietiker committed 000cf21 on 8.x-1.x authored by toncic
    Issue #2822786 by toncic, miro_dietiker: multiple-attachments.eml...
miro_dietiker’s picture

Status: Needs review » Fixed
Issue tags: -Needs tests

Committed with some comment updates.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.