Problem/Motivation

The parser should tolerate minor deviations from standards, but we still need a method to check that a message (or entity in general) is fine.

Proposed resolution

Add validate() to the entity interface.

On the Message class/interface introduced in #2416489: Extract Message as subtype to Entity?, validation should e.g. check for presence of the Subject header etc.

Remaining tasks

User interface changes

API changes

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

ModernMantra’s picture

Assigned: Unassigned » ModernMantra
Status: Active » Needs review
FileSize
2.16 KB

Made some progress, not sure whether in good direction... Patch contains implementation of validate function as well test support for it. Would be nice if some 'detailed' directives are provided...

Bambell’s picture

Status: Needs review » Needs work

Some feedbacks :

  1. +++ b/src/MIME/EntityInterface.php
    @@ -83,4 +83,12 @@ interface EntityInterface {
    +   * Check that message contains Header fields such as Subject, To, From, ID.
    ...
    +   *   Returns TRUE if message is valid, otherwise FALSE.
    

    Your documentation here should be generic to all classes implementing this interface, not specific to Message objects.

  2. +++ b/src/MIME/Message.php
    @@ -51,4 +51,16 @@ class Message extends Entity implements MessageInterface {
    +  public function validate() {
    

    You can move your documentation specific to Message's inside here.

  3. +++ b/src/MIME/Message.php
    @@ -51,4 +51,16 @@ class Message extends Entity implements MessageInterface {
    +        if (is_null($this->getHeader()->getFieldBody($key))) {
    

    You just want to know if the field exists. You can use hasField() on the Header object to avoid the overhead of returning and trimming the body.

Arla’s picture

The basic idea looks right.

RFC 5322 defines how many times a header field may occur in the message: https://tools.ietf.org/html/rfc5322#section-3.6 (the table on p. 21) Date and From are the only that are actually required. Contrary to the IS, Subject is not required (neither is Message-Id nor To). We could additionally check that the single header fields (max = 1) do not occur twice.

I think we should also add usage. Perhaps in MessageProcessor, after calling parseMessage()? Not sure if we want to abort if the message is invalid, as if it couldn't be parsed, or if we just want to log a warning or so.

Also wondering if validate() should return any hints when it is invalid, like "Missing From field". But that's added complexity and I'm not sure how it can be helpful.

ModernMantra’s picture

Status: Needs work » Needs review
FileSize
5.62 KB
5.41 KB

New patch that focuses on issues stated in comments #2 and #3

Status: Needs review » Needs work

The last submitted patch, 4: validate-2421431-4.patch, failed testing.

Bambell’s picture

Here are some feedbacks. In my humble opinion, returning an array of arbitrary strings from validate() adds too much complexity for what it's worth. Maybe we should put in some more thoughts?

  1. +++ b/src/MIME/Header.php
    @@ -213,4 +213,36 @@ class Header {
    +   * Returns the bodies of the field with the given name.
    

    A field has a single body, but you return the body of multiple fields now. "Returns the body of the fields with the given name.".

  2. +++ b/src/MIME/Header.php
    @@ -213,4 +213,36 @@ class Header {
    +    $field_bodies = array();
    

    Please use the short array syntax, always.

  3. +++ b/src/MIME/Header.php
    @@ -213,4 +213,36 @@ class Header {
    +    if ($field_bodies === FALSE) {
    

    This will never evaluate to TRUE, since you're doing a strict (type) comparison and you initialize the variable as an empty array. if (!$field_bodies) {...} will be sufficient.

  4. +++ b/src/MIME/Header.php
    @@ -213,4 +213,36 @@ class Header {
    +    foreach ($field_bodies as $key => $field) {
    +      $body[] = trim($field_bodies[$key]['body']);
    +    }
    

    Can't you just assign the 'body' array element and trim it directly in the first loop?

  5. +++ b/src/MIME/Message.php
    @@ -51,4 +51,37 @@ class Message extends Entity implements MessageInterface {
    +    // Date and From header fields are only required fields in message by IS.
    

    Please mention to which issue you are referring to by "IS" or, better, mention directly the source (RFC 5322 format).

  6. +++ b/src/MessageProcessor.php
    @@ -89,6 +89,12 @@ class MessageProcessor implements MessageProcessorInterface {
    +        $this->loggerChannel->info('Message Validation failed with message "%message" ',['%message' => implode(', ', $validate_message['value'])]);
    

    Discussion : Should we exit/throw here?

  7. +++ b/tests/src/Unit/MIME/MessageTest.php
    @@ -72,4 +72,31 @@ class MessageTest extends UnitTestCase {
    +
    +
    +
    

    Unnecessary spaces. I also think that it would be interesting to test for the case where you have all necessary fields, but duplicates. Right now you're testing the case where you are missing a field + duplicate (should be separate) and when it's fine.

ModernMantra’s picture

Status: Needs work » Needs review
FileSize
6.01 KB
3.43 KB

Status: Needs review » Needs work

The last submitted patch, 7: validate-2421431-7.patch, failed testing.

miro_dietiker’s picture

+++ b/src/MIME/EntityInterface.php
@@ -83,4 +83,12 @@ interface EntityInterface {
+   * @return bool
...
+  public function validate();

+++ b/src/MIME/Message.php
@@ -51,4 +51,38 @@ class Message extends Entity implements MessageInterface {
+   * @return array
...
+  public function validate() {

aha... :-)

miro_dietiker’s picture

Whatever validate returns should allow us to know what message is related to what field.
Thus the messages should always be keyed..

And since this stuff is hierarchical, we need to maintain key hierarchy merging with a separator.

However, i also think we should check how core does these kind of things. We are now defining something on our own, right?

ModernMantra’s picture

Status: Needs work » Needs review
FileSize
5.98 KB

Rerolling patch since it failed to apply

ModernMantra’s picture

Regarding comment #10... There are two variations of validate functions how in core it is done. In Symfony\Component\Console\Inputymfony\Component\Console\Input::validate() function only throws an exception, while in Drupal\Core\Entity\FieldableEntityInterface validate function returns list of constraint violations. So, would be nice to make agreement on which way to go further (throw only an exception, or return error messages)...

Arla’s picture

Status: Needs review » Needs work

In core, the biggest validation mechanism is the Validator Symfony component, which is used by content entities and typed data. But there are also some "custom" methods named validate() that does it simpler (e.g. ConfigImporter modifies $this->validated, ViewsPluginInterface returns string[]). FormInterface::vaildateForm() is another mechanism, where implementations typically modifies $form_state. In this case, I suggest that validate() should just return at boolean result, because it's the simplest API and we cannot really motivate anything more complex imo.

  1. +++ b/src/MIME/Header.php
    @@ -213,4 +213,32 @@ class Header {
    +   * @return null|string
    +   *   The body of the field or NULL if the field is not present.
    ...
    +    if (!$body) {
    +      return NULL;
    +    }
    

    For consistency, let's return an empty array if the field is not present.

  2. +++ b/src/MIME/Message.php
    @@ -51,4 +51,38 @@ class Message extends Entity implements MessageInterface {
    +   * Check that message contains Header fields such as Date and From.
    

    This is a bit explicit, but still vague. "Checks that the message complies to the RFC standard." would be more accurate, I think.

  3. +++ b/src/MIME/Message.php
    @@ -51,4 +51,38 @@ class Message extends Entity implements MessageInterface {
    +    // Count number of field bodies. By IS, there should be only one occurrence of
    +    // fields Received and From.
    
    +++ b/tests/src/Unit/MIME/MessageTest.php
    @@ -72,4 +72,37 @@ class MessageTest extends UnitTestCase {
    +    // the only required Header fields are From and Date. In addition, the fields
    +    // can occur only once per message.
    

    Some comment lines exceed the 80 char width limit.

    IS normally means "issue summary", i.e. the text on this issue page. I guess you mean something like "According to the RFC".

  4. +++ b/src/MessageProcessor.php
    @@ -97,6 +97,12 @@ class MessageProcessor implements MessageProcessorInterface {
    +      // Check whether message is valid by containing single occurrence of From
    +      // and Date Header Fields.
    

    This is also a bit explicit.

  5. +++ b/tests/src/Unit/MIME/MessageTest.php
    @@ -72,4 +72,37 @@ class MessageTest extends UnitTestCase {
    +    // Message triggers checking for presence of Received and From fields, as well
    +    // checking single occurrence of them.
    +    $message = new Message(new Header([
    +      ['name' => 'Received', 'body' => 'Mon, 22 Aug 2016 09:24:00 +0100'],
    +      ['name' => 'Received', 'body' => 'Fri, 19 Aug 2016 15:47:23 +0200']
    +    ]), 'body');
    +    $this->assertFalse($message->validate()['result']);
    ...
    +    // Message contains all necessary fields but duplicates.
    +    $message = new Message(new Header([
    +      ['name' => 'From', 'body' => 'Foo'],
    +      ['name' => 'From', 'body' => 'Foo2'],
    +      ['name' => 'Received', 'body' => 'Tue, 23 Aug 2016 17:48:6 +0600'],
    +      ['name' => 'Received', 'body' => 'Tue, 23 Aug 2016 17:50:3 +0600'],
    +    ]), 'body');
    +    $this->assertFalse($message->validate()['result']);
    

    These two test cases are quite overlapping.

ModernMantra’s picture

Status: Needs work » Needs review
FileSize
5.79 KB
4.14 KB

Fixed some small issues regarding comments, tests. Still needs agreement and change logic of function validate whether to return just boolean value or associative array...

Bambell’s picture

Regarding validate(), I think that it has already been decided to just return a boolean :

In this case, I suggest that validate() should just return at boolean result, because it's the simplest API and we cannot really motivate anything more complex imo.

Moving back to needs work..

miro_dietiker’s picture

Hm hm. What's the value of refurning true/false if u do all the cases checking.

If something fails, i need either info inthe api about what was the problem or if it is processing related, i would want to have a message on the logger that explains what the validation problem was.

So what is the goal of the new validate method?

ModernMantra’s picture

Status: Needs work » Needs review
FileSize
7.04 KB
5.81 KB

Changed the logic of validate function to return only boolean value and making getter method for fetching error messages. Modified a little comments to satisfy changes as well as tests.

Arla’s picture

Status: Needs review » Needs work
  1. +++ b/src/MIME/Message.php
    @@ -12,6 +12,14 @@ use Drupal\Component\Datetime\DateTimePlus;
       /**
    +   * An associative array of keys and corresponding error messages.
    +   * It contains information that is provided by validate function.
    +   *
    +   * @var array
    +   */
    +  protected $error_messages = [];
    
    @@ -51,4 +59,51 @@ class Message extends Entity implements MessageInterface {
    +  public function getValidationErrors() {
    

    This variable and the method should be moved up to Entity. The variable comment needs to have an empty line after the first line.

  2. +++ b/src/MessageProcessor.php
    @@ -97,6 +97,14 @@ class MessageProcessor implements MessageProcessorInterface {
    +      $validate_message = $message->validate();
    +      if ($validate_message == FALSE){
    

    You could inline the validate() call inside the if condition. And replace == FALSE with a !

  3. +++ b/src/MessageProcessor.php
    @@ -97,6 +97,14 @@ class MessageProcessor implements MessageProcessorInterface {
    +        $this->loggerChannel->info('Message Validation failed with message "%message" ',['%message' => implode(', ', $message->getValidationErrors())]);
    +        if($event) {
    +          $event->addArgument('Message', $message->getValidationErrors());
    +        }
    

    Let's switch to error(). And on $event, call setSeverity(RfcLogLevel::ERROR).

    Let's also switch from quotes to colon: 'Message validation failed with message: %message'

    The argument key should be something like "message validation errors".

    After logging, we need to abort with a simple return;

    There are some whitespace code style mistakes. Make sure to configure PhpStorm (Preferences - Editor - Code Style - PHP - Spaces) according to the Drupal coding standards.

ModernMantra’s picture

Status: Needs work » Needs review
FileSize
6.97 KB
3.2 KB

Status: Needs review » Needs work

The last submitted patch, 19: validate-2421431-19.patch, failed testing.

ModernMantra’s picture

Status: Needs work » Needs review
FileSize
7.37 KB
323 bytes

Bug fix due to automated test failure

Bambell’s picture

Looks good overall. Here are some feedbacks :

  1. +++ b/src/MIME/Entity.php
    @@ -131,4 +140,26 @@ class Entity implements EntityInterface {
    +    // Empty the error message storage.
    

    Just as a fyi, the word "storage" in this context might lead to confusion.

  2. +++ b/src/MIME/Header.php
    @@ -213,4 +213,32 @@ class Header {
    +    if (!$body) {
    +      return [];
    +    }
    

    The original getFieldBody() implementation would return a string or NULL, if no match was found, which is why we had that check. Makes sense to return an empty array, but then we don't really need that check anymore, since we initialized the variable as an empty array. However, we don't really want the overhead of filtering the empty array. What about :

    return ($filter && $body) ? preg_replace('/\([^)]*\)/', '', $body) : $body;

  3. +++ b/src/MIME/Message.php
    @@ -51,4 +51,36 @@ class Message extends Entity implements MessageInterface {
    +    $counter = count($this->getHeader()->getFieldBodies('Received'));
    +    if ($counter > 1) {
    +      $result = FALSE;
    +      $this->error_messages['Received'] = $counter . ' Received fields';
    +    }
    +    $counter = count($this->getHeader()->getFieldBodies('From'));
    +    if ($counter > 1) {
    +      $result = FALSE;
    +      $this->error_messages['From'] = $counter . ' From fields';
    +    }
    

    Might be more elegant to loop over the fields here. Also, we're doing these checks even if we know that the field is absent. Maybe we could move this into the foreach loop above as an else case?

  4. +++ b/src/MessageProcessor.php
    @@ -97,6 +97,15 @@ class MessageProcessor implements MessageProcessorInterface {
    +        $this->loggerChannel->info('Message Validation failed with message %message',['%message' => implode(', ', $message->getValidationErrors())]);
    

    Missing space after comma ;).

  5. +++ b/src/MessageProcessor.php
    @@ -97,6 +97,15 @@ class MessageProcessor implements MessageProcessorInterface {
    +          $event->setSeverity(RfcLogLevel::ERROR);
    

    We are not including Drupal\Core\Logger\RfcLogLevel. I think this will explode when we reach that line. Do we have a test to cover that case?

  6. +++ b/tests/src/Unit/MIME/MessageTest.php
    @@ -72,4 +72,40 @@ class MessageTest extends UnitTestCase {
    +    $this->assertFalse(empty($message->getValidationErrors()));
    ...
    +    $this->assertFalse(empty($message->getValidationErrors()));
    

    Maybe we should assert the actual validation errors?

ModernMantra’s picture

Status: Needs work » Needs review
FileSize
7.51 KB
4.23 KB

Some small changes considering comment #22

  • Arla committed 6e9e542 on 8.x-1.x authored by ModernMantra
    Issue #2421431 by ModernMantra: Add EntityInterface::validate()
    
Arla’s picture

Status: Needs review » Fixed
FileSize
1.16 KB

Nice. I'm adding a few small comment changes. Committed!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

mbovan’s picture

Adding a related bug-fix issue.