Emojis causing SQLSTATE[HY000]: General error: 1366 Incorrect string value

It is interesting to note that the first 3 times I tried to submit this bug report, I got a 5xx error.

I removed the emojis from the description and it posted.

I see an automated test with the patch I'm using as a reference, but I have limited experience with tests. I'll try, but if I can't make the time for it, I'll still post the patch and ask if someone else can contribute faster than I.

And, yes, this is not at all limited to Feeds.

In my original post, where you see the word [emoji], I had to remove two emoji chars because this d-dot-org form 500 errored out with them in.

If you want to see the specific chars, go to http://widget.websta.me/rss/n/mitadmissions and find the text "nbd just petting some ducklings at MIT Day of Play"

I have a note to report the bug elsewhere, but time is finite.

I'm working on the patch for this today, though.

I'm attaching a stripped down html file with a .txt added to the filename to allow it to be uploaded (html files are disallowed).

This is a minimal sample of the offending code.

Here's how the matching works (only grabs the first emoji if multiple):

// Regex swiped from patch #22 in issue # 1824506
$fourByteRegex = '/(?:\xF0[\x90-\xBF][\x80-\xBF]{2}|[\xF1-\xF3][\x80-\xBF]{3}|\xF4[\x80-\x8F][\x80-\xBF]{2})/s';

// Loop through my offending strings
foreach ($myevalstring as $string) {

  // Strip out all but the offending emoji
  preg_match($fourByteRegex, $string, $match);

  // If we have an offender
  if ($match) {

    // Add the string back to the array for reference
    $match['string'] = $string;

    // Spit it out
    drupal_set_message('<pre>' . print_r($match, true) . '</pre>');
  }
}

// Remove all the extraneous HTML from file by hand
// Add a .txt so I can upload to Drupal.org
// Provide a screenshot in case it looks different for other people

I am attaching a patch and have notes:

• Used code from #1824506: When importing Tweets: SQLSTATE[HY000]: General error: 1366 Incorrect string value #22
• I did not write tests, though some exist in the source from #1824506, patch in #22.
• I did not implement the config option to convert versus strip. It will always strip.
• I did test the convert and it does work. However, I had subsequent failures because the path module wasn't stripping the emojis in the alias path and then the same errors would fire for the {url_alias} table.
• I did not address this issue for any of the other Feeds plugins.

Basically, this isn't packaged up nicely for distro, but I am out of time for now. If someone else can contribute, awesome. If not, I'll try and check back again when I can.

Here is decent narrative of the issue:
http://www.jeffgeerling.com/blogs/jeff-geerling/solving-emoji-problem-dr...

@raphaeldelrosal

I'm just seeing that you PMed me to get a copy of the file after the patch is applied. I'm guessing because you aren't familiar with how to apply a patch.

Normally, it's best to test a patch by figuring out how to apply it and test then. I'd read up on patching as it's a skill you will benefit from having. For me, I usually put my patches in sites/all/patches and then cd into the module I'm patching and patch -p1 < ../../patches/thepatch.patch. That's just my workflow. It may not be anyone else's.

That said, check out the link in comment #9. Drupal 7.50 may address this issue but I haven't had a chance to test it. And I don't think I will anytime soon.