The regex used in drupagram_emoji() is too permissive, and matches valid two- and three- byte UTF-8 characters. This results in anything other than latin text being mangled.

The first byte should be restricted to the range F0-F7 rather than C0-F7, and the subsequent match fixed at three bytes.

See http://en.wikipedia.org/wiki/UTF-8#Description for the leading byte values for multi-byte characters.

CommentFileSizeAuthor
#3 drupagram-emoji_fix-2232243-3.patch1007 bytesq11q11
#1 drupagram-emoji_fix-2232243-2.patch534 bytesben.kyriakou

Comments

ben.kyriakou’s picture

ben.kyriakou commented
Status: Needs work » Needs review
StatusFileSize
new drupagram-emoji_fix-2232243-2.patch534 bytes

Patch attached.

In practice this just removes the emoji, and I think a better fix would be to explicitly strip 4-byte characters until emoji support is added with a third-party library. In practice the emoji characters are not browser- or platform- cross-compatible since many systems lack the dictionaries corresponding to these code-points. Even the Instagram web-client lacks any cross-compatible support for these characters, and on my system (FF on Mac) they display as unknown characters.

See http://apps.timwhitlock.info/emoji/tables/unicode for a good summary of emoji unicode characters.

ben.kyriakou’s picture

ben.kyriakou commented
Component: API Integration / Library » Code
q11q11’s picture

q11q11 commented
StatusFileSize
new drupagram-emoji_fix-2232243-3.patch1007 bytes

Addition to patch #1.
Location has 'name' element, and some users sometimes uses emoji for this.
Whatever, we just got this problem, so I`m just posting modified patch.

joelpittet’s picture

joelpittet
he/him
Primary language English
Location Vancouver
commented
Related issues: +#1314214: MySQL driver does not support full UTF-8 (emojis, asian symbols, mathematical symbols)

Adding a related core issue for this.

damienmckenna’s picture

damienmckenna
Location TN, USA
commented
Related issues: +#1910376: SQL error when importing tweet with emoji, +#2402153: Use php-emoji library to work around UTF-8 limitations

The Twitter module has the same bug: #1910376: SQL error when importing tweet with emoji

Rather than continuing to change the content, I suggested using php-emoji to provide a solid solution to the problem: #2402153: Use php-emoji library to work around UTF-8 limitations

damienmckenna’s picture

damienmckenna
Location TN, USA
commented

The solution is convert the 'text' fields to 'blob'.

damienmckenna’s picture

damienmckenna
Location TN, USA
commented
Status: Needs review » Needs work
damienmckenna’s picture

damienmckenna
Location TN, USA
commented
Parent issue: » #2404171: Plan for Drupagram v7.x-1.3 release
damienmckenna’s picture

damienmckenna
Location TN, USA
commented
Related issues: +#2557085: Plan for Drupagram v6.x-1.2 release

Also needs to be backported to the D6 branch.

daniel korte’s picture

daniel korte
he/him/his
Primary language English
Location Brooklyn, NY
commented
Parent issue: #2404171: Plan for Drupagram v7.x-1.3 release » #2869629: Plan for Drupagram v7.x-1.4 release