Problem/Motivation

I'm working on a site which require a field to only have Chinese/Japanese character. The pattern will include unicode pattern such as 

U+4E00–U+9FBF Kanji/Chinese
U+3040–U+309F Hiragana
U+30A0–U+30FF Katakana

The current pattern validator doesn't include unicode flag, therefore the pattern, for instance [\x{4e00}-\x{9fbf}]+cannot be used.

Proposed resolution

For the text base element, adding u flag makes the pattern valid.

// line 284 in src/Plugin/WebformElement/TextBase.php
if (preg_match('{^(?:' . $properties['#pattern'] . ')$}u', NULL) === FALSE) {...}

There are further tasks need to be resolved for the feature

  • Submission validator: I couldn't figure the place to make the submission validation compatible to the unicode pattern yet.
  • HTML5 pattern: The HTML5 input element uses another unicode format for regex which requires a conversion from the pattern field.
    For example [\x{4e00}-\x{9fbf}]+ ---> [\u4e00-\u9fa5]+
CommentFileSizeAuthor
#10 3002374-10.patch12.99 KBjrockowitz
#10 interdiff-3002374-9-10.txt11.27 KBjrockowitz
#9 3002374-9.patch2.15 KBamourow
#6 3002374-6.patch2.13 KBjrockowitz
#5 3002374-5.patch1.65 KBamourow
#2 3002374-2.patch1.47 KBjrockowitz

Comments

amourow created an issue. See original summary.

jrockowitz’s picture

jrockowitz commented
Status: Active » Needs review
StatusFileSize
new 3002374-2.patch1.47 KB

Here is the related core issue.

#2633550: #pattern fails for Unicode points

I am tempted to suggest we use HTML5 unicode regex patterns and convert them to PHP regex patterns

Here is a starting pointing for converting the PHP regex to JS. https://bytefreaks.net/programming-2/php-programming-2/php-convert-javas...

The attached patch enables unicode for PHP regex patterns.

I think this issue is blocked until the core issue is fixed.

amourow’s picture

amourow
he/him/his
commented

I did tried adding unicode flag in WebformSubmissionConditionsValidator as patch #2, but it doesn't work.

Turns out, it's still stuck in FormElement::validatePattern in the core as you mentioned in the issue #2633550: #pattern fails for Unicode points

Convert HTML5 pattern to PHP is indeed a good idea.

jrockowitz’s picture

jrockowitz commented

One immediate workaround is to specify a server and client pattern using the below element source code.

textfield:
  '#type': textfield
  '#title': textfield
  '#pattern': server-pattern
  '#attributes':
    pattern: client-pattern

The '#attributes' is editable under the element's 'Advanced' settings.

Then I can apply this issue's patch and you can create a patch for core.

amourow’s picture

amourow
he/him/his
commented
StatusFileSize
new 3002374-5.patch1.65 KB

Following the reference of the conversion between RegExp and PCRE code point. I submitted a patch to core at #2633550-11: #pattern fails for Unicode points, and also have the updated patch for the issue.

jrockowitz’s picture

jrockowitz commented
StatusFileSize
new 3002374-6.patch2.13 KB

Adding inline comments to the patch.

I think key thing is to write the patch needed to core and update \Drupal\Tests\system\Functional\Form\ValidationTest::testPatternValidation.

amourow’s picture

amourow
he/him/his
commented

Thanks for mention it. I update the issue, and just realize I totally forgot the patch file. Please review it there.

For the comment in the patch here, is it just the title of the reference article? The code in patch is actually convert to PCRE Unicode code point format
+ // PHP: Convert JavaScript-escaped Unicode characters to HTML hex references

jrockowitz’s picture

jrockowitz commented
Status: Needs review » Needs work

@amourow Feel free to improve my comments. It is important to explain what the code is doing and I like to reference the source for any confusing code snippet.

amourow’s picture

amourow
he/him/his
commented
Status: Needs work » Needs review
StatusFileSize
new 3002374-9.patch2.15 KB

Update the comment about what the preg_match() is going to do.

jrockowitz’s picture

jrockowitz commented
StatusFileSize
new interdiff-3002374-9-10.txt11.27 KB
new 3002374-10.patch12.99 KB

Once your patch is committed, it will most likely have to be released with Drupal 8.7 in May 2019.

I figured out a way to override \Drupal\Core\Render\Element\FormElement::validatePattern and add unicode support.

Once I did this, I decided to add support for a #pattern_error message.

@amourow Please review the attached patch and include an interdiff with your improvements.

The final step for this patch would be to create a change record.

jrockowitz’s picture

jrockowitz commented

Here is the change record.

  • jrockowitz committed 868aacb on 8.x-5.x 
    Issue #3002374 by jrockowitz, amourow: Allow unicode regex pattern
jrockowitz’s picture

jrockowitz commented
Status: Needs review » Fixed

I committed the patch. Please download the latest dev release to review.

amourow’s picture

amourow
he/him/his
commented

Thanks @jrockowitz,
I've tested the dev version and it works solo.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.