Problem/Motivation

In the various issues to remove misspelled words from dictionary.txt valid words are accidentally removed. This just more reviews and rerolls.

Steps to reproduce

Proposed resolution

Add a second dictionary for words that CSpell identifies as misspellings that we want to keep. These will include, at least, Drupal specific words and some technical terms.

Remaining tasks

Add change record for script changes
Commit
Update docs page, see #12

User interface changes

API changes

Data model changes

Release notes snippet

Issue fork drupal-3328741

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

quietone created an issue. See original summary.

quietone’s picture

StatusFileSize
new5.17 KB

Another other words to add?

quietone’s picture

Title: Add a dictionary for known words to keep » Add a dictionary for Drupal specific words
StatusFileSize
new4.47 KB

A reroll and an update which removed non Drupal specific words from the proposed new dictionary.

quietone’s picture

StatusFileSize
new2.68 KB
new3.6 KB

Only a reroll.

quietone’s picture

Status: Active » Needs review
smustgrave’s picture

from your comment in the other ticket should drupalism words be moved too. Example mlid.

quietone’s picture

Yes, I did not do an exhaustive search for Drupalisms. I wanted to get this started and get feedback on the idea.

smustgrave’s picture

I do think organizing them will make it easier to manage.

So +1 for RTBC

smustgrave’s picture

Status: Needs review » Reviewed & tested by the community

Actually going to go ahead and mark it as it may be hard to get all phrases in one go. Maybe once this lands there is a follow up to move drupal like words over?

catch’s picture

Status: Reviewed & tested by the community » Needs work

This is a good idea, but a few comments on the patch:

+++ b/core/misc/cspell/drupal.txt
@@ -0,0 +1,22 @@
+llamma

While llama's are very close to being a Drupalism, llamma is a real mis-spelling that we should fix to llama, and I think lammaids and lammasarelame should be handled by cspell-ignore maybe?

Also not sure about fapi, mlid, nids, tids - these could potentially be expanded to Form API, menu link ID, node IDs, term IDs etc. and removed from the dictionary, but depends on the context.

Should we call it drupal-dictionary.txt so it's easy to see that it relates to dictionary.txt?

There are also some other changes in here - is that because we need to update dictionary.txt again after a cspell/dependency update? Might want to do that in a separate issue first to get a clean diff here.

quietone’s picture

Status: Needs work » Needs review
StatusFileSize
new3.97 KB

I grepped the usages of the 'llama' type words, fapi, mlid, nids and tids. Most of those could be changed so I restored those to dictionary.txt. They will be handled in a spelling error issue in due course.

This patch add as command to build the new drupal dictionary, and renames the existing command to spellcheck:make-dict.

I have not made an interdiff because the patch is small.

quietone’s picture

If this is committed this doc page needs to be updated, https://www.drupal.org/docs/develop/development-tools/cspell

smustgrave’s picture

Status: Needs review » Reviewed & tested by the community

Does renaming a script require a change record.

catch’s picture

Status: Reviewed & tested by the community » Needs work
+++ b/core/misc/cspell/dictionary.txt
@@ -80,13 +80,11 @@ autowiring
 backreferences
-bakeware
 bangpow
 barbar

Sorry bakeware's also not a Drupal-specific term.

+++ b/core/misc/cspell/dictionary.txt
@@ -379,13 +373,11 @@ endapply
 endcode
 endembed
-endlink
 endmacro
 endofheader
 endofscript
 endpo
 endset
-endtrans
 enim

Also not sure about endlink and endtrans, I think those are from phpdoc?

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

quietone’s picture

Issue summary: View changes
Status: Needs work » Needs review
Issue tags: +Needs change record
StatusFileSize
new1.31 KB
new4.27 KB

A change record make sense to me, adding tag.

For #14. I searched and found references to @endtrans in laravel docs but I did not find any references to @endlink. However, it does seem sensible to remove the pairs of @trans and @endtrans and @link and @endlink. Either way, we can always update the dictionaries later. I think it is more important to have this in core. It will help in the other spelling error issues where people accidentally remove drupalisms.

When making the patch I rebuilt both dictionaries.

smustgrave’s picture

Status: Needs review » Reviewed & tested by the community
Issue tags: -Needs change record +Needs Review Queue Initiative

Started the change record from my understanding of the issue.

quietone’s picture

Status: Reviewed & tested by the community » Needs review
StatusFileSize
new3.56 KB
new4.32 KB

There have been recent commits that have changed the dictionary, so I am rerolling and updating the patch. It was a simple reroll.

Since we now spellcheck:core when the dictionary changes I have added a check so that it runs when either dictionary.txt or drupal.txt is modified. I also rebuilt both dictionaries.

smustgrave’s picture

Status: Needs review » Reviewed & tested by the community

Reroll appears fine

Applied #18 and ran both spellcheck scripts and got the same results.

Status: Reviewed & tested by the community » Needs work

The last submitted patch, 18: 3328741-18.patch, failed testing. View results

quietone’s picture

Unrelated failure, retesting.

1) Drupal\Tests\system\Functional\System\CronRunTest::testAutomatedCron
Failed asserting that 1691657672 is less than 1691657672.

sakthi_dev made their first commit to this issue’s fork.

sakthi_dev’s picture

As the patch test not running created a MR. Please review.

quietone’s picture

@sakthi_dev, perhaps you missed it but my comment stated that I was restarting the tests. And once an issue is using a patch work then it should stay with that workflow. Adding an MR here is not helping. I have canceled the tests and closed the MR.

alexpott’s picture

I really like this idea. This makes lots of sense. Also maybe some of the things here and in our dictionary.txt should be pushed upstream into a CSpell dictionary? For example oembed could be in the SoftwareTerms dictionary.

+++ b/core/package.json
@@ -21,8 +21,9 @@
+    "spellcheck:make-drupal-dict": "rm -f misc/cspell/drupal.txt && touch misc/cspell/drupal.txt && yarn -s spellcheck:core --unique --words-only | perl -Mopen=locale -pe '$_=lc$_' | LC_ALL=en_US.UTF-8 tr -d \\\\\\\\ | LC_ALL=C sort -u -o misc/cspell/drupal.txt",

I don't think we can have an automatic drupal dictionary maker. When we add a word to this dictionary it should be an active choice by the community via an issue on drupal.org.

+++ b/core/misc/cspell/dictionary.txt
@@ -590,7 +584,6 @@ laatste
 langcode
-langcodes

Why add langcodes to drupal.txt and not langcode?

quietone’s picture

Status: Needs work » Needs review
StatusFileSize
new2.9 KB
new4.6 KB

The patch needed a reroll.

Also, per #10 the new dictionary is now named, drupal-dictionary.txt
#27.1. Having the community validate adding words is a good point. And that should still happen if the command is used to generate the drupal dictionary. But, we also need a way to validate the dictionary and update it when words are removed. For the later reason, I have left the command in.
#27.2 Added langcode. I don't know why it was not included.

smustgrave’s picture

Status: Needs review » Reviewed & tested by the community

Reroll seems good. Going to mark but question, what will be the criteria for "drupal specific" words. Wouldn't works like lazybuilder or drupalorg qualify?

alexpott’s picture

Status: Reviewed & tested by the community » Needs work

Done another round of thinking and I'm not that sure about what I said in #27 about this change being a good idea. The reason being once we've fixed all the misspellings that are in dictionary.txt - what is the point of drupal-dictionary.txt? And maintaining the extra command package.json all makes me pause.

That said maybe it doesn't matter and it is just a case of taste. I'm not really sure that having this extra dictionary is going to speed up spelling fixes though. I'd guess that it will introduce a bigger chance of a bike shed about where a word belongs and for that reason I'm changing my opinion in #27.

+++ b/core/package.json
@@ -21,8 +21,9 @@
-    "spellcheck:make-drupal-dict": "rm -f misc/cspell/dictionary.txt && touch misc/cspell/dictionary.txt && yarn -s spellcheck:core --unique --words-only | perl -Mopen=locale -pe '$_=lc$_' | LC_ALL=en_US.UTF-8 tr -d \\\\\\\\ | LC_ALL=C sort -u -o misc/cspell/dictionary.txt",
-    "spellcheck:core": "cspell -c .cspell.json --root .. \"core/**/*\" \"composer/**/*\" \"composer.json\" \".gitlab-ci/*\" \".gitlab-ci.yml\"",
+    "spellcheck:core": "cspell -c .cspell.json --no-progress --root .. \"core/**/*\" \"composer/**/*\" \"composer.json\"",
+    "spellcheck:make-dict": "rm -f misc/cspell/dictionary.txt && touch misc/cspell/dictionary.txt && yarn -s spellcheck:core --unique --words-only | perl -Mopen=locale -pe '$_=lc$_' | LC_ALL=en_US.UTF-8 tr -d \\\\\\\\ | LC_ALL=C sort -u -o misc/cspell/dictionary.txt",
+    "spellcheck:make-drupal-dict": "rm -f misc/cspell/drupal-dictionary.txt && touch misc/cspell/drupal-dictionary.txt && yarn -s spellcheck:core --unique --words-only | perl -Mopen=locale -pe '$_=lc$_' | LC_ALL=en_US.UTF-8 tr -d \\\\\\\\ | LC_ALL=C sort -u -o misc/cspell/drupal-dictionary.txt",

The spellcheck:core command should not be changing.

catch’s picture

I think this will help to remove mis-spellings, because it'll stop people trying to remove words that we're going to keep. Maybe we could add it, then move it back to dictionary.txt when that file is empty?

quietone’s picture

Status: Needs work » Needs review
StatusFileSize
new1.35 KB
new4.48 KB

Yes, #31 is a good idea. This is definitely about making it easier to work on spelling issues. And once dictionary.txt is empty we can review the remaining words.

And since the 'drupal' dictionary is an interim step, we can add more words that are not strictly Drupalisms. This could include a variety of technical terms and perhaps the non English ones needed for testing. Some of the technical terms may need an issue upstream but it will still help if they were moved out of dictionary.txt so we can focus on the misspellings.

smustgrave’s picture

Status: Needs review » Reviewed & tested by the community

Changes to spellcheck:core have been reverted.

alexpott’s picture

Status: Reviewed & tested by the community » Needs work

The patch does not apply.

quietone’s picture

Status: Needs work » Needs review
StatusFileSize
new1.98 KB
new4.39 KB

Simple reroll.

longwave’s picture

  1. +++ b/core/misc/cspell/drupal-dictionary.txt
    @@ -0,0 +1,13 @@
    +jsonapi
    

    "jsonapi" is apparently in the cspell dictionary now as it has already been removed from dictionary.txt.

  2. +++ b/core/package.json
    @@ -21,8 +21,9 @@
    +    "spellcheck:make-drupal-dict": "rm -f misc/cspell/drupal-dictionary.txt && touch misc/cspell/drupal-dictionary.txt && yarn -s spellcheck:core --unique --words-only | perl -Mopen=locale -pe '$_=lc$_' | LC_ALL=en_US.UTF-8 tr -d \\\\\\\\ | LC_ALL=C sort -u -o misc/cspell/drupal-dictionary.txt",
    

    Does this actually make sense, or do we want to maintain drupal-dictionary.txt by hand?

edit: I see I missed @alexpott making the same point as #2, but I'm also not convinced that we need a command for it.

longwave’s picture

Status: Needs review » Needs work
quietone’s picture

Status: Needs work » Needs review
StatusFileSize
new263 bytes
new4.38 KB

I thought I left an explanation for this but I see I did not. A reason to keep that new command is to remove words like you found in #36.1. And we expect this list to grow so it will be easier to have some way to remove words that are added by any of cspells dictionaries. In fact, I used the new command to regenerate the new drupal dictionary to make this patch.

smustgrave’s picture

Status: Needs review » Reviewed & tested by the community

Reroll and json removal seems good.

xjm’s picture

Title: Add a dictionary for Drupal specific words » Add a dictionary for Drupal-specific words
Status: Reviewed & tested by the community » Needs work

This already doesn't apply again.

Once this is rerolled, I will prioritize this over other cspell issues since I think this new (and possibly temporary) dictionary will accelerate cleaning up the main dictionary. (Terms like "cacheable" and "uncacheable" that Drupal uses widely, for example, which get swept up in other scopes but are probably keepers). I think additional words should be discussed in followup issues, so the scope here is fine.

I think this is a 100% valid choice and everything in the patch (including the dictionary and command changes) seems correct and in scope to me.

Many Drupalisms should be made more global by converting them to correctly spelled and formatted English words, but others need to be kept. I suggest very stringent review before adding terms to the Drupal dictionary. E.g., "mlid" would perhaps be better converted to a snake_case_key in a perfect world, and we just need to weigh the tradeoffs of data model changes for things like that against keeping them. Whereas other terms will always be used.

smustgrave’s picture

Status: Needs work » Needs review
StatusFileSize
new1.15 KB
new4.48 KB

Rerolled and reran the dictionary scripts

needs-review-queue-bot’s picture

Status: Needs review » Needs work

The Needs Review Queue Bot tested this issue.

While you are making the above changes, we recommend that you convert this patch to a merge request. Merge requests are preferred over patches. Be sure to hide the old patch files as well. (Converting an issue to a merge request without other contributions to the issue will not receive credit.)

xjm’s picture

Status: Needs work » Reviewed & tested by the community

womp womp, randoms. Also we will want to commit this before the next bot run so the patch is fine for now.

xjm’s picture

xjm’s picture

  • xjm committed a10e0f8d on 11.x
    Issue #3328741 by quietone, smustgrave, xjm, catch, alexpott, longwave:...

  • xjm committed f48bdfcd on 10.2.x
    Issue #3328741 by quietone, smustgrave, xjm, catch, alexpott, longwave:...
xjm’s picture

Version: 11.x-dev » 10.2.x-dev
Status: Reviewed & tested by the community » Fixed

Committed to 11.x and 10.2.x. I also tried to backport it to 10.1.x, since it is a minimally disruptive coding standards tooling improvement, but it did not cherry-pick cleanly. Either the 10.1 codebase is sufficiently different from 11.x to cause non-mergeable changes in the dictionary, or we forgot to backport other CSpell issues.

I don't think a different backport version is worth the effort, though, so marking fixed. Thanks!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

dave reid’s picture

As a project that has been re-using core's dictonary.txt, which file should we be referencing going forwards as the "good list"? Because normal Drupal spellings like itok are not present in the drupal-dictonary despite being used as a query parameter for image generation as one example.

quietone’s picture

@Dave Reid, If you think 'itok' should be in the Drupal dictionary, then can you make a sibling of this issue to discuss that?

Thanks.

dave reid’s picture

I guess I'm just asking for consistency in recognizing that other projects might be re-using the dictionary file. If the idea is to slowly verify misspellings in the existing dictionary file, could those have been moved to a dictionary-unverified.txt file instead?