Problem/Motivation

If you look at the latest run https://git.drupalcode.org/project/simplenews/-/pipelines/901562, you will notice that there are failures across PHPCS, PHPStan, Cspell, and ESlint. Several of these are DI issues which make making effective automated tests difficult which is the first step towards code instability as our early warning system becomes stale.

Steps to reproduce

Proposed resolution

Run through the following attached logs and create fixes.

Remaining tasks

#3613461: How to handle DI and affects to code (backwards incompatible breaking changes)
#3613453: Fix Spelling issues defined by the pipeline
#3613455: Fix or ignore rules for PHPCS and ESLint checks

User interface changes

API changes

Data model changes

CommentFileSizeAuthor
pipeline901562-artifacts.zip16.22 KBgeneralredneck

Issue fork simplenews-3613221

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

generalredneck created an issue. See original summary.

generalredneck’s picture

Assigned: Unassigned » generalredneck

I'm taking a first stab at this but it's a BIG change.

adamps’s picture

Thanks.

I am fine with reducing the number of warnings that always occur. However this is an old and complex codebase, and much of the code came before the warnings were invented. We don't want to create instability and introduce bugs. Adding DI is a good example - it creates lots of boiler-plate code and it can be non-BC for the constructor changes. Also some of the standards (especially long lines - surely any modern monitor is way wider than 80 characters) TBH I find somewhat petty and unnecessary.

In many cases, I propose that the right answer is to ignore some of the rules entirely in our phpstan config file. As a first step, it would be good to agree which rules to fix and which to ignore.

I suggest wait until after I finish my current batch of maintainer work and make a new release. This one will touch 100s of lines and be horrible to merge with other commits going in.

generalredneck’s picture

Status: Active » Needs work

In light of what you said in your last comment, I made this a "draft" I feel like a lot of the work is valid and will be able to be rebased. I didn't want to lose it. Marking this as needs work for after a release. Here are my notes working through the codebase.

PHPStan & DI

Dependency injection (the bulk of it): refactored ~25 classes to inject services instead of calling \Drupal::… recipient handler plugins, forms (Confirm*, RequestHash, SubscriptionsBlock, etc.), controllers, actions, the views field/mail plugins, and the constraint validator. The trickiest part was Mailer/SpoolStorage/MailEntity/SpoolList, which all built MailEntity objects by hand with \Drupal::service() calls. I introduced a MailEntityFactory service to hold those dependencies centrally, and tried to break a circular dependency between (SpoolStorage ↔ Mailer) using setter injection (setMailer()) registered via a service calls: entry.

Bugs fixed while refactoring:

SubscriptionSettingsForm: removed a leftover duplicate #default_value key and reference to an undefined $database variable.
NewsletterForm::save() and Mailer::sendSpool()/sendSubscribeConfirmation(): added missing return statements/values per their declared contracts.
SubscriberHistory::t(): added StringTranslationTrait (the method didn't exist).
simplenews_entity_operation(): updated for Drupal 11.3's new CacheableMetadata parameter.
SimplenewsTestSubscriptionOutput.php was renamed to SimplenewsSubscriptionOutputTest.php: the old filename didn't match its class name pattern, so PHPUnit wasn't even discovering this test.
Fixed several class.nameCase typos (recipientHandler became RecipientHandler).
Deprecated API replacements: user_load_by_mail/user_load_by_name, user_pass_reset_url, user_pass_rehash, check_markup, field_purge_batch, filter_fallback_format, views_field_default_views_data, Renderer::renderPlain().

simplenews.drush.inc (5 errors): legacy Drush 8 functions. Going to tell PHPStan to ignore it.
Three D7 migrate source classes (6 errors): extend DrupalSqlBase, which core deprecated with no replacement. I'm going to tell PHPStan to ignore it too. We can decide if we want to keep these after a Drupal 12 release.

Fixing Monitor + ensuring tests run

Monitoring integration fixed: PendingSensorPlugin now uses the monitoring module's own $this->getService() convention instead of \Drupal::service(). Validated with SimplenewsMonitoringTest (passes) and confirmed clean in phpstan.

The SpoolStorage to Mailer circular-dependency fix I made earlier was itself broken. I'd assumed Drupal's DI container could resolve a constructor/setter circular reference the way a compiled Symfony ContainerBuilder can. It can't… Drupal ships its own lightweight runtime container that throws ServiceCircularReferenceException whenever neither side happens to already be cached. This silently killed cron-based mail sending (caught and logged by Drupal's per-hook cron exception handler, so it failed quietly rather than crashing). I reverted that one call site to the original \Drupal::service('simplenews.mailer'), documented in a comment why it's a legitimate, unavoidable exception, and confirmed via a debug trace that this really is the only way to break the cycle given Drupal's container implementation.

I also fixed a regression in SimplenewsSubscriberBulkFormTest.php where removing the flagged $this->storage property left two other usages in the same test dangling.

Verified via phpunit tests: SimplenewsMonitoringTest, the full simplenews Kernel suite (43 tests), SimplenewsAdministrationTest + SimplenewsPersonalizationFormsTest (13 tests), SimplenewsSubscribeTest (13 tests), and the bulk form test. All are green now, zero failures (only pre-existing deprecation noise from annotation-based plugin discovery, unrelated to this work). phpstan is at 12 remaining findings, all genuinely external (Drush 8 legacy functions, D7 migrate source deprecation with no replacement) or a documented, unavoidable DI exception.

Made PHPStan ignore the drush & migrate errors. Added an ignore line via code to the one place where DI is not possible, documenting in the code why.

PHPCS

adamps made a comment about how there are some rules that seem petty. It seems there was already a start to excluding rules before, so I added a couple. Specifically in the array declaration and long comment areas.

Ignored the "Possible useless method overriding detected" in the multilingual tests.

CSpell

Added our own .cspell.json. Since several of the leading developers are from Europe, it only makes sense that we add the en-gb as correct spelling. I learned a lot today about the differences in some words. That said, it's not exactly proper localization… Some of the messages that print out to the user are in en-gb… but I'm not going to gripe when UK natives have to put up with en-us BS on the regular. 😅 just pointing it out.
I didn't add Spanish as a language because there are only a few words in tests and I expect that we don't necessarily want the code to read in Spanish.

ESLint

3 optional configs have empty "content:" keys. We are just going to add { } to the mix to make ESLint be quiet. This should be functionally the same when imported. No need for an entity update either. It will be fixed if anyone imported that configuration and then hit save on any of the display modes and then they exported. Again they behave identically.

adamps’s picture

Thanks for putting your time into this. As a general summary of my comments, my job as maintainer is to keep the code stable (avoid bugs and unnecessary changes), and simple (minimise adding extra lines that have to be maintained for ever after). The patch file is currently 117kB, and adds around 1000 lines net. I feel that's about 10 times too large😃.

1) I feel the biggest difficultly is with the DI, as already mentioned in #3. Changing constructors is non-BC, in the case that a site has replaced the default class with a sub-class that overrides some functions. Also it adds a lot of code for a fairly abstract benefit - there aren't people raising issues saying "I would really like to do write some test code that injects a dummy service into this class but I'm so sad that I can't because there isn't DI". Finally, IMHO DI in Drupal isn't efficiently done - over in Mailer Plus I adopted some new styles that I would say have some clear advantages. I'm inclined to say this one is a no. Of course where the service is already injected (perhaps into a parent) and we aren't using it, then yes of course let's change that.

If you want to save your changes on a postponed issue "add DI" then I'm OK with that.

2) As you mentioned, this is a big change. Therefore I feel we should keep strictly on topic of CS fixes. You mentioned that you have fixed various bugs in passing. I also see some deprecated code fixes for D12/13 compatibility. I feel they should each go into separate issues. We need to be careful to avoid introducing bugs, and have a clear audit trail for anyone who finds a bug to try and work out why/where it comes from. I would like to reach the point where this MR, although big, consists only of trivially safe changes that are about the coding standard, i.e. how the code is expressed in terms of characters.

3) I don't necessarily see the need to change UK spelling to US😃.

4) I agree with your idea to ignore warnings in some deprecated files. Although for the Drush 8 file we could just retire it now (in a separate issue)?? Currently you have listed the warnings to ignore in specific and lengthy detail, leading to a complex file for all of us to work with. Could we instead just ignore the entire file? New warnings may appear, especially for deprecated functions. Also when ignoring, I suggest to use "identifier" not "message" where possible.

5) I don't understand why we have 2 files for spelling. If we wish to use a default file from somewhere else, why would we copy it, leading the case where every module is separately maintaining a slightly different file? It says the file is based on core so why not just use that? I feel this project should only maintain a list of words that are strictly relevant specifically to it.

generalredneck’s picture

I think the the way forward is to remove the DI based standards then because even though your code ase is set to check for them you have stated no interest in moving toward the fairly abstract benefit. Following is a detailed reasoning for this thought.

I see your point on the replacing a class in custom modules and with 23K installs (per the project page) that's a possibility. Maybe we consider this on a new version if you ever have any interest in going that direction. Specifically maybe when you do finally remove old code

The challenge here is you're going to get people who are trying to contribute and follow the rules that the Drupal community has set forth and you have continuous integration that is enforcing those. They are trying to make that continuous integration. Happy as to get a better chance of getting their code in and or make the code better where they can.

Instead, your continuous integration is currently causing several of the issues on this project to be postponed because people then have to go back and rework their solutions.

You did mention that ignoring some of the rules specifically comment lines was something you were interested in. You also mentioned that di causes some challenges, but you did not suggest ignoring those rules.

My question to you is how would you like to proceed going forward in this particular portion given your strong messaging around stability.

I'm willing to do the work and several of your thoughts above have alternative ways forward, but that one major sticking point that is affecting several of the issues is a challenge we need to address so that you're not doing this all by yourself or ignoring the module as a whole because people can't contribute in the correct manner.

adamps’s picture

Although you say "you" have this "you" did that (meaning me😃), the current situation of the code base is not as it is because of my conscious choice. The current state is the result of 20 years of evolution, and I don't feel any blame for it personally. This is hardly the only contrib module that has a load of warnings sitting there. The people who maintain the jobs that generate warnings keep adding new rules so that's the main cause of it - probably there were zero when this project was first developed. If anyone wishes to improve this module then they can raise an issue.

Perhaps I am old fashioned however my points about stability are simply the way I was taught to develop software, and my view of the right way to develop in a codebase with a large pool of contributors. It doesn't necessarily take any more time, it's just a bit of discipline, and it hopefully even saves time in the long run by cutting out fixes by one developer that turn out to cause problems for the rest.

I don't really have anything to add above my previous comments. I agree with the goal of reducing or even eliminating the warnings. I am currently giving my time right now to support your initiative to make it happen. We can fix some, and ignore others. For sure we can ignore the DI rules, I never had any problem with that. I suggested in #3 that we start by agreeing which ones were in which category😃. Well hopefully the discussion in later comments starts to develop a consensus.

Anyway, just to try and set your expectations realistically, I would like to clarify that I have used up my maintainer time on this module for now. I committed 9 issues and now I'll make a new release. I'll likely be back in about 6 months. Like any maintainer, I try to prioritise to commit the issues that are in a committable state without needing loads of discussion time. This module is gradually declining in usage, it's minimally maintained, but not abandoned. I hope that we can commit this one next time round, thanks for your contribution.

adamps’s picture

I can see that this could help other developers so if you are still interested to do it then I would aim to make some special time for it sooner. Let's keep it simple to avoid breaks like #3613373: Too few arguments to function Drupal\simplenews\Plugin\Field\FieldWidget\SubscriptionWidget::__construct()😃.

generalredneck’s picture

Adding related tasks to break this up.

I think that we can handle a few of these really easily. I need to work through your feedback on them but it shouldn't take too long. It will at least get us closer.

generalredneck’s picture

generalredneck’s picture

@adamps,

I know it's a lot of little things for review now. I hope it's easier. I honestly think that if we review them in this order:

  1. #3613453: Fix Spelling issues defined by the pipeline
  2. #3613455: Fix or ignore rules for PHPCS and ESLint checks
  3. #3613474: Retire drush.inc command file
  4. #3613483: Disable phpstan warnings that aren't suitable for this module
  5. #3613485: Fix Legit PHPStan issues

It will make it easiest. Then from there if I have time in the future I can look at the rest of the stuff moving forward.

I've got to get back to the my farm as I make contribution time in between harvesting, managing birds, and running the business. Also I worked 20 years in Web Development... most of it in Drupal Agencies, so I get the "I only have X hours of maintainer time." I too am maintainer of many many modules... just none of them this size. Was trying to make some good use of it's too hot to be outside time.

adamps’s picture

Great many thanks for your help

adamps’s picture

Just the deprecations left now. Most of those are D13 anyway - in which case I don't mind whether we fix them now or not.

Once we're done, I'd like to add this to our phpstan.neon, so that our tests will stay green. They should have finished adding D12 ones now.

    # We've already fixed all D12 deprecations. We're not concerned with D13
    # deprecations yet, and they will keep appearing. We'll deal with them
    # closer to D13 availability.
    - identifier: function.deprecated
generalredneck’s picture

Yeah... The main one that concerns me is this one

------ ---------------------------------------------------------------------- 
  Line   simplenews.module                                                     
 ------ ---------------------------------------------------------------------- 
  185    Function simplenews_entity_operation() implements                     
         hook_entity_operation but is missing the CacheableMetadata parameter  
         added in Drupal 11.3. Update the signature to:                        
         simplenews_entity_operation(\Drupal\Core\Entity\EntityInterface       
         $entity, \Drupal\Core\Cache\CacheableMetadata $cacheability).         
         🪪  drupal.proceduralHookEntityOperationMissingCacheabilityParameter  
         💡  See https://www.drupal.org/node/3533080

I'm assuming we need to allow $cacheablity to be NULL by default and that should make it work for older versions right? Guess I should make this it's own ticket.

The d12 ones should be fixed by a project bot ticket unless we need to creat a new one for that... And then lastly the ignore d13 deprecations.

adamps’s picture

The project update bot has produced a huge patch aiming to fix D13 deprecations while still supporting D10 - way too complex😃. We only need to fix the D12 ones, which AFAICS is just 2 cases. Then we can suppress the warning as discussed.

The simplenews_entity_operation() you are right we could do with NULL, and then an if test inside the function. Then later we could in theory remove that code. However I feel we could wait until we stop supporting D10, then do it the simple way. We could suppress drupal.proceduralHookEntityOperationMissingCacheabilityParameter for now with a note to come back to it later, same as the above.

I'd be happy for it all to be finished on a new fork/MR here.

adamps’s picture

Or we could move to 4.2.x, drop support for D10, requiring D11.3. Then we can fix simplenews_entity_operation() cleanly.

adamps’s picture

Or we could just leave 1 warning remaining😃

generalredneck’s picture

Hey there,
Sorry I was sick yesterday and today I'm making up for it and also got a 6 hour drive to go get feed. I'm going to have to table this until this weekend unless I get a few minutes. Let me know how you would like to handle this l. I personally like the version bump as traditionally that's how modules handle major core upgrades.

adamps’s picture

Wow fancy having to drive 6 hours to visit the store! It's OK, please take your time to get well and catch up, whenever it fits in.

Version bump is fine by me, then we don't need any =NULL it's nice an simple. D11.2 is unsupported so I reckon we can say ^11.3 which seems to be sufficient in our case. Watch out there are about 5 separate places to change it. We have a 4.x branch which can stay the same for all minor version changes so no change is needed there (if i need to backport a security fix I will make a 4.1.x, but it's not been needed yet.) I just need to remember that the next tag must be 4.2.0....

Thanks again