Written confirmation

This all makes sense, and I like it. I have some questions please.

(A) We need to have 2 templates, one for immediate confirmation, one for long-after confirmation.

c. There should be 2 kinds of confirmation event, one for immediate confirmation and one for retrospective confirmation.

I get your point and I see there are different cases. However I am concerned that we are creating dualism here and ignoring the middle way. In reality we will not get only "immediate" or "years ago" but also any point in between. So I am more inclined towards a single event and single template, with "time ago" as an available variable. We can put an if test on "time ago" in the default template to generate the same behaviour of 2 different wordings. However we get more flexibility: another site could easily alter the threshold of the if test or even split into 3 different wordings. We also get simplicity, and reduced development time.

Probably we can use twig templates as the defult text for the notifications. What Commerce does here with emails is likely a good pattern to copy.

This area is something of a speciality for me😃 due to DSM+. Commerce is an OK pattern but it could be improved. They use templates, but put all the CSS in the template. They create their own framework for sending mails (and so does simplenews and likely so does XYZ etc). They can generate HTML emails, but only through the "legacy" Core interface which doesn't formally support HTML.

Some options:

1. Symfony mailer in core is coming, but slowly - it's not even experimental yet. So that's no use.
2. We could copy some of the ideas from Commerce's framework: a system of templates, some features for translations, a few other bits.
3. We could use DSM+, which would be by far the fastest, simplest and most customisable way to do what we want. It has a much more powerful framework the Commerce, and use the symfony native HTML mail library directly. However currently DSM+ configures email transports entirely separately from Core which creates a problem for sites still on the Core transports. This option could work if DSM+ was enhanced to support fallback to the Core transports, which actually is likely reasonable easy because it already has most of the required conversion code. If you are open to this possibility then I could do some more investigation of the cost - it might be comparable to the previous option overall.

Please let me know how you feel about it...

Perhaps the required email text might vary for each of the 3 scenarios A,B,C in the IS. Indeed when you talk about immediate/retrospective, then this might actually be another aspect of the cause - immediate meaning caused by entry of a new declaration, and retrospective meaning any of the other cases.

We can add a variable to the template that indicates the cause/trigger of the confirmation. A single event and single template also seems to support this direction.

d. Have DeclarationForm dispatch the event post-save if the validity is NEEDS_CONFIRMATION and the confirmation_date field is blank.

The full decision whether to send a confirmation is likely to be complex, affected by these two factors and others

1. validity === NEEDS_CONFIRMATION
2. empty(confirmation_date)
3. maybe also changed_date > confirmation_date which relates to parts of case C: data entry error
4. maybe isNew() || has just changed a critical field else we keep sending an event endlessly even though an event subscriber is suppressing sending a confirmation (for example we don't have any contact details?) - but then maybe we need this event in case we do now have contact details
5. input from the admin: likely we need a checkbox on the form "send confirmation", which we can default based on the above which allows case B and parts of C

Case A is likely covered anyway by the first bullet validity clause, because without an explanation it should be marked as needs confirmation.

Great thanks, that seems like enough of a plan for me to start coding. I suspect I will have more detailed questions later....

It's now roughly working. I made quick progress: each of the task items corresponded to a neat isolated file or function; our design seems easy to work with.

I added some "Remaining tasks" to the IS for what I've not done yet.

I raised #3568837: Create a UI to allow staff to override whether to send a confirmation.

Code review changes done

Thanks #14 was very useful. I converted it to working code. If you try editing the policy you can see that it informs about the available variables in the "replacement patterns" section.

The generality of ConfirmationMailer to scenarios other than User account it somewhat tenuous. For the time being I kept it theoretically alive by adding two extra URL parameters.

The UserEmailConfirmation class only works properly with the 2 routes from the gift_aid_user module, so probably we should move it into that module. What do you say?

The self cancellation form still needs work: in particular it lacks any idea of retrospective cancellation.

I feel that the key here is simplicity. A confirmation is a very specific thing, which

• describes Gift Aid status including dates and charity name
• explains the "responsibility to pay any difference"
• gives a right of retrospective cancellation for 30 days and explains this right
• is tracked for audit purposes

sites may wish to send confirmations in some or all cases even where they don't strictly need to, simply to minimise audit-related risk.

The staff can simply set validity to "Valid if confirmed" for whatever cases they wish to send confirmations. We should follow the exact same logic as any mandatory confirmation, else our confirmation would not be valid.

Sites may in some circumstances wish to resend confirmations, for example if they discover mistakes in the text of past confirmations, in cases of data entry error, or updated donor contact information.

Likewise, I propose that the resend should follow identical logic. The first confirmation could easily be unacceptable, so we'd better ensure the second one is valid.

written confirmation is also an acceptable way of providing the required 'explanation' to a donor (guidance 3.6.2 says "explanation can be included on a Gift Aid declaration but can also be made separately").

This case is once again no different from the others. Surely once the Donor sees the explanation, they have a right to change their mind e.g. maybe they realise they don't pay enough tax.

(potentially long after the declaration created) validating any declaration that lacks sufficient evidence

Presumably this case still has to allow retrospective cancellation. The Donor may state they never intended to agree to Gift Aid, or that they weren't informed of the tax implications. From the perspective of this issue, it's identical logic once more. (From an admin perspective it may mean contacting HMRC to "un-claim" Gift Aid, but the "reporting" side of things is a future work item.)

===

We could imagine an "acknowledgement" - a communication sent describing Gift Aid status, but without any right to retrospective cancellation. AFAICS the Gift Aid rules don't ask for it, and care nothing about it - it would have no effect on validity, need not be tracked or evidenced, and could be worded however you like. I propose we treat that as out-of-scope of this issue.

So if you agree, it seems that we can remove the {% if not valid without confirmation %} and {% if recent %} tests from the email template, and instead show the same text to everyone.

But for validation of invalid declarations they say only:

Very interesting. But I'm not convinced😃. An alternative interpretation is that the wording difference is more accidental, not implying the significant difference you propose. My objection to your idea is that we would end up with a validation process which has no possibility of invalidation. In particular, let's say that (in 2026) the auditor insists that a declaration from 2024 isn't valid without a confirmation. If the donor responds (in 2026) with a cancellation, then I hardly see that the auditor will be impressed with your theory that the cancellation only affects future donations (2026 onwards), and that your claim from 2024 should stand.

Another consequence of your new theory is that it contradicts a fundamental premise of the IS:

Although HMRC discusses these issues separately they are in fact exactly the same in their underlying logic.

If we drop that premise, then it would vastly increase the complexity of this whole issue because we would need to distinguish the two different cases (confirmation and validation) throughout. It would mean that a statement email does not meet the conditions of a confirmation email. It would mean that DECLARATION_VALID_IF_CONFIRMED needs to be supplemented with DECLARATION_VALID_IF_STATEMENTED, isConfirmationMissing() is different from isStatementMissing(), and this would continue in very many places.

I suggest just using whether the declared date was within 30 days

The time-based dualism is trying to re-emerge😃. This seems much too simplistic. What if an oral declaration sat in a staff backlog for a month before data entry? Or if there was an oral declaration that was confirmed, then staff discovered it was sent to the wrong address and decided to resend? Or if a declaration that you are uncertain is bulletproof is also oral?

Perhaps we should have a call to discuss it??

FYI I have committed my changes from #17 for now so that I can continue to make progress.

I can reinstate something later if you do require it, but as above, I believe it would need to be much more complex than the previous code, which therefore is not useful to keep.

We shouldn't assume at the API level that a cancellation within 30 days of confirmation being sent is a retrospective retraction of the declaration

Good idea. On the cancellation form we can have two potential options: retrospective or future. Retrospective is only available if we have sent them a confirmation message within the last 30 days related to any one of their declarations. Future is only shown if the declaration is claimable (else there's no point in showing it, as the effect is the same as retrospective), which occurs after 7 days. If both options are available we show a 2-item select form element, each describing what will happen. Otherwise we can just show the text that would be on the one option as an ordinary paragraph.

here however is that when dealing with written confirmations done years after the declaration was made, the donor might wish instead to communicate something like "yeah, that declaration is fine when I made it 4 years ago, but actually I stopped being a taxpayer last year".

Yes, however it's not only "years after" though, any time above 7 days would be enough. And if they want the "last year" part they would need to contact a person as we only have the two options above.

I feel that #18 is an interesting idea, but is a non-standard process, which we can defer to another issue. I get the point, in effect you are being creative and have proposed a procedure to gather more evidence to support dubious paperwork. However let's call it something different from confirmation/validation/statement which have a formal precise meaning.

I propose that we consider confirmations/statements to be identical as per the IS. When we wish to formally validate a declaration as per the guidance (perhaps even on the authority of the auditor), then we should allow retrospective cancellation. If we don't, then we have

• non-conformant data reporting: validations which have been (non-retrospectively) cancelled within 30 days are likely not acceptable, yet the reporting UI would mark the prior declaration as valid
• a complicated implementation mess😃

How does that sound??

I raised #3570296: Improve self-cancellation UI.

In relation to that, one more thought regarding #18...

The starting point is a pile of declarations that are dubious. Likely most are correct, but some are wrong. Let's say there are 10000 claims including 100 who will cancel, of which 20 will do it retrospectively.

The goal I feel most attracted to is bringing clarity and accuracy; to be honest, and minimise audit risk. This suggests to increase the evidence for the correct declarations; to identify and withdraw the wrong ones. #3570296: Improve self-cancellation UI is very helpful in this case: it allows retroaction, but discourages it unless necessary. We have to withdraw 20. The audit is a piece of cake.

With the alternative strategy of "Anything to avoid the hassle of refunding HMRC", we end up with 100 cancellations (no option for retrospective), yet still claim for all 10000. In fact 20 of them are wrong, but we don't know which. It's possible that HMRC central computer will notice there is insufficient tax paid, and demand the money from the donor who then complains that he already cancelled. It's possible that the HMRC auditor will insist that we withdraw all 100 claims where the Donor cancelled. They might even be rather grumpy that we behaved in this way.

Maybe I'm too idealistic, it's not realistic to be so pure in the real world, and the auditors are sympathetic. In which case fine, we can have #18 as a separate issue. However I still believe it needs to be separate from confirmation/written statement. We could call it a notification. It could be applied (as an action?) to any declaration with validity "inherently valid". It grants no additional right to cancel beyond what someone already has. The process would make no change to any declaration field (in particular not confirmed date, confirmation required, validity that are altered by a true validation), other than to set some additional evidence.

Before I was resisting cron, as I felt it was complicated. Now I becoming attracted to it - true it takes a little code to set up, but once we have it we can eliminate various other difficulties.

• We create a true database field for "status" or maybe better call it "state", which is therefore highly efficient for database queries. This would provide a simple accessor to replace the current getState() function, which is fairly complex and relies on many related isXXX(). Before we had a computed field, but there were concerns over caching (or over performance if we disable caching).
• We could use state_machine module for this, which Commerce already requires.
• We use cron to trigger the state transitions that are date based: stability window, confirmation window.
• We could also potentially use cron to retry failed email sending.
• Event handlers can register for transitions. For example a site could register for a declaration reaching its end date, and send a reminder email to the donor.

How do you feel about that??

FYI Guidance "3.8.1 Invalid declarations" says

Where the donor cancels their declaration within 30 days, the declaration will be treated as if it was never made. The charity must maintain an auditable record of all written statements and cancellation notices.

I've gone about as far as I can now until we can discuss some of the decisions

• I raised #3570885: Declaration life cycle and it includes your open comment regarding isLocked() and confirmation emails.
• From before we have #3570296: Improve self-cancellation UI.

I feel that we could reasonably commit this issue as is or with minor comments, then continue the more complex cases in the other issues. This issue is now one of the biggest/most complex we've had.

Thanks Jonathan. Good, I believe we are broadly in agreement, although I would tend to express it more simply.

I'm afraid that I still don't buy your distinction of "recent/immediate" vs "retrospective" confirmation.

• I don't see how we can write code to express it, i.e. where will you draw the boundary line? Sure a confirmation email could often be truly immediate, within seconds. But maybe we had the wrong email address initially, and that is discovered after a week, or 2 or 3, or 23? And what about a posted paper confirmation? The staff might take a week to do it, or be on holiday for 2-3 weeks, or find it at the bottom of their tray 17 weeks later.
• I see nothing in the guidelines that relates to it, or describes any difference in behaviour depending on time ago.

However I do agree with you about the distinct validity states, which we have created based on the HMRC guidelines.

• "Inherently valid" means that it meets all the specified conditions. There is no need to offer retrospective cancellation, and we strongly desire not to. We could send a "notification" if we wish (without offering retrospective cancellation), but that won't help if the declaration is later corrected to "Valid if confirmed".
• "Valid if confirmed" means that it unfortunately doesn't meet the conditions. There are multiple possible cases: it is an oral declaration; we didn't yet give the 'explanation'; we don't have adequate evidence; maybe others. The cases can come from 3.8.1 "written confirmation" or 3.10.1 "written record" however it makes little difference as the wording is almost identical. In all cases, we are invited to send a confirmation/statement that includes an offer of retrospective cancellation for 30 days. If the offer is not taken, then we can treat the declaration as valid. If we don't send a message, or we send one without offering retrospective, or there is a cancellation, the we must treat the declaration as invalid, as if never made.

So I am in agreement with your "3 scenarios exactly the same" - although I only see them as 1 scenario. And I am in agreement to rule the 4th scenario out of scope - although I don't see it as "a point of debate above", I see that the guidelines 100% clearly rule it out.

Anyway we are in clear agreement how to move forward, so that is excellent and we don't need to debate further.

All comments have been done. Merge is now blocked waiting for approval.

Thanks.

I believe I might have reached a new understanding of what you are saying in #29. If we could reword "recent/immediate" as "not yet claimed", and change "retrospective" to "already claimed" then it makes sense to me.

Then the 4 states, could we understand it like this: We will allow the staff to trigger an extra confirmation if either validity == "valid if confirmed" or !hasClaimed() (which covers 3 states). In the 4th state - which has a perfectly valid declaration that already has claims to HMRC - then we really don't want to mess with it, so we don't accept an extra confirmation (or perhaps only from admin). Anyway this all feeds into #3568837: Create a UI to allow staff to override whether to send a confirmation.