SMS message FIFO dispatch queue

I intended to float the idea of doing the same thing as Courier:

Now working on this since #2677684: Add SMS message entity / storage Assigned to: dpi is committed. We can continue to change the SMS message entity as required.

There are some messages in #2677684: Add SMS message entity / storage which are better suited to this issue:

• Message retention and skip queue. Comment 12
• Gateway annotation: Gateway supports multiple recipients in a single request. Comment 12 | Better proposal in Comment 14

I should have this feature complete by the end of the weekend. Its all working as is, ready for testing.

@todo:

• the multiple recipients per gateway plugin annotation feature.
• No tests yet.
• Needs some general cleanup.
• Still undecided on how to approach the provider. Add a new 'queue' method (current PR implementation). Or use the 'send' method, and rename the current send method to something else.

PR is here https://github.com/dpi/smsframework/pull/11
Diff is here https://github.com/dpi/smsframework/pull/11/files

PR is mostly internals changes, there are two UI additions:

Adding skip queue and retention settings to gateways:

New report line items as suggested in #6:

Ive pushed the commits with max recipients-per-message chunking feature.

Would like your opinion on how the queue and send methods work/should be organised. Something to consider is the current send() method completely circumvents message chunking.

Lets see whats broken.

What remains...?

Hmm, a progress bar is simply a percentage. If I'm interpreting you correctly it could be deceptive due to retention. Eg there could always be 100% remaining if ret = 0.

This is why I went with the countdown route. There are a higher count of messages in the database, but some may have already been garbage collected.

Just testing, no need to review.

Rejiggering how skip_queue works for test, Wont pass -- seeing whats broken.

again, just for testbot

removing usages of $this->testGateway and require tests to create their own gateway if they need it. Better to do this than blanketly change send_queue value from default.

running tests - this one has a bunch of new tests

i was going through the motions to add chunking to send(), in case code was not going through the queue system. But I realise that it doesnt make sense at this time.

the return value of send() is a single result. If we are to split the message into >1 messages to accommodate chunking requirement, then multiple messages would be sent within send(). This means in more than one return result would occur. this would be an API change.

There are some things we can do:

Disallow usage of send() if the amount of recipients is over the gateway plugin limit, by thowing an exception. :( . And/or,

Force usage of queue, since it does not have a return value.

Patch as of e982cfc. Hopefully tests are fixed.

API breaks/modifications:

• Sms provider interface changed. Unless you are inheriting from DefaultSmsProvider you will get errors.
• If you have tests outside of SMS Framework relying on $this->testGateway they will need to create their own gateways with createMemoryGateway. The testGateway was removed so tests would have to explicitly opt out of skip_queue.
• Changed randomPhoneNumbers to it always returns at least 2 numbers, since the function name implies plurality. Returning random 0-1 could introduce random test failures. Also added optional phone number quantity parameter to randomPhoneNumbers.

Ive committed the PR since there's full tests coverage. I think we should still hack at the DefaultSmsProvider to improve send/queue/queueOut/queueIn.

Do you have any work in progress, or taking a time out this week?

SMS gatewaya module

Which module?

I don't think it is much of an APP change since we are in beta phase,

Its still a quirky thing to do. Pass one SMS message in, receive an unpredictable number of SMS results out.

SMS which is dispatched with crop. Since we are in crop already, what need of queuing do we have?

huh?

A good example is the scheduled SMS which is dispatched with cron. Since we are in cron already, what need of queuing do we have?

Im not sure I understand this point. Cron doesnt queue anything, cron processes whats in the queue.

This is the existing process:

Message creation page.;

This portion is message creation and storage. all message entity creation, and saving is done here, in case there is problems with sms message contraint/validation. or if there are issues interacting in bulk with the database. failures should be reduced by database transactions, but are still possible.

• create messages
• provider->queue() the messages. chunk if necessary.

Cron:

This portion is purely interacting with drupal::queue and gateway plugins. there is no new message entity creation in cron to minimise potential silent failures.

sms_cron():

• detect new messages
• add each message to the Drupal::queue system.
• clean up old sms messages.

Drupal cores' Cron::run()

• will process the drupal::queue items -- the quantity of items processed depends on factors managed by Drupal core. So not all messages may be sent in each cron run.
  • send each message via the gateway plugin
  • delete message immediately if retention = 0, easy to do since the message object is already in memory.

A way I implemented it in SMS gatewaya module was to recombine the results. I don't think it is much of an APP change since we are in beta phase, we can still do it. But we would really lose a lot if we only have to use chunked messages when we queue.

Alright since this is an API change to code not created by this issue I'll create another issue: #2707559: Chunk messages sent directly to Provider::send

They arnt queued again, they are just added to the Drupal queue system because that system manages time efficiently (Manages PHP timeouts and available system memory/PHP process max memory).

We are effectively saying everything in the {sms} table are sms that need to be sent. The Drupal queue system creates tasks to send them. Drupal will claim a task, and that task will expire. If the gateway throws an endpoint failure then when the task claim expires it will be reprocessed.

scheduled messages need to be sent at the exact time if possible.

The system timing uses minimums because cronjobs are unreliable. So if you schedule something for 6pm, it will be sent as soon as it can after that time. If super accurate timing is required then I would schedule the cron job every 15/30/60 seconds. Or just send the message directly through send().

The queue doesnt exist to serve chunking. Its there to manage time and balance load, especially if gateway endpoints are taking too long to process each request.

Im sorry, I should have linked to #2707559: Chunk messages sent directly to Provider::send. I plan to add chunking to SMS send.

So for queued messages, when they are processed by Drupal::queue, they will go through send. Send will try to chunk, but they are already pre-chunked.

What I'm trying to say is: Double. Chunk