Generation timeout too short and locking not working

Problem/Motivation

Locally there is no problem but on production with a more limited CPU and memory I'm seeing some larger (23mb JPEG) images not get processed and hit the thrown error "Image generation in progress. Try again shortly."

Steps to reproduce

I don't have exact repro steps, I guess it might be possible to set up a very limited container in terms of CPU and memory and test it there.

Proposed resolution

A better locking system that has a progressive backoff and retries. So if the generation doesn't complete in 3 seconds it tries again with a 4.5 second buffer then 6.75s, then 10s and finally 15s. This should allow adequate time for the generation of a large image to complete.

This is my hope anyway and as I can't easily replicate locally I'm going to have to use this issue as a testing ground to see if I can fix it on the staging/prod environment.

Remaining tasks

• ✅ File an issue
• ➖ Addition/Change/Update/Fix
• ➖ Testing to ensure no regression
• ➖ Automated unit testing coverage
• ➖ Automated functional testing coverage
• ➖ UX/UI designer responsibilities
• ➖ Readability
• ➖ Accessibility
• ➖ Performance
• ➖ Security
• ➖ Documentation
• ➖ Code review by maintainers
• ➖ Full testing and approval
• ➖ Credit contributors
• ➖ Review with the product owner
• ➖ Release notes snippet
• ❌ Release

API changes

• N/A

Data model changes

• N/A

Release notes snippet

• N/A