Add new feature to purge continuous jobs item

Issue fork tmgmt-3134985

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

laurentb created an issue. See original summary.

laurentb’s picture

berdir’s picture

Status: Active » Needs work
+++ b/src/Form/SettingsForm.php
@@ -59,6 +59,24 @@ class SettingsForm extends ConfigFormBase {
+
+    $form['performance']['tmgmt_purge_continuous'] = [
+      '#type' => 'select',
+      '#title' => t('Purge continuous jobs'),
+      '#description' => t('If configured, continuous translation jobs item that have been marked as accepted will be purged after a given time. The translations itself will not be deleted.'),
+      '#options' => [
+        '_never' => t('Never'),
+        '0' => t('Immediately'),
+        '86400' => t('After 24 hours'),
+        '172800' => t('After 48 hours'),
+        '259200' => t('After 72 hours'),
+        '604800' => t('After 7 days'),
+        '2592000' => t('After 30 days'),
+        '31536000' => t('After 365 days'),
+      ],
+      '#default_value' => $config->get('purge_continuous'),
+    ];
+
     $form['security'] = array(

do we really need an immediate option, we could just check for the setting being > 0? who knows if 0 could result in some weird race conditions?

Tests would be nice but could be a bit tricky, but we could manually set the changed date back in the database and then run cron, making sure it first doesn't get deleted and then it does.

huzooka’s picture

Assigned: Unassigned » huzooka

huzooka’s picture

Assigned: huzooka » Unassigned
Status: Needs work » Needs review
  1. Used patch from #2 as base
  2. Fixed some smaller nits
  3. Added option to also purge aborted job items of continuous jobs
  4. Added kernel test
  5. Also added update hook
huzooka’s picture

Assigned: Unassigned » huzooka
Status: Needs review » Needs work

Number of items purged during one cron run can be also configurable.

huzooka’s picture

Assigned: huzooka » Unassigned
Status: Needs work » Needs review
huzooka’s picture

huzooka’s picture

Assigned: Unassigned » huzooka
Status: Needs review » Needs work

After some testing in our project, I noticed that the cleanup performance varies a lot (we're using entity operations).
Deleting 5000 items sometimes needs 70 second (a bit more than 1 minute), in other cases it takes 677 sec (more than 11 minute).

I'm gonna create a queue worker, and queue the configured number of approved items; then wait until the queue is fully processed. With this approach, we can have some control on the processing time, and also, we can continue processing outside of cron, e.g. with Drush or by using Queue UI. Also, since queue workers are plugins, this approach will allow to modify the initial processing time.

huzooka’s picture

Assigned: huzooka » Unassigned
Status: Needs work » Needs review
alorenc’s picture

Assigned: Unassigned » alorenc
alorenc’s picture

Assigned: alorenc » Unassigned
Status: Needs review » Needs work

dimilias made their first commit to this issue’s fork.

dimilias’s picture

Status: Needs work » Reviewed & tested by the community

I have resolved the conflict and seems to be green :) I tested it and it works as expected so RTBC +1.

claudiu.cristea’s picture

Looks good. The only thing that I can think of, is whether we want a different threshold for the aborted job items. But that seems to be easy to implement afterwards if needed, the update path would be trivial.

claudiu.cristea’s picture

claudiu.cristea’s picture

Status: Reviewed & tested by the community » Fixed

Merged and published the change notice. Thank you

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.