Problem/Motivation

TMGMT can leave job items behind after their source content entity is deleted. For example, a site can request translation for a node and then delete that node while the corresponding job item still exists.

The job item still stores the deleted entity type and entity ID, but the source entity can no longer be loaded. This can cause problems for providers that expect the referenced entity to exist.

Note: there is also a related, but separate, problem in #2913008: Cannot abort or delete item if the source entity has been deleted
That issue is about manually handling job items whose source is already gone. This issue is about optionally removing matching job items at the time the source entity is deleted.

Steps to reproduce

- Install and configure TMGMT with a content source and a translator.
- Create a translatable content entity, such as a node.
- Create or request a translation job item for that entity.
- Delete the source entity before the job item is completed.

Actual result: The related TMGMT job item still exists and references a deleted source entity.
Expected result: Provide a way to automatically remove matching job items when their source entity is deleted.

Proposed resolution

- Add an opt-in purge_stale setting that allows TMGMT to remove job items when their source entity is deleted.
- Default the setting to FALSE, including for existing sites, so current behavior is preserved unless a site explicitly opts in.
- Keep continuous jobs even when they become empty, because they are persistent containers for future continuous job items.

Remaining tasks

- Review & merge

User interface changes

Adds a configuration option to enable purging stale job items when their source entity is deleted.
The option is disabled by default.

API changes

None

Data model changes

No database schema changes.
Adds a purge_stale key to tmgmt.settings, defaulting to FALSE.

Issue fork tmgmt-2998143

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

Loparev created an issue. See original summary.

loparev’s picture

miro_dietiker’s picture

A first feedback here:

A translator plugin needs to be made robust enough to deal with missing items. We already had many small related fixes.

loparev’s picture

I agree. But is there a reason why `ghost` job items/jobs aren't being deleted when the entity is deleted?

miro_dietiker’s picture

The idea was that we want to remember what we submitted to the provider.
I would prefer to keep it and maybe mark it stale if we know it's outdated.

loparev’s picture

StatusFileSize
new7.23 KB

What if this feature will be optional? Just as a POC, this patch provides a `Purge stale` option and TMGMT will remove outdated items/jobs only if it's enabled.

loparev’s picture

Status: Active » Needs review
miro_dietiker’s picture

Thank you for the proposal, we will need to discuss this direction.

Note also that core also doesn't support such a thing:
If you delete an entity B that is referenced from entity A, the entity A will have a stale reference item.

antoniya’s picture

+1

berdir’s picture

Status: Needs review » Needs work

Thanks for working on this. Makes sense to clean up, I'd even suggest to have it enabled by default, but then keep it disabled for existing installations with update function.

  1. +++ b/tests/src/Kernel/PurgeStaleJobItemsAndJobsTest.php
    @@ -0,0 +1,121 @@
    +
    +  protected $node1;
    +  protected $node2;
    +  protected $jobItem1;
    +  protected $jobItem2;
    +  protected $job;
    

    The properties need documentation.

  2. +++ b/tests/src/Kernel/PurgeStaleJobItemsAndJobsTest.php
    @@ -0,0 +1,121 @@
    +  /**
    +   * Test purging of stale TMGMT entities.
    +   *
    +   * Option is disabled.
    +   */
    +  function testPurgeStaleDisabled() {
    

    I think we can write the first line for those two tests so they explain what it does in a single line, something like:

    Tests purging when enabled/disabled.

  3. +++ b/tmgmt.module
    @@ -124,6 +126,38 @@ function tmgmt_cron() {
    +    $select = Drupal::database()->select('tmgmt_job', 'tj');
    +    $select->join('tmgmt_job_item', 'tji', 'tji.tjid = tj.tjid');
    +    $select->addField('tj', 'tjid');
    +    $select->condition('tji.item_type', $entity->getEntityTypeId());
    +    $select->condition('tji.item_id', $entity->id());
    

    Pretty sure this can be an entity query instead.

    Instead of loading the jobs, why not load the job items directly and delete them.

    And then I would do another query to look if there are more job items with the same job id, just doing a count query.

    The reason for that is that we have continuous jobs, which might have dozens or hundreds of job items, and getItems() would load them all.

pratik_kamble’s picture

StatusFileSize
new8.15 KB

@berdir The earlier patch was not working as 'purge_stale' config was not getting save. Updated it and made changes as per suggestions mentioned by you.

pratik_kamble’s picture

Status: Needs work » Needs review
pratik_kamble’s picture

StatusFileSize
new8.15 KB

Above patch had malformed. Attaching new one.

Status: Needs review » Needs work

The last submitted patch, 13: stale_jobs_remove_1.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

pratik_kamble’s picture

StatusFileSize
new8.15 KB

@Berdir I think even on the existing site where the module is in use. Stale Jobs remove functionality should be enabled by default.

tarasich’s picture

Tested patch from #13. Works good. Only tests should be fixed
I don't think option should be enabled by default on existing projects. It may become "surprise" for someone. Would advice against using patch from #15

Murz made their first commit to this issue’s fork.

murz’s picture

Status: Needs work » Needs review

I've adopted UI tests to the changed behavior with deleted entities, and have committed to the MR https://git.drupalcode.org/project/tmgmt/-/merge_requests/35 together with the last patch from @pratik_kamble. Please review.

odai jbr’s picture

StatusFileSize
new10.37 KB

A re-roll for #19 which I needed after applying a patch from Issue #3097660

google01’s picture

Unfortunately this patch is adding code without explicit accessCheck(TRUE/FALSE) to content entity queries.

For example in the function "tmgmt_entity_delete" in the code file "tmgmt.module". This case is really worrying because it causes errors in the removal of any content, leaving Drupal 10 websites completely inoperative.

colorfield made their first commit to this issue’s fork.

colorfield’s picture

Added accessCheck() to the MR as it is a blocker when deleting any kind of entity, including configuration, so also causes deployment issues.

We might still decide whether or not we want this behaviour to become the default one (see #16).

dewalt’s picture

Just adding diff from the MR to use with composer. The diff applies for 1.15 modules version, but has conflicts with the latest dev version.

berdir’s picture

Status: Needs review » Needs work

Yes, this conflicts badly with another feature we added.

oways23’s picture

@berdir
what are the new features that conflicts with this patch?

colorfield’s picture

Rebased with latest 8.x-1.x and fixed conflicts, had to rename tmgmt_update_8011 as tmgmt_update_8012.

If you previously applied one of the previous patch from this issue with tmgmt_update_8011, just make sure to run the one from https://git.drupalcode.org/project/tmgmt/-/blob/8.x-1.x/tmgmt.install?re..., otherwise creating a job will fail with Drupal\Core\Database\DatabaseExceptionWrapper: SQLSTATE[42S22]: Column not found: 1054 Unknown column 'file_count'.

colorfield’s picture

Status: Needs work » Needs review
redneko’s picture

Status: Needs review » Needs work

There are now merge conflicts that need to be resolved.

herved made their first commit to this issue’s fork.

herved’s picture

Assigned: Unassigned » herved

I'll work on this

herved’s picture

Status: Needs work » Needs review

I rebased and made a few key changes:
- Don't delete continuous jobs when they become empty, they are persistent containers that keep collecting items for new content (addresses the open review note on `tmgmt.module:157`).
- For existing installs, set purge_stale to FALSE, as per #16 (so the feature is opt-in and existing sites keep their current behavior)
For new installs, set purge_stale to TRUE, as per #10
- Skip tmgmt's own entities in the delete hook, so purging job items/jobs doesn't pointlessly re-enter it.
- Updated test coverage
- General tidy-up: guard clauses, return types, and simplifying the kernel test using entity_test.

---
#2913008: Cannot abort or delete item if the source entity has been deleted is a related but separate issue and shouldn't be bundled here IMHO: it addresses the issue when a job item whose source is already gone can't be aborted or deleted from the UI, whereas this issue auto-removes job items when their source is
deleted. Different problems; both still needed, but better addressed separately.

herved’s picture

Assigned: herved » Unassigned
Issue summary: View changes
herved’s picture

Issue summary: View changes
claudiu.cristea’s picture

Status: Needs review » Needs work

A question on the new settings default value on new installs

herved’s picture

Status: Needs work » Needs review
claudiu.cristea’s picture

Status: Needs review » Reviewed & tested by the community

The MR doesn't change the behavior of existing sites but applies as default for new installs. This is correct, IMHO. There is test coverage and the code looks good. RTBC!

claudiu.cristea’s picture

Status: Reviewed & tested by the community » Fixed

Thank you all for contribution. Merged

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.