Problem/Motivation

For some long text fields, the field...._format value is sometimes NULL in database for non-default languages, causing display issues.
I'm unable to identify why this is not happening to all long text fields.
After saving the entity the format is properly saved and the output issues (encoded markup in this case) is resolved.
This only seems to be happening on the download and save from the translation process.

Comments

ben.hamelin created an issue. See original summary.

penyaskito’s picture

Thanks for the report!

I will debug this ASAP. My first thought is that the callbacks are somehow running with a user that doesn't have access to those text filters.

penyaskito’s picture

Status: Active » Postponed (maintainer needs more info)

Hi @ben.hamelin, I've been trying to reproduce this issue unsuccessfully.

Did you find any more information for trying to reproduce?

ben.hamelin’s picture

Not yet. It doesn't happen on all fields for us either. Is it possible to set a default value in config and use that as a fallback option?
It's possible this is isolated to some specific nodes for some reason.

penyaskito’s picture

@ben.hamelin It's already supposed to have the same text format than the original source field had.

For discarding that this was some kind of setting change, or to see if it works as a workaround, can you maybe delete the translation of a faulty translation (if it was not modified after download) and re-download it in a test environment and check what happens?

penyaskito’s picture

Bumping to see if you can test that workaround :-)

ben.hamelin’s picture

@penyaskito - I was able to reproduce the issue in our test environment. With logging enabled, I can review the "getTranslation response received" message, and see the JOSN. There are 2 formatted text fields being returned, each with markup. The text format value is not part of the response. One saves correctly, the other does not. Reviewing the field settings both have a default value of empty, with the default text format.

To your earlier thought about permissions, maybe this is a field level permission issue? Settings for the fields look standard.

penyaskito’s picture

@ben.hamelin The format is not extracted in the JSON because it's actually not something that you want to translate. The "format" value in theory is copied from the source when \Drupal\Core\Entity\ContentEntityBase::addTranslation is called, so nothing really in the Lingotek code should be affecting this.

Would be interesting to see the differences between those two fields if any. Are the translatability and the Lingotek settings also the same?

ben.hamelin’s picture

This is a commerce product entity. The field that is working is the default "body" field for products. The field with the issue is a long text field.
The field types are different (in field.storage.commerce_product...) - "body" is "text_with_summary" while the custom field is "text_long"

Lingotek translation settings are identical.

penyaskito’s picture

Tested again with the new information and still not reproducing this issue.

@ben.hamelin Thanks a lot for your patience, I asked our Support team to contact you.*
In any case, what happens with new content of this product type? Are you facing the issue in that case?

In the cases failing, which I'm guessing is only existing content, is there any chance that you had a text format that was now removed?

ben.hamelin’s picture

@penyaskito - Sorry just circling back to this as it has occurred again on recent content translations. These are new translations (new language added to site) for existing content. It is possible we removed an old text format at some point, but since that isn't stored anywhere that we can see how would that cause this issue? If it helps for testing we could try to recreate the content in a brand new node and then test the translation again.

penyaskito’s picture

That would be helpful. If you can confirm that the format the content field had was removed that would explain why this happened, and it would be a core bug.

ben.hamelin’s picture

@penyaskito - Adding a translation for a brand new node works fine. Are you able to identify where the old format might be stored? Is it possible it is anywhere in Lingotek settings/config? We need to understand how to clean that up so that existing content can be leveraged and not have to be recreated.

penyaskito’s picture

@ben.hamelin We copy the format from the source, we don't do anything special from it. So if the source format was removed, the translation will include the same format.
If the source had already a previous translation with a different format (which might have been deleted), it's going to be preserved, as we just edit the properties we have relevant info about. So that might be the issue here.
If possible, and if the source format was fixed already, try to delete the translation before downloading the new one for confirming.

If you want to manipulate the data, check the field table in the database. E.g. for node body there's a table node__body:

+--------------+------------------+------+-----+---------+-------+
| Field        | Type             | Null | Key | Default | Extra |
+--------------+------------------+------+-----+---------+-------+
| bundle       | varchar(128)     | NO   | MUL |         |       |
| deleted      | tinyint(4)       | NO   | PRI | 0       |       |
| entity_id    | int(10) unsigned | NO   | PRI | NULL    |       |
| revision_id  | int(10) unsigned | NO   | MUL | NULL    |       |
| langcode     | varchar(32)      | NO   | PRI |         |       |
| delta        | int(10) unsigned | NO   | PRI | NULL    |       |
| body_value   | longtext         | NO   |     | NULL    |       |
| body_summary | longtext         | YES  |     | NULL    |       |
| body_format  | varchar(255)     | YES  | MUL | NULL    |       |
+--------------+------------------+------+-----+---------+-------+

There you have the "field_name"_format column.

If you want to massively update that (risky!), take into account that you might also want to touch the last revision on the corresponding field table: e.g. for node__body, that would be node_revision__body.

ben.hamelin’s picture

@penyaskito Just following up here that we continue to experience this. I will work to obtain further details on the issue and report back here.

ben.hamelin’s picture

Version: 3.8.x-dev » 3.9.2

@penyaskito - So the node this is happening on is a new node, created about a month ago. We are currently using the Lingotek module version
3.9.2.

penyaskito’s picture

What's the source field format? It's strange as we don't really deal with formats, it is Drupal itself which assigns them from the source on addTranslation()

ben.hamelin’s picture

Here is some more info after debugging this latest report. The "NULL" formats are coming from the revisions table. For this query
SELECT langcode, field_hero_text_format FROM paragraph_revision__field_hero_text WHERE field_hero_text_format is NULL

the following results:

pt	NULL
pt	NULL
it	NULL
it	NULL
it	NULL
it	NULL
it	NULL
it	NULL
it	NULL
it	NULL
it	NULL
it	NULL
it	NULL
it	NULL
it	NULL
pt	NULL
pt	NULL
pt	NULL

The main table `paragraph__field_hero_text` does not have any NULL format values.
Then, after re-downloading the translation for a particular language, the new revision is saved correctly and the format is correct.

So, this seems to be an issue with how the paragraph revisions are being stored/saved. Since the re-download fixes this, I am inclined to mark this issue as closed since it doesn't appear to be a Lingotek specific issue. Would you agree @penyaskito?

penyaskito’s picture

Status: Postponed (maintainer needs more info) » Closed (cannot reproduce)

Yes, that's good for us. I'm still not sure how the revisions table and the main table could get to have different values, but definitely doubt is a lingotek issue. Thanks for taking the time to give feedback on this!

alex.bukach’s picture

Status: Closed (cannot reproduce) » Needs review
StatusFileSize
new1.05 KB
new1.09 KB

I was able to reproduce the issue. In my case it happens when a long_text field is empty on the original (untranslated) entity (and hence on the translation too), and then it is populated. When the translation is saved, the field is saved only with "value" property, format remains empty.

Proposed solution is to check all the properties on the untralstated field value and copy to the translation ones not coming with Lingotek data.

Status: Needs review » Needs work
penyaskito’s picture

I still don't get this. Drupal is already doing that when creating the translation, by deep copying the revision.

alex.bukach’s picture

Status: Needs work » Needs review
StatusFileSize
new1.16 KB
new1.14 KB

@penyaskito Drupal does that only when a translation is created with the field being not empty. Here's the scenario I am able to catch the issue in:

  1. Create node a having a long text field, keep the field empty.
  2. Translate the node using Lingotek.
  3. Update the original node by adding some value to the long text field (make sure it contains HTML tags to be able to see the issue easily and use a text format allowing those tags).
  4. Translate the updated node using Lingotek.
  5. See the unescaped tags on the translated node page.
  6. Check the format of the translated field value in the database and see it's null.

The point is that if the field value is empty initially, Drupal neither saves the format to database. Thus when a translation is created, it neither has format for the field. And when the translation is updated, Drupal does not fetch the format from the unstralslated node.

Here's another scenario which is fixed by the patches.

  1. Create node a having a long text field, make sure to populate this field.
  2. Translate the node using Lingotek.
  3. Update the original node by changing the text format used.
  4. Translate the updated node using Lingotek.
  5. See the translated node still uses the old text format.
jahanzaib_at_straker’s picture

I am able to replicate the issues as highlighted by @Alex Bukach 's, and I have written some tests for it.
The patches contain the fix provided by @Alex Bukach and the tests.

jahanzaib_at_straker’s picture

Checking @Alex's patch without the automated tests, due to composer failure in the previous comment.

jahanzaib_at_straker’s picture

Status: Needs review » Fixed

Merged @Alex's fix with new automated tests in both 3.9.x and 4.0.x.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.