We have a feed import that fails (we have no clue why).
Looking at the "Log" tab did not help much as it crashed. It uses the locale() function to display the (translated) error text which in this case throws an PDOException:
PDOException: SQLSTATE[22001]: String data, right truncated: 1406 Data too long for column 'source' at row 1: INSERT INTO {locales_source} (location, version, source, context, textgroup) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2, :db_insert_placeholder_3, :db_insert_placeholder_4); Array ( ..[very long].. ) ) in locale() (Zeile 714 von /html/drupal/modules/locale/locale.module).
(That is line 714 in /html/drupal/modules/locale/locale.module)
It is the "else" code block "We don't have the source string, cache this as untranslated...." with a call to "db_merge()->execute" which fails.
My dirty patch is to call this "else"-block only for texts with less than 128 Bytes (mb_strlen) and calling $locale_t[$langcode][$context][$string] = TRUE; only otherwise.
It would be better to catch this kind of PDOException with SQLSTATE[22001] and probably just forget it.
The "$locale_t... = TRUE;" is needed here, too, because drupal make calls to locale() over and over otherwise.
Another question is why to try to translate an exception's message as it might hide important details. Nevertheless there'a a bug in the locale() function or the database code used by it.
Comment | File | Size | Author |
---|---|---|---|
#39 | database_sourcefield_length_20220210.patch | 909 bytes | emptyvoid |
#29 | interdiff_1885192_27-29.txt | 686 bytes | ankithashetty |
#29 | 1885192-29.patch | 955 bytes | ankithashetty |
#27 | 1885192-27.patch | 1.11 KB | ankithashetty |
Comments
Comment #1
tylernichols CreditAttribution: tylernichols commentedI have the same issue.
Comment #2
RobLoachThe size of locale_sources' source field is too small to hold the data you're inserting. Either don't insert so much data, or make the source field take more data:
The lack of patch is because it's more seeing if this helps the Drupal 7 folk first. The fix would need to get into Drupal 8 before backporting to 7.
Comment #3
worldlinemine CreditAttribution: worldlinemine commentedThis is in response to comment #2 only:
I believe that the potential negative performance impact of increasing the size of the blob outweighs the benefit. The purpose of the Locale module is to support translations in the interface in which case 16kb would appear to be more then sufficient. I would not recommend making the change suggested in comment #2.
Comment #4
kardave CreditAttribution: kardave commentedHi all,
I had this issue, and not even #2 could help me.
I had a very large string in watchdog, and dblog page failed to load because the string was forced to be saved into locales_source.source.
Not having enough time for debugging how to eliminate the large string (deleting from watchdog did not do it) I had no choice.
My only remained solution was to patch the core (I know, never ever even think of it) temporary:
/modules/locale/locale.module line 671
After some time I will remove this and see if survived the large string, but now it saved me a lot of debugging time.
I Hope this helps someone someday.
Comment #5
broonThis still happens (of course. since it hasn't been fixed/backported yet).
During development of a module, which logs some events to watchdog I made a mistake and a complete view including 25 full nodes was var_dumped into the watchdog message. Since I was on a multilingual site and currently in another language interface than English, the "Show Recent Log Posts" page fails with the above error. It works fine when switching to English.
Comment #6
ressa CreditAttribution: ressa as a volunteer commentedI am experiencing this issue now, but @kardave's temporary fix doesn't seem to work any more... Does anyone know how to fix it, also even if only temporary?
Comment #7
mecano CreditAttribution: mecano commentedSee this #2057615: Increase size of source field to hold long URLs
Comment #10
andypostThis is because mysql limit for blob field is 64k
Faced with that in d8 where config object is bigger than that
BTW looks the same limit we have for translation
Comment #11
jrockowitz CreditAttribution: jrockowitz commentedThe attached patch is port of #4 to Drupal 8. All it does is store an empty string in the 'locales_source' and 'locales_target' table if the source string is greater than 64KB.
This is only a temporary solution. The most immediate side-effect I am seeing is when translating config the source string no longer appears because it is empty.
Comment #12
jrockowitz CreditAttribution: jrockowitz commentedHere is the patch and I am setting it to needs review just get the testbot to review it.
Comment #14
jrockowitz CreditAttribution: jrockowitz commentedComment #16
andypostComment #19
Sutharsan CreditAttribution: Sutharsan at LimoenGroen commentedThe watchdog is the most likely place where too long strings get passed to
t()
, but it any code may make this “mistake”. The translation system should guard itself for this types of errors.I see two options: 1. Truncate the string; 2. Throw an exception.
I agree that increasing the table size is not the solution. Exceptional cases like these cannot justify any performance this change may have.
Comment #26
markdcI just ran into this issue on 8.9.13.
#14 applied and works.
Comment #27
ankithashettyRerolled the patch in #14, thanks!
Comment #28
andypostThere's no such function in D9, see https://www.drupal.org/node/2850048
This event needs to be logged in watchdog (warning at least) so users can catch this bugs
Comment #29
ankithashettyAddressed #28.1, thanks.
Comment #32
joseph.olstadNote: in my tests I did not have this issue with PostgreSQL, however I came across it with MySQL v8.0.28
so far I was able to mitigate my use case by:
#2844452: Export configuration YAML strings as multiline
#3271926: mysql 1406 Data too long for column 'source' at row 1 insert into locales_source
Comment #33
larowlanWith those two related items that @joseph.olstad referenced now fixed, is this still an issue
Comment #35
joseph.olstadCuriously enough, for some reason I noticed that webforms configuration that is edited by ckeditor generates
\r\n
even on the recent core release 9.3.16.Postgresql has a larger blob, doesn't have this issue but with MySQL variants I had issues importing large webform config yml , not sure why ckeditor would be outputing
\r\n
or where to turn this bad ckeditor behavior off as it affects the processing of translations that eventually go into the locales table.Sorry not much in dept here but I did check out the blob sizes for MySQL:
MySQL seems to blow up on 64K limits , the blob type is used in a few places.
Drupal running PostgreSQL 12 that I also use doesn't have this issue because it's blob field has different /higher limits.
As a workaround I manually edited my exported YML, replaced
\r\n
with\n
and everything imported correctly with MySQL, however today someone brought up to me a large webform that is unable to save a webform translation in the ui after a certain limit. Seems like two different but similar issues going on, haven't done a deep dive on this yet.Comment #38
smustgrave CreditAttribution: smustgrave at Mobomo commentedWonder if this should be reopened?
Comment #39
emptyvoid CreditAttribution: emptyvoid commentedI've got this issue in Drupal 9 and Drupal 10.
Having the body field or content fields with both editorial content as well as full HTML on larger pages causes this local field validation to fail every single time.
I have to patch each release so that the 10,000+ records and updates will actually complete in my builds.
Most recent Patch attached.
This commits to both Drupal 9.5.10 and Drupal 10.1.2