Hi all,
My question is related to showing/logging source info from an item that failed to import for some reason.
When an item fails to be processed, a message is set showing the log message. In some cases this message shows no info under "original item" or under "entity", see the attached screenshot.
This lack of info makes it impossible (for end-users) to identity which item failed to be processed, especially in the case of a very large import. In the case of my screenshot I know why the item does not process correctly because I put a non-alphanumeric character in the username field, causing it to fail on purpose. In the on-screen message or in the feeds log I see no info that could tell me which row in the source .csv file I should look at to identify the failed item import.
Does anyone have a suggestion for this? How could I get some sort of source record identifier showing up in the log? Could I use any of the hooks in de feeds API to do this in some way?
Many thanks!!
Marco
| Comment | File | Size | Author |
|---|---|---|---|
| #16 | Captura.PNG | 67.88 KB | rafaeldelrosal |
| #13 | feeds-log-failed-non-utf8-item-2730207-13.patch | 615 bytes | megachriz |
| #4 | feeds-log-failed-non-utf8-item-2730207-4.patch | 470 bytes | megachriz |
| #3 | test.csv_.zip | 578 bytes | frontmobe |
| #3 | feeds_export.txt | 3.01 KB | frontmobe |
Comments
Comment #2
megachrizI tried to import an user with a non-UTF8 character in the username (I used a character from https://www.drupal.org/files/issues/feeds-emoji_sample-2659208-6.htm_.txt). The user failed to import, but in my case the complete item was shown (see screenshot).
To exclude the possibility this is a theme issue (your theme may be hiding the contents of
<pre>-tags), can you try to view the log page using the theme "Seven" as your admin theme?If the theme doesn't appear to be the cause of the issue, could you supply an export of your importer configuration + the source file from which a row failed to import?
Comment #3
frontmobeHi MegaChriz,
Thanks for your reply, I tried your suggestion (the admin theme in use was Ember) but switching to Seven did not change anything in the output.
I included the source file I performed the test with and an export of the feeds importer that was used.
This really puzzles me so any suggestion would be more than welcome!
Thanks,
Marco
Comment #4
megachriz@Frontmobe
Thanks for your export. I found the cause of the issue! When a log message is prepared, the values of the item to import are expected to be encoded in UTF-8. It appears to be when that when an item is not UTF-8-encoded, the details of the item are emptied because of a call to
truncate_utf8().The attached patch should fix the issue. Let me know if that patch works for you.
Comment #5
megachrizChanging issue title for clarity.
Comment #8
megachrizThe failing tests have nothing to do with the patch (see #2735981: Branch tests failing because of new "administer fields" permission). Back to "Needs review".
Comment #9
frontmobe@MegaChriz,
I can conform that your patch solves the issue. The call to truncate_utf8() indeed seems to be the cause and running the string through utf8_encode() solves emptying the result.
Thanks for your help!!
Regards,
Marco
Comment #11
megachrizThanks for reporting back, Marco! I've committed the patch from #4.
Comment #12
twistor commentedThis fix is incorrect. utf8_encode() only coverts from ISO-8859-1 to UTF-8. Most text in Drupal is UTF-8, so it has the possibility to break things.
Comment #13
megachrizYou are right. When an UTF-8 encoded item fails, the char 'é' is logged as 'é'.
The attached patch uses
utf8_encode()when it detects that the string is not UTF-8. This isn't a perfect solution, but I don't think that there is a way to detect the right character set? At least, that is what I had read in #1428272-49: Added support of encoding conversions to the CSV Parser.Comment #14
twistor commentedAccording to https://w3techs.com/technologies/overview/character_encoding/all, this should handle most cases, but it will still break for other encodings. Honestly, we shouldn't be supporting this here. It's the parser's job to make sure that the encoding is correct. But, it's hard to find the bug if we can't log it :)
So, seems fine to me.
Comment #15
rafaeldelrosal commentedHello,
I have installed the patch but I still have the same problem. See attached picture. (captura.png)
It can be another problem as discussed in https://www.drupal.org/node/2659208?
I do not know what to do.
Thanks for your time!
Comment #16
rafaeldelrosal commentedComment #17
twistor commented@rafaeldelrosal, That is a different issue indeed.
Comment #19
megachrizThanks for the feedback, twistor! Committed #13.
Comment #20
rafaeldelrosal commentedThanks @Twistor,
Where should I see then? In this post?: https://www.drupal.org/node/2659208 or is there some other where talk about it?
I am fairly new to this world of drupal :-)
Thanks from Spain!
Comment #21
megachriz@rafaeldelrosal
Yes, your issue seems to deal with emoticons. You could try the patch from #2659208: Emojis causing SQLSTATE[HY000]: General error: 1366 Incorrect string value and report back if that patch works for you. According to #7 of that issue, the fix may work in your case, but is incomplete as a general solution.
Other thing you could try is to change the character set of your database from
utf8_general_citoutf8mb4(for each table and each column in each table that is utf8_general_ci), according to #1314214: MySQL driver does not support full UTF-8 (emojis, asian symbols, mathematical symbols).