I have a problem on a client site with a Feeds import that shows the correct data within the import preview but does not copy the data across to the assigned field during the import.

There are 18 columns being exported from a MySQL database, all mapped to fields within a single Product content type. Running the export 'manually' by generating an SQL to CSV file export and then uploading the CSV file to the default feeds CSV parser works a charm, no problems.

The client wants it run regularly from their product database though; so I have been trying to achieve this using either the Feeds SQL or Feeds Database parsers. Both produce the same result.

When importing with either of the direct database parsers then there are only 6 of the 18 fields that actually copy across the data. There is nothing special about any of the fields, most are text, some are decimals for prices with a few taxonomy fields also.

When viewing the data with the fetcher preview for the SQL query then the correct data is displayed.

data shown in parser preview

If however the import is viewed using the Feeds Preview module then some of the data is missing. Some of the data that IS shown on the Feeds Preview does not actually make it through the final import either, there is only data displayed for 9 fields in this preview. The full import only ever pulls in 6 of these fields.

Data not shown in Feeds Preview

There are no errors shown in the logs for any of these import fails.

Debugging Feeds isn't the easiest task but by using Feeds Debug Tamper I can see that when dealing with certain fields, their content is missing, even though it was present during the last import read, as can be seen here:

Feeds Tamper Debug output

The 'Artikel_FK' value is contained within the initial import string, but in the next import read pass then the same field is now shown as blank. This only happens on the import read for that piece of data, then it is hidden, and on subsequent reads it remains hidden.

This is happening with all of the missing items, they are present in the fetcher preview, but do not show in the preview generated by Feeds preview.

As far as I understand, the Feeds debug output is before the data is parsed, so I need to find a way of tracing at what stage that data was read and why it is showing as blank.

The client database I am importing from has been problematic for so long now anyway, using Windows-1252 encoding with a Latin1_German1_CI collation on the database I have had to previously use Feeds Tamper to convert some field data to UTF-8, HTML entities conversions for German characters, trim whitespace and perform a few search-replace on certain mis-spelled strings.

To remove any of these possibly being the culprit I have taken a copy of the database and moved onto a development server and converted all of the tables and columns to UTF-8 with a UTF-8_unicode collation. I have also added to the initial SQL fetcher script calls to convert the read data as utf8. Hopefully I have removed any 'incompatible' data formats, but the errors still persist.

The condition of the missing data is the same if I clone the importer and remove all of the Feeds Tamper plugins, so again that has no affect.

Any help, pointers or tips on further Feeds/Import debugging would be appreciated.

cheers all :)

CommentFileSizeAuthor
feed-output-ss1.png190.98 KBraynimmo
feed-output-ss3.png116.8 KBraynimmo
feed-output-ss2.png157.3 KBraynimmo
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

raynimmo created an issue.

MegaChriz’s picture

MegaChriz CreditAttribution: MegaChriz as a volunteer commented

Thanks for the detailed description of your problem!

At first glance, this looks like a problem with the Feeds SQL module. The Feeds Import Preview shows the data after it has been parsed and eventually modified by Feeds Tamper (or other modules that implement hook_feeds_after_parse()). So if you have no modules enabled that implement hook_feeds_after_parse(), then the preview basically shows what it receives from the parser; the SQL parser in this case.

As any tamper, the Feeds Debug Tamper acts also on the parsed result, thus after the parser's work is done.

Without any real data to work with, it's hard to point to the cause of the problem. It can be an issue in Feeds SQL module, a configuration error or a module that modifies the data using the hook_feeds_after_parse() hook. I think it's unlikely that Feeds Tamper is the culprit as you said you have tested also with all tampers removed, but since that module implements hook_feeds_after_parse(), you could check if you get the same results when you disable that module.