Feeds SQL Import shows data in preview but fails to import

I have a problem on a client site with a Feeds import that shows the correct data within the import preview but does not copy the data across to the assigned field during the import.

There are 18 columns being exported from a MySQL database, all mapped to fields within a single Product content type. Running the export 'manually' by generating an SQL to CSV file export and then uploading the CSV file to the default feeds CSV parser works a charm, no problems.

The client wants it run regularly from their product database though; so I have been trying to achieve this using either the Feeds SQL or Feeds Database parsers. Both produce the same result.

When importing with either of the direct database parsers then there are only 6 of the 18 fields that actually copy across the data. There is nothing special about any of the fields, most are text, some are decimals for prices with a few taxonomy fields also.

When viewing the data with the fetcher preview for the SQL query then the correct data is displayed.

If however the import is viewed using the Feeds Preview module then some of the data is missing. Some of the data that IS shown on the Feeds Preview does not actually make it through the final import either, there is only data displayed for 9 fields in this preview. The full import only ever pulls in 6 of these fields.

There are no errors shown in the logs for any of these import fails.

Debugging Feeds isn't the easiest task but by using Feeds Debug Tamper I can see that when dealing with certain fields, their content is missing, even though it was present during the last import read, as can be seen here:

The 'Artikel_FK' value is contained within the initial import string, but in the next import read pass then the same field is now shown as blank. This only happens on the import read for that piece of data, then it is hidden, and on subsequent reads it remains hidden.

This is happening with all of the missing items, they are present in the fetcher preview, but do not show in the preview generated by Feeds preview.

As far as I understand, the Feeds debug output is before the data is parsed, so I need to find a way of tracing at what stage that data was read and why it is showing as blank.

The client database I am importing from has been problematic for so long now anyway, using Windows-1252 encoding with a Latin1_German1_CI collation on the database I have had to previously use Feeds Tamper to convert some field data to UTF-8, HTML entities conversions for German characters, trim whitespace and perform a few search-replace on certain mis-spelled strings.

To remove any of these possibly being the culprit I have taken a copy of the database and moved onto a development server and converted all of the tables and columns to UTF-8 with a UTF-8_unicode collation. I have also added to the initial SQL fetcher script calls to convert the read data as utf8. Hopefully I have removed any 'incompatible' data formats, but the errors still persist.

The condition of the missing data is the same if I clone the importer and remove all of the Feeds Tamper plugins, so again that has no affect.

Any help, pointers or tips on further Feeds/Import debugging would be appreciated.

cheers all :)