Updated: Comment #274

The work for Field translation is done and has been committed!

Feeds doesn't however provide full multilingual support yet. The following is not (yet) supported:

These issues should be handled in followups. See Remaining tasks for more info.

Problem/Motivation

Currently, Feeds is hardcoded to import fields as LANGUAGE_NONE (actually technically it's hardcoded as 'und' which is even worse). This makes it impossible to import multilingual field data. Actually, there is no support for any type of multilingual imports (either using Content Translation (core) or Entity Translation (contrib)).

Proposed resolution

For Field Translation:

Field Translation is technically built in to core. The Entity Translation module merely exposes an interface for it. This means that it's relatively easy to support field translation in feeds without having any dependencies on contrib modules. The proposed resolution is to simply add language support to the mappers. This includes:

  • Adding a target configuration form to each of the mappers that allows the user to configure the language
  • Replace instances of 'und' in the mapper target callbacks with the configured language
  • Add support (but not a dependency) for the Entity Translation module -- create a translation when adding a field of a language other than the default language.

For Content Translation:

The resolution for Content Translation perhaps isn't as easy as for Field Translation. Currently (July 27, 2013) there is no agreed upon workflow for how Content Translation should work. There has been some work done by joseph.olstad in comment #76 (https://drupal.org/node/1183440#comment-7478058) but it involves installing a zipped module and patching feeds. It is perhaps a good start, but will need to be reviewed the community and the module maintainers.

Remaining tasks

For Field Translation:

DONE! The work for field translation has been finished! The patch from #268 has been committed.

For Content Translation:

A consensus on the workflow needs to be agreed upon. Does one have to create separate imports per language? Or should the language be specified in a field? Should tnids be provided in the source? Once these things are ironed out, we can move on the code.

This task should be handled in a followup issue. Candidate followup: #840142: How can the created nodes be assigned their correct language? (and other "core" fields...)

Autocreating terms in the right language:

This works only when both the taxonomy reference field and the taxonomy are multilingual. In this case, terms are created in the language specified on the Feeds target for the taxonomy reference.
When only the taxonomy is multilingual (and the taxonomy reference is not), there is no option to set the language for autocreating terms. A decision about how to handle this situation needs to be made: how do we autocreate terms in a way that respects the type of multilingual (localized, i18n translated, entity/field translated) for that set of terms? See https://drupal.org/node/1183440#comment-7579839 for a slightly longer, possibly more confusing description.

This task should be handled in a followup issue.

Respecting multilingual settings from i18n

The following multilingual settings are not respected:

  • Require language (do not allow language neutral)
  • Lock language (cannot be changed)
  • Limited amount of allowed languages

This task should be handled in a followup issue.

User interface changes

For mapping targets for translatable fields a language can be selected on the target configuration.

API changes

Original report by pcambra

Now the feeds import is no language aware for fields, that can be multilanguage, it just import as 'und'

This patch is a initial thing, but including the capability of add language property in the target definition would allow developers and other modules to add multilanguage fields in feeds.

Somehow related #1183408: Import into real_target

CommentFileSizeAuthor
#268 interdiff-1183440-263-268.txt9.75 KBMegaChriz
#268 feeds-field_translations-1183440-268.patch68.31 KBMegaChriz
#263 feeds-field_translations-1183440-263.patch67.52 KBtwistor
PASSED: [[SimpleTest]]: [MySQL] 17,099 pass(es). View
#263 interdiff.txt482 bytestwistor
#260 feeds-field_translations-1183440-260.patch67.56 KBtwistor
FAILED: [[SimpleTest]]: [MySQL] 17,032 pass(es), 81 fail(s), and 39 exception(s). View
#260 interdiff.txt7.61 KBtwistor
#258 interdiff-1183440-255-258.txt2.48 KBMegaChriz
#258 feeds-field_translations-1183440-258.patch64.06 KBMegaChriz
PASSED: [[SimpleTest]]: [MySQL] 16,841 pass(es). View
#255 interdiff-1183440-252-255.txt3.35 KBMegaChriz
#255 feeds-field_translations-1183440-255.patch63.83 KBMegaChriz
FAILED: [[SimpleTest]]: [MySQL] 16,815 pass(es), 26 fail(s), and 26 exception(s). View
#252 interdiff-1183440-249-252.txt15.39 KBMegaChriz
#252 feeds-field_translations-1183440-252.patch62.73 KBMegaChriz
FAILED: [[SimpleTest]]: [MySQL] 16,829 pass(es), 12 fail(s), and 0 exception(s). View
#249 interdiff.txt1.52 KBMegaChriz
#249 feeds-field_translations-1183440-249.patch57.25 KBMegaChriz
FAILED: [[SimpleTest]]: [MySQL] 15,634 pass(es), 27 fail(s), and 39 exception(s). View
#246 interdiff.txt38.57 KBMegaChriz
#246 feeds-field_translations-1183440-246.patch55.9 KBMegaChriz
FAILED: [[SimpleTest]]: [MySQL] 15,608 pass(es), 53 fail(s), and 66 exception(s). View
#245 feeds-field_translations-1183440-245.patch29.22 KBMegaChriz
PASSED: [[SimpleTest]]: [MySQL] 11,883 pass(es). View
#232 feeds-multilingual-1183440-231.patch30.89 KBtwistor
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds-multilingual-1183440-231.patch. Unable to apply patch. See the log in the details link for more information. View
#232 interdiff.txt3.03 KBtwistor
#228 feeds-multilingual-1183440-228.patch30.56 KBtwistor
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds-multilingual-1183440-228.patch. Unable to apply patch. See the log in the details link for more information. View
#226 feeds-multilingual-1183440-226.patch21.75 KBtobiasb
PASSED: [[SimpleTest]]: [MySQL] 7,214 pass(es). View
#225 feeds-multilingual-1183440-225.patch21.59 KBtwistor
PASSED: [[SimpleTest]]: [MySQL] 7,064 pass(es). View
#225 interdiff.txt1.07 KBtwistor
#219 feeds-multilingual-1183440-219.patch20.77 KBtwistor
PASSED: [[SimpleTest]]: [MySQL] 7,064 pass(es). View
#214 feeds-multilingual-1183440-214.patch63.37 KBtwistor
PASSED: [[SimpleTest]]: [MySQL] 7,062 pass(es). View
#212 feeds-multilingual-1183440-212.patch63.2 KBtwistor
FAILED: [[SimpleTest]]: [MySQL] 7,038 pass(es), 4 fail(s), and 12 exception(s). View
#191 feeds-field_translations-1183440-191.patch141.48 KBstefan.r
PASSED: [[SimpleTest]]: [MySQL] 6,191 pass(es). View
#181 feeds-field_translations-1183440-179-with_mapping_api_extensions.patch43.09 KBstefan.r
FAILED: [[SimpleTest]]: [MySQL] 6,072 pass(es), 99 fail(s), and 2,561 exception(s). View
#179 feeds-field_translations-1183440-179.patch14.29 KBCalystod
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds-field_translations-1183440-179.patch. Unable to apply patch. See the log in the details link for more information. View
#178 feeds-field_translations-1183440-178.patch14.29 KBCalystod
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds-field_translations-1183440-178.patch. Unable to apply patch. See the log in the details link for more information. View
#177 feeds-field_translations-1183440-177.patch29.51 KBCalystod
FAILED: [[SimpleTest]]: [MySQL] 6,162 pass(es), 20 fail(s), and 63 exception(s). View
#166 feeds-field_translations-1183440-166.patch29.48 KBtwistor
FAILED: [[SimpleTest]]: [MySQL] 6,162 pass(es), 20 fail(s), and 1,873 exception(s). View
#165 feeds-field_translations-1183440-165.patch0 bytestwistor
PASSED: [[SimpleTest]]: [MySQL] 6,191 pass(es). View
#165 feeds-field_translations-1183440-165-do-not-test.patch15.83 KBtwistor
#159 interdiff.txt3.7 KBstefan.r
#155 feeds-field_translations-1183440-155.patch35.13 KBBobík
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds-field_translations-1183440-155.patch. Unable to apply patch. See the log in the details link for more information. View
#148 feeds-field_translations-1183440-148.patch33.34 KBstefan.r
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds-field_translations-1183440-148.patch. Unable to apply patch. See the log in the details link for more information. View
#146 feeds-field_translations-1183440-146.patch24.8 KBstefan.r
FAILED: [[SimpleTest]]: [MySQL] Repository checkout: failed to create dependency directory. View
#144 Screenshot 2014-08-12 15.23.07.PNG58.8 KBBobík
#141 node_setting_language_form_field.png58.41 KBManish Jain
#141 feeds-field_translations-1183440-141.patch28.37 KBManish Jain
FAILED: [[SimpleTest]]: [MySQL] 5,925 pass(es), 5 fail(s), and 3 exception(s). View
#141 feeds-7.x-2.0-alpha8-field_translation_imports-1183440-141-do-not-test.patch20.17 KBManish Jain
#137 feeds-field_translations-1183440-137.patch30.35 KBBobík
FAILED: [[SimpleTest]]: [MySQL] 5,927 pass(es), 3 fail(s), and 0 exception(s). View
#134 feeds-field_translations-1183440-134.patch23.27 KBstefan.r
PASSED: [[SimpleTest]]: [MySQL] 5,529 pass(es). View
#130 feeds-field_translations-1183440-130.patch32.33 KBstefan.r
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds-field_translations-1183440-130.patch. Unable to apply patch. See the log in the details link for more information. View
#128 Translations_of_Das_Biest_an_deinen_Fersen___17_Hippies.jpg303.23 KBkrabbe
#128 News-Importer-DE___17_Hippies.jpg479.42 KBkrabbe
#127 dev_17hippies_de_de_dataexport.jpg644.26 KBkrabbe
#115 feeds-field_translations-1183440-115.patch31.91 KBdrclaw
PASSED: [[SimpleTest]]: [MySQL] 5,080 pass(es). View
#114 feeds-field_translations-1183440-114.patch22.53 KBstefan.r
PASSED: [[SimpleTest]]: [MySQL] 5,080 pass(es). View
#110 feeds-field_translations-1183440-110.patch22.6 KBolofjohansson
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds-field_translations-1183440-110.patch. Unable to apply patch. See the log in the details link for more information. View
#108 feeds-field_translations-1183440-108.patch22 KBolofjohansson
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds-field_translations-1183440-108.patch. Unable to apply patch. See the log in the details link for more information. View
#104 interdiff.txt670 bytesdrclaw
#104 feeds-field_translations-1183440-104.patch31.88 KBdrclaw
PASSED: [[SimpleTest]]: [MySQL] 4,698 pass(es). View
#99 feeds-field_translations-1183440-99.patch31.89 KBdrclaw
PASSED: [[SimpleTest]]: [MySQL] 4,503 pass(es). View
#97 feeds-field_translations-1183440-97.patch31.54 KBdrclaw
FAILED: [[SimpleTest]]: [MySQL] 4,522 pass(es), 65 fail(s), and 4 exception(s). View
#89 multilingual_feeds_title.png39.52 KBdrclaw
#88 feed_translation_strange_1.png113.45 KBnfavrod
#80 feeds-field_translation_imports-1183440-80.patch19.53 KBdrclaw
PASSED: [[SimpleTest]]: [MySQL] 4,503 pass(es). View
#80 feeds-7.x-2.0-alpha8-field_translation_imports-1183440-80-do-not-test.patch18.73 KBdrclaw
#79 feeds-7.x-2.0-alpha8-field_translation_imports-1183440-79-do-not-test.patch17.53 KBdrclaw
#78 feeds-field_translation_imports-1183440-78.patch19.23 KBdrclaw
PASSED: [[SimpleTest]]: [MySQL] 4,503 pass(es). View
#78 feeds-field-translation-mapper.jpg36.63 KBdrclaw
#76 feeds_translation-7.x-1.x.zip17.59 KBjoseph.olstad
#73 d6backportTNIDparserAndMapperFix-1183440-66.patch4.01 KBamanire
FAILED: [[SimpleTest]]: [MySQL] Setup environment: Test cancelled by admin prior to completion. View
#59 feeds_tnid.tar_.gz15 KBjoseph.olstad
#59 feeds_tnid_DIFFS.tar_.gz1.14 KBjoseph.olstad
#59 feeds_tnid.zip16.41 KBjoseph.olstad
#59 feeds_tnid_DIFFS.zip1.95 KBjoseph.olstad
#66 TNIDparserAndMapperFix-1183440-64.patch4.16 KBjoseph.olstad
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch TNIDparserAndMapperFix-1183440-64.patch. Unable to apply patch. See the log in the details link for more information. View
#64 TNIDparserAndMapper-1183440-59.patch4.19 KBjoseph.olstad
FAILED: [[SimpleTest]]: [MySQL] Setup environment: Test cancelled by admin prior to completion. View
#41 make_imports_language_aware-1183440-41.patch15.14 KBcolan
FAILED: [[SimpleTest]]: [MySQL] Setup environment: Test cancelled by admin prior to completion. View
#27 feeds_field_import_language_aware-1183440-27.patch12.95 KBpcambra
FAILED: [[SimpleTest]]: [MySQL] Setup environment: Test cancelled by admin prior to completion. View
#20 example_importer.txt2.43 KBdrasgardian
#17 feeds_field_import_language_aware-1183440-17.patch12.92 KBsvendecabooter
FAILED: [[SimpleTest]]: [MySQL] Setup environment: Test cancelled by admin prior to completion. View
#8 1183440-field_import_language_aware-6.patch11.24 KBsvendecabooter
FAILED: [[SimpleTest]]: [MySQL] Setup environment: Test cancelled by admin prior to completion. View
#5 1183440-field_import_language_aware-5.patch11.24 KBpcambra
FAILED: [[SimpleTest]]: [MySQL] Setup environment: Test cancelled by admin prior to completion. View
#3 1183440-field_import_language_aware-3.patch11.28 KBpcambra
FAILED: [[SimpleTest]]: [MySQL] Setup environment: Test cancelled by admin prior to completion. View
#2 1183440-field_import_language_aware-2.patch3.64 KBpcambra
FAILED: [[SimpleTest]]: [MySQL] Setup environment: Test cancelled by admin prior to completion. View
#1 1183440_fields_language_aware.patch3.61 KBpcambra
FAILED: [[SimpleTest]]: [MySQL] Setup environment: Test cancelled by admin prior to completion. View
Members fund testing for the Drupal project. Drupal Association Learn more

Comments

pcambra’s picture

Status: Active » Needs review
FileSize
3.61 KB
FAILED: [[SimpleTest]]: [MySQL] Setup environment: Test cancelled by admin prior to completion. View

Just a proof of concept by now, I have it working with a custom module that alters the fields that are marked as translatable by entity_translation module and generates a virtual field for each language-field combination, you need to apply also #1183408: Import into real_target manually to get this working

/**
 * Implements hook_feeds_processor_targets_alter().
 */
function yourmodule_feeds_processor_targets_alter(&$targets, $entity_type, $bundle_name) {
  $languages = language_list();
  foreach (field_info_instances($entity_type, $bundle_name) as $name => $instance) {
    $info = field_info_field($name);
    if ($info['translatable']) {
      $targets[$name]['name'] = $instance['label'] . ' [No language]';
      foreach ($languages as $language) {
        $targets[$name . ':' .$language->language] = array(
          'name' => $instance['label'] . ' [' . $language->name . ']',
          'callback' => $targets[$name]['callback'],
          'description' => t('The @label field of the node in @language.', array('@label' => $instance['label'], '@language' => $language->name)),
          'real_target' => $name,
          'language' => $language->language,
        );
      }
    }
  }
}

About #1183408: Import into real_target, probably the best solution is to pass the entire $targets array to the callbacks so they have much more information and flexibility to do their things instead of just the target name.

pcambra’s picture

FileSize
3.64 KB
FAILED: [[SimpleTest]]: [MySQL] Setup environment: Test cancelled by admin prior to completion. View

Ok, I've rebuilt the code from scratch, first you need to apply the patch in here #1183408: Import into real_target and then this one to get the fields language aware on import.

Now a language selector appears in order to choose the language the field is imported in.

I suppose that this patch could fit into contrib somehow overriding the theme form function for mappers ui but not without this one in #1183408: Import into real_target

pcambra’s picture

FileSize
11.28 KB
FAILED: [[SimpleTest]]: [MySQL] Setup environment: Test cancelled by admin prior to completion. View

Another iteration, now is independent of other patches.

This patch allows feeds to import language aware fields by adding a language selector in the fields admin ui module, it also adds a $mapping array that is passed from the processor to the import targets and modifies the target importers by adding a $language variable based in the mapping.

Pocketpain’s picture

I get alot of:
"Notice: Undefined index: language i theme_feeds_ui_mapping_form() (rad 803 av /modules/feeds/feeds_ui/feeds_ui.admin.inc)."

...when I patch this one. Help?

pcambra’s picture

FileSize
11.24 KB
FAILED: [[SimpleTest]]: [MySQL] Setup environment: Test cancelled by admin prior to completion. View

That's just a notice for those importers that don't have language before this patch, it should be working despite the notices.

I'm attaching a new patch that handles the language missing for old mappings.

Pocketpain’s picture

Question: by default feeds import creates node with the language set to "language natural" which means it doesn't show the translate-tab. And also it won't show that the perticular language is translated. But if you click it, it shows the translated node.

Help?

ropaolle’s picture

Sub

svendecabooter’s picture

FileSize
11.24 KB
FAILED: [[SimpleTest]]: [MySQL] Setup environment: Test cancelled by admin prior to completion. View

I tested the patch in #5 (http://drupal.org/node/1183440#comment-4631354) and it works great.
However the patch doesn't apply cleanly on mappers/link.inc, so in attachment is an update patch that applies without problems on 7.x-2.x.

When using the entity_translation module I do indeed encounter some problems with the UI: the product entities are set to language neutral and thus no translate tab is shown to edit the translation of the fields. If I change the language manually on the entity, the field translations get lost...
But if I just don't touch the product entities at all, and just go about creating product displays referencing the product entity, the fields get translated perfectly...
This is good enough for me at this point. If I find some more time I'll investigate how to work nicer with the entity_translation UI.
But apart from that this patch is ready to be committed IMHO.

Pocketpain’s picture

#8, thanks! Ill test this one.

----

Got the following errors:
can't find file to patch at input line 75
Perhaps you should have used the -p or --strip option?
The text leading up to this was:
--------------------------
|diff --git a/mappers/date.inc b/mappers/date.inc
|index 19aa65d..981ba0e 100644
|--- a/mappers/date.inc
|+++ b/mappers/date.inc

pcambra’s picture

#9 are you patching the "git way"?

patch -p1 < 1183440-field_import_language_aware-6.patch

Sven, I'm glad you've tested this!, we may need to find a solution for the entity translation issue you describe, but I'm sure that this is completely unrelated with feeds.

Pocketpain’s picture

The git-way worked. But when I import, it still won't get a specific language, tho I set the GUID to a language.

pcambra’s picture

With this patch you are able to set the language of specific fields, not to the entire entity, it doesn't make sense to set the guid to a language but the fields that are going to be multilanguage, say a description or a title, with things like guid, nid, images... you'd probably want to set to "all languages"

Pocketpain’s picture

#12, ok thanks. I have 4 different languages on every field.

1. Do I have to get all the translations in the same CSV-file or can I import from different files?

2. How do a node understand it is connected to another translation while importing? Do I need a connecting ID?

pcambra’s picture

I would recommend you to have the translations in the same csv file, i.e. name_en, name_es, name_fr.

This patch doesn't deal with "classical" node translation, this only deals with having fields in several languages so you don't need to import as many nodes as languages you have. Say node 1 with field description in 4 languages, but not 4 nodes.

Pocketpain’s picture

#pcambra: thanks! I'll try that and hope it works... Got a database which is going into about a 100 ckk-fields and 4 different languages, so it's important for me to get this right. Thanks, again.

Pocketpain’s picture

1. Another problem I have noticed is that the languages doesn't say they have been translated. You have to hit "Translate" to see the actual translation. Is there any patch to correct this?

2. Also, I don't understand if or when I need to use default languages when I'm doing the mapping. If I have 2 languages I set "Title_en" to "title" as english and "Title_es" as spanish? Or do I set the "Title_en" as "default language" because it's the main language? Sorry if the questions are stupid, but I had some problems wrong languages showing up at wrong translations...

svendecabooter’s picture

FileSize
12.92 KB
FAILED: [[SimpleTest]]: [MySQL] Setup environment: Test cancelled by admin prior to completion. View

Updated the patch from #8 for mapping/taxonomy.inc, to include support for I18N_MODE_TRANSLATE, i.e. creating a separate term per language.
I had to change the lookup process, because a term / word could be the same in multiple languages, but this should still result in a separate term per language, otherwise that term won't be shown in all the languages it applies to.
Hope that makes sense. I added a check to make sure this functionality only kicks in when I18N_MODE_TRANSLATE is enabled, to avoid conflicts with other modes.

Thanks for reviewing this! I'd like to get this language awareness into Feeds badly.

dgastudio’s picture

applied patch from #17.

i have to importers
1. hotels_all
Do not update existing nodes
PropertyId - Guid (unique) (All languages)
Description - Description (All languages)

2. hotels_es
Update existing nodes (slower than replacing them)
PropertyId - Guid - Spanish
Description - Description - Spanish

i have activated content type translation.

now, after importing CSV with HOTELS_ES, instead of updating the existing nodes, all the nodes appear like standalone documents (with language setted to Spanish) instead of becoming traslations of already existing nodes imported earlier with HOTELS_ALL

i'm doing something wrong or this is bug?

pd. Sorry for my english

pcambra’s picture

#18 the ideal thing would be to to have a unique file like this:

PropertyId - Description_en - Description_es

And have an importer "hotels"
PropertyId - Guid (unique) (All languages)
Description_en - Description (All languages)
Description_en - Description (English)
Description_es - Description (Spanish)

If you can't have this data structure, you'll need to set a unique property in the second importer somehow. Maybe propertyid would work, if not, you'll need to find the node to import with code or maybe using http://drupal.org/project/feeds_tamper

drasgardian’s picture

FileSize
2.43 KB

I've tried the patch from #17 and through the admin ui I can map the fields to the appropriate languages. But when I do the import it doesn't create any translations, it just overwrites the values of the title and body fields of the default language (english).

I'm importing all the languages in one csv file as recommended in #19.
(also tried as an xml file with same results)

Attached is an export of my importer.

Am I missing something?

pcambra’s picture

You can set fields as multilanguage using title and entity translation modules

valderama’s picture

subscribing..

I tried the patch from comment #8 and it mostly worked. However I can confirm #5, which says that the new nodes are language neutral, which seems to cause that the "Translate" tab is not displayed.

However, the fields themselves are translated correctly.

It helps a bit, if I use i18n module the set the current language as default for new nodes. Then the nodes created on import do have a language, and the "Translate" Tab is visible. When I click "Translate", there are the options to "Add Translation" for all languages. And, when I then chose to create a Translation, the fields which where translated on import are already pre-filled.

Soo, multi language field import works "a bit" at the moment, but needs a bit more work...

drasgardian’s picture

@pcambra, I am using the title and entity translation modules and have set the fields to allow translations.

after reading #22 I tried using the patch from #8 instead of the patch from #17 and I had a bit more success. As described in #22 the translate tab is available but no translations are listed as active, when I then click "Add translation" the fields are pre-populated with the imported data.

Also, the title of the base node (English) ended up with the French title.

svendecabooter’s picture

Can you elaborate on what the problem is with the patch in #17?
It's the same as in #8, only handles taxonomy fields import better, so I don't know why it should work worse, except if you encounter taxonomy specific issues.

drasgardian’s picture

Ok I tried #17 again and you're right, it is behaving the same as #8 (I'm not dealing with taxonomy fields). I guess I previously just hadn't noticed that the fields were pre-populated as I was expecting the translations to actually be active rather than needing to go to "Add translation".

Summit’s picture

Subscribing, greetings, Martijn

pcambra’s picture

FileSize
12.95 KB
FAILED: [[SimpleTest]]: [MySQL] Setup environment: Test cancelled by admin prior to completion. View

Just a little modification in #17 to check if the i18_mode is set, as I've found that sometimes is empty and the validation fails

zambrey’s picture

subscribing

zambrey’s picture

Patch #27 works great for me. Thanks guys for fixing this.
I've tested on node body & title (with title module).

iMiksu’s picture

Big thanks #pcambra for the this! #27 patch is working great. Can't wait this to be reviewed and committed!

I think this is very important to be included since I find Feeds very limited when working with multilingual content which comes from other sources... Almost lost my faith on this :S

iMiksu’s picture

One thing, I'm not able to get this work with taxonomy term processor o.O

bastnic’s picture

The patch in #27 works great but it miss to add the traduction in {entity_translation}. The effect is that the translate tab in node edit doesn't show any translation.

itamair’s picture

subscribe! Me too. I'm not able to get this work with taxonomy term processor ...

foopang’s picture

It works very well except for the title it gets replaced by the other language imported. Also, I am using Search API to index a custom field with translation enabled, but it seems that the other language is not being indexed for the field. Is there a way to work around? Thanks!

pcambra’s picture

beansboxchrispang I'd suggest you using the title module for having title as a multilanguage field (I've used it and works great) and for the Search API question, I'd say that depends on which backend are you using, not sure if solr supports entity fields with several languages yet, but if not you can create your own indexes by using hook_apachesolr_update_index and hook_apachesolr_query_prepare or alter.

foopang’s picture

Thanks pcambra, I have tried the title module but it doesn't work for me with the patch. Although i can manually create a node and make the title available in different languages, when import from feeds the original title would get replaced with an other language version.

I use Solr to make the search index, do I need to add related fields to the index? There is Translation source node, I don't know what it is different from the original node.

Also, do I need to specify a default language to use for the contents created by importing a file? I didn't specify one, so the contents created are all language neutral, and I cannot modify them in other languages unless I set the current language for them.

Thanks for your help!

foopang’s picture

Do I need to add translation for each node content after I do the import? It seems that the translated version would not be created automatically. Although the fields were imported in 2 different languages, only the default language ones were created. However, the other language version not created could be seen when going to the node view page. Is that correct?
Sorry for my English, Thanks!

Summit’s picture

Hi,
Love this patch #27!
greetings, Martijn

foopang’s picture

Can the translations get saved automatically instead of having to create them manually with pre-filled values after doing the import? Thanks!

Summit’s picture

Hi,
I reviewed the patch and it works well, thanks!
I see remarks of others that may be changed:
- #32 miss to add the traduction in {entity_translation}.
- #36 title issue.
When these things are handled, I think this patch is ready to commit!

Greetings, Martijn

colan’s picture

FileSize
15.14 KB
FAILED: [[SimpleTest]]: [MySQL] Setup environment: Test cancelled by admin prior to completion. View

The above patch gets us a good chunk of the way there. I'd like to thank everyone who worked on getting it this far.

I've been doing some work to get this initiative rolling again. Here are the other moving parts that need to work in conjunction with this issue, as alluded to by Summit:

In addition to those open issues, we need to explicitly state the prerequisites for using this new functionality effectively. I've added another paragraph of documentation to the Mapping page. This is included in the patch below, and is the only major change to the last patch (other than the previous patch not being properly Git-formatted, while this one is ;).

Prerequisites:

  • Title (When mapping, select the second "Title" in the list. The first one is the core one not provided by the Title module.)
  • Entity translation
  • Multilingual content (Both options "Set current language as default for new content" and "Require language (Do not allow Language Neutral)" need to be checked in the Multilingual settings of each content type being imported.)
colan’s picture

Tagging.

bnine’s picture

Category: feature » bug
Priority: Normal » Critical
Status: Needs review » Needs work

Locale and Content translation should be on

wusel’s picture

For Drupal 7:
I think 'Content translation' is not necessary for my site. Only 'Locale'.

I only want to import in one language (german), because my site only shows german pages to the visitors. If all imported nodes (and the views) know their language (and not 'und'), all is ok to me.

kubedan’s picture

I would like recommend module Feeds: Entity Translation for importing data through Feeds into translatable fields handled by Entity Translation.

franz’s picture

Priority: Critical » Major

Please read issue guidelines, this is not at all critical.

liquidcms’s picture

@colan

i assume this is for nodes that user translatable fields; not translatable nodes. so, re:

(Both options "Set current language as default for new content" and "Require language (Do not allow Language Neutral)" need to be checked in the Multilingual settings of each content type being imported.)

I think, if the setting is set for translatable fields for a specific node type; then it is not possible to be set.

perhaps this is bugs in one of the other modules though??

colan’s picture

@liquidcms: It's been a while, but IIRC, those are necessary to force an explicit language. It won't work otherwise. Yes, this could very well be a bug elsewhere, but I never looked into too deeply. I just made sure those options were enabled. ;)

liquidcms’s picture

too many issues still with translatable fields approach so we went back (yet again) to translatable nodes; in which case i use this new selector set as "all languages" for each field and have a column in import for language and assign this to language as a mapping field. then in one of the import hooks i set the tnid to link the translated nodes together.

mErilainen’s picture

I decided to do the same and was pretty confident that I could link the translations together with Rules since it supports Feeds. But I couldn't do it with Rules. Would you mind sharing your solution using import hooks?

liquidcms’s picture

for our particular case, the titles in FR and EN were the same so we had to after_parse to handle that.

/**
* hooks to importing questions
* 
*  - tweak the title by adding lang suffix to fool importer in to thinking they are unique records
* 
* @param FeedsSource $source
* @param FeedsParserResult $result
*/
function mymodule_feeds_after_parse(FeedsSource $source, FeedsParserResult $result) {
  foreach ($result->items as $key => $row) {
    if ($row['language'] == 'fr') {
      $result->items[$key]['title'] .= '_fr';
    }
  }
}

/**
* preocess the FR import record so we can go back and set TNID on the EN node (now that EN node has been created and we can also set TNID for impending FR node create
* 
* @param FeedsSource $source
* @param mixed $entity
*/
function mymodule_feeds_presave(FeedsSource $source, $entity) {
  if ($entity->language == 'fr') {      
    $realtitle =  str_replace('_fr', '', $entity->title);
    $query = new EntityFieldQuery;
    $result = $query
      ->entityCondition('entity_type', 'node')
      ->propertyCondition('type', 'question')
      ->propertyCondition('title', $realtitle)
      ->propertyCondition('language', 'en')
      ->execute();
    $nid = array_shift(array_values(array_keys($result['node'])));
    $entity->tnid = $nid;
    $entity->title = $realtitle;
    
    //  save tnid on english node too
    $ennode = node_load($nid);
    if ($ennode->tnid == '0') {
      $ennode->tnid = $nid;
      node_save($ennode);
    }
  }
}
twistor’s picture

Alright, reviewing this issue.

Is there a way that we could get this to work with the new mapping config options in dev?

pcambra’s picture

Assigned: Unassigned » pcambra

I'll reroll it, do you have a issue link for taking a look to this new mapping system?

twistor’s picture

Here's the original issue. #860748: Config for mappers?

There should be documentation in feeds.api.php, and you can find a simple implementation in path.inc

Not sure if this approach is feasible, or desired. I'm not terribly adept at this stuff. What are the options going forward? entity_translation vs. locale. I need to sit down and familiarize myself with this stuff.

mErilainen’s picture

For me this functionality is very important. When I'm using entity translations, there is only one node with translatable fields. feeds_et module does this, I'm not sure how is this patch different?

Currently I have all different languages in one XML file. I'm not sure is it possible to use a different importer to update the field translations from another XML file, at least the GUIDs have to match. It's better to have them in the same file I think.

pcambra’s picture

When I'm using entity translations, there is only one node with translatable fields. feeds_et module does this, I'm not sure how is this patch different?

This patch is from June '11 whereas feeds_et is most probably based in the work in this issue and was released in March '12.

I think feeds should support language import in "core" without the need of extra modules, we could include both support for node by node translation and fields translation. For what I know about future translation is based in the field approach so fields and entity translation are more future proof solution.
It's worth to note that entity translation provides interface but the capability of translate a field is core.

twistor’s picture

Looking at the patch in #41, it looks like language support would have to be added to every target field, so the config mapping api is not a good fit. Is it really just as simple as adding $language to each set_target callback?

pcambra’s picture

Ok gave it a review and try to reroll it but before doing that I'd like to get some consensus. Here are some thoughts:

  • Funny thing, some parts of the patch suggested here have been already implemented in other patches, like passing the mapping to the FeedsProcessor, so that part is already in.
  • There's no field.inc anymore and now there's a mapper per core field plus a bunch of non core ones like date or link so that part needs to be adapted.
  • The main part of this patch is obviously replacing the hardcoded 'und' by the specific language.
  • I can't express enough how cool is to have configuration by mapping, I'll explore that further in commerce feeds, that's actually the intention of this patch at the beginning and I think that the language
    should be just another setting for the mapping, here's a UI suggested:
    https://skitch.com/pcambra/ei3hk/feeds-multilanguage-ui
  • Additionally we could add a setting per processor to set the language of the node itself

I don't think we need to establish any dependency with Entity translation, multilingual content or title, multilingual capability is already in core and it's recommended to use some of those, but not mandatory.

joseph.olstad’s picture

Added tnid to mapper and parser.
please see comment #66

joseph.olstad’s picture

Using the updated plugins in post #59, you now get tnid from the Feeds UI, can now map it like you can with nid and uid and other fields.

Again, I have tested this to work using the latest dev build of Feeds as of Friday sept 28 , 2012.

I successfully imported a dataset of 1400 nodes, (700 in english, 700 in french) including tnid.

Please review and schedule for include to the next dev build asap. We will need this functionality and from my tests, it is working.

Thanks

pcambra’s picture

Please review and schedule for include to the next dev build asap. We will need this functionality and from my tests, it is working.

You need to attach your changes as a patch for review

joseph.olstad’s picture

please see comment #66

pcambra’s picture

Honestly I don't think that your patch is in the same page of the rest of the issue, but we've probably got to provide a language settings for nodes anyway so let's take a look to that.

joseph.olstad’s picture

FileSize
4.19 KB
FAILED: [[SimpleTest]]: [MySQL] Setup environment: Test cancelled by admin prior to completion. View

Hi pcambra, please see comment #66

joseph.olstad’s picture

See comment #66

joseph.olstad’s picture

FileSize
4.16 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch TNIDparserAndMapperFix-1183440-64.patch. Unable to apply patch. See the log in the details link for more information. View

pcambra, here is the good patch, I promise!

Please re-test.

This one IS working for SURE.

this patch allows tnid mapping and parsing if the module "translation" is enabled (d7 core), I pulled the latest build and patched against it and named it following the naming convention of description-issuenumber-commentnumber.patch

I've successfully tested this code several times on my system.

I've tested a feed and dataset with a tnid mapping, and a feed and dataset without a tnid mapping.

Both work when tested against this code.

mErilainen’s picture

The title says "Make field import language aware", but the last patch is all about content translation (separate nodes without field translation), right? Should they be moved to another issue? I think they are different use cases. Both useful though, I need both types of translation in my project and multilingual importing as well.

joseph.olstad’s picture

mErilainen

I need both types of translation in my project and multilingual importing as well

Hi mErilainen, it seems like there already is another issue for node.

I've added a step by step HOWTO test the patch discussed in comment #66

Alternatively I could create a feature that would help the maintainers more easily test this patch.

I'd like to help the maintainers on this.

Thanks,

mErilainen’s picture

The last comments in this issue are moving this towards content translation, which is definitely not the same as field (entity) translation. Comment number 58 is the last one I'm hoping to see some progress on.

Do we need a separate setting per processor for the node language when there is the "Language" field where we can map to? Then content should have a source language so that it won't show up as "not translated".

Horroshow’s picture

I agree with you. The patch in #66 helped me a lot but is not the same issue.

Horroshow’s picture

I can't express enough how cool is to have configuration by mapping, I'll explore that further in commerce feeds, that's actually the intention of this patch at the beginning and I think that the language
should be just another setting for the mapping, here's a UI suggested:
https://skitch.com/pcambra/ei3hk/feeds-multilanguage-ui

Cool! That's exactly what this patch needs!

abidvf’s picture

Version: 7.x-2.x-dev » 7.x-2.0-alpha7

Hi I've implemented the patch 6 mentioned in #8, I only want to import description in English using Node ID to update existing nodes, I also using the 'Language' column as well to update only English version, I follow all the steps mentioned in #8 but it still store the description in 'und', hope you understand please help.

Thanks

amanire’s picture

FileSize
4.01 KB
FAILED: [[SimpleTest]]: [MySQL] Setup environment: Test cancelled by admin prior to completion. View

Attached is a version of the code in #66 backported against Feeds-6.x-1.0-beta12

twistor’s picture

Version: 7.x-2.0-alpha7 » 7.x-2.x-dev
Assigned: pcambra » twistor

Need to make this work.

Please post patches regarding the tnid in the correct issue, thanks.

alar’s picture

For now I disabled language, I feel this may come back to bite me :P
I need to find out more about how multiple languages should be imported using a .csv file very soon!

joseph.olstad’s picture

Hi Alar, to import bilingual content that do not already have tnid's using feeds you may be interested in this method:
(Credit to S.R. and J.B. for the work ).

See zip attached
Feeds Translation
=================

Allows Feeds to import nodes and apply Content Translation to them (i.e. tnid)

Installation
------------

1. Download Feeds 2.0-alpha8 or higher
2. Apply the following patch to Feeds: http://drupal.org/files/1994654-add-plugin-alter.patch
3. Enable feeds_translation

Usage
-----

1. Create a English Feeds Mapping (or language or your choice) -> Following field needs to exsist: ```GUID``` (your unique record selector for the Feeds Mapping).

2. Create a French Feeds Mapping (or language of your choice) -> Following fields needs to exsist: ```GUID``` (your unique record selector for the Feeds Mapping) and ```Translation node id```. ```Translation node id``` Feeds source field name can be any value as it will be overwirten in step 3. Example: Source: Blank Source 1, Target: Translation node id.

3. Go to the Feeds Tamper page for the French Feeds Mapping and for the ```Translation node id``` field add the "Get TNID from GUID" plugin

4. Run the English Feed

5. Run the French Feed

- Make sure to use the Language field so the mappings have a Language applied. See table example below to store the language directly in the data.

- When the French feed runs the Tamper plugin will take the GUID and look it up in the Feeds table to find the English/original node and apply the nid of the English node to the tnid of the French node using the ```Translation node id``` field.

- Make sure to use a GUID that is unique to the specific set of language imports. If you have a unreleated feed that shares a GUID value you will end up with conflicts that has the wrong node associated.

- The English and French GUID field should have the same values. Best way to do this is have your multilingual data in a single row data structure:

ID-GUID|Title (Eng)|Title (Fre)|Description (Eng)|Description (Fre)|Language(En)|Language(Fr)
-------|-----------|-----------|-----------------|-----------------|------------|------------
1|English Title|French Title|English Description|French Description|en|fr
2|sample2|sample3|sample4|sample5|en|fr

alar’s picture

Thanks Joseph,
Feeds translation is just what I needed! Thanks for suggesting the bit about the Tamper plugin as well.
The french and english is maintained side by side on one line of excel and is loaded into Drupal as two separate nodes linked by a common GUID.
Merçi

drclaw’s picture

Status: Needs work » Needs review
FileSize
36.63 KB
19.23 KB
PASSED: [[SimpleTest]]: [MySQL] 4,503 pass(es). View

I took a stab at adding field translation to feeds (in response to @pcambra's comment: https://drupal.org/node/1183440#comment-6530976). There's no dependency on the entity_translation module, since field translation is built into drupal core, but it does support it. That is, it will create an entity translation translation record (if necessary) if entity_translation is installed when adding fields that are in a different language than the default one. The code is partly adapted from some of the outdated patches earlier in this issue as well as Feeds: Entity Translation (which no longer works either).

A quick patch overview:

  • Adds a mapping configuration form with a language option to text, number, file and link field mappings (provided that the target field is translatable).
  • Adds a language option to taxonomy field mappings (provided that the target field is translatable)
  • Uses the configured language code from the mapping configuration when importing instead of 'und'
  • I've attached a little screenshot of the mapping page as an example.

I've only had a chance to test it with the 'text' field type, but so far it's been working. It would be nice to get some feedback though to see if I'm on the right track with the code.

Something that is currently not really supported is auto-creation of new taxonomy terms (for taxonomy term reference fields). This is a tricky one and I pondered over it for a while. Suppose you had a content type with a term reference field. The term reference field isn't translatable, but the terms themselves are. Now suppose you're importing into this content type and you have an english and french column in the source for the taxonomy field. When the importer gets to a source row where the term needs to be autocreated, it will create two different terms for the english and french source values. Depending on how we're translating our taxonomy terms, this either gets us halfway there (if we're using i18n_taxonomy), or gives us an extra term we don't need (if we're using entity_translate or simple string translation). Not to mention the fact that the 'name' property of a taxonomy term isn't "field" translatable anyway (you need entity_translate AND the title module to do that). Anyway, I just wasn't sure how much we wanted to support here so I've left it out for now. Ideally it would check what type of translation the taxonomy terms are using, then do the appropriate things when creating new terms... but I thought that might need more discussion...

The patch is against the current dev release. I don't know think it will apply cleanly against 7.x-2.0-alpha8... The file.inc file has changed quite a bit and it might choke on it.

feeds-field-translation-mapper.jpg

drclaw’s picture

Here's a patch against feeds-7.x-2.0-alpha8 in case anyone needs this right away and doesn't want to use the dev release. =)

drclaw’s picture

Hello all.

I tested my changes from the patch in comment #78 today a little more rigorously and discovered a couple issues. Here's a re-rolled version that I've tested (I've uploaded one for the dev branch as well as the current alpha8 release). I can vouch for it working with text, number, file and taxonomy fields. I haven't tested the link field but from the looks of the code it should work...

See comment #78 (https://drupal.org/node/1183440#comment-7579839) for a better description of what the patch does.

drclaw

drclaw’s picture

Title: Make field import language-aware » Multilingual Feeds - Make field import language-aware

I updated the Issue Summary. Seemed like it needed one. I also think it might be worth discussion whether we split this issue off into two separate issues. One for Content Translation support (https://drupal.org/node/1183440#comment-7478058) and one for Field Translation Support (https://drupal.org/node/1183440#comment-7584721). Any thoughts?

twistor’s picture

Assigned: twistor » Unassigned

drclaw, wow, great job!

Thanks for the update, it summarizes things well.

I would add one more point, that language should be mappable, per feed item, but that's a different issue, possibly a different module.

twistor’s picture

Status: Needs review » Needs work
Issue tags: +Needs tests

Overall very nice. It is a bit of a hack, but at this point it's the only way to proceed without a major overhaul. This definitely needs some tests.

+++ b/feeds.moduleundefined
@@ -875,6 +875,70 @@ function feeds_item_info_save($entity, $entity_id) {
 /**
+ * Adds an entity translation if the entity_translation module is enabled.
+ */

Parameters need docs.

+++ b/feeds.moduleundefined
@@ -875,6 +875,70 @@ function feeds_item_info_save($entity, $entity_id) {
+/**
+ * Returns the language summary text for use in a mapper target summary_callback
+ *
+ * @see mappers/text.inc

Param docs.

+++ b/feeds.moduleundefined
@@ -875,6 +875,70 @@ function feeds_item_info_save($entity, $entity_id) {
+function feeds_mapper_summary_language($mapping, $target, $form, $form_state) {
+  if (module_exists('locale')) {
+    $info = field_info_field($mapping['target']);
+    if ($info['translatable']) {
+      $language_options = array(LANGUAGE_NONE => t('All languages')) + locale_language_list('name');
+      if (empty($mapping['language'])) {
+        return t('Language: <strong>@search</strong>', array('@search' => $language_options[LANGUAGE_NONE]));
+      }
+      return t('Language: <strong>@search</strong>', array('@search' => $language_options[$mapping['language']]));
+    }
+  }
+  return FALSE;

This isn't going to work with more complex fields, since target isn't necessarily a field name.

+++ b/mappers/file.incundefined
+++ b/mappers/file.incundefined
@@ -21,6 +21,8 @@ function file_feeds_processor_targets_alter(&$targets, $entity_type, $bundle_nam

@@ -21,6 +21,8 @@ function file_feeds_processor_targets_alter(&$targets, $entity_type, $bundle_nam
         'callback' => 'file_feeds_set_target',
         'description' => t('The URI of the @label field.', array('@label' => $instance['label'])),
         'real_target' => $name,
+        'summary_callback' => 'file_feeds_summary_callback',
+        'form_callback' => 'file_feeds_form_callback',
       );
 
       if ($info['type'] == 'image') {
@@ -29,12 +31,16 @@ function file_feeds_processor_targets_alter(&$targets, $entity_type, $bundle_nam

@@ -29,12 +31,16 @@ function file_feeds_processor_targets_alter(&$targets, $entity_type, $bundle_nam
           'callback' => 'file_feeds_set_target',
           'description' => t('The alt tag of the @label field.', array('@label' => $instance['label'])),
           'real_target' => $name,
+          'summary_callback' => 'file_feeds_summary_callback',
+          'form_callback' => 'file_feeds_form_callback',
         );
         $targets[$name . ':title'] = array(
           'name' => t('@label: Title', array('@label' => $instance['label'])),
           'callback' => 'file_feeds_set_target',
           'description' => t('The title of the @label field.', array('@label' => $instance['label'])),
           'real_target' => $name,
+          'summary_callback' => 'file_feeds_summary_callback',
+          'form_callback' => 'file_feeds_form_callback',
         );
       }

Does it make sense to map each column to a language? I'm not sure we have a choice at this point.

+++ b/mappers/file.incundefined
@@ -129,5 +138,41 @@ function file_feeds_set_target($source, $entity, $target, $value) {
 
+  feeds_mapper_add_translation($entity, $langcode);
+

Is this safe to call so often? For every column for every field.

It seems like at the very least, it will be expensive.

+++ b/mappers/file.incundefined
@@ -129,5 +138,41 @@ function file_feeds_set_target($source, $entity, $target, $value) {
+/**
+ * Mapping configuration summary for text fields
+ *
+ * @param array $mapping
+ *   Associative array of the mapping settings.
+ * @param array $target
+ *   Array of target settings, as defined by the processor or
+ *   hook_feeds_processor_targets_alter().
+ * @param array $form
+ *   The whole mapping form.
+ * @param array $form_state
+ *   The form state of the mapping form.
+ *
+ * @return string
+ *   Returns, as a string that may contain HTML, the summary to display while
+ *   the full form isn't visible.
+ *   If the return value is empty, no summary and no option to view the form
+ *   will be displayed.
+ */
+function file_feeds_summary_callback($mapping, $target, $form, $form_state) {
+  return feeds_mapper_summary_language($mapping, $target, $form, $form_state);
+}
+
+/**
+ * Settings form callback.
+ *
+ * @return array
+ *   The per mapping configuration form. Once the form is saved, $mapping will
+ *   be populated with the form values.
+ */
+function file_feeds_form_callback($mapping, $target, $form, $form_state) {
+  return feeds_mapper_form_language($mapping);

All of these callbacks that wrap feeds_mapper_summary_language() and feeds_mapper_form_language() aren't necessary, just use the functions directly.

drclaw’s picture

Assigned: Unassigned » drclaw

Hi twistor,

Thanks for the quick response and patch review! Here's some thoughts on your comments:

  • RE: Param docs - I'll get working on these. I actually didn't include them because I wasn't sure if you would want to use those functions in the way that I was using them. I figured it made sense to make the language form it's own function to avoid code duplication, but wasn't 100% on the execution.
  • RE: $target - In which case is $target not a field name? (so I can test it and account for it).
  • RE: File field column mapping - I agree, I don't think we can avoid having to choose a language for each column.
  • RE: Entity Translation support - I was a bit lazy on this one. It occurred to me a few days ago that I didn't need to set the translation for every field. Essentially we only need to do it once per entity per non-default language because all we need is the entity_translation record to be saved for the language. So, we can either keep it as it is and just check if there is already a translation for the language using $handler->getTranslations(), OR we could handle translations after the mappings are all done in a presave hook (or something). We would have to scan all the entity fields, however, for languages other than the default and build a list of langcodes, then run feeds_mapper_add_translation() on it. This would be on every row... seems a bit hacky...
  • RE: Callback Wrappers - You know, it's funny. I originally had it without the wrappers, but then changed my mind. I liked the idea of having it wrapped to keep things open for adding more configuration options down the line (similar to the way taxonomy.inc defines additional options).

And yes... Tests are definitely needed. Should we extend the current mapper tests for field, file and taxonomy or create new ones specifically for multilingual testing?

drclaw’s picture

Some things I've noticed in further testing of my patch:

The language of the entity isn't being set properly. For example, when using the Node Feeds Processor, the entity language is hardcoded to LANGUAGE_NONE ('und') in FeedsNodeProcessor::newEntity(). However, if you create a translated node normally using entity translation, the node language will always be saved as the default language (as far as I can tell). In my case this should have been 'en'.

One solution to this problem is to just add the language code to the source. However, sometimes people don't necessarily have the ability to alter their sources so I'd prefer to come up with a better solution. For now I'd suggest that we add a default language selector to the FeedsProcessor abstract class IF the site is multilingual. At least then people can set the language for the new entities. The only thing is we would then have to make sure the various processor sub classes support this new option. Additionally any contrib modules that provide custom processors (e.g. commerce_feeds) will need to add support, but I guess that was always going to be the case with multilingual.

Another bonus with the language selector is that we're probably going to need something like it for content translation support.

Anyway, if anyone has any thoughts or objections let me know. I'll be moving forward with the plan as it is, but am open to discussion.

drclaw

kenorb’s picture

Jamairi Cargos’s picture

Status: Needs work » Needs review
nfavrod’s picture

FileSize
113.45 KB

Hi
I'm currently trying to import nodes with Entity Translation enable. Two fields are enabled: Title_field and body. My default language (and you'll see reading this) : French.
I applied the patch #80: feeds-field_translation_imports-1183440-80.patchof feeds 7.x-2.x-dev (alpha8+7-dev). So I can import French and English Title and Description.
With Devel module, I can see that data is correctly inserted in title_field[en][0]['value'] and title_field[fr][0]['value']
So everything is good to render the fields. The language is correct with this.

The problem is with translation and edition. I made some test with the Target Language : try both FR and EN and array(EN;FR) (exploded with tamper) and I always have only one translation done. When I go to translate tab, I have strange cases

  1. lang: English source: (original content) : no translation and when when I add a new one : translated data of fields is replaced with the "original content" (and are the wrong one actually...
  2. lang : English source: French : but french translation does not exists.... and when I want to add a new translation the path points to [nid]/edit/Array

In all cases, I only have one translation existing for the node. No other are created even if the data exists in the DB and are correct.

I think there's a little more work on the mapper, so the values are correctely sent to the Entity.. But I don't know how to solve this.

Can someone either adapt the patch ?
or tell how Entity Translations are managed when creating the "Original Source" Content (either [Und] or for a specific language)?

So we can adapt the patch to handle this must have feature for Feeds! :-)

drclaw’s picture

FileSize
39.52 KB

@nfavrod:

I believe the issue you're having is specifically with the title module's title field replacement. I had similar issues when I was setting up my multilingual feeds. After some inspection I discovered that you still need to set a value for the title property of the node as well as the title_fields. From what I can tell, the title module does some trickery with the node form to set the old title property (which is still technically required) in a submit handler (title_field_attach_submit). We need to replicate this in our feed mapper because we're not submitting a form. The solution is pretty simple actually, just add a mapping of your default language column to the core title property of the node. See my attached screenshot for a visual example.

As for amending the patch, I'm not sure if this needs to be accommodated for in feeds, or just documented. I think for now it would be best to just document it since feeds is already big enough to maintain without supporting yet another contrib module.

drclaw

nfavrod’s picture

Hello!

Thanks for your post. I actually retried many times and it works!!

What's strange is that I did a lot of tries without Deleting the node.

Finally, the correct mapping setup was :

  • setup the hard node->title (not as unique)
  • setup the node Title Field for every language
  • AND setup a languageOtherwise the node were saved as und and it wasn't possible to edit translations, even if the data were correctly saved in fields

So the only thing that could be improved are some warnings or new logic, when you are about to import node's with translated fields. like:

  1. Setup the language as requiered, otherwise the node is stored as UND and it's not possible to assign it to a language afterward. This value must be a two-char minuscule case (Feeds Tamper can help!)
  2. Setup the Title Field in the language you're about to import is requiered.
  3. Setup node->title could be automatically done if the conditions above are correct: it could be automatically copied in the background from the Title Field of the correct language.

Here we are! Thanks guys for your job and have a nice week-end!

Horroshow’s picture

Thanks a lot for the patch in #80, I've been waiting almost for a year for this feature. Hope someone will commit this for the next alpha version.

Don't know if someone mentioned it but for those you are not seeing the language button on some fields like the Body field : it has to be translatable. Go to your Content Type and Manage Fields, modify your field and at the bottom click the Enable Translation link.

Summit’s picture

Issue tags: -Needs tests

Hi @drclaw,
Is there a new patch to review based on #80 and further?
Would love to be able to use language correctly with feeds.
Right now term-values will not show because somehow they need to be language based instead of "und".

thanks for the great work on this in advance!

Greetings,
Martijn

franz’s picture

Issue tags: +Needs tests

@Summit, any reason for removing the "Needs tests" tag?

Summit’s picture

Nope, sorry if I did this. This was not on purpose.

Summit’s picture

Issue tags: -Needs tests

Hi, Whats the status on this patch related to the review: https://drupal.org/node/1183440#comment-7589933 ?
Would love to have language awareness within the feeds!
Greetings, Martijn

drclaw’s picture

Hi,

I still need to write tests and make some adjustments based on twistor's comments. Time is tight these days, unfortunately! =(

drclaw

drclaw’s picture

FileSize
31.54 KB
FAILED: [[SimpleTest]]: [MySQL] 4,522 pass(es), 65 fail(s), and 4 exception(s). View

Okay, here's a patch re-roll with changes based on Twistor's input with tests. Here's some quick notes on the changes:

  • Fixed the field_info_field($mapping['target']); issue. I didn't understand what twistor meant at first, but I figured it out when the first run of my test failed on a file field (since it uses "sub-field" syntax like field_name:uri). +1 for tests!
  • Param docs should be all there now.
  • I added a language option to the processor form. New entities were being created either with no language set, or 'und' hardcoded as the language (depending on which processor). It defaults to 'und' so it shouldn't affect existing feeds.
  • I changed how translations are added. Instead of calling feeds_mapper_add_translation() on each mapping, I added a new method to the FeedsProcessor class (FeedsProcessor::processTranslations). It gets run before the entity is saved and figures out if translations need to be added. I think this is a far better way to do it because it a) only runs once per entity, and b) it takes the responsibility away from the mapper to manage translations
  • I also added a little support for the title module (FeedsProcessor::processEntityTitleReplacements). Basically the title module replaces un-translatable field properties (e.g. $node->title) with regular fields, but it uses some form_alter trickery that we need to mimic. Without this little fix, people have to map their default language titles twice: once to the title property, and once to the replace title_field.
  • The test is pretty simple. It just initiates a multilingual site with entity translation enabled and imports one item with multilingual field values (one for each supported field type). The test just checks that each value got correctly assigned to the correct language.

Also, just for reference, the language option and the title module support solves the issues discussed in comments #88, #89 and #90 above.

Thanks!
drclaw

Status: Needs review » Needs work

The last submitted patch, feeds-field_translations-1183440-97.patch, failed testing.

drclaw’s picture

Status: Needs work » Needs review
Issue tags: +Needs tests
FileSize
31.89 KB
PASSED: [[SimpleTest]]: [MySQL] 4,503 pass(es). View

Oops. Right, the automated test won't have entity translation... here's a re-roll that adds entity_translation as a test dependency. I also added the dependency to the test itself (in getInfo()). That should do it...

drclaw’s picture

Issue summary: View changes

Issue summary update by drclaw

GaëlG’s picture

If you read this issue, you may also be interested in #2146719: Create aliases in all languages when importing with Feeds.

joseph.olstad’s picture

The last submitted patch, 66: TNIDparserAndMapperFix-1183440-64.patch, failed testing.

AdamGerthel’s picture

I've tried the patch in #97, but I can't get the importer to create new translations of the nodes. If I manually create a translation of an imported node, and then run the import again, the field values of that language are being imported properly. So it's working to some extent.

I'm using:
Feeds 7.x-2.0-alpha8
Entity translation 7.x-1.0-beta3

Update:

I found the issue. The site had both english and swedish activated. The default language was set to english, but the nodes were set to be created in swedish (which was correct). However, I noticed that the function "processTranslations" from the patch looked at the default language. After changing default language to swedish and running the import again, everything worked. The imported nodes were created as swedish nodes, and english translations of them were added by the importer. The fields we're also filled properly.

In essence, the patch probably needs to be updated in order to handle this situation. Default language may very well be different from the language of the imported items.

drclaw’s picture

Issue summary: View changes
FileSize
31.88 KB
PASSED: [[SimpleTest]]: [MySQL] 4,698 pass(es). View
670 bytes

Hm. That's a good point. Seems like we should use the language configured on the feed processor here. Here's an updated patch that should do it. Would you mind giving it a try?

Thanks!
drclaw

stefan.r’s picture

drclaw, are there any caveats to keep in mind while using this patch?

drclaw’s picture

I don't think so, but it's sort of still pending more review and testing by the community so there might still be things I haven't considered. The only way to know for sure is to test it out! =)

justafish’s picture

The patch in #104 worked great for me, thanks!

olofjohansson’s picture

FileSize
22 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds-field_translations-1183440-108.patch. Unable to apply patch. See the log in the details link for more information. View

Modified the patch at #104 for the latest dev version, and...

- Remove translations for languages without any field values. By calling removeTranslation() of the appropriate translation handler, we'll remove existing field values properly from the database.

- Check the language property of the entity when determining the default language instead of calling getLanguage(), since the entity of the language may have been specified during import.

Status: Needs review » Needs work

The last submitted patch, 108: feeds-field_translations-1183440-108.patch, failed testing.

olofjohansson’s picture

FileSize
22.6 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds-field_translations-1183440-110.patch. Unable to apply patch. See the log in the details link for more information. View

Updated for the dev from february 11th, and..

- Use the entity_language() function when checking for the entity language rather than checking the entity property. This property doesn't exist for every entity type, such as taxonomy terms. This has been wrapped within the getEntityLanguage() function, which returns the language that has been configured for the processor if the entity object doesn't have any language support. This means that I renamed the getLanguage() function to getConfigLanguage().

- Don't add translations for undefined field languages.

hluup’s picture

Patch in #110 worked fine for me.

colan’s picture

Status: Needs work » Needs review

Fixing status (which will trigger the test bot).

Status: Needs review » Needs work

The last submitted patch, 110: feeds-field_translations-1183440-110.patch, failed testing.

stefan.r’s picture

Status: Needs work » Needs review
FileSize
22.53 KB
PASSED: [[SimpleTest]]: [MySQL] 5,080 pass(es). View

Re-roll of #110

drclaw’s picture

FileSize
31.91 KB
PASSED: [[SimpleTest]]: [MySQL] 5,080 pass(es). View

Nice work guys! I tested the most recent patch out and it works. The tests I added a while back got lost somewhere along the way though. I've added them back. =)

soraver’s picture

I tried the patch from 115 but it didn't create the translation.
I selected the Language: English and the Language: Greek from the mapping.
I'm not sure if I did something wrong. Could you please provide some instructions?

PedroMiguel’s picture

This patch dont work with custom entity's (like the ones created with ECK). Always get a undefined language index and processor settings form wont save.

drclaw’s picture

@soraver: Gonna need a little more info to help

@PedroMiguel: That's interesting. I haven't tested it against ECK entities so I can't really help you there. I'll give it a try when I have some time though and see what I can do.

soraver’s picture

@drclaw: I can pm you the url and the logins of the site if this helps you

PedroMiguel’s picture

@drclaw, no problem, I end up using other "not conventional" solution for my needs, but as the last dev have support (experimental) to any entity the patch is not ready to apply because break that implementation.

I was using feeds + feed xpath parser, importing xml.

Ps: just to be documented to anyone interested how I did, I just split my feeds and make imports for each one with a default language assigned.

drclaw’s picture

@soraver sure let's try that.

nielvrom’s picture

stijndmd’s picture

patch in #115 seems to work just fine.
Thanks for this, good job!

If I run into anything later on, I will definitely let you know.

colan’s picture

If we can get one or two more positive reviews with basic use cases, I'd like to see this RTBCed soon. (I worked on this issue 2 years ago!)

For additional issues, such as getting this to work with ECK, let's open follow-up tickets. This thread is quite long, and we're already doing quite a bit in the latest patches.

tvl’s picture

About Patch #115, Tested on multilingual site (English, Welsh) using Drupal 7.24 and 7.28 Feeds 7.x-2.0- alpha8+33-dev (also Feeds XPath Parser 7.x-1.0-beta5)(also Entity Translation 7.x-1.0-beta3)

Case 1 (working):
Content type with text-field translatable (let’s say field_foo)
Feeds importer + patch, for single lang. I configure: Language:English OR Welsh
Import the xml -> perfect!

Case 2 (no working):
Content type with text-field translatable (let’s say field_foo)
Feeds importer + patch, but I create two mappings for field_foo, one for English and one for Welsh. At XPath XML parser Settings I map the first field_foo to field-foo-eng (the xml tag is < field-foo-eng >) and the second field_foo (it appears twice because we map it two times, one for each language) to field-foo-cym
Try to import the xml->
An AJAX HTTP error occurred. HTTP Result Code: 200 Debugging information follows. Path: /batch?id=23&op=do StatusText: OK ResponseText: Fatal error: Allowed memory size of 201326592 bytes exhausted (tried to allocate 65488 bytes) in \includes\utility.inc on line 29

Tested with File and http fetcher

stefan.r’s picture

Can anyone reproduce the issue in case 2? Perhaps @tvl would be willing to post a link to the feed, with any sensitive info blanked out?

krabbe’s picture

I've tried patch #115 and it seems to work, except I still have an issue with the title-field. I'm using three languages, DE, EN and FR. The original language is DE. I'm using xpath-xml-parser. I'm mapping the original title to the node title, and the translated titles to the title-field (each language one).
But I only get the german an french title imported correct. The english title field is filled with the german title...
I changed the order of the fields in mapping, but no difference. I don't get the english title imported.
I'm doing the same with the body field and it works perfect.

krabbe’s picture

I forgot the other two screenshots. (Still confused with the "new" style of posting things here)

krabbe’s picture

Mmmh, don't know what I did, but now its working.
I don't know, if I have to activate internationalization. I first didn't, but after activating it (without any of its submodules) I get all three titles in their correct translation.

stefan.r’s picture

FileSize
32.33 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds-field_translations-1183440-130.patch. Unable to apply patch. See the log in the details link for more information. View

It seems $this->config['language'] is defined in FeedsProcessor but not in FeedsEntityProcessor. Adding this so that FeedsEntityProcessorTest passes (it appears the testbot is not running this test?)

--- a/plugins/FeedsEntityProcessor.inc
+++ b/plugins/FeedsEntityProcessor.inc
@@ -135,6 +135,7 @@ class FeedsEntityProcessor extends FeedsProcessor {
       'skip_hash_check' => FALSE,
       'bundle' => NULL,
       'values' => array(),
+      'language' => LANGUAGE_NONE,
     );
   }
stefan.r’s picture

@krabbe (#129), I don't know that Internationalization should be a dependency necessarily. If you weren't running the Title module previously, could you see if using that instead of i18n fixes your issue?

sdrycroft’s picture

Status: Needs review » Needs work

The last submitted patch, 130: feeds-field_translations-1183440-130.patch, failed testing.

stefan.r’s picture

Status: Needs work » Needs review
FileSize
23.27 KB
PASSED: [[SimpleTest]]: [MySQL] 5,529 pass(es). View
sdrycroft’s picture

Thanks for that Stefan. I'm impressed with the work on this patch so far, although I now need to go and update my own Excel import module to ensure it works with this patch.

Greg Boggs’s picture

I believe this patch needs a reroll. It fails to apply for me against the latest git clone.

Hunk #2 FAILED at 44.
Hunk #3 FAILED at 63.
2 out of 3 hunks FAILED -- saving rejects to file mappers/text.inc.rej

Bobík’s picture

FileSize
30.35 KB
FAILED: [[SimpleTest]]: [MySQL] 5,927 pass(es), 3 fail(s), and 0 exception(s). View

Rerolled patch to latest git.

Status: Needs review » Needs work

The last submitted patch, 137: feeds-field_translations-1183440-137.patch, failed testing.

Bobík’s picture

Patch works, but tests are failing because they expect LANGUAGE_NONE key in fields.

Suggestion to improve mappers config:
Currently it is possible to select any globally enabled language or LANGUAGE_NONE. It will be more foolproof if it checks field language configuration if field is translatable (require to select language) or not (use LANGUAGE_NONE and hide field).

MegaChriz’s picture

  1. I think tests are failing because it partly undoes the work in #2224643: Support for input format configuration on a per-field basis :
    +++ b/mappers/text.inc
    @@ -70,10 +80,10 @@ function text_feeds_set_target(FeedsSource $source, $entity, $target, array $val
    -        $field['und'][$delta]['format'] = $mapping['format'];
    +        $field[$langcode][$delta]['format'] = $format;
    
  2. The patch in #137 also adds two files called text.inc.orig and text.inc.rej which should not be in there:
     mappers/text.inc.orig            | 139 +++++++++++++++++++++++++++++++++++++++
     mappers/text.inc.rej             |  62 +++++++++++++++++
    ...
    diff --git a/mappers/text.inc.orig b/mappers/text.inc.orig
    new file mode 100644
    index 0000000..235aea3
    --- /dev/null
    +++ b/mappers/text.inc.orig
    @@ -0,0 +1,139 @@
    ...
    diff --git a/mappers/text.inc.rej b/mappers/text.inc.rej
    new file mode 100644
    index 0000000..88e8d29
    --- /dev/null
    +++ b/mappers/text.inc.rej
    @@ -0,0 +1,62 @@
    
  3. +++ b/mappers/text.inc
    @@ -25,6 +25,8 @@ function text_feeds_processor_targets_alter(&$targets, $entity_type, $bundle_nam
    +        'summary_callback' => 'feeds_mapper_summary_language',
    +        'form_callback' => 'feeds_mapper_form_language',
    

    This will not work when "Text processing" is set to "Filtered text (user selects text format)", because then these callbacks will be overwritten later on:

    if (!empty($instance['settings']['text_processing'])) {
      $targets[$name]['summary_callback'] = 'text_feeds_summary_callback';
      $targets[$name]['form_callback'] = 'text_feeds_form_callback';
    }

Currently it is possible to select any globally enabled language or LANGUAGE_NONE. It will be more foolproof if it checks field language configuration if field is translatable (require to select language) or not (use LANGUAGE_NONE and hide field).

Checking the field configuration for translatability sounds like a good idea, but on the other hand, wouldn't that make this feature more limited?

Manish Jain’s picture

FileSize
20.17 KB
28.37 KB
FAILED: [[SimpleTest]]: [MySQL] 5,925 pass(es), 5 fail(s), and 3 exception(s). View
58.41 KB

Hello All,

I am using the latest stable release of the module and i applied patch in #80 seems to work fine for me, except when creating new nodes on import. It does not set the node language to the default language so had to make some changes to the FeedsNodeProcessor.inc. I created a form field for language in the node settings form which defaults to the default language see the attached image. The language set in this form field is used for all the translatable fields, so we just need to set the language only once and not for every field.

Also adding a patch for the latest dev release, problem addressed in #136 has also been fixed in this patch.

Thanks
-Manish

kenorb’s picture

Status: Needs work » Needs review

Status: Needs review » Needs work

The last submitted patch, 141: feeds-field_translations-1183440-141.patch, failed testing.

Bobík’s picture

Manish Jain: This is not good solution because we need to import multiple languages at once. For example my current configuration looks like this:
screenshot

Bobík’s picture

Btw, patch #137 does not support body field.

stefan.r’s picture

Status: Needs work » Needs review
FileSize
24.8 KB
FAILED: [[SimpleTest]]: [MySQL] Repository checkout: failed to create dependency directory. View

@Bobík: perhaps the body field works only when using Entity Translation?

Updated patch, let's see if this passes tests...

MegaChriz’s picture

@stefan.r
I think you uploaded the wrong patch, there are only line deletions in there.

At least one of the reasons that the test failed is that the patch in #137 (and the patch in #141 too) partly undoes the work in #2224643: Support for input format configuration on a per-field basis , like I said in #140. Also in that issue, target configuration was added for text fields that have filtered text enabled (which the body field has by default). This conflicts with target configuration that is added for field language, as in the current design only one callback for target configuration per target can be defined.

I hope that this clears things up.

stefan.r’s picture

FileSize
33.34 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds-field_translations-1183440-148.patch. Unable to apply patch. See the log in the details link for more information. View

Yes that was the wrong file, here's another try.

The last submitted patch, 146: feeds-field_translations-1183440-146.patch, failed testing.

Status: Needs review » Needs work

The last submitted patch, 148: feeds-field_translations-1183440-148.patch, failed testing.

Status: Needs work » Needs review
stefan.r’s picture

@MegaChriz this patch passes tests but perhaps anyone could do a little code & functionality review to be sure this works well with #2224643: Support for input format configuration on a per-field basis ?

MegaChriz’s picture

@stefan.r
I'll plan to do so, but it might take a while as I'm less experienced in the multilingual thing. With a first glance at the patch in #148, it looks like the language configuration would work in combination with the text format configuration for filtered text enabled text fields. Thanks for your work.

Bobík’s picture

@stefan.r: Bad diagnosis, sorry. It did not worked with formatted long text fields. But with your latest patch it works :-)

Bobík’s picture

FileSize
35.13 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds-field_translations-1183440-155.patch. Unable to apply patch. See the log in the details link for more information. View

I've added option to use $entity->language property as field language to support use case with import table like this:

lang	title
-----------------------
en		English content
cz		Český obsah

Then user can map "lang" column to Language mapper and select Use language property in Target configuration.

I did not wrote test for this feature, because I am quite lost in this tests.

dpico’s picture

Hi! You seem to be doing a great job. Thanks!!! In my opinion, this is a very necessary feature to feeds.

I'm new to this and I also need to import some bilingual feeds. Nothing too sophisticated: title and description in Spanish and English. I found some hope in this thread but after reading 155 comments I can't make myself clear on what patch I should apply to the feeds module that would make the thing. I'm trying Bobík's last one, but I don't know what's the file I should provide to the patch. Is there another module I should install as well?

Thank you very much for any clue on this.

designguru’s picture

#155 Seems to be doing the trick for me! Wonderful :) I can import nodes with default languages!

q./

stefan.r’s picture

@dpico you can apply the patch against the dev version of Feeds.

stefan.r’s picture

FileSize
3.7 KB

Here's an interdiff for Bobik's patch in #155.

I have not tried it but just so I understand his explanation correctly, one field in the feed will assign a language to a record and any other fields within that record that have the "Use language property" option selected will use that same language? And just to make sure this this is something we want, there is no other way to accomplish that, without changing the feed?

Bobík’s picture

@stefan.r: Correct. It is useable for feeds where language of every record is in one of rows. But not only that, you can still set other fields by setting exact language in mapper.

stefan.r’s picture

If there is no other way to do that, it seems useful in order to import feeds with records in multiple languages.

In any way the current description is: "You can select particular field language (or language nautral) or use language code saved in entity's language property (e.g. $node->language) you set by some mapper above.", the select option is called "Use language property", and the "All languages" option is being renamed to "Language neutral". The wording of these is confusing, maybe someone can think of something clearer? This feature will need tests too.

Lastly I am wondering if this is really is a must-have in terms of this issue? If this is only a "nice to have", or there already is a possible workaround (Feeds Tamper?) I'd suggest o get something more basic basic RTBCed (such as the patch in #148), and put this feature in a separate issue as it is introducing quite some further complexity and still needs a bit of work (and testing). This issue is 3 years old already and I would like to see something committed at some point :)

Bobík’s picture

Why not to commit current state? (my patch :-))

Yes it needs tests, but is it problem?

twistor’s picture

twistor’s picture

The current mapping API is a mess and we shouldn't make every mapper have to deal with this.
#2333029: Extend mapping API to allow for defaults and multiple callbacks.
This is "postponed" on the one above, which I should hopefully have done in the next couple days.

twistor’s picture

FileSize
15.83 KB
0 bytes
PASSED: [[SimpleTest]]: [MySQL] 6,191 pass(es). View

Just experimenting to see if I'm on the right track with #2333029: Extend mapping API to allow for defaults and multiple callbacks..

As I'm browsing through the code, I'm struck that there are too many things happening at once.
It would be nice if we could break out, title field, and entity_translation into separate issues. I'm not sure about the entity_translation, I have to dig into processTranslations() to see what it's trying to do.

Consider this a git stash.

twistor’s picture

FileSize
29.48 KB
FAILED: [[SimpleTest]]: [MySQL] 6,162 pass(es), 20 fail(s), and 1,873 exception(s). View
twistor’s picture

Is there a reason we're using locale_language_list() in some places and language_list() in others. It seems to me, that if locale.module isn't enabled, there should be no need for translation at all. I'm probably wrong though.

Status: Needs review » Needs work

The last submitted patch, 166: feeds-field_translations-1183440-166.patch, failed testing.

partdigital’s picture

Why were the hooks_feeds_processor_targets_alter() function changed to hook_feeds_processor_targets()?

See text.inc, taxonomy.inc etc. These aren't invoked anywhere and doing this breaks the import.

Edit: Disregard this comment, I see what's happening.

onotic’s picture

Can anyone help me out? I'm not so familiar with applying patches.

For some reason the patch won't apply to my Feeds.
I used feeds-field_translations-1183440-165-do-not-test.patch

I'm running Feeds 7.x-2.0-alpha8 (not the DEV version)

When I scan the patch and compare it to my files, the code is completely different.

Tried aswell with DEV but same results.

error: patch failed: feeds_ui/feeds_ui.admin.inc:635
error: feeds_ui/feeds_ui.admin.inc: patch does not apply
error: patch failed: mappers/taxonomy.inc:96
error: mappers/taxonomy.inc: patch does not apply
error: patch failed: plugins/FeedsProcessor.inc:669
error: plugins/FeedsProcessor.inc: patch does not apply

Thanks in advance

The last submitted patch, 155: feeds-field_translations-1183440-155.patch, failed testing.

stefan.r’s picture

@onotic:

git clone --branch 7.x-2.x http://git.drupal.org/project/feeds.git
cd feeds
git checkout a6abe50
wget https://www.drupal.org/files/issues/feeds-field_translations-1183440-155.patch
patch -p1 < feeds-field_translations-1183440-155.patch 

This gets the git repo from back in august and applies the patch in #155 in it (I think the ones in #165 and #166 were just experiments)

@twistor: any pointers on further work you would like to be done on the patch in #166?

onotic’s picture

@stefan.r

Thanks a bunch! It works :) Also happy to see it works with commerce products.
You have any idea if and when there will be a new version of the feeds module implementing this patch?

valderama’s picture

I've tested the patch (#155) and tried to import taxonomy-terms with a name_field in 4 languages. Its works fine as long as the default language of the site is set to English. If I set German as the default language, then all English fields are filled with the German term-name.

Any ideas?

Thanks!

valderama’s picture

It seems the problem is caused by title module - title_entity_presave() sets current active language as the source language for the title field. Some really complicated crap, imho.

In practice one can avoid this problem, if you start the feeds import when your default language. E.g. If German is your default language, trigger the import from de/import/your_importer

Calystod’s picture

FileSize
29.51 KB
FAILED: [[SimpleTest]]: [MySQL] 6,162 pass(es), 20 fail(s), and 63 exception(s). View

I test the patch #166. It doesn't work for Inhereted Language.

In FeedsProcessor.inc, the patch add:

foreach ($mapping['preprocess_callback'] as $preprocess) {
          $preprocess($source, $target_item, $mapping['target'], $value, $mapping);
}

which must be

foreach ($targets[$this->id][$mapping['target']]['preprocess_callback'] as $preprocess) {
          $preprocess($source, $target_item, $mapping['target'], $value, $mapping);
}

The patch works correctly for me after this correction (including with targets of others modules).

But, apparently, this same code is add by the patch on issue 2333029 too. So, the patch of issue 2333029 must be apply before the patch #166 and the code between these issues shouldn't be repeat.

I add the patch #166 with my correction for now but isn't the best practice.

Calystod’s picture

FileSize
14.29 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds-field_translations-1183440-178.patch. Unable to apply patch. See the log in the details link for more information. View

Patch without the modifications provided by issue 2333029. You must apply the patch number 18 of this issue before to apply this patch.

I've always 2 bugs: In administration page of importer, I can't configure some targets like textfield with html filter, I have an AJAX error. I suppose it's because I have 2 parameters on the target's configuration.

If I use Feature to configure this information, I have an other bug when I verify the content of $entity: The field have 2 languages. The inheret language and the undefined language. But for undefined language, value is empty.

Calystod’s picture

FileSize
14.29 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds-field_translations-1183440-179.patch. Unable to apply patch. See the log in the details link for more information. View

Just a little correction in the name of preprocess.

catopuma’s picture

For my understand i use the latest Feeds dev release and i need for first Patch #18 and and then your latest Patch?

I have an issue with updating Nodes/Entities to add translations with a second import.
https://www.drupal.org/node/2350443

I use at the moment the Feeds Entity Translation module

stefan.r’s picture

FileSize
43.09 KB
FAILED: [[SimpleTest]]: [MySQL] 6,072 pass(es), 99 fail(s), and 2,561 exception(s). View

@neofelis1985 yes but that is patch #18 from this issue: #2333029: Extend mapping API to allow for defaults and multiple callbacks.

Alternatively you can use the patch in this comment which combines both patches (so that the testbot can pick it up).

stefan.r’s picture

Status: Needs work » Needs review

The last submitted patch, 178: feeds-field_translations-1183440-178.patch, failed testing.

The last submitted patch, 177: feeds-field_translations-1183440-177.patch, failed testing.

The last submitted patch, 179: feeds-field_translations-1183440-179.patch, failed testing.

Status: Needs review » Needs work
ali_b’s picture

i am confused - which patch can i use?

stefan.r’s picture

@ali_b, I would go with #148 as that is the last patch that passed tests.

The last submitted patch, 148: feeds-field_translations-1183440-148.patch, failed testing.

stefan.r’s picture

Status: Needs work » Needs review
FileSize
141.48 KB
PASSED: [[SimpleTest]]: [MySQL] 6,191 pass(es). View

re-roll of #148

ali_b’s picture

thanks stefan, after apllying patch 148 I got error:
Parse error: syntax error, unexpected ''term_search'' (T_CONSTANT_ENCAPSED_STRING), expecting '(' in .../sites/all/modules/feeds/mappers/taxonomy.inc on line 300

joseph.olstad’s picture

Hi Ali_b , I also noticed that the recent commit 45152d374e6 (committed friday Jan 16th) breaks patch 191 from this issue

The workaround is to apply patch 191 on commit 006234fc3cc859e (Tue Dec 23 13:54) rather than the latest commit.

Here is what you have to do to test it:

git clone --branch 7.x-2.x http://git.drupal.org/project/feeds.git;
cd feeds;
wget https://www.drupal.org/files/issues/feeds-field_translations-1183440-191.patch;
git checkout 006234fc3cc859e4fec0cb261675577efcf389b8 --hard;
git apply < feeds-field_translations-1183440-191.patch

What has to happen now is for someone to refactor (reroll) patch 191 but using the latest commit from dev.
That doesn't mean we can't test patch 191 , if you want to test it, use the steps above with the previous commit from dec 23.

Ali_b, please test out the patch using the above steps, and let us know how it works.

colan’s picture

Status: Needs review » Needs work
Issue tags: +Needs reroll

Updating status based on #193.

nsurround’s picture

I applied patch 191 on commit 006234fc3cc859e (Tue Dec 23 13:54) as was suggested and tested using taxonomy term processor. Having two name_fields (Title module) with different language assigned produces 'An AJAX HTTP error occurred. HTTP Result Code: 500 Debugging information follows. Path: /batch?id=.....' error when trying to import. Using only one name_field the import works including using a custom entity field wilth different translations mapped. Looked at the multilingual.csv from the patch (test) and see that the title should be translatable etc. Not sure multilingual taxonomy terms has actually been tested yet with the patch 191 or is a seperate issue.

Breakerandi’s picture

When will this go into the dev version? I have no git and i need it urgent..

flyke’s picture

Does anyone have an update on nsurround's question ?

I have a taxonomy using entity translation.
Title and description of the terms ar translated using the Title module.

I would like to be able to use feeds to import terms for this taxonomy from a csv file with these fields for example:

  • term_id
  • term_name_en (this is the English term name)
  • term_name_nl (this is the Dutch term name)
  • term_description_en
  • term_description_nl
flyke’s picture

When using content translation (not entity translation) I can comfirm that everything stated in #76 works.
I also found these instructions where I found the feeds translation module: https://github.com/CityofOttawa/feeds_translation

So following the instructions I was able to import and translate nodes from a csv file using Feeds and Feeds Tamper.

Next I will try to tackle my taxonomy with entity translation problem and see how that goes.

flyke’s picture

I'm happy to say that, after testing, using entity translation and Feeds also works for me :-). Using latest dev of Feeds, Feeds Tamper and also the Feeds Translation module.

For content translation you need to create 1 importer per language an follow instructions on #76. Each field must be included only once but need to have the correct language set.

For entity translation you only need 1 importer, and readd each translatable field for each language in that importer.

nsurround’s picture

flyke, were you able to use entity translation with only 1 importer and be able to translate the taxonomy term name (title property). I was able to translate any custom field added to a term but not its term name title. For instance EN term name 'Red' could not translate to ES 'Rojas' etc using entity translation, feeds and taxonomy term processor. If you were successful with this could you explain a bit more the method or module versions used? Thanks for you input.

JamesAn’s picture

I'm hoping to also confirm the method in #1183440-199: Multilingual Feeds - Make field import language-aware with Entity Translation, of using the latest dev snapshot of Feeds and Feeds Tamper along with the Feeds Translation module attached in #1183440-76: Multilingual Feeds - Make field import language-aware. Fingers crossed.

I won't be looking at the Content Translation implementation as that's not the route being taken in this multilingual client project.

flyke: just to clarify, no actual patches were applied, correct?

1mundus’s picture

I can confirm that core (Content Translation) translation method works with Feeds if you follow #76. Actually, you can even make it easier and skip Feeds Tamper part. In that case, here are the steps necessary:

Usage

1. Create a English Feeds Mapping (or language or your choice) and use the following fields in this example:
GUID (if you don't use Title as unique field), Title, TNID, language

2. Create a French Feeds Mapping (or language of your choice) -> and use the following fields in this example:
GUID (if you don't use Title as unique field), Title, TNID, language

3. Run the English Feed

4. Run the French Feed

- Make sure to use the Language field so the mappings have a Language applied.

Here is an example of the csv file without GUID being used:

Title (En)|Title (Fr)|Description (En)|Description (Fr)|Language(En)|Language(Fr)|TNID
English Title|French Title|English Description|French Description|en|fr|100

sboutas’s picture

I have tried the #76 but the tnid of the French nodes is 0.
I have noticed that the Feed items' GUID is a URL (unique for each item).
The English feed item and its French translation have the same GUID though.
Does the GUID has to be numeric in order for the plugin to work?
If yes, how it can be modified to check for strings?

ekes’s picture

I've been reading up on this thread, and I'm missing what about when the feed, or feed element itself is specifying the language?

RSS allows specifying the language of the whole channel.
ATOM uses xml:lang which can be granular to the element.
iCal has a LANGUAGE property that can also be granular to the component.

Unlike CSV, or similar, where you have set up to have one column/field for one language, and another for another, and where you can then make multiple mappings; you have to map your DESCRIPTION to Body, or whatever, and allow the feed to specify the language - which can change per feed or field - all using the same configuration.

What seems obvious coming from this perspective is that the FeedsElement should have a property for language that could be set by the parser. Then the mapper can use this (as appropriate for the field).

How would this fit with the above configuration solutions?

stefan.r’s picture

@ekes that would work, but it assumes people have control over what their feeds look like. Multilingual elements don't always have a language property.

ekes’s picture

@ekes that would work, but it assumes people have control over what their feeds look like. Multilingual elements don't always have a language property.

At which point the FeedsElement::language remains LANGUAGE_NONE and the mapper continues as it is.
With the exception of the above discussion where the user is configuring this when configuring the Processor Mappings?

1mundus’s picture

@ekes

I believe you can use Feeds tamper to set a default value on a language field. Feeds Tamper's plugin name is "Set value or default value".

stefan.r’s picture

I meant if the feed looks like this, maybe we don't want the language of all the elements to be LANGUAGE_NONE:

<feed>
  <item>
    <element_en>Hello</element_nl>
    <element_es>Hola</element_es>
  </item>
</feed>

We'd need to use something like Feeds Tamper to transform the format of the feed...

In any case this issue seems a bit stuck, maybe the Feeds maintainers can give some direction?

MegaChriz’s picture

I think that #2333029: Extend mapping API to allow for defaults and multiple callbacks. first should get in, in order to fix this issue properly. I think we need to have the possibility to have multiple summary/form callbacks, for the reasons noted in #147.

Another reason this issue is hold back is because getting a new release out has more priority, see #2290465: [Meta] Create beta 1 release. To get this out sooner, a proposal is made to remove the generic entity processor, see #2469219: Remove the Generic Entity Processor.

virusakos’s picture

With latest dev feeds and feeds_et I have managed to import a multilingual taxonomy vocabulary without applying any patch.
I just had to create a taxonomy with localize translation mode and replace the name field with an entity field.

The source for my taxonomy vocabulary has the following format:
term name | translation 1 | translation 2

The importer uses term name for both target Term name (name) used as unique and Name [English] (name_field:et:en)

sboutas’s picture

I have two feeds, one in English and one in French where the GUID in both languages is the same. When I import these feeds I create nodes of a specific content type. I want somehow to link the English and French nodes and make them aware of the translation so I can use the language switcher when I read the article to switch to its translation.

None of the above patches or custom code worked for me so I created my own module and got my feeds translated.
I only used feeds tamper module so I can add the language to the target. I used blank source in the source and language field at the target and then in feed tamper I used a rewrite for that field with the value set to en for the English one and fr for the French one.

The fields that I used for both feeds (Source -> Target):
Item GUID -> GUID (Used as unique), Blank source -> Language, Title -> Title, Description -> Body, Published date -> Published date.

Here is my module's code. Sorry but I was too lazy to tidy it up :) It works very well though.


function my_module_feeds_after_save(FeedsSource $source, $entity, $item, $entity_id) {
  if ($entity->type == 'my_content_type' && $entity->language == 'en') {
    //When importing from the English feed
    $nodeid = db_query("SELECT entity_id FROM {feeds_item} WHERE guid =" . $entity->feeds_item->guid . "")->fetchField();

    if ($nodeid) {
      $node = node_load($nodeid);

      if ($node->language == 'fr' && $node->tnid == '0') {
        $ennode = node_load($entity->nid);
        $ennode->tnid = $ennode->nid;
        node_save($ennode);

        _add_link_item($ennode, 'your_parent_link_id', 'en');

        $node->tnid = $ennode->nid;
        
        /* Updates the French node's alias path with the English one. On my site we use only aliases in English.
        Skip this line if you want your alias in French. */
        $node->path = array('alias' => $ennode->path);
        
        node_save($node);
      } else {
        _add_link_item($node, 'your_parent_link_id', 'en');
      }
    }
  } else if ($entity->type == 'my_content_type' && $entity->language == 'fr') {
    //When importing from the French feed
    $nodeid = db_query("SELECT entity_id FROM {feeds_item} WHERE guid =" . $entity->feeds_item->guid . "")->fetchField();

    if ($nodeid) {
      $node = node_load($nodeid);

      if ($node->language == 'en' && $node->tnid == '0') {
        $node->tnid = $node->nid;
        node_save($node);

        $frnode = node_load($entity->nid);
        $frnode->tnid = $node->nid;
        node_save($frnode);

        _add_link_item($frnode, 'your_parent_link_id', 'fr');
      } else {
        _add_link_item($node, 'your_parent_link_id', 'fr');
      }
    }
  }
}


//This function adds a menu link to the feed item
function _add_link_item($node, $plid, $lang) {
  $linkitem = array(
    'menu_name' => 'main-menu',
    'link_title' => $node->title,
    'link_path' => 'node/'. $node->nid,
    'plid' => $plid,
    'language' => $lang,
  );

  menu_link_save($linkitem);
}

twistor’s picture

Status: Needs work » Needs review
FileSize
63.2 KB
FAILED: [[SimpleTest]]: [MySQL] 7,038 pass(es), 4 fail(s), and 12 exception(s). View

Let's see what happens.

Status: Needs review » Needs work

The last submitted patch, 212: feeds-multilingual-1183440-212.patch, failed testing.

twistor’s picture

Status: Needs work » Needs review
FileSize
63.37 KB
PASSED: [[SimpleTest]]: [MySQL] 7,062 pass(es). View
twistor’s picture

Assigned: drclaw » twistor
Issue tags: -Needs reroll

This patch includes #2333029: Extend mapping API to allow for defaults and multiple callbacks.. If someone wants to review that, I could get to this fairly quickly.

planctus’s picture

I've applied successfully the patch at #214 to the dev version of the module.
I got this, though, in an installation with many other drupal plugins:
PHP Fatal error: Call to protected FeedsImporter::__construct() from context 'FeedsPlugin' in ...feeds/plugins/FeedsPlugin.inc on line 59
In a clean install i'm not able to reproduce the error and feeds seems stable, instead.
Thanks,
Da.

twistor’s picture

@planctus, are you sure that was from this patch at not updating the dev version of Feeds? It might be related to #2497507: Pass plugin definition to FeedsPlugin objects..

Can you post the full error message in that issue? It should be easy to track down.

planctus’s picture

I think you're perfectly right @twistor, i also had to upgrade feeds to apply the patch, indeed.
Da.

twistor’s picture

FileSize
20.77 KB
PASSED: [[SimpleTest]]: [MySQL] 7,064 pass(es). View

Alright! #2497507: Pass plugin definition to FeedsPlugin objects. was committed, that makes this patch much smaller.

Still needs tests.

Could really use some people manually testing as well.

planctus’s picture

The patch applies flawlessy, i've no clues about how to test a multilingual import with this new patch, though. Thanks.

MegaChriz’s picture

@twistor There is a test called FeedsMapperMultilingualFieldsTestCase in the patch from #191.

@planctus
Things to test include:

  • Behaviour with locale module turned on and off.
  • Behaviour with Content Translation module turned on and off.
  • Behaviour with Entity Translation module turned on and off.
  • Behaviour when importing to content (nodes, terms, users) that are *not* multilingual.
  • Behaviour with the various multilingual settings, for example:
    • Require language (do not allow language neutral) setting turned on and off.
    • Lock language (cannot be changed) setting turned on and off.
    • Limited amount of allowed languages vs all languages are allowed.

There are lots of things to test. Maybe it would be a good idea to come up with a testplan that describes a few use cases with steps to follow. Or, as a start, only describe the use cases and let the testers figure out the steps to take.

twistor’s picture

@MegaChriz, Good to know, I'll roll that in.

As for what to test:

  • This patch doesn't deal with Content/node translation at all.
  • It does support locale and entity translation.
    • Locale support allows you to assign a language to nodes and other entities will need minor patches to add support.
    • Entity translation support will allow you to assign a language to any translatable field.
  • I haven't worked on the "Require language", "Lock language", or limited languages at all yet.
1mundus’s picture

@Twistor and MegaChriz

Were there any changes in the previous releases that have made the patch in #76 redundant?

I'm using Feeds 7.x-2.0-alpha8+85-dev and I have managed to update TNID's just by installing Feeds translation module from #76. Patch wasn't necessary and translated nodes are connected properly to original nodes.

If this newest patch (#219) deals only with entity translation, please consider adding core translation support as well. #76 seems like an approach that's working well.

codefactory’s picture

i d like to thank twistor for the patch

I applied the patch to the latest dev version (2015-Jun-04) and when i tried to use it it threw this error
An AJAX HTTP error occurred. HTTP Result Code: 500 Debugging information follows. Path: /batch?id=42&op=do StatusText: Service unavailable (with message) ResponseText: Recoverable fatal error: Argument 3 passed to locale_feeds_preprocess_callback() must be an array, string given in locale_feeds_preprocess_callback() (line 19 of /sites/all/modules/feeds/mappers/locale.inc).

i ve checked with watchdog debug prints and the arguement is string indeed.
I ve removed the array type in front of the $target argument (file feeds/mappers/locale.inc)
function locale_feeds_preprocess_callback(FeedsSource $source, $target_item, array $target, array &$mapping) {
to
function locale_feeds_preprocess_callback(FeedsSource $source, $target_item, $target, array &$mapping) {

and the module seems to work as expected
I m not quite sure what the implications are however

twistor’s picture

FileSize
1.07 KB
21.59 KB
PASSED: [[SimpleTest]]: [MySQL] 7,064 pass(es). View

Thanks, that is a bug indeed.

tobiasb’s picture

FileSize
21.75 KB
PASSED: [[SimpleTest]]: [MySQL] 7,214 pass(es). View

Updated patch for the current dev.

stefan.r’s picture

So this just needs tests now? Any pointers on what would need to happen for this to support content translation as well?

twistor’s picture

Issue tags: -Needs tests
FileSize
30.56 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds-multilingual-1183440-228.patch. Unable to apply patch. See the log in the details link for more information. View

Any pointers on what would need to happen for this to support content translation as well?

That should be a separate issue. I honestly haven't thought about it yet. Since content translation uses separate nodes, it sounds like it would be a very different solution.

Can anyone figure out why testMultilingualFieldMappings() needs to do $this->drupalGet('node/1/translate'); before the French translation will show up in the test? I'm stumped.

Status: Needs review » Needs work

The last submitted patch, 228: feeds-multilingual-1183440-228.patch, failed testing.

MegaChriz’s picture

Hi twistor, if you can wait two days with making major changes to the patch, that would be great. I'm nearly done with reviewing the patch in #228, I just need to wrap it up a bit to make my review good followable. I have tested a lot of cases, some of them might be good to use for automated tests. I did not touch the code yet nor did I looked at the automated test: it won't harm my review if you only would change the tests.

twistor’s picture

Status: Needs work » Needs review
FileSize
3.03 KB
30.89 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds-multilingual-1183440-231.patch. Unable to apply patch. See the log in the details link for more information. View
twistor’s picture

Opps. Cross post. You can see the interdiff, it's pretty small. I'll leave this for a bit then.

MegaChriz’s picture

Status: Needs review » Needs work

It took me a while to get through this (> 12 hours), but here is my review. This review mainly focusses on testing the patch in action. I've glanced through the code, but I skipped reading the details of mappers/locale.inc, mappers/entity_translation.inc and tests/feeds_mapper_multilingual_fields.test for now. Maybe I will review the code more in detail later.
I started with testing using the patch from #225 first, then later continued with the patch from #228.

First, I had to figure out in which possible ways entities or fields can get a language assigned with Feeds. I found out that there are three possible ways:

  1. Setting a language on the processor
    Provided by the patch
  2. As target configuration
    Provided by the patch
  3. As a mapper
    Already available

Processor: testing imports using the language setting on the processor

I did some tests using only the language setting on the processor, not using the language setting on the targets.

Setup

  • Locale module enabled
  • Three installed and enabled languages: English, Dutch and German
  • Fetcher: file upload
  • Parser: CSV parser
  • Processor settings:
    • Language: German
    • Update existing items
    • Skip hash check
  • Source: a CSV file with two items

Node processor

To get the language setting available on the node processor settings, the Locale module must be enabled first. This is good.

Scenario Result Status
Check if the language setting on the processor is available with the Locale module enabled. Language setting is available. Correct
Check if the language setting on the processor is available with the Locale module disabled. Language setting is not available. Correct
Import new items. Two nodes are imported in German. Correct
  1. Change processor language from "German" to "Dutch".
  2. Update existing items.
Two nodes are updated, but their language is still German. Uncertain
  1. Disable Locale module.
  2. Import new items.
Language setting on processor is no longer available. Two nodes are still imported in German. Uncertain
  1. Uninstall Locale module.
  2. Import new items.
Language setting on processor is no longer available. Two nodes are still imported in German. Uncertain
  1. Disable language "German".
  2. Import new items.
On the processor settings, the language "German" is no longer available. Two nodes are still imported in German. Uncertain
  1. Remove language "German".
  2. Import new items.
On the processor settings, the language "German" is no longer available. Two nodes are still imported in German. Uncertain
  1. Enable Content Translation module.
  2. Disable multilingual support for content type.
  3. Import new items.
The language settings are still available on the processor. Two nodes are still imported in German. Incorrect, but unsupported?
  1. Add a mapping target to language.
  2. In the source, set language of first item to "nl" (Dutch), leave language for second item empty.
  3. Import new items.
The first node is imported in Dutch (correct). The second node doesn't have a language at all, not even LANGUAGE_NONE (incorrect). I would have expected that it got the German language assigned. Incorrect, but out of scope for this issue?

Taxonomy term processor

For the Taxonomy term processor, the module Taxonomy translation (i18n_taxonomy) had to be enabled. This module is part of the Internationalization project.
I also tried if the language setting became available with enabling the Entity translation module (instead of Taxonomy translation), but that didn't happen. On admin/config/regional/entity_translation I had turned on "Taxonomy term" for translatable entity types.

Scenario Result Status
  1. Enable Entity Translation module.
  2. Enable translation for taxonomy terms on admin/config/regional/entity_translation.
  3. Check if the language setting on the processor is available.
Language setting is not available. Incorrect
  1. Enable Taxonomy translation module.
  2. Check if the language setting on the processor is available.
Language setting is available. Correct
  1. Enable Taxonomy translation module.
  2. Set processor language to "German".
  3. Set translation mode for vocabulary to "Localize".
  4. Import new items.
Two terms are imported in LANGUAGE_NONE (should have been German). Incorrect
  1. Enable Taxonomy translation module.
  2. Set processor language to "German".
  3. Set translation mode for vocabulary to "Translate".
  4. Set website language to "Dutch".
  5. Import new items.
Two terms are imported in the current active language, Dutch (should have been German). Incorrect

I did not test importing terms further because I didn't manage to import them in the correct language in the first place.

Target configuration: testing imports by setting a language on a target

Setup

  • Modules Locale and Entity Translation enabled
  • Three installed and enabled languages: English, Dutch and German
  • A content type with the following fields with translation enabled:
    • Long text and summary (Body)
    • Date (Unix timestamp)
    • Image
    • Integer
    • Link
    • Term reference
    • Text
  • Fetcher: file upload
  • Parser: CSV parser
  • Processor settings:
    • Language: Language neutral
    • Update existing items
    • Skip hash check
  • Mapping targets:
    • GUID (Used a unique)
    • Title
    • Body (German)
    • Date: Start (German)
    • Image: URI (German)
    • Image: Title (German)
    • Image: Alt (German)
    • Integer (German)
    • Link: URL (German)
    • Link: Title (German)
    • Term reference (search by name, auto create, German)
    • Text (German)
  • Source: a CSV file with two items

Test results per field

Field Result Status
Body The body field gets the German language assigned. Correct
Date The date field gets the German language assigned. Correct
Image The image field gets the German language assigned, including all subfields ("URI", "Title", "Alt"). Correct
Integer The integer field gets the German language assigned. Correct
Link The link field gets the German language assigned, including all subfields ("URL", "Title"). Correct
Term reference The taxonomy term reference field does get the German language assigned, but automatically created terms are in LANGUAGE_NONE, even if the taxonomy is set as multilingual (with Taxonomy translation or Entity Translation modules). Incorrect
Text field The text field gets the German language assigned. Correct

Other tests using language setting on the target configuration

Scenario Result Status
  1. Import new items in German.
  2. For each field mapper, change the language from "German" to "Dutch".
  3. Update existing items.
The fields now gets the Dutch language assigned, but the German version of the node is completely removed. I don't think that this is right. See next scenario. Incorrect
  1. Create a node, set language to Dutch, fill in the text field.
  2. Remove all existing mappers.
  3. Add a mapper for the text field and set language to "German".
  4. Add a mapper for title and set this as the unique target.
  5. Import a CSV file with one item that has the same title as used for the created node.
A German version of the existing node is created, but the Dutch version of the node is removed. I think this is incorrect because from an user perspective you only wanted the feed to control the German version of the language. Incorrect
  1. Remove all existing mappers.
  2. Add a mapper for the text field and set language to "German".
  3. Add another mapper for the text field and set language to "Dutch".
  4. Import a CSV file with one item.
One node is created. For the text field, two entries are created (Dutch and German). Correct
  1. Disable Entity Translation module.
  2. Import new items.
All fields get the German language assigned. On the target configuration, the "language" option is still available. I think this is correct because I read that the Entity Translation module is mainly an UI module. Correct
  1. Uninstall Entity Translation module.
  2. Import new items.
All fields get the German language assigned. On the target configuration, the "language" option is still available. I think this is correct for the same reasons as in the previous scenario. Correct
  1. Disable language "German".
  2. Import new items.
All fields get the German language assigned. The German language is no longer selectable when changing target configuration for each field mapper. On the mapping page, the following notice is shown:

Notice: Undefined index: de in locale_feeds_summary_callback() (line 48 of feeds/mappers/locale.inc).

The fact that the notice is displayed seems incorrect to me, even though it is caused by a misconfiguration. A more user-friendly message would be nice.

Incorrect (but caused by misconfiguration)
  1. Remove language "German".
  2. Import new items.
Two nodes get created but any data for the missing language is not imported. However, the images get created on the file system. Incorrect (but caused by misconfiguration)

Code

When looking at the code I only found one minor things so far:

  • There is one leftover occurence of "und" on ± line 226 of tests/feeds_tests.module.

Other things that I saw which do not have to be an issue:

  • The patch from #191 contained specific code for the Title module, and that is no longer included in the latest patch. I'm not sure if this is still needed as I didn't include the Title module during testing.

Summary

  • Importing multilingual nodes using the Node processor's language setting works as expected.
  • Importing multilingual terms using the Taxonomy term processor and the module Taxonomy translation does not work as expected.
  • For every field that is tested, the language setting on the target configuration works as expected, except for the Term reference field.
  • Auto created terms do not get imported in the language that is set on the target configuration.
  • Feeds will still try to import content in a specific language even if that language is no longer available (disabled or removed).
  • Feeds will remove content in other languages when importing content for one specific language, using the language setting on the target configuration.

What's next?

I would like to add some additional automated tests for some of the scenario's above, though I first need to know what the expected behaviour is for the scenario's which status I had set to "Uncertain" (the blue ones). I would also like some discussion about the yellow marked scenario's before I eventually write tests for them. I also hope to review the code more in detail.

twistor’s picture

Oh, wow, MegaChriz++.

It will take me a day or two to address these things.

cilefen’s picture

This is making sure MegaChriz gets credit for the megareview.

twistor’s picture

Status: Needs work » Postponed
Related issues: +#2529538: Generic entity language support.

Postponing on #2529538: Generic entity language support.. It's really a different issue, and one that can be solved pretty simply, while making this simpler in the process.

MegaChriz’s picture

@twistor, cilefen
Thanks for your nice comments! Now I feel my review is even greater than I already thought it was. :)

pwiniacki’s picture

Wow @MegaChriz great review, I wish all patches to be tested like that :P

Status: Postponed » Needs review

Status: Needs review » Needs work

The last submitted patch, 232: feeds-multilingual-1183440-231.patch, failed testing.

LukM’s picture

Following. Is any development still going on?

MegaChriz’s picture

Assigned: twistor » MegaChriz
Status: Needs work » Postponed

@LukM
Yes, there is still development going on. The first part of this feature, which is being worked on in #2529538: Generic entity language support., looks finalized. The first part is about configuring a language on the processor. I will go through its code one more time and then commit it. I'm already working on automated tests for the second part of this feature: configuring a language per field. After these tests are complete I expect that I'll need a couple of more weeks to fix all the issues that I noted in #234. Targeted deadline is mid January 2016, but I can't promise that it will be ready by then as I can only work on this feature in my free time.

  • MegaChriz committed 20f741c on 7.x-2.x
    by MegaChriz: add entity_translation as a test dependency for issue #...
MegaChriz’s picture

Status: Postponed » Needs review
FileSize
29.22 KB
PASSED: [[SimpleTest]]: [MySQL] 11,883 pass(es). View

The first part of language support, where was being worked on in #2529538: Generic entity language support., has been committed! Now you can configure a language on the processor.

In this issue the work will continue to also support language per multilingual field. A field can be made multilingual in the UI using the Entity Translation module.

We start off with a reroll from the patch in #232. This is not a full fix for the issue. Problems reported in #234 for field translation still apply. I just hope that by rerolling first it will be easier to follow changes that are posted later.
The patch will only apply to dev, not on beta1.

MegaChriz’s picture

FileSize
55.9 KB
FAILED: [[SimpleTest]]: [MySQL] 15,608 pass(es), 53 fail(s), and 66 exception(s). View
38.57 KB

Great, the reroll broke nothing. Phew.

Okay, hang on. Here is a patch that is a complete rewrite of the tests for field language that were posted earlier on. The tests pretty much covers every flaw that I spotted during my review in #234. Fixes are not included yet, so the FeedsMapperMultilingualFieldsTestCase should fail (if it doesn't, I'm gonna whine).

"When this baby hits 88 miles per hour, you're gonna see some serious shit."

Status: Needs review » Needs work

The last submitted patch, 246: feeds-field_translations-1183440-246.patch, failed testing.

The last submitted patch, 246: feeds-field_translations-1183440-246.patch, failed testing.

MegaChriz’s picture

Status: Needs work » Needs review
FileSize
57.25 KB
FAILED: [[SimpleTest]]: [MySQL] 15,634 pass(es), 27 fail(s), and 39 exception(s). View
1.52 KB

In the method FeedsProcessor::map() every target is emptied. I think that means that the target is also emptied for every language. So would it work to only empty the target for the configured language, if available? And would it possible break something else (emptying targets, for example)?

Let's watch.
(Tests should still fail, by the way: not all spotted flaws are addressed with this patch.)

Status: Needs review » Needs work

The last submitted patch, 249: feeds-field_translations-1183440-249.patch, failed testing.

The last submitted patch, 249: feeds-field_translations-1183440-249.patch, failed testing.

MegaChriz’s picture

Status: Needs work » Needs review
FileSize
62.73 KB
FAILED: [[SimpleTest]]: [MySQL] 16,829 pass(es), 12 fail(s), and 0 exception(s). View
15.39 KB

Worked on:

  • When disabling/removing a language that was configured for a target, values will be imported in LANGUAGE_NONE. On the mapping screen, on the target, the following message will be displayed:

    Language: Error: Language @lang not available

  • Fixed PHP notice that appeared on the mapping screen after disabling a language that was configured for one of the targets.
  • Added a test for clearing out values: when clearing out values for a single language only the values for that language should be cleared.
  • Added a test for clearing out values when a language gets disabled: when a language gets disabled, values are imported in LANGUAGE_NONE. When importing empty values later, values for LANGUAGE_NONE should be cleared. This doesn't happen yet, so the test currently still fails.
  • Figured out why going to node/1/translate was required for the node to contain the French translation in FeedsMapperMultilingualFieldsTestCase::testMultilingualFieldMappings(). That was because not all languages were available when field_available_languages() gets called during node_load(). Solution: reset static cache after fields are made multilingual.

Still to do:

  • When importing terms using the taxonomy reference field target, terms should be imported in the language specified on the target when the taxonomy is multilingual.
  • Values should be cleared out for LANGUAGE_NONE when importing empty values for a language that gets disabled. This is because values get imported in LANGUAGE_NONE too when the configured language for the target is no longer available.

The test "FeedsMapperMultilingualFieldsTestCase" should still fail.

Status: Needs review » Needs work

The last submitted patch, 252: feeds-field_translations-1183440-252.patch, failed testing.

The last submitted patch, 252: feeds-field_translations-1183440-252.patch, failed testing.

MegaChriz’s picture

Status: Needs work » Needs review
FileSize
63.83 KB
FAILED: [[SimpleTest]]: [MySQL] 16,815 pass(es), 26 fail(s), and 26 exception(s). View
3.35 KB

Completed:

  • When importing terms using the taxonomy reference field target, terms are imported in the language specified on the target when the taxonomy is multilingual.
  • Clear values out in LANGUAGE_NONE when the specified field language is not available and the field is translatable.

There is one thing that bothers me: values are cleared out for a language before the language to use is finally decided.
This is the situation right now (in the patch):

  1. In FeedsProcessor::map() the target is emptied. If the mapping option "field_language" is known and the field is translatable, the target will only be emptied for the specified language.
  2. In mapToTarget() for each target preprocess callbacks are called. locale_feeds_preprocess_callback() sets the mapping option "language".
  3. In mapToTarget() is ensured the mapping option "language" is always set.

As you can see we can not safely empty the target as long as the language isn't finally decided. If an other module specifies the language in a target preprocess callback, we may be emptying the target for the wrong language or emptying the target completely.

Possible solutions to this problem:

  • Invoke the preprocess callback before emptying the target. The con of this solution is that you can no longer use this callback to alter the target on the entity, as it will then be wiped out later.
  • Accept that preprocess callbacks should not alter the mapping language for field targets.

Thoughts?

Status: Needs review » Needs work

The last submitted patch, 255: feeds-field_translations-1183440-255.patch, failed testing.

The last submitted patch, 255: feeds-field_translations-1183440-255.patch, failed testing.

MegaChriz’s picture

Status: Needs work » Needs review
FileSize
64.06 KB
PASSED: [[SimpleTest]]: [MySQL] 16,841 pass(es). View
2.48 KB

Fixed a bug in entity_translation_feeds_presave(). When the language of the translation handler is LANGUAGE_NONE, it shouldn't empty the value for LANGUAGE_NONE.

Let's see if this passes tests.

MegaChriz’s picture

Great, tests are passing! Now the last thing to do is the following:

Values for translatable fields are cleared out for a language before the language to use is finally decided.
Thus if the final language is changed later via a target preprocess callback, values are emptied for the wrong language (or even worse: the target is emptied for all languages).

My suggestion: run preprocess callbacks before the target is emptied. Con: alterations on the entity's target in the preprocess callback will have no effect.

See #255 for more details.

twistor’s picture

FileSize
7.61 KB
67.56 KB
FAILED: [[SimpleTest]]: [MySQL] 17,032 pass(es), 81 fail(s), and 39 exception(s). View

Status: Needs review » Needs work

The last submitted patch, 260: feeds-field_translations-1183440-260.patch, failed testing.

The last submitted patch, 260: feeds-field_translations-1183440-260.patch, failed testing.

twistor’s picture

Status: Needs work » Needs review
FileSize
482 bytes
67.52 KB
PASSED: [[SimpleTest]]: [MySQL] 17,099 pass(es). View
MegaChriz’s picture

Thanks for your work twistor!

I've spotted one minor thing: in feeds.api.php, in the documentation for hook_processor_targets() the documentation still says that values on the entity can be prepared:

// Preprocess callbacks are called before the actual callback allowing you
// to prepare values on the entity or mapping array.

While I was at it, I figured the documentation for this hook could be improved a lot: I've opened an issue for this: #2638722: Improve documentation for hook_feeds_processor_targets(). If we get that one committed first, then we don't need to correct the documentation for preprocess callbacks in this issue.

As discussed on chat, I still find it a pity that the feeds source and entity can no longer be used as context to set/change mapping options in the preprocess callbacks. I do understand that the implementation in #263 is simpler and more efficient, as these callbacks only have to be called once in a request, rather then for every item and target. We could debate about this some more, but we could also call this thing to be finished.

I'll be working on the change record for this issue.

MegaChriz’s picture

I have written a draft for the change record:
https://www.drupal.org/node/2638802

The change record should not be published until a patch is committed.

See comment #264 for what is still open for this issue.

MegaChriz’s picture

rcodina’s picture

I want to share what I've being doing for content translation for a long time:

For each content type that I want to import I create a custom field to store "tnid" value from source web (in this case "field_tnid_web_antiga"). Then I modify the importer to map source web "tnid" to that field. Then all the magic is done by a custom module named "feeds_link_translations" with attached code.

With this method you can import nodes with multiple languages at the same time.

/*
 * Links translations together
 */
function feeds_link_translations_feeds_after_import($source) {

  //TODO: make this a configurable parameter
  $field = 'field_tnid_web_antiga';

  //Source language: default site language nids are used in order to link node translations together with tnid columns
  $source_lang = language_default();
  $source_lang = $source_lang->language;
  

  //I. Building hash table with default language nodes: given a source tnid it gives you the value of target tnid
  $result = db_query('SELECT n.nid,f.field_tnid_web_antiga_value FROM {node} n, {field_data_field_tnid_web_antiga} f WHERE f.entity_id = n.nid and n.tnid=0 and n.language = :language', array(':language' => $source_lang));
  $node_def = $result->fetchAll();

  $array_source = array();
  $array_source_new_tnid = array();
  if(is_array($node_def) && count($node_def)>0) {
    foreach($node_def as $n) {
      $array_source[] = $n->field_tnid_web_antiga_value;
      $array_source_new_tnid[$n->field_tnid_web_antiga_value] = $n->nid;
    }
  }

  //II. For each language, we check all nodes and set their tnid column with its correct value if we have the translation for old tnid
  $list = array_keys(language_list());

  $link_counter = 0;

  foreach ($list as $curr_lang) {

    $result = db_query('SELECT n.nid,f.field_tnid_web_antiga_value FROM {node} n, {field_data_field_tnid_web_antiga} f WHERE f.entity_id = n.nid and n.tnid=0 and n.language = :language', array(':language' => $curr_lang));
    $node_curr_lang = $result->fetchAll();

    if(is_array($node_curr_lang) && count($node_curr_lang)>0) {
      
      foreach($node_curr_lang as $n) {

        if(($n->field_tnid_web_antiga_value != '0') && in_array($n->field_tnid_web_antiga_value, $array_source)) {

          $nid_def = $array_source_new_tnid[$n->field_tnid_web_antiga_value];
          $nid_curr = $n->nid;

          if($nid_curr != $nid_def) {

              db_update('node')
                ->condition('nid', $nid_def)
                ->fields(array('tnid' => $nid_def))
                ->execute();

              db_update('node')
                ->condition('nid', $nid_curr)
                ->fields(array('tnid' => $nid_def))
                ->execute();

              $link_counter++;
           }
        }
      }
    }
  }

  drupal_set_message(sprintf(t('%d nodes have been linked with their translations'), $link_counter));
}
MegaChriz’s picture

I noticed that the automated tests for multilingual didn't include the boolean field. In the attached patch I've added this field in the tests. This patch is also a reroll which was required after committing #2602508: NULL imported to Boolean field as 0.

  • MegaChriz committed d4463d3 on 7.x-2.x authored by twistor
    Issue #1183440 by twistor, MegaChriz, stefan.r, drclaw, pcambra,...
MegaChriz’s picture

Status: Needs review » Fixed

Committed #268. Thanks to all!

Now instruct contrib to add support for multilingual mappers as well. See the change record.

stefan.r’s picture

Woohoo! Thanks!

Let's update the issue summary?

joseph.olstad’s picture

Good job everyone!
I think this merits a 7.x release, and (probably ?) porting to 8.x?

crystaldawn’s picture

I was expecting a new "tnid" in the mapper fields with the Dev version after this was closed/fixed but I do not see such an animal. IE: If the source data has the parent nid (AKA tnid), then it should be able to import translations for that the parent node by creating new nodes and then setting the tnid field to the source nid that is contained in the source data set. Did that not get put into this commit or am I looking at this to soon before it's been tarballed? If not, then what is the issue link for the Content part of things (non-field related). I'm going to check the git repo as well I spose, I could be to early :P

MegaChriz’s picture

Issue summary: View changes

I've updated the issue summary. For things that haven't been addressed yet (for example: support for Content translation), followup issues should be created.

@joseph.olstad
Yes, a new release (7.x-2.0-beta2) will be made shortly. I'm working on writing the release notes and updating the changelog. This functionality should indeed be ported to the 8.x-3.x version at some point, though I think that version is not ready enough to do that now (correct me if I'm wrong). I think that first the base functionality of Feeds should be completed and, after that, that tests from the 7.x-2.x version should be ported. See #1960800: [meta] Feeds 8.x roadmap.

@crystaldawn
The fix does not include support for Content translation. Getting only Field translation support done took long enough! :) This issue should be handled in a followup issue. It looks like we could use #840142: How can the created nodes be assigned their correct language? (and other "core" fields...) for that, though that issue is a bit a mess as it tries to address a lot of unrelated things as well. So maybe it's better to close that one and create a new issue for Content translation instead?

Since the signature for the target preprocess callbacks changed, I've added a change record for that as well: https://www.drupal.org/node/2667016

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

klausi’s picture

This caused unexpected behavior with entity_translation and field default values that suddenly appear for all languages. See #2850474: Entity Translation support creates text field entries for all languages with default values. Any input there would be appreciated.

joro78’s picture

A question when using i18n with localization of terms, emerges. Feeds is version: 7.x-2.0-beta3. I am importing nodes, containing a few taxonomies. Import is via CSV file.

I have set the feed import language properly and all the other imported fields populate accordingly. When importing translated terms with feeds, they do not map correctly, but even did not at all. My terms were translated in the corresponding vocabularies and I am not creating new ones in the import.

An interesting observation is, that terms match if they have the same term name (as the source language), but not when translation string differs. Terms are created if I choose create new in Feeds import, but they appear in the source and not the desired language. Moreover I need mapping of the translated term to their sources and importing as translated ones.

I was thinking of using different terms per language, and haven't tested it. It may work but I have many terms in site in a few languages.

Is it possible or is there any solution to the case?
Any help will be appreciated.