Improve documentation of field multiple language system

I'd need to know what in http://api.drupal.org/api/drupal/modules--field--field.multilingual.inc/... needs to be improved (high level descritpion, internal descritpions, missing pieces), and please don't answer everything. That would not help ;)

crosspost :)

Yeah, the data access has become so convoluted that I see many devs using EntityDrupalWrapper (from the Entity API module) to avoid the mess of language prefixes, etc. Would be nice if there was a cleaner API for it.

chx, can you please tone down the rhetoric? It's not constructive, and is tearing down hard work that folks like plach have done. Post suggestions for improvement, not criticisms.

Here's a shot at a skeleton for that page, which totally needs work, but is a place to start (I know we can't do headings in doxygen but this is just to organize.)

About Multilingual Fields

Fields natively implement multilingual support, which is optionally set per-field. This is useful in situations where ... . Situations where you might not want a field translatable are ... .

(Basically, provide an overview of the feature and the thinking behind it.)

Multilingual Field Structure

All fields use the following structure:

$entity->{$field_name}[$langcode][$delta][$column_name]

* For untranslatable fields $langcode is ...
* For translatable fields $langcode is ...

Accessing Multilingual Fields

As a module or theme developer, here's the proper way to access a field's property.

// Example code, and rationale.

Multilingual Field API

// The stuff about _field_invoke() etc.

Didn't know field_get_items existed.
/me refactors his code.
Definite +1

Having a single-value userreference called field_user, who needs to be added to a multi-value userreference called field_user_history, but only if s/he is not yet there. The best place to achieve this seems to be nook_node_presave(). I was struggling with forging the set by cycling through $node->field_user_history[$node->language], when I was pointed to a blog post of @davereid. Then I switched to using field_get_items('node', $node, 'field_user_history') which I do understand is a more straightforward way to do so. Now I can properly build the set for my aforementioned use case.

OTOH the question remains unanswered: what is the setter counterpart of it? IOW: how to properly assing value to a field? I was trying $node->field_user_history[$node->language] = $set, but this didn't work (I have even tried hook_field_attach_presave() and basically every node API hook I thought to be to-the-point, without any success). Then I tried $node->field_user_history[LANGUAGE_NONE] = $set and my code began working, even as simple as in a hook_node_update().

So: how should one assign value to a field properly? and why should one do it that way?

In IRC, davereid suggested $node->field_foo[field_valid_language($node->language)] = $items.
Also, subscribe.

As I see field_valid_language($node->language) is only something like a safety measurement to ensure that specific language code (still) does exist, but it does not explain me too much. Why should I be using $random_language (safeguarded by this function or not), or why am I enforced to use LANGUAGE_NONE? Basically that's what I want to know from the smallish coder's point of view.

The best docs I found on this so far are in http://api.drupal.org/api/drupal/modules--field--field.multilingual.inc/... (its one of the few topics from the API navigation screen).

First thing to note from there is that there can be two types of fields: translatable and not translatable. Unless you have the entity_translation module enabled and you run the latest D7 release (where related bugs where fixed), all entities will be created untranslatable. Older Drupal 7 releases had a bug where fields got created translatable.

A key thing to understand from there is that the set of languages you might be able to save data to depends on a setting on the field (whether it is translatable or not). If it is translatable, then it also depends on another set of settings (the list of languages on the site). So when saving a field, you need to consider the field type and then whether the language you are saving into is on the site (if a translatable field).

http://api.drupal.org/api/drupal/modules--field--field.multilingual.inc/... is then the API function to gather a list of languages that one field can be saved in. It considers if the field is translatable and if it is, will consider the available languages on the site. So you get a list of language codes that you can save your field in.

Then which one of these you save your field in depends on your context. If you know the language of the data that is being saved, use that. Informed from node_form() (http://api.drupal.org/api/drupal/modules--node--node.pages.inc/function/...) (or locale module's field translation code), you can see that $node->language is used to gather input on the node data in core. That is since Drupal core does not have a UI for entering translated fields, so it will always display a UI for the field in the node's language. The entity_translation module has a UI and hooks to save fields in other languages than the main language of the node.

At this point I'm not sure you actually want this in a node form submission process, but if you do, entity_translation module should be investiagated for how it allows you to plug into that process (and in that case I'm sure it will tell you the exact language code to use, like node_form does for field_attach_form()).

So in short:

$list_of_language_codes = field_available_languages('node', $field_data);
$language_code = somehow_pick_what_i_want($list_of_language_codes);
$node->field_foo[$language_code] = $items;

Now, field_valid_language() is interesting in that it would default to the content display language selected for the page . That information in itself is useful for somehow_pick_what_i_want() but depending on the nature of the data, it could be wrong. If you have a CSV with data in 5 languages, and you need to import that, you just need to know the 5 language codes and use them specifically. If you have some kind of custom form that then updates a field value on an entity, the page content language (global $langugae_content) selected for that page sounds like a safe bet to default to though.

However, the sole use of field_valid_language($node->language) is not good, since for non-translatable fields, the field value cannot be saved in arbitrary languages, it will always be LANGUAGE_NONE. And whether a field is translatable or not is a per-field configuration thing, which field_valid_language() does not care about.

Strong disclaimer: I did not participate in the building of this feature, neither I say I understand how it works, this is merely a recollection of what I'm seeing in the existing code and docs and would welcome corrections. I'm definitely not an authoritative source. Awaiting verification.

#1214472: Undefined index: und in taxonomy_access_form_alter() -- incorrect assignment of language key seems like a good issue where better understanding of the system would be very helpful. Just throwing in as a use case to cover in docs.

Not to get pedantic but this is a task, not a bug.

Thanks for posting examples here, they are really useful to understand which are the scenarios in which the Field API is being used. I'd like to see more code exploiting the Field API to craft better examples on how to manage multilingual fields correctly.

No time to expand on this now, just wanted to say that #13 is more or less correct. I should be able to provide details later today.

Patterns for specific use-cases would be very helpful. E.g., how to properly handle language codes:

• On generic entities (not necc. nodes)
• In a form alter
• During form validation and submission
• In hook_field_*()
• In hook_node_*()
• Etc.

Re: Examples of how Field API gets used: I have some half-broken examples and some crazy workarounds in TAC. They're probably too big to post snippets here, but I attempt to handle language codes in the following places. (Warning: high WTFs.)

• taxonomy_access_form_alter() (Currently broken, as per #1214472: Undefined index: und in taxonomy_access_form_alter() -- incorrect assignment of language key). Hoping to replace this entirely with #1204230: Missing hook_field_widget_form_alter()...)
• taxonomy_access_field_attach_validate()
• _taxonomy_access_pre_validate() (Form validation callback.)
• _taxonomy_access_entity_fields() (Helper function that looks for taxonomy fields in an entity and aggregates info about them.)
• _taxonomy_access_compare_fields() (Helper function to validate field changes.)

drupalcontrib.org seems to be a month or more stale, so I can't link them there. Repo link:
http://drupalcode.org/project/taxonomy_access.git/blob/refs/heads/7.x-1....

One thing that I got seriously stuck on (hours of debugging) is that one needs to pass an array keyed by langcode to use form_set_value() on a field element in a form. Quite different from D6 there.

Tagging.

Uh, tag fix.

I'd like to stress on one thing that maybe obvious, but perhaps that has not been sufficiently outlined: fields may hold multiple items array, one for each available language. The details are explained in the opening paragraph of the current documentation, but the point is that if a field is translatable any operation, except viewing and adding an array of items to it for a particular language, needs to be done on every language available in the field structure. That's how the Field API works: if you pass it an $entity object where fields are populated with as much as languages as you want, it will know how to handle it, as it will simply act on every value available. The only exceptions to this behavior are:

• field_attach_view(), as one usually needs to display one language value at the time. It accepts an optional language parameter, if this is not present the current language is used. field_get_items() is the per-field counterpart, and should be used in the view/render phase, since it defaults to the language the field would be dispayed in.
• field_attach_form(), since our core UIs allow only one value to be entered for a particular language at a single time. Altough multilanguage field forms could be introduced in contrib, this is by far the most common use case. Here we are missing a per-field counterpart, basically because the assumption is that one knows in which language a value is being entered, but this is true only when editing values, not when adding them. A field_set_items() function might help here in providing a sensible default for the source language if it is unknown/unspecified.

Another thing to keep in mind is that nothing prevents any field to eventually become translatable, it's just matter of switching a flag in the database. AAMOF translatable (and this can be misleading terminology) means that it can have a different value for each available language, not that its type is inherently translatable: a map might not be "translatable" in itself but I might want to have a different map for every available language. Hence every field should be treated as potentially translatable, relying on it being created as non-translatable might lead to having to fix its behavior later.

However, after talking with @chx in IRC some days ago, we agreed that to bring people's understanding of field multilingual support to an acceptable level we need several steps:

• Improve the available documentation, the suggestion in #7 seems like a good start.
• Provide code samples on d.o. explaining the available best practices.
• Try to fix core tests, which are really a poor example at the moment, since they uniformly use hard-coded LANGUAGE_NONE values almost everywhere. This might not be feasible after all, since if you know which language the field is supposed to have, it's correct to use it: in most core tests we don't enable Locale, hence LANGUAGE_NONE is the only possibile value while asserting results.
• Improve Field API introducing field_set_items(), we need a separate issue for that.

Meanwhile, if people has more example to post, they are really welcome. They will be useful to design examples and field_set_items() itself.

Can someone please update the issue summary so I don't have to read all the comments to figure out what the problem and proposed solution is here? Thanks...

Er, tagging?

API improvements issue: #1260640: Improve field language API DX.

Tagging for the content handling leg of D8MI.

+++ b/core/modules/field/field.multilingual.incundefined
@@ -10,20 +10,59 @@
+ * Fields natively implement multilingual support by holding multiple values ¶

trailing whitespace... a lot of

I'm going to fix some spelling and whitespace

Good job Ryan! I have some feedback below:

Handling of multilingual fields.

Fields natively implement multilingual support by holding multiple values of the field array. All fields use the following structure:
$entity->{$field_name}[$langcode][$delta][$column_name]

By default a single-language site in English without the locale module enabled uses LANGUAGE_NONE as the $langcode, and there is always only one language variant for each field.
$entity->body[und][0][value];
$entity->langcode = 'und';

This looks good, but I would change "locale" to "Locale" for consistency with the next section and to use the human-readable module name.

With Locale module enabled (as well as with Content Translation), fields will not have their own language information, but LANGUAGE_NONE is used to signify that we don't have that specified. The whole entity with all its properties and fields is assumed to have the entity language code
assigned (conceptually).
$entity->body[und][0][value];
$entity->langcode = 'en';

1) Minor: I would change "With Locale module" to "With the Locale module".

2) I'm a bit confused with this part of the first sentence:

fields will not have their own language information, but LANGUAGE_NONE is used to signify that we don't have that specified

This makes me think that either:

* it should be "...fields will have..." (because of the "but")
or
* it should be "..., and LANGUAGE_NONE is used..." (because of the "will not")

My guess is it is the latter.

Field translation is pluggable, but the only known implementation is the contributed Entity translation module that provides a user interface to field-translation enable entities and change the translatability on fields.

Looks good I would probably change the last part to:

...provides a user interface to enable field translation for entities and change the translatability on fields.

When we enable field translation, a field that does not have translation enabled will always be LANGUAGE_NONE, and no other language values will be present. The meaning of this is different from the above. It means that the field will have the same value across all translations. This is also called a shared field. The $entity->langcode value will persist to represent the language of the properties of the entity, such as the title, author as well as signify the original language that the entity was saved with. That information can be useful for permission and workflow purposes.
$entity->body[und][0][value]; // not translatable (shared)
$entity->langcode = 'en'; // applies to properties of the entity

Minor:

Formatting for "entity, such as the title, author as well as" should probably change to something like "entity, such as the title and author, as well as".

Also:

Using the "title" as a property example might be misleading as often we would enable the Title module in order to get a translatable title field (which is separate from the property title) for node entities. You might want to use "author and status" as examples for properties.

For a field with translation enabled in field settings, multiple variants of the value may be available in any of the enabled languages. Data that was created prior to enabling the translation will remain, however, it will be ignored by Drupal. The LANGUAGE_NONE version of the field in this case
serves only to preserve the historic configuration and can be safely deleted when it is no longer used.
$entity->body[en][0][value]; // translatble
$entity->body[und][0][value]; // ignored by Drupal
$entity->langcode = 'en'; // applies to properties of the entity

Looks good except "translatble" should be "translatable" in comment. Interesting to know about the historic 'und' data!

The available languages for a particular field are returned by field_available_languages(). Whether a field is translatable is determined by calling field_is_translatable(), which checks the $field['translatable'] property returned by field_info_field(), and whether there is at least one
translation handler available for the field. A translation handler is a module registering itself via hook_entity_info() to handle field translations.

Looks good.

Multilingual Field API

By default, _field_invoke() and _field_invoke_multiple() are processing a field in all available languages, unless they are given a language suggestion. Based on that suggestion, _field_language_suggestion() determines the languages to act on.

I would change "are processing" to "process".

Most field_attach_*() functions act on all available languages, except for the following:

- field_attach_form() only takes a single language code, specifying which language the field values will be submitted in.

- field_attach_view() requires the language the entity will be displayed in.

Since it is unknown whether a field translation exists for the requested language, the translation handler is responsible for performing one of the following actions:

- Ignore missing translations, i.e. do not show any field values for the requested language. For example, see locale_field_language_alter().

- Provide a value in a different language as fallback. By default, the fallback logic is applied separately to each field to ensure that there is a value for each field to display.

The field language fallback logic relies on the global language fallback configuration. Therefore, the displayed field values can be in the requested language, but may be different if no values for the requested language are available. The default language fallback rules inspect all the enabled languages ordered by their weight. This behavior can be altered or even disabled by modules implementing hook_field_language_alter(), making it possible to choose the first approach. The display language for each field is returned by field_language().

It is not obvious to me what "the first approach" is in the second-to-last sentence.

Accessing Multilingual Fields

@see field_get_items()

See @link field Field API @endlink for information about the other parts of the Field API.

Looks good.

:)

+++ b/core/modules/field/field.multilingual.incundefined
@@ -10,20 +10,59 @@
+ * to signify that we don't have that specified. The whole entity with all ¶
+ * its properties and fields is assumed to have the entity language code ¶
...
+ * translatability on fields. ¶
...
+ * When we enable field translation, a field that does not have translation ¶
...
+ * language of the properties of the entity, such as the title, author as well ¶
+ * as signify the original language that the entity was saved with. That ¶

Line end whitespace, among other places.

+++ b/core/modules/field/field.multilingual.incundefined
@@ -10,20 +10,59 @@
+ * Data that was created prior to enabling the translation will remain, however, it will be ignored
+ * by Drupal. The LANGUAGE_NONE version of the field in this case serves only to preserve the ¶

These are over 80 chars.

+++ b/core/modules/field/field.multilingual.incundefined
@@ -10,20 +10,59 @@
+ * enabled uses LANGUAGE_NONE as the $langcode, and there is always only

lets use LANGUAGE_NOT_SPECIFIED as that's how it's named in d8.

only fixed whitespace and spelling from comment 29. this patch does not fix the things pointed out in comments 32 or 34.

still has a line wrap problem

line wrap fixed

Kristen brought together the above docs text with two figures I've created today to help document this and posted at http://drupal.org/node/1500308 - I've made some slight edits to the final part on the API functions to try and make some more sense of it. I think we should cycle back any improvements from here that are not there yet and then decide if we want to have it all in the API docs as well or having it on d.o is better and should be linked from the API docs instead.

Also, of course, several things might be better explained then our current version does, I think the a way better understanding of wtf that $langcode should be that chx was looking for in the opener should be achieved with the two added figures and the explanation :) I hope :)

I would suggest that since that doc on d.o has images etc., and it's difficult to maintain documentation in two places, that we have a minimal version in the API docs, with a link to that page.

No worries :)

Leave for D8, it can be rerolled once committed. If you'd like to keep uploading them, append -do-not-test.patch to the end.

Looks good! I looked at the documentation and applied the patch with no issues.

Put on the sprint tracker for D8MI.

As someone who spends a lot of time working in off-line mode (e.g. airplanes), I'm not a big fan of moving documentation out of the code to drupal.org. I'm not sure what the rationale is; it isn't mentioned in the comments or in the issue summary.

@Dries: we posted the text that was originally destined for the API docs inside code at http://drupal.org/node/1500308 with two figures that should help understanding the concepts a huge deal. It makes sense to "duplicate" it in our API docs, but we don't have any way/process to include supporting figured in our API docs shipped with core, so we need to keep the d.o page in any case. I've thrown up this question of whether we want to have the text in the Drupal-shipped API docs (which I agree would be valuable), and jhodgdon commented it looks best to have it at one place and link there. See http://drupal.org/node/1232120#comment-5782440. We did not get this back to the summary. What do you think?

My take on drupal.org vs. in-code/api.drupal.org is this:

Function basics (params, return, description of what it does, etc.) definitely belongs in-code, and needs to be maintained with the code. This also applies to classes, methods, constants, hooks, etc.

Reference information, such as the much-loved and much-hated Form API reference, also belong in the code and on api.drupal.org.

Several-paragraph overviews of groups of functions also belong in-code, to give "road maps" (this uses @defgroup, and makes the Topics list on api.drupal.org). For example, a list of all of the hooks that are invoked during each node-related operation, or a list of the hooks you would need to implement to create a field module (in both cases the hook pages would give you more details about each hook).

Tutorials and lengthy examples for how to do things with the Drupal API belong on drupal.org or in the Examples for Developers project. For example, a step-by-step "how to create a field module" tutorial, or a sample field module.

Right?

So in this case, I think we want to distill the essentials of the multiple language system into the group/topic that is in the code, but we don't want to give a full tutorial there, and as noted above, we don't have a good way to include images (and I don't think we would want to include API docs images in the Drupal Core code base).

Of course, the dividing line between "tutorial and lengthy example" vs. "several-paragraph overview" may not be clear... we can definitely tweak it until we figure out what is "just right".

I think it would be good to include the essence of the docs in code, and have the extended docs with images on d.o.

Honestly I don't see the reason to dump the current in-code documentation. IIRC it holds fairly lower-level info than the (definitely good) online docs. What about adding the reference without removing the current docs?

@plach: the goal of the online docs page at http://drupal.org/node/1500308 was that it would have *more* information then the API docs, not that it would have *different* information. Basically Kristen worked on setting that doc page up to have a better intro to what field language means even (it means different things in different cases), and then provide API pointers for different cases.

If the online docs page does not have all the information that we have in the API docs, then that is a mistake. I think the information should be cross-merged (ie. improvements from the doc page vs. what is in API should go to the docs page and vice versa). I've said this above in #38.

Ok, I'll have a deeper look to differences. However, FWIW, I think keeping online docs and in-code docs should be a non-issue, in the sense that as @jhodgdon was pointing out they should be different, and the intended audience should be too, probably. When I reviewed the new docs I thought they were a higher-level complement to the in-code docs. Having PHP docs living outside code does not really make sense to me.

Again, I find the new docs great but I don't think they should replace the existing ones: maybe we want to merge the last part, which is more technical, in the PHP docs and leave the reference to the online docs for a more high-level description?

However, I'll crosscheck and see if I can make more concrete proposal.

@plach: well, there is nothing stopping from online docs from being deeply technical, the database API docs at http://drupal.org/developing/api/database are pretty darn technical (and there is not much in the code docs about them).

Anyway, our goal here was to have good docs on what multilingual fields are about and how people can use them via the API. If both the API docs and the online docs satisfy that (and/or at least they are cross-linked so people can fill in the details from one to the other), I think we are good. After all it is up to the issue openers and people who did not understand this before to tell if the new docs are good.

@chx? @Dave Reid? @Booba?

First of all, thanks for turning on the light bulb in my head (whilst it's a bit dim as of now, but definitely there's hope).

Secondly, it seems to me that the debate is not about the documentation itself (which is pretty well-to-the-point and easy-to-understand, congratulations), but the format/style/availability of it. @Dries (and some others, including me) wants documentation that's right in the code (to be able to have a look at it while flying high). Anyway, working with Drupal means always-be-connected (for me), since the (API) docs are only one source of information–and there are times when it's not even the best source. I think this particular debate (about having an image as an (API) docs) belongs to this category: it's easier to understand if it's drawn, but it would be great if it would be available offline.

OTOH, most of the stuff available at http://drupal.org/node/1500308 could definitely be in-code; particularly the flowchart is the only thing that I can't imagine how to reproduce in-code. So, I for one am with @Dries this time to have in-code documentation as much as possible, but with a reference to the only thing that cannot reside there: the flowchart. (This way the documentation could be versioned _with_ the code itself, and not _besides_ the code.)

Additionally, I had a really off-topic and far-reaching idea. If there were some automation that generates offline-browsable format of the online documentation (CHM, bunch of HTML files and stuff, or even PDF), then we could have the values from both the worlds: the images and offline availability. Anyway, I think being online while flying will be available much more sooner than this tool… :S

The handbook doc is way better than nothing, but it would also be helpful to explain how to handle the langcodes for field widgets in entity forms, where you are dealing with a FAPI array rather than the entity. (As I recall, field_language() also did not work properly when I was trying to alter certain fields on entity forms.)

@Ryan Weal: are you working on this?

@chx: so in other words you are asking the following: "I have a piece of data; I asked the API and this type of data is stored on this entity for French, German and Spanish separately; Now which language should I save my data with?". Did I get your question right? Well, the answer to that is: you should know. It is pretty impossible to tell what language you data is in without you knowing it I think. It can very much depend on your data. Eg. you got that data from a 3rd party service to update the entity, you got that data from a user on a form, etc. You should consider the language information you got with your data (the language information from the 3rd party service, the language of the page the user used to enter the data, etc), and use that language. I don't see what kind of code snippet could we provide for deciding on that language, since it depends on your data.

Well, Dries' comment was that he was looking for more of the online docs built into the text as it makes sense, no?

Thanks @Ryan Weal. Couple notes regarding the patch:

• Any @see should go at the end of the docblock. Reference: http://drupal.org/node/1354#see
• We could also use @link/@endlink to give the link a title and more context.
• Adding a link is a good idea, but I think we should probably add more actual text from the handbook page to the API documentation as well. (Links to external resources should be used sparingly and should never be relied on as the sole source of information. The codebase should be self-contained without need for the internet.) I'm completely fine with committing a small addition like this first and then keeping this issue open to port the updated docs into the codebase, but a complete patch is easier to review. :)

Setting status per #65

Ryan: are you still working on this? I think its would be prudent to remove the sprint tag since there is no ongoing work on this :/ I'd like to use this tag for stuff that is actively worked on and people can help out / review.

Added a note about the new entity_language() function in http://drupal.org/node/1500308.

Should be back on sprint at least for the week.

Ryan: still working on it? It's been a month :/

The field system is undergoing massive changes, so this needs to be postponed until http://drupal.org/node/1696640 is finished.

Resetting status per #65, and besides it is a D8 patch.

The D7 and D8 documentation in this section of the code look almost identical right now, so I'm pretty sure this needs to go in D8 first.

Also, "critical" doesn't seem like the right status for a documentation issue... though it would definitely be great to see this improved.

Updated issue summary.

This is back to 7.x I think since #2067079: Remove the Field Language API.

I think there is a real need to improve the documentation here, particularly what @xjm mentions in #17 (how to use language codes properly in different scenarios).

It might make sense to postpone that on #1260640: Improve field language API DX, which aims to add missing API functions for some of those scenarios.