If you create a multi-lingual site, and try to use Views on it, you'll run into various problems.

This meta-issue exists to collect all of the various problems in one place.

Overall Plan

  • In the presence of multiple languages, each row in an entity view should represent one translation of one entity item. So each row has an entity ID and a language associated with it, for the base table.
  • Any filters/sorts/arguments related to fields on an entity should filter/sort based on the translated field values associated with that language on that entity. In other words, the field tables need to be joined based on language, for translated fields.
  • You need to be able to filter the entities in the views rows by language of the translation (should also be a field, sort, and argument). Choices:
    - Specific languages (enabled languages on the site, plus "undefined" and "all languages")
    - Current language (the language chosen by URL, language switcher, and other negotiation)
  • Entities should also have a field/filter/sort/argument for Original Language.
  • If there are relationships/joins in the view to other entities, their language can be filtered independently. However, they need another language choice, which will filter so that their language matches the language of the base entity table (only applies if the base table is an entity).
  • When making Views output for a given entity, the user should be given a choice of languages:
    - Specific languages (enabled languages on the site, plus "undefined" and "all languages")
    - Current language (the language chosen by URL, language switcher, and other negotiation)
    - - Row language (the fields/entities/etc. from each entity will be displayed in the language associated with the translation of that entity in the row)
    This choice should be a setting on the Display. Whether the output is fields (Field module fields and entity properties like title), entities, links, or whatever, they should all obey the Display language setting.

Individual issues

Here is a table of the individual issues that need to be resolved in order to realize the overall plan.

Problem Issue(s) Discussion/explanation Follow-up
[1] Some entities lack data tables[DONE!] In order for each row in a view to represent a translation, every fieldable, translatable entity needs to have a table with the language field in it, which is the data table. Two Core entities lack this table currently: User and Custom Block.
[2] Join field tables to entity data table[DONE!] #2218065: Need to join fields to the entity field data tables, not entity tables, or filtering increases number of results Currently, the Field module's Views integration is joining field data tables to the entity base table. It needs to join to the entity field data table instead, so that the languages on the fields are connected to the language of the views row. For (rare) entities without data table, join as we are now.
[3] Add entity-level language filters [DONE! Some followups may be needed...] If view rows are entity translations, you need to have some way to filter views by language of the translation. Views should also be filterable by original/source language. Need sort, field, and argument for both of these as well. Note that the default Views entity data class proposed in #1740492: Implement a default entity views data handler would take care of this for most entities in a reasonable way. Views provided by Core need to use these filters:

Also: #2162051: Decide if views should default to having a language filter.

[4] Deal with missing translations Note: Depends on [3] If you have filtered a view by entity translation language, and a given entity item doesn't have a translation in that language, should there be a fallback so the entity still appears in the view?
[5] Remove field-level language filters [DONE] #2218025: Field language filter/sort/etc. for Views do not work and are not needed Field-level language filters exist now but don't make sense if rows are translations and fields are connected correctly to the rows (picking out translations)
[6] Translate the display properly Remaining main issue: #2393339: [META] Make sure Views base fields are using Field API for formatting, and do not lose functionality and related/sub issues (see right column).

This will make sure that all fields can be translated (currently base fields cannot be). The solution is to make the Entity field plugins support base fields, which means that the base fields will be using regular Entity formatters instead of Views-specific formatters, and change the base fields in their Views data classes to use Entity field handlers instead of Views-specific handlers.

Past/closed/duplicate issues on this subject:
- #2006606: Views rendering ignores entity language, eg: Identical nodes rendered in views when nodes have translations
- #2149649: Entity rendering/theming does not use the active entity language to render links
- #2394041: Row language settings from entity views should be on display level for all views
- #2217569: Fields row plugin: Translation is non-uniform for base fields, Field UI fields, links; no way to choose "this row's language"
- #2407801: Views generic field handler does not work with base fields
- #2394883: Language setup for entity and field based rendering in views is independent, confusing UI, lacking test coverage
- #2429447: Use data table as views base table, if available.

When this issue was originally filed, if your Views display used entity view modes, they were displayed using your choice of the row's language, the original language of the entity, or the page language, and this setting was on the Row plugin. Field-based views had a setting for Field Language, which was hidden in the "Advanced" setting on the Display plugin, with options only for static languages (English, French, etc.). And only Field UI fields, not base fields like Title, obeyed the Field Language setting. So... The plan:
(a) Unify the language settings and their options for Field and Entity views [this is done]
(b) Put the settings on the Display (not the Row plugin) [this is done]
(c) Make translation work for Base fields as well as Field UI fields [still to do].
Once this issue is done, we need to do a few final manual tests (and file issues/add test coverage if problems are located):
  • Make sure that the display is actually working right for both Entity and Field based views: base fields, Field API fields, entity view modes.
  • Test using static language choices, plus "original language of entity" (or whatever that is called in the UI, "row language" (or whatever that is called) and "page language" (negotiated/selected language)
  • Test field-based views with relationships, such as Entity Reference fields, Taxonomy fields, and node author, and make sure these are working right with respect to number of rows in the result and display language. All fields from all entities should be displayed in the same language.
  • Test links: should be going to the entity consistent with the data display (so if this row is in Spanish the link should go to es/node/1 or whatever). Types of links to test: stand-alone (like the Edit link and View link pseudo-fields), automatic (if you have an entity and you check the box that says "include links" or something like that), on fields like Node title that have their own specific plugins and a "link this back to the node" option, and on short string generic entity fields with "link this back" box checked.

(I did part of this testing on 7 Oct 2015 on #2450195: Original language of entities not accessible in views anymore comments 17 and onwards that day).

Additional related issues, child issues, and follow-ups:
- #2582535: Duplicated rows on admin/content if author is translated
- #2582499: User views - user links are not localized
- #2450195: Original language of entities not accessible in views anymore
- #2450205: Translation settings don't appear for node in standard.
- #2420737: Differences in dynamic language names are confusing in views, content, etc.
- #2508892: "Content language of view row" rendering option does not always ensure row language is present

[7] UI: Language choice labels DONE! #2037979: "Current user's language" views filter label is named very misleading The language choice called "Current user's language" really means "The negotiated language". This label is confusing and needs an update.
[8] Update help and documentation The help and documentation needs to explain this stuff
[9] Field-based multilingual views with relationships do not work Fixed as part of a different issue. #2446681: Error "Column 'langcode' in field list is ambiguous" thrown due to TranslationLanguageRenderer not rendering a field from a relationship If you create a field-based multilngual view with relationships, and try to display Field UI fields in the view, you get an SQL exception.
[10] Views with non-translatable fields seem to be somewhat broken #2451657: Views should not condition joins on the langcode of fields that are not translatable If you have a translatable entity with a non-translatable field on it, and use that field in a view, it will try to use a nonexistent langcode database field.
[11] Views with bulk operations (admin views) are confusing Done! Since views show translations and entity bulk operations operate on the entities as a whole, it could lead to confusion and loss of data
[12] Glossary Ajax doesn't work with multilingual #2600804: AJAXified glossary view arguments fail when Language URL detection enabled

Comments

jhodgdon’s picture

Issue summary: View changes
Gábor Hojtsy’s picture

Version: 8.1.x-dev » 8.0.x-dev
Issue summary: View changes

It is very reasonable to target these for 8.0.x

jhodgdon’s picture

Yeah, definitely 8.0.x -- I meant it to be there, just a finger slip.

Meanwhile: Adding background information that was formerly in the summary of #2218025: Field language filter/sort/etc. for Views do not work and are not needed...

----

There are currently two different types of translatable entities: those with a data table (like Node) nad those without (currently just User and Custom Block).

For entities with a data table, such as nodes:

For entities without a data table, such as User, it's a real mess:

  • Translations are currently being done at the field level and there is no equivalent of the node_field_data table, so a view of Users doesn't have any definite language associated with each row, and one row == one user.
  • If you use a field in a view (for display, filtering, sorting, etc.), the only table we can join on is the base users table, so if the field has translations, each join is going to multiply the number of rows in the view. So, for users, we would need to filter by language for each translated field that is being used in the view in any way, to get back to the desired number of rows (unless we actually did want to show all the translations as separate rows the view for some reason).
  • Then when we get to display... It seems like logically, if we are doing a field-level display, then each field in the query output for a given row does have a language on it (either because we filtered by language, or because we're displaying all languages as separate rows on purpose), so we should display that field in that row with that row's that-field language. We are definitely not doing that right now -- we are instead using that global field display language setting, which I don't think we want or need for this use case either. On the other hand, if we are using a view-mode-based display for the rows instead of field-based display, we would need to figure out how to choose the language for each row... and I think in this case, it should probably be a setting on the display, shouldn't? So when you choose to display "Teasers" or whatever the view mode is called, you would select what language to display it in? (I don't think field-level language filters could apply to a display mode, because they are on each field, and there is no guarantee that you have exactly one in a view -- you could have zero, or multiple of these filters and they do not have to have the same values).
jhodgdon’s picture

Issue summary: View changes

Just had a short discussion with plach and Gábor Hojtsy about what to do if there are joins/relationships in the view.

The problem is this: if you have a view on base table A (an entity), then join to table B (another entity), then there are two entities, each associated with a language, in the view.

So, we need to expand the plan a bit:
a) When rendering a view, render each entity in the language of that entity's value in the row (if it has languages)
b) Each entity gets filtered by language separately
c) We need another choice for entity language filtering, which would be "Match the language of the entity that is the base table for the view"

Added this to the summary.

jhodgdon’s picture

Issue summary: View changes

typo

Gábor Hojtsy’s picture

Issue summary: View changes

Rescoped #2161845: Node views (front page, admin) do not use the proper language filter to cover the admin view also because it was much easier to cover both at once vs. doing it separately. Also now have a very elaborate test on our expectations of default and specific language behaviors for each. Please review.

jhodgdon’s picture

Issue summary: View changes

Summary updated -- highlighted places where we need more issues to be filed eventually. I'm also going through child issues and updating titles/summaries...

jhodgdon’s picture

Issue summary: View changes

Add numbers and status

jhodgdon’s picture

Issue summary: View changes

Minor wording adjustment in the master plan, sorry for all the updates, I'm editing issues to make them clearer...

jhodgdon’s picture

Issue summary: View changes

Added #2317975: Update Views help to explain multilingual to make sure we update the help on this.

jhodgdon’s picture

Regarding "[3] Add entity-level language filters" from the issue summary... I looked into the existing implementations of hook_views_data() in Core.

Many of them are not entities, or don't have language (like File).

The only ones I think need language filtering are Comment, Node, Taxonomy, and User:
- We took care of Node on #2217943: Views cannot be filtered by language of translation.
- It looks like Comment already has the same filters/fields we added for Node on that issue.
- Taxonomy and User have this too, but they're not wrapped in:

 if (\Drupal::moduleHandler()->moduleExists('language')) {

which probably should be added, because it's kind of silly to show/filter/sort by language if there is only one language on your site.

So I tested this. The goal is that for each of these entities, if you make a view of them, there is one row per ID/language, and then you can filter by language. Here's what I found (note: "with language module" means I turned on Language and Content Translation, added a language, made these entities translatable, and translated them; I also had to add a field to User in order to make it translatable):

Comment

No language module

Fine, no language fields/filters/sorts are visible; probably shouldn't be, see above.

With language module

- Initially, View shows all translations of all comments, good!
- I added a language filter. It doesn't work -- it is trying to filter on comment.language but it should be on the field data table (there is no language database column on the comment table).

Filed: #2320277: Views comment language field/filter is for original language code, no translation language field/filter

Taxonomy

No language module

Could not really test. See #2320275: Taxonomy views do not work (fatal error) Will need to revisit once that is fixed.

With language module

Could not really test. See #2320275: Taxonomy views do not work (fatal error) Will need to revisit once that is fixed.

User

Note: probably depends on #1498664: Refactor user entity properties to multilingual being fixed...

No language module

View works. Language displays as English. Shouldn't have the language field/filter/etc. available without Language module, as noted above.

With language module

Rows are just the users, not users + language. Probably should retest once #1498664: Refactor user entity properties to multilingual is fixed.

I've updated the issue summary...

plach’s picture

jhodgdon’s picture

Agreed on postponing on the entity controllers for Views data.

So, following-up on #11... The Taxonomy error was closed as a duplicate of #2252739: Taxonomy field handler fatal error in linking to terms, cannot create taxonomy views with defaults.

So I tested Taxonomy with the patch for that issue, which works by the way. Everything is cool with Taxonomy in multilingual views: when I translated terms, the view showed all translations of all terms, and I was able to use the language filter and language field with no problems.

So the only thing we should probably fix for taxonomy is the minor issue that it is pretty silly to display the language fields/filters if there is no Language module.

Although I have been thinking about this. I think we should probably still have those filters/fields available (they work OK) -- and we should for Node and Comment as well. The reason is that if someone imports a view from elsewhere and it has those fields, I think you'd end up with some kind of problem.

To test this, I made views of Node and Comment [which both have this if(language is installed) check in their hook_views_data] and then uninstalled the Language and Content Translation modules. I ended up with "Broken/Missing Handler" in the field/filter I had created... The views still mostly worked though -- the filter just wasn't applied, and the field displayed the label only. I guess that's not a tragedy.

So I've added a note to the Comment issue, and filed #2320521: Follow-up: Node language views filters need label adjustments for Node.

Gábor Hojtsy’s picture

The language field/filter was moved recently to the views module I think in #2301045: Standard profile has views which include elements dependent on uninstalled modules, not valid in config, with the explicit idea that because it is used so widely, it should not be living with language module. Content can have language even without language module, although the language selector UI on content will only be visible once language module is enabled (unless some other module alters it in, which is while possible, not very likely).

jhodgdon’s picture

Filed #2320743: Taxonomy views needs filter/field on original language for taxonomy (needs to have a filter on Original Language). Summary updated.

Gábor Hojtsy’s picture

#1498664: Refactor user entity properties to multilingual just landed, so point one all done? :)

Gábor Hojtsy’s picture

Oh, well, that has some follow-up tasks to do in the last column, still todo.

jhodgdon’s picture

I talked to Gábor in IRC just now... I think the bullet point about "enforce that each entity has a data table" can be scrapped. On #2218065: Need to join fields to the entity field data tables, not entity tables, or filtering increases number of results we can fall back to the join we are doing now if an entity doesn't have a data table, and if a contrib entity is multilingual/fieldable without a data table, they will run into this filtering problem... so be it.

But we should document somewhere that a translatable fieldable entity should have a data table. Filed: #2321381: Document that fieldable translatable entities should use data tables

plach’s picture

Issue summary: View changes
jhodgdon’s picture

Issue summary: View changes

Wow, a lot of progress in the past week! Updated summary.

jhodgdon’s picture

Issue summary: View changes
jhodgdon’s picture

In item #3 in the issue summary, the follow-up to making sure all entities have the language filters (translation and original) is to survey the provided default views configs and make sure they are appropriately filtered.

I just did a survey, and filed #2323899: Provided default Node views need language filtering to take care of the views provided by Node and Taxonomy, which (aside from the Home view and Node Admin, are lacking filters).

The default Comment view (recent comments block) will also need to be updated -- it has the filter but currently that filter is for the original comment language, not the translation... and there is some question in my mind as to what the correct thing to do for comments is anyway. That probably requires more thought.

I didn't look at the User views either, because we don't have filtering yet there.

jhodgdon’s picture

Tested item #3 in summary for User entities. User views currently only have a filter available for the preferred language of the user, although for users, all translations of all users are now being shown in the results. Filed #2323939: Views user language field/filter is for original language code, no translation language field/filter.

jhodgdon’s picture

Issue summary: View changes
jhodgdon’s picture

jhodgdon’s picture

Issue summary: View changes

Yippee! All of #3 is done! Updating summary. May need some follow-ups but I'm not sure about that actually.

jhodgdon’s picture

Updated #6 to add additional issue link that's blocking progress on this.

jhodgdon’s picture

Updated #6 to add another new issue to the fray.

jhodgdon’s picture

Issue summary: View changes

One more item added to #6, just a note that when all the issues are fixed, to retest and make sure it is all working. I know we have automated tests but... just need to double check it.

jhodgdon’s picture

Issue summary: View changes

Updating #6 due to one of the issues being resolved! I like it when it gets smaller. :) Actually it's down to 2 open issues now, both of which are being worked on.

jhodgdon’s picture

jhodgdon’s picture

Issue summary: View changes

Updating #6 again, to add note about relationships.

jhodgdon’s picture

Updating #6 again, with fixed issues! We're getting close...

jhodgdon’s picture

Gábor Hojtsy’s picture

jhodgdon’s picture

Yeah, I was just going to add that second one to this meta. Didn't know about the first one. Adding both as part of item 6.

jhodgdon’s picture

Issue summary: View changes

Just updating status etc.

jhodgdon’s picture

Issue summary: View changes

Summary update. #9 was fixed.

jhodgdon’s picture

jhodgdon’s picture

jhodgdon’s picture

Issue summary: View changes

Uh oh, I was just testing another issue and noticed that after adding a translation of a User entity, the People page now shows the user twice. We need an issue for this... similar to what is going on for admin/content page currently... which also doesn't seem to be part of this issue.

Adding to summary so it doesn't get forgotten.

plach’s picture

Gábor Hojtsy’s picture

Issue summary: View changes
jhodgdon’s picture

Issue summary: View changes

Adding that one to the summary. Also combined #11 and #12 in the summary.

dasjo’s picture

jhodgdon’s picture

Issue summary: View changes

OK, adding to summary in #6.

jhodgdon’s picture

Issue summary: View changes

#11 has been fixed. Yahoo!

jhodgdon’s picture

Issue summary: View changes

Found a new issue, added to #6.
#2582499: User views - user links are not localized
One step forward, one step back.

jhodgdon’s picture

Issue summary: View changes

Found another one: #2582535: Duplicated rows on admin/content if author is translated -- adding it to [6] because it's related to testing that was listed on [6].

xjm’s picture

Category: Bug report » Plan

#2600804: AJAXified glossary view arguments fail when Language URL detection enabled might be another issue to add here, although it appears to also exist in D7.

jhodgdon’s picture

Yes, sounds like a good one to add here. Thanks! As a note, many of these issues were also broken in D7. I doubt very many of them are new, really. Anyway, adding to summary and related issues.

Version: 8.0.x-dev » 8.1.x-dev

Drupal 8.0.6 was released on April 6 and is the final bugfix release for the Drupal 8.0.x series. Drupal 8.0.x will not receive any further development aside from security fixes. Drupal 8.1.0-rc1 is now available and sites should prepare to update to 8.1.0.

Bug reports should be targeted against the 8.1.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.9 was released on September 7 and is the final bugfix release for the Drupal 8.1.x series. Drupal 8.1.x will not receive any further development aside from security fixes. Drupal 8.2.0-rc1 is now available and sites should prepare to upgrade to 8.2.0.

Bug reports should be targeted against the 8.2.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.6 was released on February 1, 2017 and is the final full bugfix release for the Drupal 8.2.x series. Drupal 8.2.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.3.0 on April 5, 2017. (Drupal 8.3.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.3.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

_Archy_’s picture

I am not quite up to date with this issue, but just found a new one related to this. This might already be in progress / duplicate. Would be nice if someone checked it out: https://www.drupal.org/node/2862511