Problem/Motivation
Drupal core exposes translation language fields for the entities it supports. Most entities just use the LanguageField provided by views but there are also some entity fields that override that in user, node and taxonomy. User has a link to user option, node has a link to node option and taxonomy always links to the taxonomy term. These specialized field handlers are not universally applied, some language fields of users and taxonomy do not use their handlers but the default views handler.
The views default language field handler and the node language field handler both have a "Native name" option which is a no-op. So if you configure a node or a view field handler to display in native name, that will never work. There is a @todo at both places which points to #1616594: META: Implement multilingual CMI which has long been closed.
Proposed resolution
1. Unify the language field handling by having one field handler instead of entity type specific handlers.
2. Extend the language formatter from StringFormatter, so we get linking to entity for free.
3. Remove the special cased entity specific language field handlers.
4. Restore the native name display functionality in language formatter.
Note that original language handling in views is broken and will not be possible to solve the same way, see #2450195: Original language of entities not accessible in views anymore for an issue solving that.
Remaining tasks
Review. Commit.
User interface changes
Consistent translation language field options across entity types.
API changes
Specialized views handlers are removed from node, user and taxonomy. Views using those would need to be updated.
Comment | File | Size | Author |
---|---|---|---|
#168 | interdiff.txt | 1.62 KB | Gábor Hojtsy |
#168 | 2384863-168.patch | 26.58 KB | Gábor Hojtsy |
#166 | interdiff.txt | 5.89 KB | Gábor Hojtsy |
#166 | 2384863-166.patch | 26.57 KB | Gábor Hojtsy |
#164 | interdiff.txt | 867 bytes | Gábor Hojtsy |
Comments
Comment #1
Gábor HojtsyComment #2
Gábor HojtsyComment #3
vijaycs85Comment #4
Gábor HojtsyComment #5
Gábor HojtsyCovering the other case and the config schema as well. No test yet.
Comment #6
Gábor HojtsyConflicts in part (config schemas) with #2325269: Test and fix views in test_views directories against their configuration schema, postponed.
Comment #7
vijaycs85extend/reuse++
@todo--
This is good to go.
Comment #8
Gábor HojtsyWell, this does need its own tests to ensure the option actually works :)
Comment #9
Gábor Hojtsy#2325269: Test and fix views in test_views directories against their configuration schema landed, unpostponing.
Comment #12
Gábor HojtsyRerolled.
Comment #13
Gábor HojtsyComment #14
Gábor HojtsyA quick attempt at test coverage.
Comment #16
Gábor HojtsyThe language names will not be displayed translated if the translation is not there, hahaha.
Comment #17
Gábor HojtsyEven more tests to cover both behaviors of the other language field and to test the original behavior of the view, so all two field handlers in both variants.
Comment #19
Gábor HojtsyIf we remove the link_to_node setting then this becomes more applicable to the more general language field handler too.
Comment #20
vijaycs85hopefully, it won't come again as I can see it is there in core views configuration.
Comment #21
Gábor Hojtsy@vijaycs85: that setting makes sense for the field as in the view. It does have a FALSE default if not provided in the view, so the effective version is the same. However it would not be valid for a language type field, so instead of removing it in the test where I convert the field type, I removed it altogether because it works the same way.
Comment #22
jhodgdonI tried this out in a Node view and it seems to be working fine.
I tried a Taxonomy view and it doesn't have this setting though on its language fields. Should it? Probably taxonomy has its own plugin, ugh. Is that out of scope for this issue? It seems as though if we have the option on generic language plugins and node plugins, we should have it for all of the language plugins.
And... Can we get rid of the specific entity language plugins and unify them possibly, rather than having 5 different ones or whatever we have?
Comment #23
Gábor Hojtsy@jhodgdon: I consider that a different problem from this issue which is about the plugins not providing the advertised features.
Comment #24
Gábor Hojtsy@jhodgdon: for the views data / field types question, NodeViewsData overrides the language fields to use the node_language plugin but EntityViewsData defaults language fields to the language plugin. So entities with a language field other than node should offer this field handler up for their language field. For example for taxonomy terms:
Comment #25
jhodgdonI didn't see that taxonomy setting. If you use the field "Translation language" instead of "Original language", the Native Language box is NOT in the settings. ?!?
Comment #26
Gábor HojtsySo node, taxonomy and user seems to have their own language field handlers that extend from their respective main module handlers. And then the general language handler. Only node and the general one has native language support. While this issue was opened to fix the falsely advertised native feature that did not work before this issue, we can also unify the fields here, but that would require some examination that those special field features are actually not needed.
Comment #27
jhodgdonOK. So I got interested in the question of what the individual entity-specific Language plugins are doing that the base one in \Drupal\views\Plugin\views\field\LanguageField is not doing. (This plugin is the default field plugin used for all language fields on entities unless the specific EntityViewsData class overrides the plugin choice).
Here is a survey:
a) Taxonomy
- Uses \Drupal\taxonomy\Plugin\views\Field\Language (ID "taxonomy_term_language") as the plugin for the Translation Language field, but not for the Original Language field (which in itself is odd; the Original Language field uses the base Views plugin default).
- This plugin extends the Taxonomy field plugin. This gives it options to link to the taxonomy term page and to convert spaces in the name (in this case, the language name?!?) to hyphens.
- The taxonomy language plugin does not currently have the native language display option on it.
- To me, neither of the options from the Taxonomy plugin makes any sense for a Language field and I think we should get rid of this override and the \Drupal\taxonomy\Plugin\views\Field\Language plugin class entirely, and use the default base views plugin instead, which would give us the native language option for free, and make the two language fields for Taxonomy behave the same.
b) Node
- uses \Drupal\taxonomy\Plugin\views\Field\Language (ID "node_language") for Translation language, Revision Translation language, and Revision Original language fields.
- This plugin extends the Node field plugin, with the addition of a native language option.
- The Node plugin gives you the option to link to the node. Like Taxonomy, I don't really think it makes a lot of sense to link the language field to the node.
- So again I think we should just get rid of this language plugin and use the Views default language plugin instead.
c) User
- uses \Drupal\user\Plugin\views\Field\Language (ID "user_language") for the "Original language of the user information" field, but not for the other language fields.
- As usual this gives you a "link to user" option, see (a) and (b).
- Again I think we should drop this, and that would also make all language fields on User behave the same, since the others use the default plugin already.
d) Other entities: there are no [field][id] plugin overrides for user language fields that I can see in the other classes that extend EntityViewsData: AggregatorItemViewsData, AggregatorFeedViewsData, CommentViewsData, or FileViewsData,.
So... If we think that it's important for Language fields to have the "link to entity" option, we could add that to the base Language plugin. That was actually done for regular short text fields recently on #2387019: String field formatters cannot link to their parent entity. But that is all we would lose if we got rid of the 3 specific language plugins and switched to using the one default Views plugin. To me, this option doesn't seem all that important to have, but if we need it, it could certainly be provided. The way to do that, actually, would probably be to make a new field plugin called "entity language" with this option, and use that as the default for all language fields on entities in the base EntityViewsData class. (The base language plugin for Views as a whole doesn't know about entities.)
Anyway, thoughts? I don't think losing the "link to entity" option is a major problem, and unifying everything in one central plugin, and getting rid of 3 classes and several lines in those EntityViewsData customizations would be good.
Comment #28
Gábor HojtsyI can see use cases for example to display a translation of a node with a link using the original language name back to the original language for example. Is there a a use case of a language field that is NOT on an entity? Should we have a basic_language and a language formatter like with strings now?
Comment #29
Gábor HojtsySoliciting feedback at https://twitter.com/gaborhojtsy/status/545605810509070336
Comment #30
jhodgdonAh, good point. If the links actually are produced in the correct language, that would be a good use case. (I am not sure they do. At some point we should test this, once we're closer to having views actually working for multilingual... they may be working right but I'm just not sure. Actually I'll add this to the giant #6 on the Views meta.)
So, I think yes we should have a basic_language and language formatter. While we have no use cases in Core for the non-entity language, that is not conclusive because all our Views integration in Core is entities. Oh wait, we do have several things in Core that are not entities or fields that have views integration, like dblog and tracker... I don't see any language fields in any of them, but in theory someone could have a non-entity db table with views integration and a langcode column and want to have a language field, so yes I think we should keep a non-entity version around. Making its name basic_language and keeping language for the entity version would mean not having to change Core, so that does sound like the best idea with the least impact, since I think most views integration in d8 contrib is likely to be entities, right?
Onward!
Comment #31
jhodgdonAdding the ability to link back to the entity would also be consistent with the string decision: just put the checkbox on all "short strings". If it isn't used, it doesn't cost much.
And I think... wouldn't it be as simple to implement as having the entity language plugin extend the new entity-string-with-link plugin, rather than extending plugin base or the non-entity language plugin?
Comment #32
Gábor HojtsyRolled the patch with removal of the "special" language fields than, just using one common language field. (No linking yet, we should discuss if this is an 80% use case given that the views content rewrite features can ALWAYS be used to link it).
Comment #33
jhodgdon+1000 on this patch! I love that almost all of it is "remove stuff", and that most of the added lines are in a test, with a few lines added to make the "native language" functionality on the default Views language field actually work.
One thing I think this patch is missing is to remove also the 'user_language' custom field from UserViewsData? I see Node and taxonomy but not User.
And agreed we still need to decide whether the Language field needs "link to entity" on it. You can definitely use rewrites to do this... well probably anyway. I am not sure that rewrites would be easy to use to get a link to the entity in the correct language?
Comment #34
vijaycs85Looks good to me.
Comment #35
jhodgdonWe need to do user as well as node and taxonomy, see #33. I guess I forgot to change the status.
I'm also not sure we decided it was OK to remove the ability to link language fields to their entity, which was present in the special fields for node, user, and language.
Comment #36
vijaycs85Ok, here is a patch that removes user_language field.
Still need to address this one.
Comment #37
Gábor HojtsyLOL on the duplicate definition, haha....
All right, if we are to do the link feature, is this the kind of stuff we want done there? The string formatter does not check if you have access to the parent entity. (This may actually be a problem of the StringFormatter?)
Comment #38
jhodgdonRight now there is no access checking at all for base entity fields, see #2385443: Test that base entity fields on views respect field level access. Once that is taken care of, which actually would be on #2342045: Standard views base fields need to use same rendering as Field UI fields, for formatting, access checking, and translation consistency, which will make all base entity fields rendered using entity formatting, I think we'll be OK.
The reason I think so is that if you've got a row that is entity X and you have no permission to view entity X, then you shouldn't be seeing *anything* in that row at all, including the language field, whether or not it is linked to entity X. So I don't see how that is a problem? It may not even be a problem now; hopefully views is not letting you see any results for entity X if you don't have entity view permissions.
Comment #39
vijaycs85@Gábor Hojtsy, reg: #37.2 - Looks like it is a copy/paste from parent class
core/modules/user/src/Plugin/views/field/User.php
. It doesn't look like related to language.Comment #40
Gábor Hojtsy@jhodgdon/@vijaycs85: indeed, it makes sense to not need this on the field formatter level. So the only remaining question is indeed the linking feature loss with this patch.
Comment #41
jhodgdonIf we assume (not saying we should!) that we need to not lose this functionality of linking the field to the entity, how much would it take to make it happen? It seems like the answer is "not much":
- We'd need a specific entity language field handler class, which would either extend the existing "short text with link to entity" handler that was created recently and make it into a language renderer, or extend the existing "language that isn't entity" handler and make it link to the entity.
- We'd need to make this the default handler for language fields in the EntityViewsData base class, either by (a) giving this new one the "language" ID and giving the old non-entity one a different ID, or (b) by changing EntityViewsData. Probably (a) is easier.
I think that rather than debate whether we need this functionality or can afford to lose it, we should just do this.
Comment #42
Gábor HojtsyWell, we'll also need test coverage for the non-entity language field then. It is true that we can still test it on entities. So may not be a biggie to add that.
Comment #43
dawehner... for those kind of access checking we always rely on query rewrite, so for example the hook_node_grants/records() system. This is taken care of.
Great work and great cleanup here!
Do we have field formatters for that? ... we would at one point need that, in case we really want to support it.
Comment #44
Gábor Hojtsy@dawehner: we already extended the scope based on what @jhodgdon suggested, I would avoid unifying it with field formatters as well here. The original scope was some views exposed settings that lied because they did not have any effect.
Comment #45
dawehnerWell sure, it was more of a general question ... especially in the context of the base field formatter issue.
If I read the patch we already add a test.
Comment #47
Gábor HojtsyRerolled. It did not apply on the user.views.schema anymore because the user_language duplication was removed earlier. So we only need to remove one now. Still needs work for the restoration of the linking to entity feature. I looked into how this can be restored by a user on the UI and that looks very confusing. You need to add the entity id to the field view, then hide it and then know the URL pattern used for the entity at hand. If we are fine with that workaround, then this can go in as-is.
Comment #48
rodrigoaguileraI did a manual testing
-Enable all 4 multilingual modules
-Add a new language
-Translate the native name of the new language using config translation
-Add a new language field to the content listing view checking "Display in native language"
-Create nodes in different languages
-Check the language translation
without the patch the languages are shown in English
With the patch the languages are translated.
I also reviewed the code and fixed a couple of comments and an indent problem.
Comment #49
Gábor Hojtsy@rodrigoaguilera: thanks! Can you please post an interdiff (https://www.drupal.org/documentation/git/interdiff) so we can review the changes made? Thanks!
Comment #50
rodrigoaguilerahere it is
Comment #51
jhodgdonGreat! #45 marked it RTBC. #46 was a reroll. Interdiff to the latest patch looks good, and there was a manual test even. And I agree that the patch looks good, so I'm also +1 on RTBC.
However:
- The issue summary doesn't seem to match the patch.
- We need a beta evaluation added to the summary.
Comment #52
Gábor HojtsyUpdated the summary. Looking at how 3 removed field handlers had linking features (user and node configurably and taxonomy automatically), not sure we should get this in without that. What is so wrong with having an EntityLanguage field handler and no Language field handler for non-entities? That is to only support the entity case for language fields?
Comment #55
jhodgdonLooks like the patch doesn't apply now...
Regarding not having a non-entity Language field available in Views... I am not sure whether or not that is OK. For other fields, Views has maintained non-entity versions for base database data. But maybe Language is so entrenched in our entity system that having a langcode field in a non-entity table doesn't even make sense?
Comment #56
Gábor HojtsyLooking into implementing the entity and the entityless one... I think we spent enough time debating if we need it, we should just go ahead and do it or we'll agonize on it for eternity. There are core tables with langcode that are not entities, eg. locale and path alias tables.
Comment #57
Gábor HojtsyComment #58
Gábor HojtsySo I went to add entity linking support and by researching how best to do that realized, that it would be better to just extend the existing string FIELD formatter which requires a minimal subset of #2342045: Standard views base fields need to use same rendering as Field UI fields, for formatting, access checking, and translation consistency, only the parts absolutely needed here. It makes sense to have the same features in entity field displays as in views field displays and not to need that implemented twice. So we can get rid of the views language field entirely hopefully in favor of the field formatter which can then get the linking feature as inherited from the string formatter easily. Looking forward to what testbot thinks of this one :)
Comment #60
Gábor HojtsyRestoring the language views field, no specific tests yet (only restored pre-existing changes from #56 in views.field.schema.yml and LanguageField.php in views). @dawehner points out we should support locale like use cases. See eg. #1853534: Reintroduce Views integration for locale.module.
Comment #61
Gábor HojtsyResolving fails by undoing some misguided plugin_id changes in shipped views. Filters, sorts, etc. for languages should not be switched to field, just fields :)
Comment #62
Gábor HojtsyUse the same native_name naming for the views field and add tests for that too.
Comment #64
dawehnerSo I guess we still need to drop this change for now, right?
Comment #65
Gábor Hojtsy@dawehner: no, why? Then we loose the linking to the entity feature which the field formatter implements.
Comment #67
Gábor HojtsyForgot interdiff for the last change.
Comment #69
Gábor HojtsyResolve the schema fails. The field needs a type key to specify the field formatter. I messed up a view and the schema fix discovered a bug in another view (minor).
Comment #70
Gábor HojtsyFixing more fails. The summary need to be returned so it can be altered and the entity views data test should expect the new thing.
Comment #71
jhodgdonLooking good!
Nitpicks:
a)
This last line should be "Displayed in native language". oops.
b) I don't think settingsSummary() is returning a value? oops. [Oh, you fixed that in the latest patch while I was reviewing, good!]
c) I'm unclear on why core/modules/views/src/Plugin/views/field/LanguageField.php changed the setting machine name from native_language to native_name? I guess it was for consistency with the Field API plugin, but it seems like most of the previous specific Views plugins like node_language used "native_language" as the setting name too. Why make this change? I don't see any discussion above, and I don't see "native_name" anywhere in Core prior to this patch.
Comment #74
Gábor HojtsySo I renamed to native_name because the option toggles to display the native name of the language, not the native language per say. We can also think it displays the name in the native language of the language, which is slightly more complex but explains native_language :D I opted to do the rename because it seemed like we are getting rid of the views field and at that point there was no precedent for the setting. Then I added back the views field, so now it again has a precedent. There is no precedent for native_language btw outside of this issue either.
Comment #75
Gábor HojtsySo now this seems to be down to two fails:
1. NodeLanguageTest fails on no 'entity_type: X' value on the field. That should be a relatively easy fix I hope :)
2. HandlerAllTest fails with
which is to me a huge WTF. I found core/modules/views/src/Plugin/views/field/Boolean.php uses it as UtilityXss for some reason (also puzzling), but that sounds like may have been the workaround for this WTF. Why is that happening?
May be able to continue work on this tomorrow, not tonight anymore.
Comment #77
Gábor HojtsySo the HandlerAllTest fail is due to how there is a core/modules/views/src/Plugin/views/field/Xss.php, so the Xss class used in core/modules/views/src/Plugin/views/field/Field.php leads to the conflict. We should rename the Xss utility class used in Field.php to UtilityXss like in Boolean.php. Thanks @dawehner for the guidance. Should now only have one test failing.
Comment #79
Gábor HojtsyThe remaining fails need changes in how views handles field formatters. There are several details not provided for them to work. Postponing on #2407801: Views generic field handler does not work with base fields.
Comment #80
Berdir@dawehner suggested that I add a comment here that it would be useful to be able to print out the language code and not the label (For example, in feeds and other kind of data exports). Maybe add that as an option here while all this stuff is being worked on. Or some sort of output rewriting, does not have to be easy, just possible :)
Comment #81
Gábor Hojtsy@Berdir: maybe I made a mistake by doing this with the field formatter instead of copying the thing in views, but that does not mean I would not love to keep scope control somehow :) I suspect views output rewriting may support that already, or if not, it would be a great addition in another issue. This was supposed to resolve the falsely advertised option in views and is already adding new field formatter options because that was the fix to make less copy-paste code necessary.
Comment #82
Gábor Hojtsy#2407801: Views generic field handler does not work with base fields now landed. Should pass now, although it will not apply yet.
Comment #84
Gábor HojtsyRerolled. Should hopefully pass now given prior fails were all from #2407801: Views generic field handler does not work with base fields.
Comment #87
Gábor HojtsyFirst fix config immutability.
Comment #89
Gábor HojtsyPart of the reasons for the fails is that EntityViewDisplay.php receives the right rendered value BUT then it does not get the right value for access in buildMultiple():
So although the built view value will be right, the access nulls that out. I don't get why would that happen, the entities where we have the language field on are published, otherwise they would not show up in my view. What else should decide field visibility on this item?
Comment #90
Gábor HojtsyIt would be amazing to get one more pair of eyes on this, I feel like I spent way too much time on this :(
Comment #91
dawehnerI think the problem is
language_entity_field_access()
, see interdiff.It at least let more of the tests pass, some still have a different behaviour, but I have seen similar problems in the other issue as well.
Comment #92
Gábor HojtsyAh, that makes sense. Retitling and elevating as a sub-issue of #2407801: Views generic field handler does not work with base fields because this implements that for the language base field basically. Not sure of sub-issues of that like this one should be critical or not.
Comment #93
Gábor HojtsyUpdated patch includes the one line fix from @dawehner with some docs updates and another one line cleanup I found yesterday while debugging.
Comment #95
Gábor HojtsyComment #96
Gábor HojtsyDoh. So the remaining fails seem to be two things:
1. The field is not actually rendered in the language of the row found. That to me looks like the territory of #2394883: Language setup for entity and field based rendering in views is independent, confusing UI, lacking test coverage. We cannot really decide here which translation to take. Field::process_entity() has some logic, but it is in the realm of #2394883: Language setup for entity and field based rendering in views is independent, confusing UI, lacking test coverage to resolve that to support row language. We can of course avoid to test for a nonexistent feature that we are not going to resolve either by not having translated entities in the view. Going to try that next.
2. When switching the config to the views built-in field, it does not actually use that to render the language names, it keeps using the prior setup (I assume due to some kind of caching). Will look into how to resolve that.
Comment #97
Gábor HojtsySo this fixes #96/1 by not testing for the values of a translated node, because #2394883: Language setup for entity and field based rendering in views is independent, confusing UI, lacking test coverage is not yet resolved. #97/2 is still intact and that is not due to caching. The formatter based field handler is used all the time instead of my specified one. Don't know yet why.
Comment #98
yched CreditAttribution: yched commentedI don't really feel qualified to vet the approach or actual code, just one minor remark :
The comment would now be more relevant in the class' implementation of viewValue() :-)
Comment #99
Gábor HojtsyComment #101
Gábor HojtsyPostponing on #2394883: Language setup for entity and field based rendering in views is independent, confusing UI, lacking test coverage so we can properly test this.
Comment #102
Gábor HojtsyThat one landed. Back here.
Comment #103
tstoecklerSettings summary should simply return an array of strings, not a render array.
See FormatterInterface::settingsSummary().
Comment #104
Gábor HojtsyWell, the existing settingsSummary() on StringFormatter (the base class) returns a render array *before this patch*. The API does not define if it would be a render array or a simple string array?!
Comment #105
Gábor HojtsyWell looking at https://api.drupal.org/api/drupal/core%21lib%21Drupal%21Core%21Field%21F... string formatter is the only outlier. Duh. Will fix it here then.
Comment #106
Gábor HojtsyFirst rerolling #93. Ignore #97 as the issue that those fails were due to is now in :) So we can avoid removing some of the test coverage as #97 did and instead move back to the fuller patch in #93.
Did not yet address the feedback from @tstoeckler and @yched but intend to do so.
Comment #108
vijaycs85In this patch:
1. Addressed both #98 and #103.
2. Re-exported views.view.language_test.yml
TODO: Fix test fails.
Looks like the new language field always gets the current language instead of the item's language at below $item:
Comment #110
Gábor HojtsyGiven you fixed the base class, this will never happen :) We should just add another item to the summary array (only keep the else case of this one).
The reason I did not have these was that when you turn these between a plugin_id field and language, not all options will be valid. As you can see in the fails, you started to get config schema fails due to these. Instead of needing to remove a whole bunch of items, I removed the ones that were used with their default values, so the test is cleaner. If you add them back, you also need to add removal code to the test when you switch the plugin id of the field.
Comment #111
vijaycs85Thanks @Gábor Hojtsy.
#110.1 - Just updated the if condition, so the summary gets same text.
#110.2 - Yep, rolled back changes I made to views in #108.
Fixed few test fails as per our discussion on IRC.
Comment #113
Gábor HojtsyI don't think you implemented what I suggested here?
This should be done conditional on a flag in a Drupal state. If you do this unconditionally, then the first part of the test is futile, the field formatter is never tested.
Also the comment is outdated here.
Comment #114
Gábor HojtsyHere is a short update for those.
Comment #116
Gábor HojtsyFix the state checking in the test module code, so it fails again "the right ways". Still problematic with row language :/
Comment #117
Gábor HojtsyDuh, patch was somehow missing all NodeLanguageTest changes. Huh.
Comment #120
Gábor HojtsyDebugging the fails.
1. So langode is the worst animal to do this on :D First it is not translatable, so the translation langcode was always
und
. Fixing that in Field::getFieldLangcode().2. However the langcode field value on the entity is still the original one and not the translation. Not sure how best to make that one use the appropriate langcode for translation. Added a @todo for that. I'll try to summon @plach for that to help :(
Comment #122
Gábor HojtsyRerolled. The user language field was changed for URL API changes, but we remove it anyway :D
Comment #123
jhodgdonI'm trying to organize the "fix this field" issues and make sure there are issues for all of them, so making sure they're all listed on #2393339: [META] Make sure Views base fields are using Field API for formatting, and do not lose functionality as a start.
Comment #125
vijaycs85clearing viewData cache seem to fix the 4 fails...
Comment #127
Gábor Hojtsy@vijaycs85: yay, thanks! Now only the rendering language problem for the langcode needs to be fixed. @plach: do you have ideas?
Comment #128
jhodgdonSo I'm a bit confused here about the rendering problem.
There are actually 2 language fields that Views is allowing us to add to views: the "original language" of the node (from the base node table) and "translation language" (from the node field data table).
If we're using Field UI rendering... can we actually get at both of these database fields? Is the problem that Field rendering says "Oh, you want a field called 'langcode'. I see that is on the node table, let's use that" rather than checking the field data table? Really it needs to figure out that in the View you have chosen either the one on node base or the one on the field data table, and get that data, rather than trying to know which one to choose.
I'm just kind of guessing here, based on what you said in #120, trying to understand what's happening.
Comment #129
Gábor HojtsyAs far as I understand, the entity field API will not render a translation of a non-translatable field, that does not make sense. The langcode field is not translatable. To display the proper "translation" of the langcode field, that would need special casing in the entity field API. In #120 I already fixed Field::getFieldLangcode() to special case the langcode field when looking up the translation language (second part of https://www.drupal.org/files/issues/interdiff_10167.txt). Still when we attempt to actually render the field in that translation, it is not different in that translation because its not translatable. That still needs fixing.
The same problem should obviously not appear for other fields, because if they are not translatable they will *actually* not be different in the translations. Unlike language which is a special flower.
Comment #130
jhodgdonAh. So maybe we shouldn't use the basic 'field' handler for language fields then? We may just need to subclass that handler and do something different?
Comment #131
Gábor Hojtsy@jhodgdon: I think it would be more uniform if Field.php could handle it, but either way it will need to have that code for the language field somehow either there or in a subclass.
Comment #132
jhodgdonYeah, it does seem like a special case.
Comment #133
Gábor HojtsyPatch did not apply anymore, rerolled.
Comment #134
Gábor HojtsyI tried my best to butcher my way into rendering the right language. Once again the obvious reason for the fails is that the field langcode is different from the entity langcode. ContentEntityBase keeps only one copy of non-translatable fields and even if we attempt to change that one copy of the langcode field it freaks out because such a translation already exists of course. So I went to remove the translation but that cannot be removed because its the default langcode... Duh. So this really needs some adult supervision. I cannot even butcher my way into a success on output let alone any meaningful code solution.
This would not work:
Comment #135
jhodgdonI'm confused about what is working and what is not working here.
Comment #137
Gábor Hojtsy@jhodgdon: so the problem is the language field itself is not translatable; so it does not have values for translations; you cannot therefore render a language field in a translation, it will always use the base value (original language of entity) as it is not translatable.
I fixed Field::getFieldLangcode() with this code to special case the langcode key so it is capable of looking up the right translation:
So that works around the case, that the langcode field is not translatable. So that makes Field::getFieldLangcode() find the right translation. Then Field::process_entity() uses that to:
So we have the right translation from then on. And then the actual rendering is in Field::getItems():
The Entity::get() here will skip looking for translations for this field, because .... its not translatable. Yeah. So although the entity language is cool here, the field will never have that language, because its never translatable. The butchering I tried above is to try and set the field value anyway temporarily, but you cannot do that because the entity default language would change, and you already have a translation in that language.
Comment #138
jhodgdonIt seems like the only way around this is to make a special subclass of the Field views-field-handler-plugin class to handle the Language field(s), rather than trying to make the generic Field handler work.
Comment #139
Gábor HojtsyYeah my problem is I don't know what code would go in that class. Once we know the code, its easy to put it into Field or a subclass of Field, either way :) I am not sure how to get around the very tight and otherwise right protection around the language field :)
Comment #140
plachI started looking into this, not done yet...
Comment #141
plach@Gabor:
I tried a few experiments without luck, I'm afraid this is telling us there's something wrong with our current model. Basically here we want to display the translation language, but there is no such field defined. Given that any piece of data attached to a (content) entity needs to be a field, we are basically missing a (translatable) field definition for the translation language.
Now, if we look at our API, we already have the translation language in the form of the
ContentEntityInterface::language()
method, and we have the default language viaContentEntityInterface::getUntranslated()
. The concepts are there, but they are not backed by corresponding field definitions.So what I tried to do (I'm not really done yet, wanted to get some feedback before coding too much), is going a pretty radical way and make our
langcode
definition translatable. If we look at our standard schema the{entity_field_data}.langcode
is actually translatable, as we have a different value for each translation. The missing part is adefault_langcode
untranslatable field definition allowing us to tell which is the default translation.After overcoming the initial shock, I realized this change seems to fit pretty well our current architecture: for instance this would allow us to have a language selector on content translation forms letting us change the translation language. This is not a big use case probably, but I think it clarifies pretty well that the direction should be correct.
Another example is that this way Rules would be able to distinguish between the various translation objects natively, by simply relying on field definitions, without needing to special case the translation language.
If you agree this is an approach worth trying we should open a separate issue and postpone this on that.
Btw, I tweeted about this at https://twitter.com/plach__/status/568937859844956161
Comment #142
BerdirNot sure I fully understand this yet.
This would essentially mean the field definitions would be the way the storage already is, and it wouldn't actually change the storage, correct? So less magic needed in the storage that makes the non-translatable langcode field actually stored per translation and the non-field default_language column.
Sounds like a change that makes sense, I've wondered before why it is not translatable, $node->getTranslation('de')->langcode->value isn't what you expect it is, as you said yourself.
Translatable vs. localizable, given we make the language field translatable, how does content_translation know that it is not actually a field that can be "translated" by the user? Or can it, in a way, based on your comment?
As TMGMT maintainer, I have yet another meaning of "translatable" to think about: The parts of an entity that I can extract and send to a translator, based on the tell that I sent you, my current code started picking up the content_translation_source field (with a value 'und') as a translatable text because it was translatable. But I think that can be solved by adding an options provider, which is also required for validation, @fago already opened an issue for that.
Comment #143
yched CreditAttribution: yched commented+1 on #141, makes a lot of sense IMO.
(if, by any chance, the refactor happened to also clean up the fact that ContentEntityBase internally stores the field values of the default language under 'und' rather that the actual langcode, I'd be double yay :-) But chances are this is an orthogonal refactor ?)
Regarding content_translation :
I have lost track a bit on whether BFD::setTranslatable(TRUE) means
"the field *is* translatable on all bundles, no config will ever change that"
or
"the field *supports* translatability, and content_translation UI can configure it to actually be translatable or not on a given bundle".
Because the latter doesn't sound like the behavior we'd want for that "translatable language field". If the entity type & bundle are configured to be translatable, I shouldn't be able to keep the "language" field untranslatable, that would be very broken.
Comment #144
plachSounds good, created #2431329: Make (content) translation language available as a field definition and replied there. Marking this postponed again.
Comment #145
fagoyeah, the change makes a lot of sense to me as well - so the field definitions really show you how things are stored. Clarity++. It's quite a bit of change for the meaning of $entity->langcode but given language() already works different, I think aligning them to work the same way makes it work as developers would expect it.
However, I do think I miss what it buys us here. A possibility to easily render the used display language?
Comment #146
Gábor Hojtsy@fago: yes, the problem this is blocked on is the translations' langcode cannot be rendered as a field (because its not a translatable field). See #137.
Comment #147
Gábor HojtsyComment #148
jhodgdonLike fago, I'm still a confused about why we need to make langcode a translatable field.
It seems like what we need for this issue is
a) An entity-aware method to use as a Views formatter, which would print out the language of the translation being displayed, with the option to display the language name either in the defined Views display language for that row, or in the language's native language.
b) The same as (a) but to print out the original language of the entity.
The solution proposed currently in this issue is to use the generic Entity Field views formatter for these values, but this is not working because langcode is not currently a translatable field. So the proposal here is to make langcode be a translatable field.
It seems to me that an alternative solution would be to make two new formatters for "entity-aware translation language" and "entity-aware original language" that would use the existing language methods on ContentEntityInterface to discover what language to print, and then print it out. I don't understand why that wouldn't work?
We may have other reasons, as discussed above, for making langcode a translatable field, but it seems like the "make a non-generic formatter for Views" solution would be much easier, and would solve the problems of this issue?
Comment #149
Gábor Hojtsy@jhodgdon: so the fields are exposed as individual fields supported in views (original and translation language). If we are to make individual formatters for them too, then how do you tie them so they are only ever to be used with the original and translation language, so you cannot render your original language with the translation language formatter, etc? Field formatters are tied to field types not views exposed data blobs, no?
Comment #150
jhodgdonYes, technically formatters are independent. However, they are specified in the entity views data, not by users, so this isn't a problem.
So technically, you could put anything in for the plugin ID for a field handler in your entity views data, like assigning the custom 'node' formatter to be the formatter of the 'status' field on the User table. But you choose not to do that, because it wouldn't work if you did.
Similarly, we would choose to use the correct formatters in the entity views data for the language fields.
Comment #151
Gábor HojtsySo one of the points of this issue was to use the field formatter so that we don't need to redo the same code in views. If we are to have separate classes for the original language and translation language, and then need to reimplement the formatting code there, that sounds like a lot of duplication no?
Comment #152
jhodgdonThat is a point... As I see it, I think the reasons we need this issue are:
a) There are currently several custom language filters for specific entities, which have duplicated code, mostly so they can link to the entity and also hopefully display the language name in the native language (which is most likely not working).
b) Any entity without a custom language filter is using the base Views handler for non-entity-aware language fields. This is not correct, because:
- entity access is not checked
- it will just be displaying the language code from the database that is in the views query result, rather than the language of the translation that was selected for display for the entity via the Language settings of the view.
So the two proposals for solving this are:
1. Use the generic entity Field handler for all entity-related Language fields. But this will only work for the "language of this translation" field I think, and only if we also make the "language of this translation" field be a real, translatable field, which it isn't currently, and also if we make the "original language" be a real, non-translatable field, which it also isn't currently.
2. Make two specific Views handlers to handle the "language of this translation" and "original language" pseudo-non-fields. We might even be able to combine them into one handler with a setting to say whether to output the translation or original language, since the only thing that differs between the two cases is how to read this value from the $entity object.
I am not sure which of these is easier, but my guess is that (2) is easier, and also I think that (2)'s fix is more localized to solving the problem of dealing with language fields of entities in Views, while (1) is a more widespread fix. I think that both of them would solve the problem of this issue; we may have other reasons for wanting to do (1) but getting it actually done is, I'm pretty sure, going to be more work and have a lot of other implications on Drupal Core.
Comment #153
plachIf we want to keep the current approach and avoid special-casing the (translation) language field, I don't see other solutions than making the
langcode
field translatable. I wouldn't mind if a different solution were adopted here, but I think this issue uncovered a flaw in our current architecture so I will work on #2431329: Make (content) translation language available as a field definition anyway.Comment #154
Gábor HojtsyAll right, let's see if #2431329: Make (content) translation language available as a field definition is not possible "in time" for this one and then return to resolve it differently.
Comment #155
jibran#2431329: Make (content) translation language available as a field definition is in.
Comment #156
plachNo need to start from scratch I'd say ;)
Comment #157
Gábor HojtsyFirst a reroll.
Comment #158
Gábor HojtsyWhile rerolling the patch I found #2450195: Original language of entities not accessible in views anymore.
Comment #159
Gábor HojtsyWe can roll back the Field.php changes now.
Comment #160
Gábor HojtsyUpdated title and issue summary to set scope of this to translation language only given that we need to resolve default language in #2450195: Original language of entities not accessible in views anymore.
Comment #161
Wim LeersNote that this was added because the docs for
::getName()
say this:So if that's wrong, the interface should be updated as well.
Comment #162
Gábor HojtsyFixed the interface. It really depends on the construction of the object. Most languages are config entities which may or may not be loaded in some translation.
Comment #163
Wim LeersThis is incapable of returning associated cache contexts & tags… which AFAICT means
LanguageFormatter
cannot actually extendStringFormatter
and implement::viewValue()
.Alternatively, make this return a render array with
#markup
containing the string, then#cache[context]
can be set as necessary.StringFormatter
must then merge the cacheability metadata.That will then allow you to resolve the language cache context @todo.
Setting the necessary cache contexts for the link (in case the link setting is enabled) will be taken care of in #2449459: [PP-1] Make URLs cache context aware — no work needs to be done for that here.
Comment #164
Gábor HojtsyDiscussed this with @WimLeers and I in fact started to refactor viewValue() for render arrays. But turns out we don't need to do that. The language value is either displayed in the original stored configuration value or the native name of that language. It does not depend on a language context, regardless of displaying the languages on a Chinese page, the formatter would either display the language in its configured original name or in its native name. Not in Chinese.
When the field formatters change the caches are already invalidated, so since this only varies by the data and the formatter setting, we are fine as-is. Removing the @todo.
Patch should be complete.
Comment #165
Wim LeersI wonder if we want to keep a comment like this, that uses the German/Deutsch example, explaining that you'd always see EITHER of those two strings depending on the field settings, and not a translation of it (such as "Allemand" in French or "Duits" in Dutch).
Should or shouldn't we inject these?
Wow, nice catch!
I think this points at missing test coverage?
s/translated/in native form/
Missing leading backslash.
Comment #166
Gábor Hojtsy1. Added.
2. Injected in the formatter. I have no idea how would a views handler allow for injecting services (minus adding a method for it specifically), so did not do it there.
3. Well it failed when we did not make this change, so it does have test coverage :) We did not use the language field access checking for view operations before, only for edit. We don't have a view specific limitation, it should always be visible.
4. Good point, fixed.
5. Fixed.
Comment #167
Wim LeersOnly nits and nitty nits. Patch looks good, but I can't really sign off on the Views stuff, so keeping at NR.
LanguageFormatter :)
Nit: we usually sort these alphabetically.
Nit: Asserts.
Comment #168
Gábor HojtsyAll those changes make sense.
Comment #169
dawehnerDo we still need the toggle?
Comment #170
Gábor Hojtsy@dawehner: that is to test the views built-in handler (that does not use a field formatter, because it is not for entity based views) so we have a solution for a language field on a non-entity like locale or URL alias data. You requested that above with a reference to #1853534: Reintroduce Views integration for locale.module.
Comment #171
dawehnerAh, this makes sense.
This looks perfect for me.
Comment #172
alexpottNice patch. This issue addresses a major bug and is allowed per https://www.drupal.org/core/beta-changes. Committed 8d87a92 and pushed to 8.0.x. Thanks!
Comment #174
Gábor HojtsyThanks all!