Language negotiation overhaul

If I may, I'd like to try to re-phrase your issue:

It seems like you would like separation between:

· the language of the drupal interface
· the language of the displayed content

So that the profile's language-choice sets the interface language, and
the language switcher sets the language of the displayed content.

As opposed to the current behaviour, which is that the interface language
changes to match the language of the content being viewed.

EDIT: This module might help you with your problem:
http://drupal.org/project/preserve_language

If I'm correct in re-stating your issue, I think you'd need to change this to a
feature request for D7 rather than a D6 bug-report. Maybe that's why Damien
closed the other thread?

@alliax, -Anti-: I (tried to) closed the discussion in both #222401: Code block never executes in language_initialize() and #282189: Code never executes in language_initialize(), creates ambiguity regarding design choice of language logic because user htalvitie there keeps refering to a "programming error" that does not exist.

I have no hard feeling whatsoever about the design choice behind that implementation. The behavior changed between Drupal 5 and Drupal 6 and it should probably be better documented.

But, quoting Gabor from #222401 (comment #29):

Unfortunately you have not been around when this language system was designed, so you have not been able to provide your feedback. If we fall back on the user preference when "no language negotiation" is selected that does not only breaks the definition of "no negotiation", but also gets to different language content for the same URLs depending on user preferences, right?

The idea behind only switching languages based on path prefixes or domain names is to have different language content under different identifiable URLs.

The behavior you see and hate is by design. Any change will have to be via a feature request, and will have to be against Drupal 7. Reassigning appropriately.

Kick.

If we make the interface language independent from the content language, users may override the site's default interface language without messing with the content language. The content language will be defined by the settings at /admin/settings/language/configure and the interface language with a new checkbox at the same page allowing administrators to (dis)allow users to override the site's default interface language.

I wrote a patch for Drupal 6.6 which seems to solve the original description of this.

http://drupal.org/node/335699

i don't think this is a problem that core really needs to address... i think it can be done in contrib very easily... active translation module does basically this.

Personally I think core _does_ need to address this feature, since core used to do this and it Locale already does similar things.

My core patch removes the use of URL prefixes to select a language, in stead looks at the user account setting (or browser language in case of anonymous users).
Active Translation can't do that, but it's a perfect companion (and I've been recommending it to people).

I suggest keeping the URL prefix fallback intact, since it is a very handy feature. Instead, I'd like to see a new option "Allow users to override the interface language", just like there used to be in D5.

#325074: Hard error with locale was a duplicate.

Why can not fix this problem in 6.x? :(

#282178: Interface language independent from content language was a duplicate.

A duplicate of which issue?

I do not understand, too...
The same bug in version 6.x #325074: Hard error with locale

can someone give me a detailed guide how to completely replicate the percieved problem, and how the auto-negotioation works.

also, in my opinion, all non-content (aka interface) texts should follow the setting in the user settings, while all content texts should follow the user settings, unless accessed trough an auto-negotiated url.

for anonymous users, all content including non-content (aka interface) texts should always follow the default site language, unless is is accessed trough an auto-negotiated url, in which case it should follow that setting for all parts

my opinion behaviour:

*) Using site default as fallback.

afaik these setting are feasible with the new caching system. which only does page-caching for anonymous users.

Explanation

I don't distinguish between anonymous and authenticated users, so a contrib may provide a method for anonymous users to save the language of their choice in a cookie, for instance. A user's preference should not influence the content language, because of SEO, unless there is no other way to select another content language (e.g. no path prefix or domain name). Otherwise users would never be able to select another content language. The (e-mail and) interface language is may be defined by the user. If a user wants to see the content in another language, he should use another URL if available. If this option isn't available, the content language is based on the user's preference.

Proposal for changing language settings

None

The used language is solely based on a user's preference.

Path prefix

The requested path is checked for an existing language code prefix. Throw a 404 if none is found.

Domain name

The used language is solely based on the domain name Throw a 404 if no match can be found.

Using an extra checkbox administrators may decide if Drupal should use the site's default language if Path prefix and Domain name are unable to find a match.
Note: these options are about the content language only.

Alliax, you really need to stop whining. If you can't come up with a decent proposal that doesn't include rants on people or Drupal, I suggest you simply don't come back here. Instead of telling people what you want over and over again, you might want to take a look at the proposals Alexanderpas and I posted in #13 and #14, and give us some proper feedback.

If you continue whining I can guarantuee you less people will cooperate here and you won't see this feature in core for at least the next year. So if you want this proposal to be accepted, quit messing around.

alliax, i'm with Xano whining is totally unproductive in the end. you might get a short term result but in the end you'll just be putting yourself on everyone's ignore list.

Xano, if you look closely, you'll see that we're proposing almost the same, just in your case: User's preference - Site default == anonymous user.

just a small note: throwing a 404 is not always nice, therefor, i suggested fallback to site default if content does not exists in user language and not accessed by specific url.

Actually, I took your scheme and started to rebuild it. Two things: User's preference may be for an anonymous user, but also for an authenticated user. Lots of sites will want to provide anonymous users with a way to set a preferred language. This doesn't have to be in core, but we shouldn't make it impossible either. 404's are not always nice indeed, that's why I propose to add a checkbox to allow Drupal to fall back to the site's default language. It is much more flexible than the current settings (since all options will work either with and without the fallback, depending on the administrator's needs) while it shouldn't make the administration panel more complicated. Basically my scheme shows how both content and the interface can be multilingual based on flexible preferences.

/me agrees! great work!

now, the only thing we need is a patch (or a better proposal!)

I don't mind writing a patch for this, but that won't happen until at least next weekend (which means Wednesday at the earliest :-P )

I have the same problem (in 6.x)
http://www.drupaler.ru I can not upgrade to version 6 because of this error. (I can not find a reasonable explanation for this mistake...)

Default locale on this site En. And all users (Russian, Ukrainians, Germans, Belarusians, etc.) should see an interface only in En?

we'll try to add this first in 7.x, and later might backport to 6.x

keeping this on bug report, as this is changed behavior from 5.x to 6.x (albeit by design) and users are actively having problems!

also, a small note: the behaviour we're suggesting is similar to wikipedia!

THIS IS NOT A BUG! IT IS A FEATURE THAT HAS PARTLY BEEN REMOVED FROM DRUPAL 6. WE'RE TRYING TO GET IT BACK IN ALONG WITH SOME OTHER IMPROVEMENTS.
This has been said before (and) in other issues as well. Bugs are when the code does something it should not be doing according to its makers, which is not the case here. Thank you for your time :-) This seriously is no bug, so technically we cannot say it is. I understand people are having trouble with the behaviour in Drupal 6 (I have too), but it's just like that. Language selection in D6 works exactly like the (rather fuzzy) descriptions in the forms tell you. If you have any requests that have not yet been posted, please do let us know!

It will not be backported, since only the upcoming release will get new features. When D5 was under development, D4.6 and D4.7 were only being maintained in that only bugfixes were commited. The same happens with D5 and D6 now D7 is under development.

Changing the title once again. We're now going to change the entire language selection system, which has got little to do with only the interface language or the content language.

Something just occured to me: how are we going to save and load users' preferred language? Are we going to use a separate table or do we store it in the data column of {user}. If we're going for the separate table, do we load the data using hook_user() (for flexibility) or just once per page load for the active user only (for performance)?

xano, {user}.language... and that ends up in the global $user object.

Ah. Hadn't taken the time to look carefully as I was just about to go to bed :-P Anyway, saving the preferred language won't be a problem then.

I just started working on a patch. Here's a little to-do list, primarily for myself:

• Update language selection form at account edit page.
• Update language configuration form at /admin/settings/language/configure.
• Change language negotiation to a general fallback option.

The attached patch updates existing language negotiation constants (bootstrap.inc), the language negotiation configuration page (locale.inc) and the help for this page (locale.module). I have changed the form at that page to a system settings form. This saves a bit of (unnecessary) code and makes the form more consistent with the rest of Drupal.

NOTE: This patch is only a partial fix for this issue. This is what I have been able to do so far. As soon as I have more time I'll continue working on it.

#335699: Language negotiation "language fallback" only has been marked as duplicate

oh and by the way, since we now have a partial patch...

Sounds good...
Thank you for work
Can I see, these fixes in 6 version to?

Shall we change "presentation language" to "content language"? It makes more sense to me, especially now the interface language is configurable as well.

@pvassili: Nope. New features are only added to the upcoming Drupal release, in this case version 7.

Oops!

it doesn't really matter... letś just go for content language, as it removes jargon.

And here's version 2. I have rewritten the language negotiation functions. The global $language variable is now an array, where the keys 'content' and 'interface' both contain the language object of the language used for respectively the content and the interface (What a surprise :-P ). What have I done so far?

• The language selection form at account edit page has been updated. The descriptions have been changed and a 'default' option has been added.
• The language configuration form at /admin/settings/language/configure has also been updated.
• Changed language negotiation to a general fallback option.
• The $language global now is an array with the keys 'content' and 'interface', each one containing a language object.
• I completely rewrote the negotiation functions.
• I also updated common.inc and path.inc, so they wouldn't give me fatal errors.

This patch didn't cause any fatal errors on a Clean Drupal installation using the minimal installation profile. It will, however, cause several Drupal errors messages to pop up. If you guys agree with my solution for saving two languages in $language, I will start to update the rest of core.

Can I try apply this patch for version 6?

Looks good.
Although I don't quite understand why anyone would want to set language for interface and content seperately. Or is the idea to replace modules such as Active Translation?
Seems to me we don't want to give users too many selection boxes on their profile either.

@pavsili: Sure, but don't expect it to work. Like I said before, this will not be made available for Drupal 6.
@guyver800: A lot of people want to be able to differentiate between content and interface language. This patch is not meant to replace Active Translation, although there might be some overlap. We're also not adding nieuw fields to user profiles, just a single option to the already available list of radio buttons, so users can choose to view the site in the default language.

I am still in doubt if we actually need to allow users to choose to view the site in the default language, since this option will only be set if users manually update their language settings. When they do, it's just as easy to choose an 'absolute' language, like English or French. If we remove this option from the user profiles, the scheme will still work the same, because not all users have a preferred language set (anonymous users in 99% of all cases).

ow! Here is the music.
Got to read this later - tracking

Patch for d6 is very needed! I think better to leave http://drupal.org/node/325074 open and produce patch here.
Suppose this patch can never be official but very essential for multilingual sites on d6.

If people want a patch for D6 I suggest they wait until I have finished this one and then backport it to D6. Don't expect that you're allowed to create a feature request at d.o though.

Anyway, could you please stop polluting this issue with requests for Drupal 6?

2 Xano
You said that #325074: Hard error with locale duplicate.
You said

"Like I said before, this will not be made available for Drupal 6"

Here are discussing about version 7.
You moved here my problem and not solve it. Where is the logic?

we will first try make this availble for 7.x

as long as this feature is not implemented for 7.x, don't even try thinking about 6.x

remember: if we don't do this path, and go for 6.x first, this feature might not get into 7.x, and the problem stays active, while when we first implement it in 7.x, it might get into 7.x and D7 will be the killer release we're all wanting it to be, including this feature.

sorry to be this harsh, but we're having the problems too, that's why we're trying with all power to implement it in 7.x first, then we can think about some solution for 6.x

I WILL KILL A KITTEN FOR EACH COMMENT REQUESTING THIS FOR 6.x BEFORE IT IS COMPLETELY COMMITTED TO 7.x
So please, think about the kittens!

Sorry, imho I think: you need to correct first the old mistakes(6.x) (many people have a big problems...), and then make a new DEV versions (7.x)...

Thank you for your understanding and work

@Xano: I've seen the requests for interface/content language seperation, but I was just curious about a practical use case for it. I guess interface language consistency when browsing different language content on a site is a good reason.
About my settings comment, I got confused when I saw the $language array in the patch, sorry about that. I get it now, and the table in #14 makes perfect sense :)

As for your comment about the 'site default' user setting, it indeed doesn't seem necessary to have it selectable. If a user hasn't selected a language, behaviour should be the same as with an anonymous user.

The next guy posting here asking for a 6.x bugfix first will be rewarded by a block from drupal.org. Thanks for the hard work of those who keep the proper developer process and the understanding and patience of others.

$language as array... I can't think of anther way... besides including a language object in $content and $user (which i think is horrid!)

about the site default setting... mabye we should just set the user language to the site default language when registering... and i mean setting it to "en" when registering on an site with "en" as default language. (unless ofcourse the user set's something else.)

@alexanderpas, GuyveR800: Currently (and in D6 as well IIRC) a user's preferred language is set to the current page language during registration, i.e. if a user registers at example.com/en, his preferred language will be English, while this will be Dutch if the user registers himself at example.com/nl. Because of this I don't see any real need for an extra 'default language' option, because the user might be better off selecting his language of choice directly.

@GuyveR800: Example of the use of this patch: You're English and are visiting a German site. Your German is good enough for a basic understanding of the comments people post, but an English interface would make you feel a little more comfortable. Or take Wikipedia. You're Dutch and you'd like the interface to be Dutch as well, but you would also like to be able to browse different translations of articles without having to look at translated interfaces as well.

//Edit: when I have completed this patch I'll need somebody else to create new tests and update existing ones, since my knowledge of writing those tests roughly equals zero.

These two issues should be reviewed in relation to language negotiation. They are both bugs that result from the fact that, when we have user preferred languages or browser preferences, currently we can have more than one version of a given page/path. The first of the two issues will require significant changes to language negotiation and/or Drupal bootstrapping to solve. Probably, we should solve these bugs before making major new changes to language negotiation.

#339958: Cached pages returned in wrong language when browser language used
#342349: User cannot switch to default language with language negotiation fallback option

Good point! IMO, the second issue can be fixed without breaking this one. The first issue, however, needs some more thought. I replied to it proposing a 'solution' that prepares Drupal for the patch in this issue.

A while ago I saw an issue proposing to ship drupal with a prefix for english set, but I can't find it right now. That, and adding a warning or other form of awareness if someone removes a prefix should fix the second issue that seems to be popping up in a lot of duplicate issues. (Which is a good sign that it the behaviour wrong, or at least non-intuitive.)

If you want to discuss the prefix problem, please take a look at #338055: Ensure non-default languages always have a path prefix.

I'll see if I can complete the patch this weekend.

With two possible languages being used on webpages there is also the possibility of two text directions being used on the same page. If this happens a lot of dir="ltr" and dir="rtl" attributes are needed. Dozens of such attributes one a page (because content and interface are mixed) make the markup rather messy. Is there a way to keep this simple?

I think we need three basic levels of text-direction: Global level, Block level and Item level, and hide them when not applicable. (btw: What about the lang attribute.)

I have no idea what you're talking about. What do you mean with "levels of text direction" and how and when do you want to hide them?

well... if both languages (interface and content) have the same direction, we don't need to display all those dir="ltr" and dir="rtl"

what i mean with levels, can be seen in the themes.
global = page.tpl.php
block = block.tpl.php etc.
item = the actual content items themself (like a menu item, or a single node)

about the exact method of hiding... don't ask me...

That's not really a problem. The problem is that both core and contrib may need to set the language direction multiple times for the same web page. The current ways of providing a page with the right information about text direction - both the attributes in the html tag and RTL CSS files - cannot be used like they are now with two languages at the same page.

In short: Core and contrib need to provide text directions. What's the easiest (and best) way for maintainters to do this?

I think they only need to set in the area they're responsible for, based on the selected language... and weither it's part of the interface or content

the actual storage of the direction is ofcourse in the language object ;)

however, we might also want to solve that in a seperate issue, as even wikipedia seems to haven't solved that problem, and it's not a showstopper...

also, we don't have the problem when both have the same direction.

This is Drupal and if we do something, we try to do it the right way :-)

Let me explain it again: If we start using two languages on a single page, we need to define text directions more often than just in the HTML tag. Defining the direction for every piece of content and interface may get messy, no matter how many times any random module has to define it, the total picture will get messy. The ultimate question is: how do we make sure it's as little messy as possible?

I agree with alexanderpas, text direction is a separate issue. Let's keep focus on landing the actual language negotation changes.

This is not a bug! (dixit Xano). Hum.
At work I've just been doing the testing for a new application which I specified so I'm sensitive to the question. I regret not having been around when you were discussing how to do this language negotiation and I appreciate all the hard work that goes into thinking about it (expecially since I have thought for some time that Drupal would really be much much better for real language capability that allows us Europeans to handle multi-language sites more easily.
However I also have some sympathy with alliax (I didn't think he was whining that much actually), since while the change to the language setting may have been by design, it was also a change in functionality which people like alliax (and me) have been counting on without thinking ever since D4.

Anyway, since I can't contribute code, perhaps I could contribute two use cases, one critical IMHO and the other nice-to-have.

Critical use case
==========
Site admin does not understand one or more of the languages that the administered site displays (for example, I manage a site in French, no problem I can speak French. But supposing I were asked to manage a site in German, which I don't speak?). It should therefore be possible for the site admin(s) to work in another language than the one calculated by browser or whatever, or in the Drupal default.

Nice-to-have use case
=============
I'm a native English speaker living in France and I would like to bypass the language chosen for me automatically to use my native language.

We discussed that with Xano on the IRC today.

It turns out there seems to be a misunderstanding of how interface and content language are managed in Drupal 6. In a nutshell, the interface language is determined by the language negotiation setting, ie. from the prefix in the URL or from the domain name. The content language, on the other hand, is determined by the nid of the node being viewed.

So at this time, you *can* view a French node in English, or an English node in French. The confusion comes from the fact that the Content translation module automatically match interface language and content language in the URL it generates.

To clarify: let's say node/1 is an English node and node/2 is its French translation. If you are viewing the English node at en/node/1 clicking on the "French" link will jump you to fr/node/2. We could add a checkbox to make this behavior configurable. This way you could choose to never change the interface language when you jump to node written in a different language (ie. we will jump the user to en/node/2 in that case).

The "let user select its preferred language" is a different issue. Solving it could be as simple as redirecting the user to the correctly prefixed url. This way, if I selected French as my preferred language and visit en/node/1, I will be automatically redirected to fr/node/1.

(2 Damien Tournoud ) I'd like to see translated only interface for registered users.
On the site http://www.drupaler.ru people work in 11 languages
All search engines will see 11 copies of each page (in each language)!
en/node/1 = ru/node/1 = uk/node/1 .... instead of node / 1
Search engines will consider it spam.

I don't know if we should use path prefixes for the interface language, both because it's not SE friendly and because site administrators may not like to add another piece of text to the URL for this.

1. URLs should only be used for content, not for the interface, since content is what webpages are all about. Publishing the same content at different URLs is not appreciated by search engines. I therefore suggest removing all negotiation options. Different paths for different translations may be set using Pathauto.
2. We may also add a function to load a node by translation set ID. This function will load the translation matching the current content language if it exists. Otherwise it loads the translation matching the site's default language and if this fails too, it falls back to the node's original language. Although this is something that belongs in another issue, it might be something to take into account if we're doing a rewrite of the language negotiation.
3. Since nodes are not the only content a page may offer (I consider a view content too) I suggest we split up $ language, so it contains both the interface and the content language, allowing modules to translate their strings to either of those two languages. The hard part will be to determine what exactly is content and what belongs to the interface, when converting core if we indeed split up $language.
4. If a user's preferred language is blank/default, the interface should match the content langage.

I just came across this and I am confused and worried. I have a packet of sites which I manage and whose language I do not understand at all (eg Chinese even!).
Some of the sites use the subdomain to show which language is which (eg the French site is fr.mysite.com) but others have no indication of the language at all. So I have done some experiments and set the language negotiation to "None" because I didn't have any before, and the site only has one language (the site language), and English (for the administrator). How do I deal with this? I don't want suddenly to have "/fr/mycontenturl" instead of "mycontenturl" because otherwise I will muck up all my search engine indexing.
It looks to me as if the way things are now it is simply impossible for me to upgrade to D6

@joel_guesclin: set the primary language prefix to '', set the english language prefix to en and configure language negotiation to "Path prefix only.". Finally, access the website with a /en prefix when you administer it. Easy enough?

Could everybody stay on-topic?

@Damnein Tournoud: What do you think about #70?

I'm with Xano.

@Damien, that's a choppy workaround.

also, the behavior we're requesting is not weird, let me give you a small simulation:
1. log-in at the arabic wikipedia (http://ar.wikiopedia.org/)
2. set your language to english. (good luck)
3. click the (Manage your global account) link.
4. Now set your language back to arabic.
5. click the (أدر حسابك العام) link (it's the same link.)
6. try making sense of that interface page ;)

as you can see, changing the language to english made your actions a lot easier.

but we need to do this in small steps, if we want this patch to land.

can someone write a tasklist?

Well, because I thought language negotiation was for the content and not for the interface we're pretty much back to square one now: we need to decide what exactly do we want? After we can decide how to modify Drupal accordingly.

I don't know if we should use path prefixes for the interface language, both because it's not SE friendly and because site administrators may not like to add another piece of text to the URL for this.

1. URLs should only be used for content, not for the interface, since content is what webpages are all about. Publishing the same content at different URLs is not appreciated by search engines. I therefore suggest removing all negotiation options. Different paths for different translations may be set using Pathauto.

I'd like to see some citations supporting the claim that it's not SE friendly. I think leaving it as an option is best. On sites I've built I want anonymous visitors to be able to create a link to an article an send it to a friend who will see exactly the same content.

2. We may also add a function to load a node by translation set ID. This function will load the translation matching the current content language if it exists. Otherwise it loads the translation matching the site's default language and if this fails too, it falls back to the node's original language. Although this is something that belongs in another issue, it might be something to take into account if we're doing a rewrite of the language negotiation.

I think this is the best suggestion offered, content needs to be grouped and loaded by tnid so that if the localized version isn't available the original is shown.

3. Since nodes are not the only content a page may offer (I consider a view content too) I suggest we split up $ language, so it contains both the interface and the content language, allowing modules to translate their strings to either of those two languages. The hard part will be to determine what exactly is content and what belongs to the interface, when converting core if we indeed split up $language.

I'm not seeing the win here.

4. If a user's preferred language is blank/default, the interface should match the content langage.

Again I disagree, I want users to be able to send a link that's in a specified language. If I want someone in another country to view a page in their language without having to login or change languages how would I do that with your proposal?

I think your proposal basically removes flexibility that some sites want without providing an alternative. You can already select the language and content separately. As Damien Tournoud has pointed out several times in this issue the nid loads a node and the language setting specifies the interface. I want to be logged in in English and load a Thai node to check that the translator didn't screw up the formatting or compare a Chinese Traditional and Simplified version...

The most realistic suggestion that's been presented in this thread is to not automatically rewrite the URLs to switch languages based on the interface... that bit should be optional.

Guys, you must understand that the design decisions that lead to Drupal 6 language subsystem has not been taken lightly. In particular, the idea that we must have the language in the URL is a very strong one, and will probably stay.

Please try to understand the current system in-depth, and if you fell that we need to change it, please read all the documentation that was produced by the i18n team during the D6 development cycle. Most of that documentation is in the i18n group. Please especially have a look at:

• Gabor's Roadmap for i18n in Drupal 6 core,
• the discussion on why Language have to be in the URL,
• and the description of the Solutions drafted after Budapest Internationalization Meeting

I'd like to see some citations supporting the claim that it's not SE friendly. I think leaving it as an option is best. On sites I've built I want anonymous visitors to be able to create a link to an article an send it to a friend who will see exactly the same content.

Two words: duplicate content. Search engines don't like sites having multiple pages with the same content.

Again I disagree, I want users to be able to send a link that's in a specified language. If I want someone in another country to view a page in their language without having to login or change languages how would I do that with your proposal?
A user's interface language is based on several parameters in this order: Preference > Browser language > Site default. A user's preferred language can be set in his user profile or using sessions/cookies if he is not logged in. I would not base the interface language on the URL, because IMO the URL should be related to the page's main content, not to the interface.

I'm not seeing the win here.

What's your definition of content? Just nodes? What about views, blocks, taxonomy, etc? IMO these are content too. Menu items are interface items, for instance.

I'd like to see some citations supporting the claim that it's not SE friendly.

http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=66359

@Damien
1. AFAIS the roadmap only speaks about content, not interface and administration.
2. Language have to be in the URL - while i agree with this, I would even go that far to say, that the current implementation is broken (at least from a usability POV) for logged in users, as they can't decide the language on language neutral pages.
3. How does it work? 2.4 even explicitly states:

If a path without a language prefix is found (eg http://example.com/) a negotiation algorithm runs. It tries to check for the user language (set on the user profile page), then falls back to the browser preferences, then falls back to the default language if others fail.

(emphasis mine)

i'm inclined to put as a critical bug, if it indeed doesn't match specification.

alexanderpas, looking at your link it doesn't seem like it's really that big of an issue:

Google tries hard to index and show pages with distinct information. This filtering means, for instance, that if your site has a "regular" and "printer" version of each article, and neither of these is blocked in robots.txt or with a noindex meta tag, we'll choose one of them to list. In the rare cases in which Google perceives that duplicate content may be shown with intent to manipulate our rankings and deceive our users, we'll also make appropriate adjustments in the indexing and ranking of the sites involved. As a result, the ranking of the site may suffer, or the site might be removed entirely from the Google index, in which case it will no longer appear in search results.

I've got sites with multiple language and haven't seen them dropping out of the indexes. Mostly because the links are re-written to be consistent and keep you in the same language. It's not going to link from /en/node/1 to /ja/node/1 it'll go to /ja/node/23.

Duplicate content on a site is not grounds for action on that site unless it appears that the intent of the duplicate content is to be deceptive and manipulate search engine results. If your site suffers from duplicate content issues, and you don't follow the advice listed above, we do a good job of choosing a version of the content to show in our search results.

Since Drupal put the language into the headers I think Google would be smart enough to choose the correct version based on the language.

I think the burden is still on you to prove that the current system hurts search rankings.

I don't know if we should use path prefixes for the interface language, both because it's not SE friendly and because site administrators may not like to add another piece of text to the URL for this.

1. URLs should only be used for content, not for the interface, since content is what webpages are all about. Publishing the same content at different URLs is not appreciated by search engines. I therefore suggest removing all negotiation options. Different paths for different translations may be set using Pathauto.

I'd like to see some citations supporting the claim that it's not SE friendly. I think leaving it as an option is best. On sites I've built I want anonymous visitors to be able to create a link to an article an send it to a friend who will see exactly the same content.

if we would be using the url for interface language, it would be very harmfull, as we have pages with the same content and only translated interface.

but, that's not the root of this issue.

Guys, let's separating the content and interface...

Roadmap suggestion:

- let's first make it match this specification:

If a path without a language prefix is found (eg http://example.com/) a negotiation algorithm runs. It tries to check for the user language (set on the user profile page), then falls back to the browser preferences, then falls back to the default language if others fail.

this also includes things like http://example.com/admin

this'll leave a lingering bug, that interface will be in a user-strange language when accessed with a language specific url, which we'll solve in the next steps:

- Split off $language, in $user-language and $content-language (leaving $language, with current behaviour, temporarily existing for the time being to ease this updating.)
$user-language will now be similar to D5 language behaviour.

- implement negotiation for $user-language when accessed with a non-language specific url, AND no, explicit language preference has been set.

now we have the behavioural code complete, we can now move towards the first implementation.

- fix everything to use $user-language and $content-language, (t() will use $user-language probaly).

this'll open up the codebase for true node translations, so jp/node/1 is the japanese translation of the english en/node/1

- eradicate $language.
we don't want stale data....

we already noticed there will be language direction problems, we won't fix them until we have some base on which we can fix them (also easying up the previous items)

- fix language direction problems.

Splitting $language has no use if we don't decide that content is more than just nodes.

wow... crosspost lol ;)

- let's first make it match this specification:

If a path without a language prefix is found (eg http://example.com/) a negotiation algorithm runs. It tries to check for the user language (set on the user profile page), then falls back to the browser preferences, then falls back to the default language if others fail.

this also includes things like http://example.com/admin

First, the document you read in the group is not a specification, only a description of the implementation at one point. The specification is the strings displayed in the user interface, and you will see that the negotiation behavior is perfectly conform to the specification :)

I already explained how easily you can have the administration in the language you want. For the question of the user preference, please see my last pont in #68. We don't want any URL to return different content when accessed by different (anonymous) users. That will break a lot of things, including caching.

this'll leave a lingering bug, that interface will be in a user-strange language when accessed with a language specific url, which we'll solve in the next steps:

That's not a bug. That's a feature. If the user access a language-specific URL, it should show the page in that language, except if we build a redirect mechanism as I explained in #68.

we already noticed there will be language direction problems, we won't fix them until we have some base on which we can fix them (also easying up the previous items)

- fix language direction problems.

That's actually the only thing to fix.

@Damien,

I already explained how easily you can have the administration in the language you want. For the question of the user preference,

as i already said, that's a tacky fix, if a site administrator needs to do that, the core of D7 is usability.

if an admin wants to admin his site, which has completely french content, in english, it should be as simple as setting the user language.

please see my last pont in #68. We don't want any URL to return different content when accessed by different (anonymous) users. That will break a lot of things, including caching.

well... I don't think that's fully true... there will probaly be changes, however, since we have page cache and block cache, they won't be that big.
- default anonymous users (no language selected trough contrib) will get the default page-cache, as interface and content are the same.
- logged-in users, will get block cache, which'll get similar treatment as now (with t() handling).

The "let user select its preferred language" is a different issue. Solving it could be as simple as redirecting the user to the correctly prefixed url. This way, if I selected French as my preferred language and visit en/node/1, I will be automatically redirected to fr/node/1.

we should only use en/node/1 for actual enlish content, and fr/node/1 for french content, no matter what the interface language is.

Remember this is a feature request, and what i called a bug in #83, is a bug in the implementation of this feature.

@Damien: Thankyou for your reply to joel_guesclin which also addresses my problem. I'm impressed that does seem to work fine. Also, I can activate the "Language switcher" block and switch from one to the other easily. Obviously there is a lot of thought gone into this and there is more to language issues than meets the eye (I will have to experiment a bit and get a better feel for how it works).

Can I try to use the patch in message #40?
Are there any changes?

don't use patch in #40, solution in #72 might help you.

I've taken a break from this issue. (will be coming back.)

alexanderpas, thank you for your help. I tested in my case: http://drupal.org/node/222401#comment-844115

minddump:

- read this now: W3C I18N FAQ: When is it appropriate, or not, to use language negotiation?

Despite its limitations, language negotiation is a useful function and it is desirable to implement it in multilingual sites. But the shortcomings must be addressed. In short, it is important to provide means for visitors to override the automatic choice of language when it is wrong.

- user-language takes precedence over negotiation, and page (content translation) language.
- only negotiate on language independant pages, when no user language has been set.
- no negotiation on language specific pages, use page language (or user language) instead.
- url part does not influence interface.

- page = one node
- translation = another node
- all nodes and all translations are availble in all languages (login, or explicit language setting (for anonymous, contrib) might be required.)

- node/4 = french page
- node/4 = permalink = redirect to fr/node/4, no negotiation.
- nl/node/12 = dutch translation of (french) fr/node/4
- nl/node/4 = redirect to nl/node/12, no negotiation.
- es/node/4 = no spanish translation availble = redirect to fr/node/4, language to spanish.

- node/6 = language independant page
- node/6 = negotiation url.
- de/node/6 = redirect to node/6, language to german, no negotiation!

translation redirection is optional!

this should keep problems at minimum, keep the negotiation and solve most of the problems the users are experiencing.

comments are welcome.

An interesting thing about the W3C page:
http://www.w3.org/International/questions/qa-when-lang-neg returns its content in German for me, the same content as
http://www.w3.org/International/questions/qa-when-lang-neg.de.php
Also available:
http://www.w3.org/International/questions/qa-when-lang-neg.en.php
http://www.w3.org/International/questions/qa-when-lang-neg.fr.php
http://www.w3.org/International/questions/qa-when-lang-neg.pl.php

For me as the user of their site this is great but they might have problems with duplicate content issues. They might not care but I think we should care avout SEO and not return different content to anonymous users under one url.

http://www.google.de/search?hl=de&client=firefox-a&rls=org.mozilla:de:of...
vs
http://www.google.de/search?q=FAQ%3A+When+to+use+language+negotiation&ie...
see also
http://www.w3.org/International/questions/qa-when-lang-neg.php

you also might want to see the solution displayed on this (non existing) page: http://www.w3.org/International/questions/qa-when-lang-neg.nl.php

- minddump response:

As we know which node we're accessing (nid) we can also display a page like that when no translation is availble for that explicit requested language. (nl/node/7 when it is a french node, and no dutch translation is availble)
(valid, installed, language, but no content translation, optional feature, http status code 300, no duplicate content problems.)

"This page is not availble in your language, but is availble in the folowing languages"

we can solve the duplicate content issue by redirecting (302/303/307) /node/8 to /xx/node/8 (where xx is the content language), and negotiate the language used (for the interface, as content is "invariant" from this aspect), giving no duplicate content issues.

on language independant pages, we do the exact opposite ( ;) ) redirect /xx/node/14 to /node/14 and set (interface) lanuage to xx, (no negotiation) again leaving no duplicate content problems.

note that the language the user have set in their settings, is ALWAYS used before any negotiation is done, but only after redirection.
if negotiation (including page language) FAILS (e.g. googlebot, on language independant pages), the default language WILL be used.

the redirection will ensure that the final displayed url's always state the language of the content, independant of the interface language.
/de/node/10 is always german content,
/node/18 is always language independant

s.Daniel, i think the SEO argument is a red herring as i outlined in comments #76 and #80.

minddump:

- SEO is a red herring, especially if you do proper 300/302/303/307 (no actual duplicate content).

- all content is (invariant and) always accessible in all languages, the only thing actually negotiated is the interface.

order of preferability
- user language
- session language (aka. previously negotiated)
- page language
- browser language
- default language

internal actions: (insecure pseudo-code)

if($URL language != $page_language) {
  drupal_set_session('session_language', $URL_language);
  drupal_goto($page_language . '/' . $GET['q']);
}

drewish: I am not sure I understand your point. Duplicate content on websites has resulted in ranking penaltys in the past and even though search engines try to deal with it they often do misinterpret. For example these two URLs show the same content to Googles robot: http://www.w3.org/International/questions/qa-when-lang-neg.en.php vs http://www.w3.org/International/questions/qa-when-lang-neg and are both in the google index: http://www.google.de/search?hl=de&client=firefox-a&rls=org.mozilla:de:of... vs http://www.google.de/search?q=FAQ%3A+When+to+use+language+negotiation&ie...

However as far as I can tell http://www.w3.org/International/questions/qa-when-lang-neg.en.php is in the suplemental index and is competing with http://www.w3.org/International/questions/qa-when-lang-neg possibly hurting its possibilitys to rank propperly.

The current system does not hurt SE rankings normaly but it could in cases where Links are set wrong as no redirect is being done from the multible urls a node can be accessed through to the one correct aliased url.

I've read this entire thread. Obviously, a lot of thought has gone into how to get the multilangual sites both versatile for admins (for various types of multilingual sites) as well as end users.

From my experience, there are 2 kinds of multilingual sites:
A) Translation based -- ie where content is genrated and managed by admins or a very small group of people.
B) User based -- ie where multilingual content is generated by a VERY LARGE number of users at a very fast pace, to an extent that translating and syncronising multilingual content can become practically unmanageable.

For the reason of the SECOND type of multilingual websites, I have raised a feature request with i18n here http://drupal.org/node/328132 and it was marked as being duplicate of this discussion.

My feature request was that:
a) Interface Language be treated as separate from Content Language
b) allow users (anonymous and authenticated) to select their Interface Language (radio-buttons - only 1 choice allowed) SEPARATELY from their Content Language (checkboxes - mutiple selections are possible). This is best implemented on www.Reddit.com

As a result the user (authenticated AND anonymous) would be able to see Interface in her prefered language (eg English), and also see content and content listings in multiple languages of her choice simultaneously (eg English AND French AND Italian, but not Portugeese or Spanish).

Emails sent from the system and Messages should be displayed in user's chosen Interface Language.

Please note: I'm not saying that this is the ONLY way that it should be, but it should be an additional option for the admins to configure their sites, because it seems that this is an evolving web standard for sites like Reddit.com and YouTube.com

I was wondering if this thread will address this issue???

PS: as for language negotiations, I totally agree with Alexadreapas: it should be a hierarchical check that checks the user language (set on the user profile page OR in Session cookie for Anonymous users -- see BBC.co.uk or Reddit.com for example), then falls back to the browser preferences, THEN fall back to IP address, then falls back to the default language if others fail. But this should be done only upon the first visit to the site and be immediately written somewhere (maybe cookie for Anonymous) for later user, because if this kind of calculations are done with every click, it would slow does the site dramatically.

IMHO, Admins should also have an option to present the choice of Interface language and Content languages during user registration.

subscribing

subscribing - I realize I haven't been a part of this discussion but I completely agree with drupalina's request, and alexanderpas' language negotiation scheme is the one that makes the most sense to me as well.

There is a patch.

Xano, alexanderpas, others:

Are you available to participate in the internationalization in core code sprint, http://groups.drupal.org/node/18443? If so, we may be able to tackle this as one of our focus issues. Please sign up at the groups site if you can make it. Thanks!

Having not been part of the discussion before now, but having now read through the entire thread, I agree that #94, #96 (and #98) make a lot of sense to me too; I hope to see this sort of approach implemented and maybe I will be able to contribute to it as well.

Having not been part of the discussion before now, but having now read through the entire thread, I agree that #94, #96 (and #98) make a lot of sense to me too; I hope to see this sort of approach implemented and maybe I will be able to contribute to it as well.

nedjo That was able to fix?

Absolutely agree with #98 -> #96 -> #94 - we talk about this from start with @pvasili

If user choose Language in user-preferences so interface lang should not depend on content language (node lang)

I have read through all posts of this issue now. Thoughts:

The goal or actual problem is not entirely clear to me. It might make sense to split this issue into multiple, because it seems that people are talking about completely different things:

a) Support preferred user languages (probably using ranked priorities, e.g. FR -> DE -> EN).

b) Don't change interface language when displaying content in a different language. Or in other words: Allow to view content in other languages while keeping interface language, but still allow to use links to display certain content in a certain language.

c) Solve some SEO issues, which I did not understand. Those SEO issues may only exist, when a language-specific URL does not display the content in the requested language, but falls back to the original language (=> duplicate content). However, I'm not sure whether this issue is valid at all.

d) Load content by tnid and negotiate language, URL alias and stuff afterwards only.

e) Don't use separate nodes for translations of nodes.

Yes, most of these are worth to discuss. And, some of these are in line with recent discussions on translatable fields during a i18n core sprint, where we identified that D7 might be able to support translated content without separate nodes, but using field translations instead.

So, if someone of you would be so kind to split this issue up into multiple issues (some of them might exist already), that would be great.

Reading my summary again, this issue is now fully covered by

- #367595: Translatable fields (already committed)

- #282191: TF #1: Allow different interface language for the same path - needs a mission critical momentum, better review + test NOW. But please do not hi-jack that issue with "+1" follow-ups, thanks.

- #533924: Allow different language types to be enabled separately - ditto

I have a website for a bilingual country, nodes and comments of which can be mixed everywhere, but users prefer their choice of UI language. Language negotiation can be very good for other situation, but one of the kind of website I am talking about. In my case it really does not matter for users of which language is content, but UI does. And they could easily select their choice of language back in Drupal 5, now this feature very much desired and often asked one. I am sorry that this issue still have been discussed with no concrete solution.

Anyone could achieve Drupal 5 behavior for Drupal 6 preserving common URLs for all languages activated?

@yngens take a look at http://drupal.org/project/languageinterface

S. Daniel, thanks for the link, but http://drupal.org/project/languageinterface does require language negotiation, so changes URLs just as well as another modules does http://drupal.org/project/preserve_language. The users of my site copy and reference the URLs for the same pages very often, but at the same time they would prefer to use interfaces of different languages. It will be messup if they start to use URLs of the same page with different language prefixes in them.

I would really appreciate if someone could point me the way of configuration Drupal 6 the old Drupal 5 way, so that users could select their choice of language for interface on their accounts (currently it changes only for the language of e-mails) and nothing else, URLs for the nodes should remain the same for all the languages.

http://drupal.org/project/preserve_language is what you're looking for, as the language in the url is representative of the content language, and not the users preferred interface language.

also, next time, consider opening a new support request, or posting on the forums instead of adding to a already closed issue.

alexanderpas, thank you for your response, but http://drupal.org/project/preserve_language does change the URLs. I need to provide users interface language of their choice, but the URLs throughout the site should remain absolutely the same regardless of any conditions.

I know this is a fairly old thread, but it is the perfect topic and it was based on Drupal 7.

Does anyone know of a module that can do what this one can do, just in Drupal 7?: http://drupal.org/project/preserve_language

I do think that the interface should be preserved to the user's setting. It's horribly scary for a user to have their language switched on them! They don't know how to get out of it!

My bad. Looks like there is a division in the locale module. You just need to scroll down on the DETECTION AND SELECTION tab at #overlay=admin/config/regional/language/configure and you'll see both "User interface text language detection" and "Content language detection."

If you turn off URL, Session and Browser for the User interface and just leave User (for their preference) and the Default as the fall-back (you don't have a choice; default stays), then it will only use their setting for the interface. Very cool.

subscribe