One of the many problems with the outdated locale module code is that it does not do good separationn of its many tasks (language list management, language negotiation config, translation, .po file import/export). For one it mixes the language list handling with the translation handling and does not provide a useable API *at all* for language CRUD. *The* locale language API consists of one function in Drupal 6 and 7 called locale_add_language(). Yeah, locale_add_language(). And it has a monster list of arguments. While the rest of Drupal grew up and use standardized objectname_save() functions which return the updated value and work with the object structure, there is no such language API in place.

The attached patch is a humble first attempt at introducing a locale_language_save() function that takes a real language object. That is "real" in a sense like node_save() needs a real node object. Defaults are filled in, so only the language code is needed practically. It does some automation for new languages for reusing our built-in language metadata and language updates to enable them when you set to default for example. It does not go the whole way to do things like ensure there can only be one default language for example though.

One more thing this does introduce is hooks for reacting when a new language is added or an existing one is updated. This replaces the more generic and all-encompassing module_invoke_all('multilingual_settings_changed') that was there in Drupal 7. Now modules can react to the concrete action and the concrete language that was added or saved.

Note that I did *not* add an alter hook for good measure before it is saved, but it could make sense. Looking for feedback there.

I'm not sure locale_language_save() actually belongs in locale.inc. The role of locale.inc is very blury anyhow, and I hope to move out most if not all code from there in subsequent patches, so maybe this will move to either a locale.languages.inc or the locale.module itself (although language CRUD is not really needed on all page views).

Finally, I've only updated the places where the addition API was used before, and on a cursory test run, some main tests covering that ran fine on my machine. I did not update the places where languages are updated and the patch does not include a deletion API function yet. I also think that the loading/listing functions for languages should be looked at and standardized, but that might not fit into this patch after all. We'll see.

Feedback *very welcome*. I have lots of improvement plans like this so if we can progress with these, locale can be a really nice and modern module in Drupal 8 finally :)

Parent issue

#1260488: META: Introduce real language APIs

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Status: Needs review » Needs work

The last submitted patch, locale-languages-crud-first-take.patch, failed testing.

catch’s picture

Rather than alter could add presave? Only did a cursory review but looks good from that.

Gábor Hojtsy’s picture

Sure, its an API design question :) Some modules do alter, others do presave. Agreed presave might make more sense here.

catch’s picture

Should've qualified my comment a bit. I think it would be good to move towards using presave hooks for things that are being stored persistently, and alter hooks for things changed on runtime (or sometimes changed then cached) - at least while they offer much the same thing, I think about those two different hooks quite differently.

Gábor Hojtsy’s picture

Status: Needs work » Needs review
FileSize
12.81 KB

All right, here is a little update:

- some test files in locale.test and field.test still used locale_add_language(), fixed those, this should hopefully make it pass now :)

- realized using $language for these can be dangerous, we have the global $language, and should probably avoid using $language generally for other purposes... that would be a huge cleanup in locale module though... for now used $language_obj in the patch at places invoking the API, better suggestions welcome

- added a presave hook and made is_new a property of the object, so the presave hook knows if the language was new or not

- added logic to handle default language change in the API, so it updates the default language too if a new one is to be set the default

Status: Needs review » Needs work

The last submitted patch, locale-languages-crud-first-take.patch, failed testing.

Gábor Hojtsy’s picture

Status: Needs work » Needs review
FileSize
12.28 KB

Ups, one inadvertent edit caused test fails. Removing that part.

Gábor Hojtsy’s picture

BTW opened #1216094: We call too many things 'language', clean that up for the $language an an object vs. $langcode vs. $language as a string vs. $language as a non-global vs. $language_obj that looks ugly problem.

sun’s picture

Issue tags: +API change

Makes sense.

However, I'd expect the language CRUD API functions in language.inc, not locale.inc.

Gábor Hojtsy’s picture

@sun: well, logically I agree. However, language.inc is full of API functions which can be used on a site that does not have locale.module enabled, while locale.inc only has code to be used if locale module is on. (I do think it is a very bad idea to have module specific code in Drupal's main "includes" and want to solve this for Drupal 8). However, that would also mean that this will not get into language.inc after all.

We've just talked briefly in IRC about moving the language configuration parts of locale module (vs. the UI translation and gettext support) to a language.module. Then we'll own the language_* namespace both in terms of a module providing stuff for that and language.inc. This pattern is used elsewhere for API functions that should work without modules. Such as the filter_* functions that are outside filter.module. I think then it would make much more sense to have language module API functions in language module, would it?

Gábor Hojtsy’s picture

I've documented how this fits to overall Druapl 8 plans at http://groups.drupal.org/node/161589.

plach’s picture

Gábor Hojtsy’s picture

Any more comments on this? I don't know of any I have not acted on yet. So what do you think is holding this up to get in?

sun’s picture

Status: Needs review » Needs work
+++ b/includes/install.core.inc
@@ -1380,11 +1380,21 @@ function install_import_locales(&$install_state) {
+    $language_obj = (object) array(
...
+    );
+    locale_language_save($language_obj);

+++ b/includes/locale.inc
@@ -423,84 +423,82 @@ function locale_string_is_safe($string) {
+ * @param $language
+ *   Language object with properties corresponding to 'languages' table columns.
+ */
+function locale_language_save($language) {

+++ b/modules/field/tests/field.test
@@ -2642,7 +2642,12 @@ class FieldTranslationsTestCase extends FieldTestCase {
+      $language_obj = (object) array(
...
+      locale_language_save($language_obj);

+++ b/modules/locale/locale.admin.inc
@@ -350,14 +350,25 @@ function locale_languages_predefined_form_submit($form, &$form_state) {
+    $language_obj = (object) array(
+      'language' => $langcode,
...
+    locale_language_save($language_obj);

I'm not particularly happy about the $language_obj variable name. For one, that variable is most often called $language elsewhere. Second, the abbreviated _obj suffix is even worse than the abbreviated $langcode variable.

We should use $language for this object everywhere, and leave the global renaming of $language or $langcode to the dedicated issue and ongoing discussion in #1216094: We call too many things 'language', clean that up

In case there's a name clash with $language in a function body somewhere, I'd recommend to rename the existing $language to $langcode if necessary. Doing so will help to get a potential renaming in aforementioned issue done more easily.

9 days to next Drupal core point release.

Gábor Hojtsy’s picture

@sun: no, existing clashes are there between global $language and this $language. So I think we should at least migrate away from global $language as a pre-requisite to this issue. I've opened #1222194: Rename global $language to $language_interface to make that happen at least. Then we'll still need to figure out the language identifier problem in #1216094: We call too many things 'language', clean that up so we can actually make a _load() operation happen. I entirely agree with $language_obj is lame, that was totally a heads-up to work around global $language.

Gábor Hojtsy’s picture

Status: Needs work » Postponed

To make sure not to branch the conversation, marking this postponed on #1216094: We call too many things 'language', clean that up.

Gábor Hojtsy’s picture

Title: Implement a sane CRUD API for languages » Introduce locale_language_save()
Status: Postponed » Needs review
FileSize
23.26 KB

Here is an update to the patch based on discussion in #1216094: We call too many things 'language', clean that up. Until/unless #1222194: Rename global $language to $language_interface is committed, the best we can do to eliminate conflict with the global $language is to use $GLOBALS['language'] (which Drupal core already does at places), so applied that. Where $language was used to reference language code in the tests, I've updated to say $langcode. That had ripple effects for other related variable names, so applied to those too. So the attached patch applies the current short term best practice.

Additionally to that, I've made the following changes:

- removed the group definition for this single function; discussed with jhodgdon and sun in IRC that most of the locale groups don't make sense and we'll reintroduce a group as it makes sense on a larger scope when we have stuff to explain... the bigger API is still forming
- added API docs for the three new hooks, language presave, insert and update; added note to the mutlilignual settings changes hook that it is going away soon

Retitling the issue to keep the task more focused, given that it turns out that implementing a full CRUD is way bigger then what fits comfortably in an issue. I'm hopeful we can move incrementally from here and implement a sensible loader next to be able to do the update and delete parts as well. Currently both update and delete are tightly baked into the Form API. Fun, yeah.

Therefore I expect the full use of this function as well as the removal of the multilingual settings changed hook in a followup.

This should be reviewable and committable as-is now without dependeny on any other issue.

Status: Needs review » Needs work

The last submitted patch, locale_language_save_api.patch, failed testing.

Gábor Hojtsy’s picture

Status: Needs work » Needs review
FileSize
22.66 KB

The checkUrl() change was incorrect, url() takes a language object in its options key, not a langcode (like t()). Ugh. Fixed that which should fix the tests.

Gábor Hojtsy’s picture

sun’s picture

Issue tags: +API clean-up
Gábor Hojtsy’s picture

Patch applied with offsets. Rerolled. Also included a little fix for the phpdoc on the new function. I think this looks good, please review! Thanks!

Status: Needs review » Needs work

The last submitted patch, locale_language_save_api_1.patch, failed testing.

Gábor Hojtsy’s picture

Status: Needs work » Needs review
FileSize
23.46 KB

Looks like a newly added test portion that was not yet there in the previous roll caused this to fail. Converting that too to the new API. Please review so we can get this in and move on :)

attiks’s picture

I had a close look at the code, but it all looks good, only thing that looks strange is the use of $languages inside function install_select_locale_form($form, &$form_state, $locales, $profilename) {, it contains the following

  $languages = _locale_get_predefined_list();

but _locale_get_predefined_list actually returns a list of country codes/names

BTW not really sure if this has to be in this patch or in #1216094: We call too many things 'language', clean that up

Gábor Hojtsy’s picture

Yes, that. We have #1231402: Drupal does not use ISO language codes, iso.inc is misleading which has a similar problem but we could not yet figure out how to name things here separately really. I think it will be the job of followups to #1216094: We call too many things 'language', clean that up once we figure out what is a "language" :) This patch IMHO is already pretty sizable and we can do the overall cleanups once we reach conclusions on language terminology.

attiks’s picture

Status: Needs review » Reviewed & tested by the community

Completely agree with #26

plach’s picture

I ain't sure I get why the language API functions cannot live in language.inc and get the language namespace, even if we don't introduce a language API module. Is this because they rely on the language table? From a DX perspective I'd rather delegate the language schema handling to system and get language_* functions than stick with locale_language_* which are IMO pretty confusing.

After all the language concept is baked into bootstrap.inc so having system (always if we don't introduce a dedicated module) handle the language schema should not be an issue.

Gábor Hojtsy’s picture

@plach: I think @sun et al would want to cut down on system module, and move its responsibilities to separate modules instead of tacking all that stuff together, so "bloating" system with the language list sounds counter to the goals of #1224666: [meta] Unofficial Drupal 8 Framework initiative (which Dries yesterday says he considers making official). We can obviously then following that thinking create a language module (which you and I disputed elsewhere for usability reasons).

All-in-all I think its better to introduce these things now even if moving them to their own module/place we keep on our radar. I think conceptually and according to the #1224666: [meta] Unofficial Drupal 8 Framework initiative ideals, language.module would be the place eventually (where most if not all of the language_* namespace would migrate as well). My approach to this work is as always faster, smaller and reviewable chunks, so we can fire and move on quick. That of course means the chunks should make sense on their own, but also means we don't need to solve every concern at once :)

BTW this problem is even more apparent with #1260510: Introduce a language_load($langcode) and also with #1260528: Introduce API to delete languages.

plach’s picture

Ok, as long as the possibility of eventually moving everything into the language_* namespace stands, I'm fine with this. I still have to review the code, though.

Edit: Just a note: I considered delegating the language schema to system a temporary measure too, until we'd figure out how to deal with core API modules in the framework initiative. @eaton mentioned the possibility to make them hidden.

franskuipers’s picture

Language is only part of the locale. Other parts of locale are country, numbers, time & date format, calendar defaults and currency.

Maybe we should go one step further and create an object $locale where we store all locale settings. Then we get an object like described by plach in http://drupal.org/node/1216094#comment-4726208.

If we go this way, the CRUD functions should be locale_save(), locale_load(), locale_load_multiple(), locale_update(), and locale_delete().

The locale->languages object is only used for the interface languages, the content translation language should have its own settings. Likewise locale->numbers->decimal_separator should used for input validation and display of floating point fields.

Gábor Hojtsy’s picture

@franskuipers: definitely not going to happen in this issue. This is about the problem of no standard whatsoever with other APIs when it comes to adding or updating languages and no way to hook into the process. That is it. We do not manage country lists or number formats on these admin pages. Once those have UIs and concepts in core, we can think about merging or not merging them. This has been discussed elsewhere as well, if you want to read up on those discussions: http://groups.drupal.org/node/161589 as well as #1216094: We call too many things 'language', clean that up both of which should be very fun and insightful reading.

franskuipers’s picture

sorry... crosspost

BTW: it could mean we have a default (minimum) locale where the defaults are handled (country, first day of month and date format would be moved out of system.module), this would be an required core module.

locale/language can exist as a submodule which can be enabled if you want a multi-language (interface) site.
locale/numbers can be used to define the default widgets for floating numbers.

attiks’s picture

I'm in favor of letting this pass as is, so the rest of the API can be build as well in a much cleaner way then before. If necessary we can always move to a common locale structure later on.

franskuipers’s picture

Patch looks complete to me and passes all tests. Ready to commit.

svendecabooter’s picture

Issue tags: -API change, -API clean-up, -D8MI

#24: locale_language_save_api_2.patch queued for re-testing.

Status: Reviewed & tested by the community » Needs work
Issue tags: +API change, +API clean-up, +D8MI

The last submitted patch, locale_language_save_api_2.patch, failed testing.

svendecabooter’s picture

Status: Needs work » Reviewed & tested by the community
FileSize
22.77 KB

Rerolling this patch against the latest status of core.
Had to remove a few lines of code in a function that disappeared since #1219236: Locale module includes 1350+ lines of unneeded code on all page loads.

svendecabooter’s picture

Status: Reviewed & tested by the community » Needs review
svendecabooter’s picture

Let's try again for the testbot

svendecabooter’s picture

Ugh... I shouldn't submit patches this early...
Apologies for the spam.

Gábor Hojtsy’s picture

@svendecabooter: well, they definitely did not disappear, they just moved to other places. Did you keep the pieces for moved-around code?

svendecabooter’s picture

Not good... this one should be better.
Thanks for being alert ;)

svendecabooter’s picture

After an IRC discussion with Gabor we decided to merge in the patch at #1260520: Apply language_save and language_load to the locale management UI into this one as well.
This patch now applies the proper usage of the locale_language_save() API function, and goes halfway into deprecating hook_multilingual_settings_changed(). This still gets invoked via the delete function (see #1260528: Introduce API to delete languages) and should probably be cleaned up there.

Thanks for reviewing

Gábor Hojtsy’s picture

The changes look good to me, and it indeed spreads use of the save function as far as it can in the locale admin API. It also deprecates the multilingual_settings_changed which was part of the goal. Final removal can happen with the delete API linked.

Status: Needs review » Needs work

The last submitted patch, locale_language_save_api_5.patch, failed testing.

svendecabooter’s picture

Status: Needs work » Needs review
FileSize
28.46 KB

Here is an updated version of the patch, which fixes a problem with the "default" property.
Upon saving the language, the property $language->default is expected, but not set when languages are loaded in language_list().
This latest patch adds support for this.

Status: Needs review » Needs work

The last submitted patch, locale_language_save_api_6.patch, failed testing.

svendecabooter’s picture

Status: Needs work » Needs review
FileSize
28.54 KB
svendecabooter’s picture

Latest patch makes sure $language->default is always set to a value (TRUE / FALSE)

Status: Needs review » Needs work

The last submitted patch, locale_language_save_api_7.patch, failed testing.

Gábor Hojtsy’s picture

Status: Needs work » Needs review
FileSize
29.06 KB

Here is an update that simplifies some property logic a great deal and fixes the remaining test fail by dropping our magic to update the old default language from the save function. That was a side effect of the function that caused problems elsewhere, namely in the form submit function that used more up to date data and then was overwritten with the cached data from language_default(). We should avoid that and let the caller deal with updates with up to date data. Therefore the removal of that magic. This should be cleaner and actually pass. :)

podarok’s picture

subscribe
#52 works good for me

svendecabooter’s picture

#52 seems to do the trick perfectly :)
Nice one!

Gábor Hojtsy’s picture

Status: Needs review » Reviewed & tested by the community

Good to go then I guess? :) The latest patch is just a merge of the RTBC patch from above and some application of the code to other places with testbot and reviewers happy.

Gábor Hojtsy’s picture

Status: Reviewed & tested by the community » Needs work

With the language delete patch from #1260528: Introduce API to delete languages, this will not apply anymore. @svendecabooter, can you help with a reroll?

svendecabooter’s picture

Status: Needs work » Needs review
FileSize
28.92 KB

Here is a rerolled patch that now works with the latest changes.
This patch doesn't change any of the add / delete lines compared to the previous patch, except the @todo task that the previous patch added to field_multilingual_settings_changed() in field.multilingual.inc, because this function now no longer exists.

So if this passes the testbot, this can probably go back to RTBC

podarok’s picture

Status: Needs review » Reviewed & tested by the community

looks like good patch

RTBC

Gábor Hojtsy’s picture

Agreed. Cross-checked with the delete patch and looks like we are on track to finish the plan to introduce real APIs for save/update/delete for languages with this one. Should go in.

Gábor Hojtsy’s picture

Please commit.

sun’s picture

Thanks, looks good to me, too.

catch’s picture

Issue tags: -API change, -API clean-up, -D8MI

#57: locale_language_save_api_9.patch queued for re-testing.

Status: Reviewed & tested by the community » Needs work
Issue tags: +API change, +API clean-up, +D8MI

The last submitted patch, locale_language_save_api_9.patch, failed testing.

Gábor Hojtsy’s picture

Priority: Normal » Major

Rerolling. Other patches are waiting on this being committed, so marking major.

Gábor Hojtsy’s picture

Status: Needs work » Needs review
FileSize
28.85 KB

Same patch rerolled for the latest predefined language list changes.

catch’s picture

Status: Needs review » Needs work
+++ b/includes/locale.incundefined
@@ -429,85 +429,69 @@ function locale_string_is_safe($string) {
+    // We rely on the caller to update the old one as well to avoid conflicts

I didn't understand what this comment referred to. It is mainly for contrib modules to react if the default changes, but that seems more like a @todo to provide notification of changes of defaults but as long as an issue is open I'd be happy removing the comment entirely.

+++ b/includes/locale.incundefined
@@ -429,85 +429,69 @@ function locale_string_is_safe($string) {
+  // Changing the language settings impacts the interface.

We've been removing cache_clear_all() from CRUD functions to allow callers to control this themselves. Also depending on language negotiation settings it feels like the interface might not change at all (say if you have everything with language prefixes). Could this move to the submit handler for now?

+++ b/modules/locale/locale.admin.incundefined
@@ -119,60 +119,35 @@ function theme_locale_languages_overview_form($variables) {
-
-    $language->weight = $form_state['values']['weight'][$langcode];
-
-    db_update('languages')
-      ->fields(array(
-        'enabled' => $language->enabled,
-        'weight' => $language->weight,
-        'prefix' => $language->prefix,
-        'domain' => $language->domain,
-      ))
-      ->condition('language', $langcode)

Bye bye!

Can't see anything else to complain about.

0 days to next Drupal core point release.

Gábor Hojtsy’s picture

Status: Needs work » Needs review
FileSize
28.79 KB

Thanks for the review. We discussed the first comment in IRC. The basic summary is follows:

We tack on the ->default property to languages as the locally cached $language->langcode == language_default()->langcode value. This property is not saved to the DB however. We probably want to rework that in the future,and there are various issues around default language handling that would be good to clean up. Our basic problem is that Drupal needs to support language even without locale module enabled and needs to support it differently once locale is enabled. The issue for default language handling improvement is at #1272862: Clean up default language handling.

Notifications on default changes are already achieved, since when the _insert or _update hooks are called the global default is not yet updated, so you can check if the ->default property is TRUE but != language_default()->langcode, in which case you see the new default language. You can update your records accordingly. All this is only relevant for contrib modules, no core use of this. Since we have this functionality in, I've removed the comment.

We've been discussing the cache problem on IRC extensively and could not figure out a sane way to abstract that out, so on request, opened an issue at #1293252: locale_language_insert() and locale_language_update() should only have local cache side effects and added a @todo to the code to figure that out.

Here is a reroll with these (minimal) changes.

catch’s picture

Status: Needs review » Fixed

Reviewed this in irc with Gabor.

For easy reference here's the interdiff from #65-#67.


diff -u b/includes/locale.inc b/includes/locale.inc
--- b/includes/locale.inc
+++ b/includes/locale.inc
@@ -473,9 +473,6 @@
     // Set the new version of this language as default in a variable.
     $default_language = language_default();
     variable_set('language_default', $language);
-
-    // We rely on the caller to update the old one as well to avoid conflicts
-    // with cached data updates stepping on each others toes.
   }

   // Update language count based on enabled language count.
@@ -484,9 +481,9 @@
   // Kill the static cache in language_list().
   drupal_static_reset('language_list');

+  // @todo move these two cache clears out. See http://drupal.org/node/1293252
   // Changing the language settings impacts the interface.
   cache_clear_all('*', 'cache_page', TRUE);
-
   // Force JavaScript translation file re-creation for the modified language.
   _locale_invalidate_js($language->language);

The previous re-roll was just s/iso/standard and this had been RTBC for 2-3 weeks beforehand, so I've committed this to 8.x. Hope this helps to get some of the other issues moving.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

Gábor Hojtsy’s picture

Issue tags: +language-base

Tagging for base language system.

Gábor Hojtsy’s picture

Issue summary: View changes

Add parent.