Installer auto-selects languages that don't have a translation, causing a requirements error [#1904214]

Problem/Motivation

After selecting British English during core installation, the installation is halted and an error is reported:

The English, British translation is not available
The English, British translation file is not available at the translation server. Choose a different language or select English and translate your website later.

This error is caused by a missing translation at the localize.drupal.org server. For British English (and some other languages) no translations are available at all: http://localize.drupal.org/translate/downloads. Missing translation are usually the result of translation teams which have not (yet) started. The druapl installation uses the languages returned by standard_language_list(). This list roughly matches the languages available at l.d.o, but does not take into account the languages for which no translations are available at l.d.o. Therefore not all languages presented at installation are actually available for installation. This is not a good user experience.

Proposed resolution

One of these solutions:
1. #568986: Dynamically update standard language list from localization server
2. Add an extra field to standard_language_list() to indicate the availability of translations at l.d.o
3. Create a new function that return a list of not translated languages.

Not translated languages can be removed from the language list at installation (solution 2 and 3). But can also be used to ignore during automatic language import.

Remaining tasks

to be determined

User interface changes

The list of languages presented during installation, will only contain the languages for which at least a drupal core translation is available.

API changes

to be determined

Comment	File	Size	Author
#21	1904214-language-list-translation-21.patch	3.2 KB	vijaycs85
#21
#16	1904214-language-list-translation-16.patch	16.76 KB	vijaycs85
#16

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Comment #1

Sutharsan CreditAttribution: Sutharsan commented 31 January 2013 at 12:35

These languages have no D7 core translation or have no translations at all:

function locale_not_translated_languages() {
  return array(
    'en-gb', // English, British
    'br',    // Breton
    'ga',    // Irish
    'hy',    // Armenian
    'mfe',   // Mauritian Creole
    'mi',    // Maori
    'mt',    // Maltese
    'or',    // Oriya
    'os',    // Ossetian
    'ps',    // Pashto
    'pt',    // Portuguese, International
    'rw',    // Kinyarwanda
    'sd',    // Sindhi
    'sw',    // Swahili
    'ti',    // Tigrinya
  );
}

This is the function I propose with solution 3.

Comment #2

Sutharsan CreditAttribution: Sutharsan commented 31 January 2013 at 12:45

Solution 2, IMO adds a substantial data overhead to the existing list (below). And I doubt this is the proper place to add it to; l.d.o language properties don't belong in a generic language list.

function standard_language_list() {
  return array(
    'af' => array('Afrikaans', 'Afrikaans'),
    'am' => array('Amharic', 'አማርኛ'),
    'ar' => array('Arabic', /* Left-to-right marker "‭" */ 'العربية', LANGUAGE_RTL),
    'ast' => array('Asturian', 'Asturianu'),
    'az' => array('Azerbaijani', 'Azərbaycanca'),
    'be' => array('Belarusian', 'Беларуская'),
    'bg' => array('Bulgarian', 'Български'),
    'bn' => array('Bengali', 'বাংলা'),
    'bo' => array('Tibetan', 'བོད་སྐད་'),
    'bs' => array('Bosnian', 'Bosanski'),
    'ca' => array('Catalan', 'Català'),
    'cs' => array('Czech', 'Čeština'),
    'cy' => array('Welsh', 'Cymraeg'),
    'da' => array('Danish', 'Dansk'),
    'de' => array('German', 'Deutsch'),
    'dz' => array('Dzongkha', 'རྫོང་ཁ'),
    'el' => array('Greek', 'Ελληνικά'),
    'en' => array('English', 'English'),
    'en-gb' => array('English, British', 'English, British'),
    'eo' => array('Esperanto', 'Esperanto'),
    ...

Comment #3

catch

he/him

English

CreditAttribution: catch commented 4 February 2013 at 14:54

Title:	Can't install in British English	» Installer auto-selects languages that don't have a translation, causing a requirements error
Priority:	Normal	» Major

Bumping this to major since the language is also selected by default. Presenting an option that's not available isn't great, but selecting it by default means you have to go back and undo that default selection to get anywhere which is considerably worse.

Comment #4

Gábor Hojtsy

he/him

Hungarian

Hungary

CreditAttribution: Gábor Hojtsy commented 8 February 2013 at 09:53

@catch: well, we wanted to pull the language list from localize.drupal.org and that could be filtered for which ones have translations for a given major version. That was shut down in #568986: Dynamically update standard language list from localization server. It would also introduce a HTTP request before even the first Drupal install screen shows which might time out or at least make it look like Drupal is loading slow and have other fun stuff, you know :)

So maybe our best bet for performance is to include some kind of static dump data in Drupal core releases and keep it up to date.

The underlying problem BTW is that localize.drupal.org can have any number of translation teams but not all of them are actually active or translate anything for a given version of core. The teams listed by Sutharsan above are essentially dead currently, since they never translated at least one single string that shows up in some release of Drupal 7 (or something is seriously broken with the translation file generator tool, but I checked some languages earlier and even in interactive search they could not find any strings for core 7.x).

Comment #5

vijaycs85

London, UK

CreditAttribution: vijaycs85 commented 8 February 2013 at 15:47

The teams listed by Sutharsan above are essentially dead currently, since they never translated at least one single string that shows up in some release of Drupal 7 (or something is seriously broken with the translation file generator tool, but I checked some languages earlier and even in interactive search they could not find any strings for core 7.x).

Are we sure, this is applicable for English, British(en-gb). Because I can see(here: http://localize.drupal.org/translate/languages/en-gb/translate) it has lots of translation strings but no change in what we have in nativeEnglish (en) which might be an issue?

Comment #6

Gábor Hojtsy

he/him

Hungarian

Hungary

CreditAttribution: Gábor Hojtsy commented 8 February 2013 at 15:55

The filter to use is the latest Drupal 7 release with translated strings only. http://localize.drupal.org/translate/languages/en-gb/translate?project=d... you get that there are no translated strings. That is why there is no .po file. Substitute for other languages.

Comment #7

Sutharsan CreditAttribution: Sutharsan commented 8 February 2013 at 17:39

If we agree that standard_language_list() is a list of languages available at l.d.o and not a universal list of languages, we can simply remove those languages without an active translation team from the function (i.e. those listed in #1).

Comment #8

vijaycs85

London, UK

CreditAttribution: vijaycs85 commented 8 February 2013 at 17:52

Can I suggest something here? Why can't we have a po file for all language with minimal words? i.e. just one word in all language, so that we can always assume that all language available. Adding a filter list and updating them seems bit integrity and manageability issue. Any thoughts?

Comment #9

Sutharsan CreditAttribution: Sutharsan commented 8 February 2013 at 19:58

@ vijaycs85, the thing with #568986: Dynamically update standard language list from localization server and your suggestion, is that they need to establish an internet connection before the very first page is displayed. This brings two problems. Af first the risk of not being able to establish the connection and the possible time delay caused by establishing the connection and download of data. Second is the fact that these solutions that they need to call home without consent. Not everyone likes to ping/connect without giving explicit confirmation. My proposal is to remove the languages without translation from the list. IMO the list was intended to reflect the available languages. Not translated languages are not available.

Comment #10

Gábor Hojtsy

he/him

Hungarian

Hungary

CreditAttribution: Gábor Hojtsy commented 8 February 2013 at 20:02

Yeah, making a .po file with one string available would not actually make the languages available practically. Also currently any language added will need the list updated too. Localize.drupal.org still gets new languages.

Comment #11

jbrown CreditAttribution: jbrown commented 8 February 2013 at 23:58

Adding the error message for googlability:

The service definition "http_default_client" does not exist.

Comment #12

Gábor Hojtsy

he/him

Hungarian

Hungary

CreditAttribution: Gábor Hojtsy commented 9 February 2013 at 07:17

@jbrown: wrong issue? :)

Comment #13

jbrown CreditAttribution: jbrown commented 9 February 2013 at 11:34

@gabor thats the error I get when I try to install as British: #1912422: Can't install Drupal as "English, British"

Comment #14

Sutharsan CreditAttribution: Sutharsan commented 9 February 2013 at 13:27

@jbrown, see #1887046: Convert drupal_http_request usage in install.core.inc to Guzzle. for a solution of your issue.

Comment #15

vijaycs85

London, UK

CreditAttribution: vijaycs85 commented 18 February 2013 at 20:59

So seems we can't have proposed solution #1 anytime near by. IMHO #2 looks good except removing languages those have no translation (considering we won't list them even after installation?). We can have one more element to indicate core translation of a language and disable them on language select list, if indicator=0. Make sure on every stable release sync with l.d.o status.

Comment #16

vijaycs85

London, UK

CreditAttribution: vijaycs85 commented 19 February 2013 at 10:56

Status:

Active

» Needs review

File	Size
1904214-language-list-translation-16.patch	16.76 KB

Below patch does following:

1. Adds flag to standard_language_list to indicate the translation availability (though, couldn't find few lanaguage listed in #1 in standard_language_list()).
2. Remove the languages without translation from installation language page: personally I think, we need to disable the options instead of removing. I have a sandbox module http://drupal.org/sandbox/vijaycs85/1340952 which allows to do this. But not sure how feasible to get this into core.

Todo:
- Alter description of language field to reflect this change ("i.e. If your language is not this list, it doesn't have translation in l.d.o etc).

Comment #17

Sutharsan CreditAttribution: Sutharsan commented 19 February 2013 at 14:22

Re 1. Should we really keep these languages in the list? For me only languages which can be used out of the box should be in the list. The languages listed in #1 have no translations because they don't have enough community support. I prefer them to be removed from the standard_language_list().
Re 2. A disabled language can not be used, but nothing explains why it can't be used and there is nothing a user can do to change this. Keep it simple: only list languages that can be used.

If we really insist on adding this extra option to the standard_language_list() data array, we should modify the Language class for this too (Drupal\Core\Language\Language).

Comment #18

Gábor Hojtsy

he/him

Hungarian

Hungary

CreditAttribution: Gábor Hojtsy commented 19 February 2013 at 14:25

Agreed with Sutharsan, let's remove those and revisit on release.

Comment #19

catch

he/him

English

CreditAttribution: catch commented 19 February 2013 at 14:58

@catch: well, we wanted to pull the language list from localize.drupal.org and that could be filtered for which ones have translations for a given major version. That was shut down in #568986: Dynamically update standard language list from localization server.

@Gabor, you marked that issue won't fix yourself, and the use case here is different to storing this in CMI?

The HTTP request could have a one second time-out added, if we're particularly worried about performance on this screen it could potentially be done in an AJAX request which updates the SELECT list dynamically. I'm a lot more concerned about file system scans and database table creation in terms of installer performance generally.

I'm also OK with just removing those languages from the list as a stop-gap fix though.

Comment #20

Gábor Hojtsy

he/him

Hungarian

Hungary

CreditAttribution: Gábor Hojtsy commented 19 February 2013 at 15:12

@catch: given that we got no support to store the language list in a sensible local format that is not yet another made-up thing for D8MI only, it did not make sense to pursue. While updating the select list on the fly works for *removing* languages, that does not solve adding languages, which would need some kind of local language list storage outside of a list in a PHP file that we cannot modify from code. That got no support. I don't think we only want to solve *removing* languages here, that would still require us to revisit this list manually whenever a new language is added on localize.drupal.org.

The conclusion in #1632236: Convert built-in language list to CMI was that the best technology in Drupal 8 is manual updates with each release. Nobody cared to support any other position.

Comment #21

vijaycs85

London, UK

CreditAttribution: vijaycs85 commented 19 February 2013 at 16:02

File	Size
1904214-language-list-translation-21.patch	3.2 KB

Seems below languages removed from the list already:

    'br',    // Breton
    'os',    // Ossetian
    'ps',    // Pashto
    'rw',    // Kinyarwanda
    'sd',    // Sindhi

Below patch removes remaining in the list (at #1).

Comment #22

Gábor Hojtsy

he/him

Hungarian

Hungary

CreditAttribution: Gábor Hojtsy commented 19 February 2013 at 16:05

Status:

Needs review

» Reviewed & tested by the community

As per above, this should work as a stop-gap fix. We can reopen #1632236: Convert built-in language list to CMI or figure out some other method to store language list dynamically if we want to keep it updated outside of the release cycle.