Problem/Motivation

The installer was recently fixed in #2113955: Rely on proper server side version fallback for translations to support a more seamless translation fallback system implemented on the server side (where all the info on versions is available). However, locale module still uses some local data to try to find translations when a concrete version is not available and therefore is not consistent with the installer behavior. It also results in a confusing dependency between update and locale modules. You only have up to date data in localization update if update module is enabled, but the dependency is not officially enforced due to locale being one module doing many other things.

Proposed resolution

Just be consistent with the installer and use the same server side version fallback scheme. This also removes the update module dependency and makes checking for translation updates faster.

Remaining tasks

None.

User interface changes

Minor. The y.x translations are now looked for when dev versions are encountered (instead of older translations). If there was an older translation, the y.x file should be available as well. This is apparent on the UI:

API changes

None.

#2030775: Reduce dependency of Locale on Update module

CommentFileSizeAuthor
#114 interdiff.txt646 bytesGábor Hojtsy
#114 1832946-114.patch24.13 KBGábor Hojtsy
#108 interdiff.txt2.16 KBWim Leers
#108 locale-update-1832946-108.patch23.5 KBWim Leers
#96 d8-locale-translation-updates.png70.97 KBJose Reyero
#85 locale-update-1832946-85.patch23.53 KBSutharsan
#85 interdiff-1832946-82-85.txt4.55 KBSutharsan
#82 locale-update-1832946-82.patch21.73 KBSutharsan
#82 interdiff-1832946-77-82.txt6.25 KBSutharsan
#77 interdiff-1832946-75-77.txt13.56 KBSutharsan
#77 1832946-local-update-fallback.77.patch17.41 KBSutharsan
#75 interdiff-1832946-72-75.txt1.76 KBSutharsan
#75 1832946-locale-update.75.patch5.48 KBSutharsan
#72 1832946-locale-update.72.patch5.41 KBJose Reyero
#57 1832946-project-api-56.patch40.42 KBSutharsan
#57 interdiff-1832946-55-57.txt9.8 KBSutharsan
#55 1832946-project-api-55.patch48.65 KBJose Reyero
#44 1832946-project-api-44.patch57.2 KBSutharsan
#36 interdiff-1832946-31-34.txt47.68 KBSutharsan
#35 1832946-project-api-34.patch70.74 KBSutharsan
#35 1832946-project-api-34.patch70.74 KBSutharsan
#35 interdiff-1832946-31-34-no-white-space.txt1.02 KBSutharsan
#31 1832946-project-api-31.patch113.13 KBvijaycs85
#25 project_api-1832946-25.patch67.46 KBSutharsan
#25 interdiff-1832946-15-25.txt45.55 KBSutharsan
#15 project_api-1832946-15.patch65.05 KBSutharsan
#15 interdiff-1832946-5-15.txt48.1 KBSutharsan
#11 project_api-1832946-11-do-not-test.patch74.14 KBSutharsan
#11 interdiff-1832946-5-11.txt18.15 KBSutharsan
#5 project_api-1832946-05.patch59.04 KBSutharsan
project_api-xxxx-01.patch58.92 KBJose Reyero
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Status: Needs review » Needs work

The last submitted patch, project_api-xxxx-01.patch, failed testing.

Jose Reyero’s picture

Status: Needs work » Needs review

project_api-xxxx-01.patch queued for re-testing.

Status: Needs review » Needs work

The last submitted patch, project_api-xxxx-01.patch, failed testing.

Jose Reyero’s picture

Status: Needs work » Needs review

project_api-xxxx-01.patch queued for re-testing.

Sutharsan’s picture

Issue tags: +D8MI, +sprint, +language-ui
FileSize
59.04 KB

Re-rolled the patch.

Added a project definition to locale_test_translate module. Strange, how tests ever passed without it.

--- a/core/modules/locale/tests/modules/locale_test_translate/locale_test_translate.info
+++ b/core/modules/locale/tests/modules/locale_test_translate/locale_test_translate.info
@@ -3,8 +3,8 @@ description = Translation test module for locale module testing.
 package = Testing
 version = 1.3
 core = 8.x
+project = locale_test_translate
 hidden = TRUE

Status: Needs review » Needs work

The last submitted patch, project_api-1832946-05.patch, failed testing.

Sutharsan’s picture

Bot fails on Updgrade Path test, but test passes locally. Although with an exception, but it is the same exception as when the test runs on 8.x without the patch). I'll give the bot one more spin.

Sutharsan’s picture

Status: Needs work » Needs review

#5: project_api-1832946-05.patch queued for re-testing.

Sutharsan’s picture

The patch make some changes to .info file properties:

  • 'project status url' now accepts a 'FALSE' value.
  • 'interface translation project' is removed in favor of the existing 'project' property
  • 'project translation server pattern' was previously known as 'interface translation server pattern'

Some examples from the patch:

+ * If a project is not intended to be checked for code updates, add this data
+ * data in the info file: 'project status url = FALSE'
+; Hide this project from update status.
+project status url = FALSE
 ; Definitions for interface translations.
-interface translation project = locale_test
-interface translation server pattern = core/modules/locale/test/test.%language.po
+project translation server pattern = core/modules/locale/test/test.%language.po

IMHO 'project translation server pattern' is not a significant improvement compared to the old 'interface translation server pattern'. Especially now locale module is named 'Interface translation'. If we make any change, I suggest 'interface translation url' as it is an url, and replacement patterns are not required.

Sutharsan’s picture

I like to use this API as a starting point to create project objects. Project objects can be used to objectify the interface translation code as in #1842380: Convert $source object to a TranslatableProject class. But I'm struggling with the right way to instantiate the project objects. I don't have experience in this field and I can't find proper examples. Below is a scetch of my ideas how to do it.

  • SystemProjectFactory is a class which generates and handles SystemProject objects.
  • SystemProjectFactory::scan() processes project .info files as project_process_info_list() currently does.
  • SystemProjectFactory::getAllProjects() returns array of SystemProject objects.
  • SystemProject objects contain project data of a single project as currently returned by project_list().
  • class LocaleProjectFactory extends SystemProjectFactory for locale module.
  • class UpdateProjectFactory extends SystemProjectFactory for update module.

Examples in code:

// IN CURRENT PATCH ============================
function project_list() {
  if (FALSE && $cached = cache()->get('project_list')) {
    $projects = $cached->data;
  }
  else {
    $module_data = system_rebuild_module_data();
    $theme_data = system_rebuild_theme_data();
    project_process_info_list($projects, $module_data, 'module', TRUE);
    project_process_info_list($projects, $theme_data, 'theme', TRUE);
    project_process_info_list($projects, $module_data, 'module', FALSE);
    project_process_info_list($projects, $theme_data, 'theme', FALSE);
    // Allow other modules to alter projects before fetching and comparing.
    drupal_alter('project_list', $projects);
    // Cache the site's project data for at most 1 hour.
    cache()->set('project_list', $projects, REQUEST_TIME + 3600);
  }
  return $projects;
}

function locale_translation_project_list() {
  $projects = &drupal_static(__FUNCTION__, array());
  if (empty($projects)) {
   $include_disabled = config('locale.settings')->get('translation.check_disabled_modules');
   $projects = array_filter(project_list(), function ($project) use ($include_disabled) {
     return ($include_disabled || !empty($project['project_status'])) &&
       (!isset($project['info']['project translation server pattern']) || $project['info']['project translation server pattern']);
   });
  }
  return $projects;
}

function update_get_projects() {
  $projects = &drupal_static(__FUNCTION__, array());
  if (empty($projects)) {
   $include_disabled = config('update.settings')->get('check.disabled_extensions');
   $projects = array_filter(project_list(), function ($project) use ($include_disabled) {
     return ($include_disabled || !empty($project['project_status'])) &&
       (!isset($project['info']['project status url']) || $project['info']['project status url']);
   });
  }
  return $projects;
}

// PROPOSAL ============================
function locale_translation_project_list() {
  if (FALSE && $cached = cache()->get('locale_translation_project_list')) {
    $projects = $cached->data;
  }
  else {
    $module_data = system_rebuild_module_data();
    $theme_data = system_rebuild_theme_data();
    $factory = new LocaleProjectFactory('locale_translation_project_list');
    $factory->scan($module_data, 'module', TRUE);
    $factory->scan($theme_data, 'theme', TRUE);
    $factory->scan($module_data, 'module', FALSE);
    $factory->scan($theme_data, 'theme', FALSE);
    // Allow other modules to alter projects before fetching and comparing.
    $factory->alter();
    // Cache the site's project data for at most 1 hour.
    cache()->set('locale_translation_project_list', $projects-getAllProjects(), REQUEST_TIME + 3600);
  }
  return $projects;
}

function update_get_projects() {
  if (FALSE && $cached = cache()->get('update_projects')) {
    $projects = $cached->data;
  }
  else {
    $module_data = system_rebuild_module_data();
    $theme_data = system_rebuild_theme_data();
    $factory = new UpdateProjectFactory('update_projects');
    $factory->scan($module_data, 'module', TRUE);
    $factory->scan($theme_data, 'theme', TRUE);
    $factory->scan($module_data, 'module', FALSE);
    $factory->scan($theme_data, 'theme', FALSE);
    // Allow other modules to alter projects before fetching and comparing.
    $factory->alter();
    // Cache the site's project data for at most 1 hour.
    cache()->set('update_projects', $projects-getAllProjects(), REQUEST_TIME + 3600);
  }
  return $projects;
}
Sutharsan’s picture

This is working code to replace update_process_info_list() (or project_process_info_list() if you wish). I like to have comments on the objectification of the code. Pointer to good examples are appreciated too. A know omission is a missing ProjectInterface class, but that should be easy to extract. Also like to have comment on the above usage of this ProjectFactory class. The implemententation is slightly changed as you can see below.

The two debug calls give the same data.

  // New using ProjectFactory
  use Drupal\Core\Project\ProjectFactory;
  $factory = new ProjectFactory('locale_translation_project_list');
  $factory->parse($module_data, 'module', TRUE);
  $factory->parse($theme_data, 'theme', TRUE);
  $projects = $factory->getAllProjects();
  debug($projects);

  // Existing using update_process_info_list();
  module_load_include('compare.inc', 'update');
  debug(update_get_projects());
dww’s picture

Status: Needs review » Needs work

Thanks for moving this forward, this is probably a good idea so we can share more code between update manager and other parts of core. I don't have time to do a thorough review, but this is now on my radar and I hope to look more closely Soon(tm).

However, do we really want this to consume the "project" namespace? That's not going to be any fun at all for the project module, which is kinda important for d.o and our community. :( I guess in the past the attitude has been "if core wants a namespace, contrib be damned", so perhaps we're just going to have to somehow deal with this in D8 and rename that module (or something). But holy crap, that's going to be a HUGE amount of work, and perhaps core could save us the trouble of basically rewriting the whole thing (again) by using some other namespace for this...

Thanks,
-Derek

Jose Reyero’s picture

Some thoughts:
- We need to develop the concept of 'project' first with an API as simple as possible. Factories, etc just add complexity, don't add anything at this point.
- Though a Project class may be something to aim at, that should start with component classes (Module, Theme) etc or otherwise we end up with a class that is still a huge array of arrays.

IMO, what we need to focus on here is some project_list() that can provide a list to be reused by both update and locale, and not running with different lists, that you need to cache independently, for each, and then rebuild all of them after we rebuild the list of modules or themes.

In order to give projects some proper entity (I mean existence, not Drupal Entity) and information, maybe we should consider too getting them their own info file, for what we need some namespace that doesn't overlap with module.info. Thinking of this I would propose something like
[projectname].project.info
That should contain all the project-specific data like name, version, update server for the modules in that project, etc...

The major issue atm is compiling the list of projects form the list of modules is really a 'dirty' (as not well defined) process and some important waste (duplicated info for modules in a project, name clashes, etc, etc..).

@dww,
How does [projectname].project.info as a unique file per project sounds for the drupal.org packager?

About namespaces, I don't think we want to take over project_* namespace. We could very well use system_project_* or component_project_*. Maybe a component.api (that can be used for modules, themes, projects would make more sense.)

clemens.tolboom’s picture

I agree with @Jose Reyero to make Project as simple as possible. But by adding classes for Modules and Themes would make this a Drupal 9 issue. But we should add those eventually.

By adding a [projectname].project.info we can speed things up.

As we are adding an API we should add a class for Project like \Drupal\Core\Project so there is no namespace problem with the Project project. We do have \Drupal\Core\SystemListingInfo.php so I'm a little confused as that comes close to our needs.

In discussing with @Sutharsan I proposed composition design pattern for update / locale like

class Project {
  function getName() {
    return spl_object_hash($this);
  }
}

class TranslationProject {
  var $_langcode;
  var $_project;

  function __construct($project) {
    $this->_project = $project;
  }
  function getProjectName() {
    return $this->_project->getName();
  }

  function setLangcode($langcode) {
    $this->_langcode = $langcode;
  }
}

My 2cents

Sutharsan’s picture

Update module and Locale module have different needs for project data. Where Locale only requires as list of projects with name and release version, Update module needs more details such as sub-modules(/themes) project time stamp, package, etc. IMHO class inheretance/override is the ideal for this differentiation. The attached patch tries to combine this with a minimum added complexity. A Project class holds the project data and a ProjectBuilder class contains the logic to process the module and theme data and build a project object. Locale and Update have their own inherited builder class to differentiate. LocaleProjectBuilder filters only the required data, UpdateProjectBuilder does some additional data processing on top of the ProjectBuilder results.

I have the following OO related questions:

  • Is the ProjectBuilder class a correct way of creating objects? I've chosen to build an array of object inside the class because of the nature of the building process. It needs to build lists of sub-modules for each project, therefore it can only generate all projects at once and not one a the time.
  • The ProjectBuilder class instantiates Project objects. But should be able to determine which (derived) project class to use. I've use a protected property $project_class. Is this a proper way to do it?
  • The UpdateProjectBuilder adds a method processVersionInfo() which does not exist in the base ProjectBuilder class. Is this good practice?
  • Does Project need an abstract base or interface class?

Some more notes on the code: I changed the project parsing code in ProjectBuilder as little as possible. Its complexity holds me back, but I would rather break it up to make the base project class more lean and move the Update module specific stuff into the UpdateProjectBuilder. In other words, more differentiation between Locale and Update project data. The UpdateProject is not used, but I left it there for now to show the kind of data Update modules calculates. I did not want to modify the Update module code which uses the $project array and convert it into a $project object. As a temporary solution I introduced a getAllProjectsAsArray() method to the UpdateProjectBuilder which is now used by Update module to retrieve the project data.

I like the idea to add a *.project.info file to each project. If it includes the data we need to build projects (including a list of sub modules/themes), it would make the project building process much easier.

Though a Project class may be something to aim at, that should start with component classes (Module, Theme)

I don't agree with @Jose Reyero on this. A project is a container of components, where the only (required) relation between them is that the project must know the names of the included components. Ideally these are not names, but component objects. Currently only Update module has the requirement for this data and I have the impression that this data only required for displaying a list of included modules/themes in the interface. I don't want to postpone the introduction of project objects until Component objects are introduces. I need the project objects to start with improvements on automatic translation import.

@ClemensTolboom, I don't want to jump into TranslationProject yet. Getting the base Project class right is difficult as it is ;) But thanks for your cents.

Sutharsan’s picture

Status: Needs work » Needs review

Status: Needs review » Needs work

The last submitted patch, project_api-1832946-15.patch, failed testing.

Jose Reyero’s picture

Update module and Locale module have different needs for project data. Where Locale only requires as list of projects with name and release version, Update module needs more details such as sub-modules(/themes) project time stamp...

This is where I start not liking this patch because id does away with the concept of 'project' list api.

I think a project is a project independently of which other part of Drupal uses it so if we cannot have a single list and a single object for both locale and update I don't see the purpose of this issue anymore.

Ok to Project class, though, just don't make it too complex.

katbailey’s picture

I promised some feedback on this patch, but first a disclaimer: I have nfi about the inner workings of either locale module or update module :-P This feedback is just about the class design.

Also, I think it might be a good idea to get chx to look at this, because he has done a lot of work around module/theme discovery and has further plans in that area I believe. It would be a shame if there were efforts being duplicated.

Here are my comments on the classes...

Project class:

+++ b/core/lib/Drupal/Core/Project/Project.phpundefined
@@ -0,0 +1,94 @@
+  /**
+   * Array of project info, as defined in the .info file.
+   *
+   * @var array
+   */
+  public $info = array();
+
+  /**
+   * The timestamp of project release.
+   *
+   * @var integer
+   */
+  public $datestamp = 0;
+
+  /**
+   * Array of modules included in this project.
+   *
+   * @var array
+   */
+  public $includes = array();
+
+  /**
+   * Array of disabled modules in this project.
+   *
+   * @var array
+   */
+  public $disabled = array();
+
+  /**
+   * The type of project. e.g. module, theme.
+   *
+   * @var string
+   */
+  public $project_type = '';
+
+  /**
+   * The project status. 1 = enabled, 0 = disabled.
+   *
+   * @var string
+   */
+  public $project_status = '';

It looks like most of these properties can get set on instantiation. Use setters for the rest instead of having public properties.

ProjectBuilder class:

+++ b/core/lib/Drupal/Core/Project/ProjectBuilder.phpundefined
@@ -0,0 +1,303 @@
+  /**
+   * Parses module .info files data to collect project information.
+   */
+  public function parseModules() {
+    $module_data = system_rebuild_module_data();
+    $this->parse($module_data, 'module', TRUE);
+    if ($this->include_disabled) {

We generally want to avoid calling out to procedural functions within our classes - and seeing as we are always going to need this information, why not pass it as a parameter to the constructor?

+++ b/core/lib/Drupal/Core/Project/ProjectBuilder.phpundefined
@@ -0,0 +1,303 @@
+      if (!$project = $this->getProject($project_name)) {
+        // Only process this if we haven't done this project, since a single
+        // project can have multiple modules or themes.
+        // @todo dynamically define a class.
+        $project = new $this->project_class($project_name, $project_display_type);

A nicer way of dealing with this might be to use a ProjectFactory, which could also be passed in the constructor of the ProjectBuilder, so then you would do something like...

$project = $this->projectFactory->getInstance($project_name, $project_display_type);
+++ b/core/modules/locale/locale.compare.incundefined
@@ -127,73 +121,29 @@ function locale_translation_build_projects() {
+    $builder = new LocaleProjectBuilder();
+    $builder->parseModules($include_disabled);
+    $builder->parseThemes($include_disabled);
+    $projects = $builder->getAllProjects();

Looking at this calling code to see how the ProjectBuilder class gets used, and taking the above comments into account, I think what you want to end up with is calling code that looks like this:

$module_data = system_rebuild_module_data();
$theme_data = system_rebuild_theme_data();
$builder = new LocaleProjectBuilder($module_data, $theme_data, $include_disabled, new ProjectFactory());
$projects = $builder->getAllProjects();

... assuming it never makes sense to have a ProjectBuilder object that has not parsed the data, i.e. so you would just call the parse() method in the constructor and be done (the parseModules() and parseThemes() methods would then be unnecessary.)

I hope this helps!

Sutharsan’s picture

@katbailey, thanks for your review. Regarding public properties, I see a number of classes using public properties such as file, node and (other) entities. I guess this is for backwards compatibility only, or am I missing something? Also I have not found any rules or coding standard for this in Drupal yet.

katbailey’s picture

This is just bog standard encapsulation. Is there a reason you think those properties should be public or that they can't be set in the constructor?

Sutharsan’s picture

I used public properties because this required the least amount of changes when re-using existing code ($project['foo'] changed to $project->foo) and I found public properties used in various places in core. I'm not defending my code, I will rework it without hesitation to comply with best practice, but I'm searching for the right and the proper (and all the exceptions to that ;) ) and trying to understand it all.

katbailey’s picture

I don't think there's any need to be overly tentative about the extent of the changes you'll be making here. We are embracing OOP in D8 and this has already entailed enormous changes to the way we write code for Drupal. One huge advantage to embracing a very commonly used set of design principles is that we don't have to maintain coding standards documentation about things like the question of public/protected/private properties - we can just follow general OOP best practices. We *want* Drupal to not be special in this regard.

The only reason I can think that you might want to use public properties would be as a micro-optimization, i.e. to avoid the method call, as discussed here: http://stackoverflow.com/questions/6215398/calling-the-variable-property...

But in this particular case, as I said the majority of those properties look like they can be set on instantiation and will not be changed again. And even if they did need to be set again, using setters is still the best practice - the kind of micro-optimization mentioned above is definitely not the norm.

Sutharsan’s picture

Possibly related: #1833592: [META] The road out of module build circular dependency hell I checked with chx, this is the issue where the work for 'module/theme discovery' is going on. See katbailey's remark in #19.

Sutharsan’s picture

Status: Needs work » Needs review
FileSize
45.55 KB
67.46 KB

This patch now uses project classes as katbailey suggested:

  • Class Project.php: protected variables. Does this need an underlying interface class?
  • Class ProjectBuilder.php: Building array of classes from supplied module and theme data. Changed to use the Project getters and setters.
  • Class ProjectFactory: Very minimal. Should we use this or use a Project::create() method instead and call ProjectBuilder::__construct() with the required class name?

@katbailey, I would appreciate your review on the above.

Some observations on the system_project_list() as a single source of project data.

  • system_project_list() is now the only project list source. This is now back as how it was in the initial patch by Jose Reyero. system_project_list() return an array of objects. This however has some consequenses:
  • Locale and Update module both still expect the project to be an array. In locale_translation_project_list() and update_get_projects() I've added code to convert the data to an array. This needs a follow-up.
  • In update_process_project_info(), update_calculate_project_data() and update_calculate_project_update_status() add some 14 more parameters to the (current) project array. If we modify the update module to use a project object instead of an array, these 14 parameters must be added to the Project class or the parameters must be added to an UpdateProject class which extends the Project class.
  • For Locale module, the introduction of a project class is required for #1842380: Convert $source object to a TranslatableProject class. The Project class must be extended into a TranslatableProject class. I think this requires TranslatableProject to extend Project class, but perhaps there are different ways.
  • system_project_list() makes a list of all projects both enabled and disabled and it caches the result. Disable projects are filtered out in update_process_project_info(). This however results in the update page to show both enabled and disabled modules included in for example Drupal Core. Since both Update and Locale module use their own 'show/hide the disabled project' setting, changing this back to the original behaviour will lead to removing the cache from system_project_list() and using individual caches in Update and Locale. Any other ideas?

@Jose Reyero, I understand your desire for a single list to be used by both Update and Locale. But IMO the arguments for separate data, are growing in number. I'm trying to make this patch according to your single source idea, but I am running out of ideas to combine it in a decent way with one single project object.

Status: Needs review » Needs work

The last submitted patch, project_api-1832946-25.patch, failed testing.

Gábor Hojtsy’s picture

Category: feature » task

I think this would be an API refactoring, not a feature. Marking as such. @katbailey: thank you help review the recent updates? Sounds like we are on the right path.

podarok’s picture

agree with #27 - this is task!

katbailey’s picture

Sorry for taking so long to look at this again. This is looking much better. Regarding the questions above, yes we need a ProjectInterface and a ProjectFactoryInterface. Then the advantage of passing in an actual factory rather than a string representing a fully-qualified class name for a Project class is that we get much better error handling, i.e. if you pass in anything other than an object implementing the ProjectFactoryInterface it will blow up with a nice helpful message.

Below are just some minor points about method names.

+++ b/core/lib/Drupal/Core/Project/Project.phpundefined
@@ -0,0 +1,284 @@
+  /**
+   * Gets human readable project name.
+   *
+   * @return string
+   *   Human readable project name. Falls back to machine name if not available.
+   */
+  public function getProjectName() {
+    return isset($this->info['name']) ? $this->project->info['name'] : $this->name;
+  }

I'd call this getHumanName() as then it's clearer why you're not just returning $this->name.

+++ b/core/lib/Drupal/Core/Project/ProjectBuilder.phpundefined
@@ -0,0 +1,283 @@
+  /**
+   * Stores a project in the internal project array.
+   *
+   * @param object $project
+   *   Project object.
+   */
+  protected function setProject($project) {
+    $this->projects[$project->getName()] = $project;
+  }

This should be addProject() as you're not setting the $project property but adding to the $projects property.

Edit: I thought dreditor had lost my changes initially, but it hadn't - removing the duplication.

Sutharsan’s picture

@katbailey, thanks for the review. I've been offline for D8 work for a while. But I hope to get back to this issue shortly.

vijaycs85’s picture

Status: Needs work » Needs review
FileSize
113.13 KB

Stright re-roll of #25

Status: Needs review » Needs work

The last submitted patch, 1832946-project-api-31.patch, failed testing.

Sutharsan’s picture

Issue tags: -sprint, -language-ui

The syntax error was due to a missing closure parenthesis in update_get_projects(). Also fixed an error with the use path of LocaleProjectBuilder.
Patch follows

Sutharsan’s picture

For evaluation: interdiff-1832946-31-34-no-white-space.txt
For git apply: interdiff-1832946-31-34.txt
Needs more work to process katbailey's comments, but first I want to agree with Jose on the direction.

Sutharsan’s picture

FileSize
47.68 KB
aturetta’s picture

Status: Needs work » Needs review

Trigger testing for the latest patches...

Status: Needs review » Needs work

The last submitted patch, 1832946-project-api-34.patch, failed testing.

Gábor Hojtsy’s picture

Status: Needs work » Postponed

We have been discussing this with Sutharsan, and there are more important things to work on now and this did not get the kind of interest we hoped from outside our little bubble, so marking this postponed.

Berdir’s picture

That's unfortunate as I had to look at the locale "file" handling stuff and it made my head hurt, but we'll have to live with that I guess ;)

Gábor Hojtsy’s picture

I think we all agree this is a great cleanup effort. In the meantime, we did not manage to sign up a reasonable amount of people for this to work on it and we have important things to work on unfortunately. This one being pretty important right now: #1998056: Automatically update interface translations using cron.

Sutharsan’s picture

Status: Postponed » Active
Issue tags: +sprint

Setting back to sprint and active for focus to get this in before the code freeze.

Sutharsan’s picture

Assigned: Unassigned » Sutharsan

I've discussed yesterday with Jose to revive this feature. We will go back to the root of the issue and create a simple API with minimal changes and aimed at removing the dependency of Locale upon Update module. Re-roll of the original issue will be the first step.

Sutharsan’s picture

Status: Active » Needs review
FileSize
57.2 KB

Re-roll of patch in #5.

Status: Needs review » Needs work

The last submitted patch, 1832946-project-api-44.patch, failed testing.

plach’s picture

Issue tags: +language-interface

Tagging for the rocketship.

Gábor Hojtsy’s picture

Fix tags.

Gábor Hojtsy’s picture

Issue tags: -sprint +API change, +API clean-up

This looks like a massive API change. I believe we did the moving of functions around already. Is there anything else left here to do?

Gábor Hojtsy’s picture

@Sutharsan: are you still working on this? Is this still needed? What kind of API changes are needed?

webchick’s picture

The issue summary could really use an update that explains what the API changes introduced in this patch are in order to expedite approval.

xjm’s picture

An explicit list would be especially helpful, e.g.: https://drupal.org/node/2056513#summary-api-changes

Sutharsan’s picture

I received your comments, but I may need a few days to find the time to write something about the status of this issue.

jair’s picture

Issue tags: +Needs reroll

Needs Reroll

Jose Reyero’s picture

Status: Needs work » Needs review
FileSize
48.65 KB

Rerolled against latest HEAD.

Status: Needs review » Needs work

The last submitted patch, 1832946-project-api-55.patch, failed testing.

Anonymous’s picture

Issue summary: View changes

Updated issue summary

Sutharsan’s picture

I've updated the API summary.

The below functions in #55 are no longer needed as their functionality is now included in \Drupal\Core\Utility\ProjectInfo. This patch removes them.

  • project_process_info_list()
  • project_file_get_name()
  • project_filter_info()
Gábor Hojtsy’s picture

@dww was concerned in #12 for core taking on some of the project_* namespace. From the issue summary, this looks like still being the case? This should be reviewed with a core maintainer for API change approval before substantial further work is put into it.

dww’s picture

Yup, still concerned, but don't have the time/energy to follow closely or keep fighting. Thanks for bringing that back up!

dww’s picture

Issue summary: View changes

Updated API changes

sun’s picture

I agree that we badly need to clean up this code. Like @Berdir, I also had a very hard time in debugging test failures in the current project-related Locale code.

When proceeding here, it would be a good idea to move the generic parts out of Utility and into Drupal\Core\Extension\Project\* instead (or perhaps even Drupal\Core\Project), since this entire code heavily depends on the extension system, so various services should be injected.

Jose Reyero’s picture

I've just bumped into this one once again with latest beta13. Any chance we can get this thread back to life?

I'd say this is a huge usability issue. That a module silently depends on another one is so ugly...

There's only this API documentation about the issue and it is still confusing for me:

"For full functionality this function depends on Update module. When Update module is enabled the project data will contain the most recent module status; both in enabled status as in version. When Update module is disabled this function will return the last known module state. The status will only be updated once Update module is enabled."
From https://api.drupal.org/api/drupal/core%21modules%21locale%21locale.trans...

Gábor Hojtsy’s picture

Jose Reyero’s picture

Version: 8.0.x-dev » 9.x-dev

@Gábor,

It may be possible if it does not involve API changes...

Or in other words, it won't be possible... :-)

Ok, then let's focus on adding some warning in the UI, that related issue you mention.

I'm recategorizing this one for 9.x.

Gábor Hojtsy’s picture

Note that when I wrote that I was going by https://www.drupal.org/contribute/core/beta-changes which is the policy. Sounds like this may or may not be argued under the "reduced fragility" umbrella even in D8 but ideally should be discussed with core committers first.

penyaskito’s picture

API additions are allowed at 8.x.y. Do we really need to change underlying APIs? Couldn't we keep those as wrappers and deprecate them?

webchick’s picture

Version: 9.x-dev » 8.0.x-dev

Right, moving back to 8.0.x unless it's shown that there's no way to do this in a BC way. (And possibly even then; needs a beta evaluation.)

Jose Reyero’s picture

Luckily I've been working on this for some client's project, I've been looking into the API again, and running some tests. Some notes about this:

- We already have something similar to this API, as we have some tools that locale can be using without update, namely Drupal\Core\Utility\ProjectInfo

- Running locale module's translation updates without Update Manager is perfectly possible. The only issue is we won't get translations for installed development versions of modules, and may not get translations right away for newly released modules / core versions. What Update Manager does is getting the last tagged version for a module that we may have installed as -dev version. (Dev versions won't have translations available so we need to fall back to the last tagged version for that module).

- The current status is Locale module has some explicit check for update module and if it's not enabled, it just won't return any project information at all, then the "Available translation updates" page though still there, won't display any information.

So while it could be possible and easy to provide some partial project translation information and download it without update manager, I am not sure it is worth the trouble, as end users will get more or less available project translations depending on whether Update Mgr is enabled or not, and this may be difficult to communicate -see the second note.

IMO an 'all or nothing' (with update/without update) option is way easier to understand than 'you only get updates for some projects beacause of this and that...'. Then the only thing worth fixing is the UI / Usability issue of not getting any update / not knowhing why, which is being addressed in #2501035: Without "Update Manager" module enabled, "Available translation updates" does not get updated, should be documented

In summary I think we can close this thread as "won't fix"... Well, we could argue whether it's fixed or not because we do have some API, only it never gets used by locale without update manager enabled and also they use different wrappers to get the project list.. but that won't make any difference..

Sutharsan’s picture

So while it could be possible and easy to provide some partial project translation information and download it without update manager, I am not sure it is worth the trouble

It was not much trouble, working code was already available in Localization Update module. I've created a patch at #2550137: Improve interface translation fallback if Update module is disabled

IMO an 'all or nothing' (with update/without update) option is way easier to understand than 'you only get updates for some projects beacause of this and that...'.

Generally I agree, but when I saw the practical implication in #2501035: Without "Update Manager" module enabled, "Available translation updates" does not get updated, should be documented. That we are going to tell the user that a status page is not available...

I have no opinion on the direction of this issue (yet).

Jose Reyero’s picture

There has been some interesting development on the localization server that allows us to run translation updates entirely without depending on Update manager, see https://www.drupal.org/node/2501035#comment-10327233

I'm giving this issue one more try, I hope a patch will follow soon.

Jose Reyero’s picture

@Gábor,

Just got started with the patch, but the issue I'm facing is that there are no translations at all for modules without tagged releases, which are almost all of d8 contrib atm.

Also, which would be the right URLs to check for so they get redirected to the fallback? From current system, once we don't filter out dev releases I get URLs like these:

http://ftp.drupal.org/files/translations/7.x/devel/devel-7.x-1.x-dev.es.po

http://ftp.drupal.org/files/translations/8.x/mollom/mollom-8.x-1.0-alpha...

(This one, Mollom, at least has some tagged releases, just I don't know how to build the right URL form the dev version)

Gábor Hojtsy’s picture

#2113957: Build server side version fallback system for translations has some examples, eg /files/translations/6.x/gallerix/gallerix-6.x-1.x.ja.po

Jose Reyero’s picture

Status: Needs work » Needs review
FileSize
5.41 KB

This patch:

- Removes dependencies on update manager.
- Uses new server URLs (with server side fallback) for dev|alpha|beta versions of modules

Status: Needs review » Needs work

The last submitted patch, 72: 1832946-locale-update.72.patch, failed testing.

The last submitted patch, 72: 1832946-locale-update.72.patch, failed testing.

Sutharsan’s picture

Title: Create a small project API to be used by Update and Locale » Use serverside translation version fallback for translation update
Issue tags: -API change, -API clean-up, -Needs reroll +Needs tests
FileSize
5.48 KB
1.76 KB

Merged the solution from #2550137-5: Improve interface translation fallback if Update module is disabled. Making the regex less specific and adding a check for '-dev' in the issue string for performance.

Patch needs cleanup and tests. Not changing the status yet.

Gábor Hojtsy’s picture

Looks cool on a short review :)

Sutharsan’s picture

Assigned: Sutharsan » Unassigned
Status: Needs work » Needs review
FileSize
17.41 KB
13.56 KB

Next iteration:

  • Renamed LocaleUpdateNotDevelopmentReleaseTest to LocaleUpdateDevelopmentReleaseTest
  • Renamed module locale_test_not_development_release to locale_test_development_release
  • Update the test for locale_translation_build_projects() in the above test class
  • Small clean-ups

Todo: Add a new assertion. See the @todo in the patch.
Probably all the test that now fail, fail because locale_translation_build_projects() does return data when Update module is disabled. So test must be adapted to this new, desired, behavior.

Gábor Hojtsy’s picture

Issue tags: +sprint

Bring on D8MI sprint for tracking :) Thanks for working on this.

Status: Needs review » Needs work

The last submitted patch, 77: 1832946-local-update-fallback.77.patch, failed testing.

The last submitted patch, 77: 1832946-local-update-fallback.77.patch, failed testing.

Sutharsan’s picture

Assigned: Unassigned » Sutharsan
Sutharsan’s picture

Assigned: Sutharsan » Unassigned
Status: Needs work » Needs review
Issue tags: -Needs tests
FileSize
6.25 KB
21.73 KB

Got the tests working, by adding locale_test module to language related tests. The module prevents core from being translated when for example a language is added. Lets see if the bot agrees.

Status: Needs review » Needs work

The last submitted patch, 82: locale-update-1832946-82.patch, failed testing.

Sutharsan’s picture

Issue summary: View changes
Issue tags: -Needs issue summary update
Sutharsan’s picture

Sutharsan’s picture

Status: Needs work » Needs review

The last submitted patch, 57: 1832946-project-api-56.patch, failed testing.

The last submitted patch, 57: 1832946-project-api-56.patch, failed testing.

The last submitted patch, 75: 1832946-locale-update.75.patch, failed testing.

Status: Needs review » Needs work

The last submitted patch, 85: locale-update-1832946-85.patch, failed testing.

The last submitted patch, 75: 1832946-locale-update.75.patch, failed testing.

Status: Needs work » Needs review

The last submitted patch, 82: locale-update-1832946-82.patch, failed testing.

Sutharsan’s picture

Last patch tested Ok :)

Jose Reyero’s picture

Issue summary: View changes
FileSize
70.97 KB

The patch seems to work but it doesn't apply cleanly anymore, it needs a re-roll, will post it in a minute.

Attaching screenshot, we can see it finds the right translations (it also works when clicking on Update).

Jose Reyero’s picture

Status: Needs review » Reviewed & tested by the community

Finally it doesn't need a re-roll, my mistake, wrong branch I guess.

So, this works, it looks good to me, moving to RTBC.

Gábor Hojtsy’s picture

Title: Use serverside translation version fallback for translation update » Runtime translation download fallback works different from installer translation download fallback
Category: Task » Bug report
Issue summary: View changes
Priority: Normal » Major

Updated issue title and summary significantly.

Status: Reviewed & tested by the community » Needs work

The last submitted patch, 85: locale-update-1832946-85.patch, failed testing.

Gábor Hojtsy’s picture

Status: Needs work » Reviewed & tested by the community

Status: Reviewed & tested by the community » Needs work

The last submitted patch, 85: locale-update-1832946-85.patch, failed testing.

plach’s picture

Status: Needs work » Reviewed & tested by the community

Annoying bot

Status: Reviewed & tested by the community » Needs work

The last submitted patch, 85: locale-update-1832946-85.patch, failed testing.

The last submitted patch, 85: locale-update-1832946-85.patch, failed testing.

Gábor Hojtsy’s picture

Looks like legitimate needs work now.

  1. +++ b/core/modules/locale/src/Tests/LocaleUpdateInterfaceTest.php
    @@ -7,6 +7,7 @@
    +use Drupal\Component\Utility\FormattableString;
    
    @@ -87,8 +88,12 @@ public function testInterface() {
    +    $release_details = new FormattableString('@module (@version). @info', [
    +      '@module' => 'Locale test translate',
    +      '@version' => '1.3-dev',
    +      '@info' => t('File not found at %local_path', array('%local_path' => 'core/modules/locale/tests/test.de.po'))
    +    ]);
    

    Test fails due to no FormattableString.

  2. +++ b/core/modules/locale/tests/modules/locale_test_development_release/locale_test_development_release.module
    @@ -0,0 +1,42 @@
    +}
    \ No newline at end of file
    

    No newline.

Wim Leers’s picture

Status: Needs work » Reviewed & tested by the community
FileSize
23.5 KB
2.16 KB

Fixed both points in #107, which are trivial changes, so back to RTBC.

Status: Reviewed & tested by the community » Needs work

The last submitted patch, 108: locale-update-1832946-108.patch, failed testing.

The last submitted patch, 108: locale-update-1832946-108.patch, failed testing.

The last submitted patch, 108: locale-update-1832946-108.patch, failed testing.

The last submitted patch, 108: locale-update-1832946-108.patch, failed testing.

Gábor Hojtsy’s picture

Status: Needs work » Needs review
FileSize
24.13 KB
646 bytes

The language switching test is again a victim of the language adding leading to translation problem.

Wim Leers’s picture

I don't understand how adding the locale_test module fixes the problem?

Gábor Hojtsy’s picture

@wimleers See #82.

function locale_test_locale_translation_projects_alter(&$projects) {
  // Drupal core should not be translated. By overriding the server pattern we
  // make sure that no translation for drupal core will be found and that the
  // translation update system will not go out to l.d.o to check.
  $projects['drupal']['server_pattern'] = 'translations://';

There is also the import_enabled killswitch in locale.settings (testing profile ships with its own locale.settings to kill automated imports) but that does not seem to play nice with the config system when we actually enable locale in the testing environment. (Note that this problem did not pop up before because the code that identified the projects to deal with was dependent on update module, and this patch obviously removes that dependency, so it starts downloading translations where locale is enabled but update is not, but that is a good thing mostly, except some tests :).

Status: Needs review » Needs work

The last submitted patch, 114: 1832946-114.patch, failed testing.

Wim Leers’s picture

Status: Needs work » Reviewed & tested by the community

Thanks!

RTBC if green.

Gábor Hojtsy’s picture

Gábor Hojtsy’s picture

Issue summary: View changes
Gábor Hojtsy’s picture

  • effulgentsia committed bb8be56 on 8.0.x
    Issue #1832946 by Sutharsan, Jose Reyero, Gábor Hojtsy, Wim Leers,...
effulgentsia’s picture

Status: Reviewed & tested by the community » Fixed

Pushed to 8.0.x.

Status: Fixed » Needs work

The last submitted patch, 114: 1832946-114.patch, failed testing.

webchick’s picture

Status: Needs work » Fixed

Grumble.

Gábor Hojtsy’s picture

Issue tags: -sprint

Yay, thanks all!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.