Problem/Motivation

If one of the *.info files in a module, a theme or a profile is encoded in something other than UTF-8 or contains invalid byte-sequences, the 'Verify' task will fail with a bunch of warnings:

and it will be impossible to add new sites to the broken platform because Aegir won't have any information about the available install profiles:

Steps to reproduce

What happens is this:

  1. a hosting-task Drush command will be dispatched when a user clicks 'Verify' button in the Aegir UI
  2. a new Drush process will be invoked in the backend to execute provision-verify command
  3. provision module will look for all available modules, profiles and themes on a platform
  4. provision_parse_info_file() function will parse info-files and keep any values as-is without modifications
  5. when all the work in provision-verify is done, the list of discovered packages has to be communicated between two Drush processes, so the data has to be serialized
  6. to do that, _scrub_object() function is called on the list of packages and it looks like this:
    function _scrub_object($input) {
      return json_decode(json_encode($input), TRUE);
    }
    
  7. if $inputcontains invalid UTF-8 sequences, json_encode() will return FALSE
  8. as the result, Aegir does not see any packages on the platform.

Proposed resolution

provision_parse_info_file() should strip any invalid UTF-8 sequences from parsed values. There is no better place to do this.

Remaining tasks

  • Find a way to check encoding when 'mbstring' extension is not available.
  • Tests maybe?

User interface changes

None.

API changes

None.

Data model changes

None.

Comments

20th created an issue. See original summary.

20th’s picture

You may argue: 'But everybody always use UTF-8 in 2017!' I thought so as well, and yet I've just spent two days worth of time figuring out why can't I install any sites on a new platform…

20th’s picture

Issue summary: View changes
Status: Active » Needs review
StatusFileSize
new2.29 KB
20th’s picture

StatusFileSize
new1.36 KB

Huh, the UNICODE_MULTIBYTE constant is not available in provision module because Drupal is not fully bootstrapped, my bad.

helmo’s picture

My first though is ... can't the source of this invalid encoded string be fixed....

But I can see how this is a practical solution.
Do you have a link to an example I can use to test this?

20th’s picture

StatusFileSize
new522 bytes

Sure, just add this module to any platform in Aegir and run 'Verify' task.

You mean fix this problem by actually resaving all files in UTF-8? Sure, it will fix it. The thing is, there is no way to tell that you have an encoding problem - all you see as is a bunch of unrelated warnings. It took me two days of digging in Aegir and Drush source to trace this back to a json_encode() call.

helmo’s picture

Status: Needs review » Reviewed & tested by the community

Tested .. it does the trick :)

  • helmo committed 0430fa6 on 7.x-3.x authored by 20th
    Issue #2850197 by 20th: 'Verify' task fails when module *.info file...
helmo’s picture

Status: Reviewed & tested by the community » Fixed

Committed, thanks.

20th’s picture

Thanks, @helmo!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.