Hi @all,

Problem/Motivation

The problem I have is, that backups made by this module don't restore emoji unicode characters. After restoring a backup it's displayed as ??? e.g. inside a body field.

Steps to reproduce

The Database of my Drupal 8.7.3 supports collation utf8mb4_unicode_ci

  • Insert in a body field a glyph of an emoji (Example)
  • Create a backup with Backup and Migrate
  • Restore the backup with Backup and Migrate
  • At the place where the emoji should be displayed, there's only ???? shown
CommentFileSizeAuthor
#8 backup_migrate-n3065017-8-5.x.patch1.04 KBDamienMcKenna
#8 backup_migrate-n3065017-8-4.x.patch1.12 KBDamienMcKenna
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

dercheffe created an issue. See original summary.

dercheffe’s picture

dercheffe
he/him
Primary language German
Location Sersheim, Germany
CreditAttribution: dercheffe commented
Issue summary: View changes
ikit-claw’s picture

ikit-claw CreditAttribution: ikit-claw as a volunteer commented

What exactly are you using to add emoji's? Or is this just drupal core?

dercheffe’s picture

dercheffe
he/him
Primary language German
Location Sersheim, Germany
CreditAttribution: dercheffe commented

In this case I pasted it for testing into a body field directly, only using core's CKEditor.

In general it works, until a backup is restored via BAM.

But I also observed the same behavior earlier using CKEditor Emojione module on another site (but it has a bug currently, so its not usable in general for now).

dercheffe’s picture

dercheffe
he/him
Primary language German
Location Sersheim, Germany
CreditAttribution: dercheffe commented

Okay, following I observed in the meanwhile:

  • The same error occurs, when I create a backup with the drush sql:dump command. I don't know if BAM code relies on the same code base.
  • Using mysqldump command for creating the backup (not with BAM) without any parameter in my shell also brings the same error result
  • Using mysqldump command with --default-character-set=utf8mb4 it's all working as it should. Emojis are still there after restore of the backup

Found this here: https://stackoverflow.com/questions/20216849/mysqldump-with-utf8-can-not-export-the-right-emojis-string

When I analyze the BAM MySQL dump I've observed following:
SET NAMES utf8; although there should be a utf8mb4_unicode_ci?

Promo-IL’s picture

Promo-IL CreditAttribution: Promo-IL commented

+1
I've done /backup_migrate/lib/backup_migrate_core/src/Source/MySQLiSource.php

/*SET NAMES utf8;*/
SET NAMES utf8mb4;
DamienMcKenna’s picture

DamienMcKenna
Location NH, USA
CreditAttribution: DamienMcKenna at Mediacurrent commented
Version: 8.x-4.0 » 8.x-4.x-dev
Priority: Normal » Critical
Parent issue: » #3102502: Plan for Backup and Migrate 8.x-4.2

Oh, that's a great find, thank you Promo-IL! Bumping this to critical because it causes data problems.

To anyone who wants to fix their backups, just change the line "SET NAMES utf8" in the SQL file to "SET NAMES utf8mb4" and you should be able to reimport the database correctly.

DamienMcKenna’s picture

DamienMcKenna
Location NH, USA
CreditAttribution: DamienMcKenna at Mediacurrent commented
Status: Active » Needs review
FileSize
backup_migrate-n3065017-8-4.x.patch1.12 KB
backup_migrate-n3065017-8-5.x.patch1.04 KB

Ok, this should resolve it for new files, existing backup files will need to have that line in the output changed.

  • DamienMcKenna committed 94db1e2 on 5.0.x 
    Issue #3065017 by DamienMcKenna, dercheffe, Promo-IL: Restore emoji...

  • DamienMcKenna committed a0957ae on 8.x-4.x 
    Issue #3065017 by DamienMcKenna, dercheffe, Promo-IL: Restore emoji...
DamienMcKenna’s picture

DamienMcKenna
Location NH, USA
CreditAttribution: DamienMcKenna at Mediacurrent commented
Title: restore emoji unicode characters not possible » Restore emoji unicode characters not possible
Status: Needs review » Fixed

Committed.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.