Drupal 11 compatibility

I've updated method signatures to align with upstream interfaces. But now I'm seeing:

InvalidArgumentException: Property entity is unknown. in Drupal\Core\TypedData\TypedDataManager->getPropertyInstance() (line 204 of core/lib/Drupal/Core/TypedData/TypedDataManager.php). Drupal\Core\TypedData\Plugin\DataType\Map->get() (Line: 50)
Drupal\content_sync\Normalizer\EntityReferenceFieldItemNormalizer->normalize() (Line: 152)
Symfony\Component\Serializer\Serializer->normalize() (Line: 24)
Drupal\serialization\Normalizer\ListNormalizer->normalize() (Line: 152)
Symfony\Component\Serializer\Serializer->normalize() (Line: 25)
Drupal\serialization\Normalizer\ContentEntityNormalizer->normalize() (Line: 92)
Drupal\content_sync\Normalizer\ContentEntityNormalizer->normalize() (Line: 148)
Drupal\content_sync\Normalizer\FileEntityNormalizer->normalize() (Line: 152)
Symfony\Component\Serializer\Serializer->normalize() (Line: 131)
Symfony\Component\Serializer\Serializer->serialize() (Line: 36)
Drupal\content_sync\Exporter\ContentExporter->exportEntity() (Line: 133)
Drupal\content_sync\Form\ContentExportForm->processContentExportFiles() (Line: 297)

As far as I can tell, here it's misinterpreting the "id" field (integer) as an entity reference field, and failing to load the referenced entity.

@star-szr pointed out that, in content_sync.services.yaml, the content_sync.normalizer.entity_reference_field_item service points to #2575761: Discuss a better system for discovering and selecting normalizers. That ticket notes that:

[...] relying on the priority to just sort the services and pick the first match is very problematic. You need to get one weight wrong and everything explodes completely if you end up with a combination of normalizers or denormalizers that don't play nicely together.

We tried commenting out that service, and then see:

Call to undefined method Drupal\Core\Field\Plugin\Field\FieldType\IntegerItem::getUrl() in Drupal\content_sync\Normalizer\LinkItemNormalizer->normalize() (line 57 of modules/contrib/content_sync/src/Normalizer/LinkItemNormalizer.php). Symfony\Component\Serializer\Serializer->normalize() (Line: 24)
Drupal\serialization\Normalizer\ListNormalizer->normalize() (Line: 152)
Symfony\Component\Serializer\Serializer->normalize() (Line: 25)
Drupal\serialization\Normalizer\ContentEntityNormalizer->normalize() (Line: 92)
Drupal\content_sync\Normalizer\ContentEntityNormalizer->normalize() (Line: 148)
Drupal\content_sync\Normalizer\FileEntityNormalizer->normalize() (Line: 152)
Symfony\Component\Serializer\Serializer->normalize() (Line: 131)
Symfony\Component\Serializer\Serializer->serialize() (Line: 36)
Drupal\content_sync\Exporter\ContentExporter->exportEntity() (Line: 133)
Drupal\content_sync\Form\ContentExportForm->processContentExportFiles() (Line: 297)

Here's it's trying to use the LinkItemNormalizer on an IntergerItem.

I implemented the change from $supportedInterfaceOrClass to ::getSupportedTypes() in Normalizer classes, with promising results.

The nature of the error has now changed:

Error: Undefined constant Drupal\Core\Database\Database::RETURN_AFFECTED in Drupal\content_sync\Content\ContentDatabaseStorage->cs_doWrite() (line 44 of modules/contrib/content_sync/src/Content/ContentDatabaseStorage.php). Drupal\content_sync\Content\ContentDatabaseStorage->cs_write() (Line: 149)
Drupal\content_sync\Form\ContentExportForm->processContentExportFiles() (Line: 297)

Looks like this is the relevant change record: https://www.drupal.org/node/3185520

Removing the deprecated 'return' query option resolved that issue and appeared to allow a bunch of exports, before running into:

Drupal\Core\File\Exception\InvalidStreamWrapperException:  in Drupal\Core\File\FileUrlGenerator->doGenerateString() (line 106 of core/lib/Drupal/Core/File/FileUrlGenerator.php). Drupal\Core\File\FileUrlGenerator->generateString() (Line: 32)
Drupal\file\ComputedFileUrl->getValue() (Line: 17)
Drupal\serialization\Normalizer\TypedDataNormalizer->normalize() (Line: 152)
Symfony\Component\Serializer\Serializer->normalize() (Line: 37)
Drupal\serialization\Normalizer\ComplexDataNormalizer->normalize() (Line: 152)
Symfony\Component\Serializer\Serializer->normalize() (Line: 24)
Drupal\serialization\Normalizer\ListNormalizer->normalize() (Line: 152)
Symfony\Component\Serializer\Serializer->normalize() (Line: 25)
Drupal\serialization\Normalizer\ContentEntityNormalizer->normalize() (Line: 92)
Drupal\content_sync\Normalizer\ContentEntityNormalizer->normalize() (Line: 151)
Drupal\content_sync\Normalizer\FileEntityNormalizer->normalize() (Line: 152)
Symfony\Component\Serializer\Serializer->normalize() (Line: 131)
Symfony\Component\Serializer\Serializer->serialize() (Line: 36)
Drupal\content_sync\Exporter\ContentExporter->exportEntity() (Line: 133)
Drupal\content_sync\Form\ContentExportForm->processContentExportFiles() (Line: 297)

This latest issue appears to be due to an unconfigured private filesystem, which is the default in DDEV. Configuring it in settings.php allowed all of the files to be written:

$settings['file_private_path'] = '../private';

Perhaps it'd be worth adding a check for this setting in Drupal\serialization\Normalizer\ContentEntityNormalizer::normalize(), before trying to write to the private filesystem?

This allowed the batch process to complete, but then result in a WSOD:

TypeError: Drupal\Component\Utility\Html::escape(): Argument #1 ($text) must be of type string, null given, called in /var/www/html/.local-dev/web/core/lib/Drupal/Component/Render/FormattableMarkup.php on line 238 in Drupal\Component\Utility\Html::escape() (line 431 of core/lib/Drupal/Component/Utility/Html.php).
Drupal\Component\Render\FormattableMarkup::placeholderEscape() (Line: 187)
Drupal\Component\Render\FormattableMarkup::placeholderFormat() (Line: 195)
Drupal\Core\StringTranslation\TranslatableMarkup->render() (Line: 15)
Drupal\Core\StringTranslation\TranslatableMarkup->__toString()
strpos() (Line: 16)
Drupal\Core\Logger\LogMessageParser->parseMessagePlaceholders() (Line: 63)
Drupal\content_sync\Logger\ContentSyncLog->log() (Line: 127)
Drupal\Core\Logger\LoggerChannel->log() (Line: 50)
Drupal\Core\Logger\LoggerChannel->error() (Line: 276)
Drupal\content_sync\Form\ContentExportForm->finishContentExportBatch() (Line: 458)

That last error looks like a false positive. I had previously commented-out the initial snapshot taken in hook_install(). Re-enabling that and re-installing the module has resulted in a successful export.

The "Synchronize" tab no longer shows any deleted items, so that's encouraging too.

I guess testing importing would be the logical next step.

Running imports via Drush, I got:

Unable to decode output into JSON: Syntax error
  TypeError: Drupal\content_sync\Normalizer\FileEntityNormalizer::denormalize  
  (): Return value must be of type ArrayObject|array|string|int|float|bool|nu  
  ll, Drupal\file\Entity\File returned in Drupal\content_sync\Normalizer\File  
  EntityNormalizer->denormalize() (line 144 of /var/www/html/.local-dev/web/m  
  odules/contrib/content_sync/src/Normalizer/FileEntityNormalizer.php).

This was fixed pretty easily by allowing file entities to be returned from FileEntityNormalizer::denormalize().

I got this error when trying to import:

  Unable to decode output into JSON: Syntax error                                                                                                  
                                                                                                                                                   
  TypeError: Drupal\Component\Utility\Html::escape(): Argument #1 ($text) mus                                                                      
  t be of type string, null given, called in /var/www/html/.local-dev/web/cor                                                                      
  e/lib/Drupal/Component/Render/FormattableMarkup.php on line 238 in Drupal\C                                                                      
  omponent\Utility\Html::escape() (line 431 of /var/www/html/.local-dev/web/c                                                                      
  ore/lib/Drupal/Component/Utility/Html.php).

Which really is not terribly useful. The stacktrace doesn't mention any code in content_sync at all...

I was eventually able to import all of our content by specifying all of the entity types in a particular order:

drush content-sync:import --yes --skiplist --entity-types=user,file,media,node,taxonomy_term,commerce_store,commerce_shipping_method,commerce_product_variation,path_alias,block_content,commerce_product

So perhaps there's something wrong with the dependency graph, or whatever is used to determine the order of import.

At any rate, this is working for our use case. We will continue to use and test it as we proceed with the project. I'll provide updates here if/when I run into any other issues. However, I'm going to set this issue to "Needs work", so others can weigh in.

Here's a patch of the MR branch that should work against the most recent release on the 4.0.x branch.

We've been successfully using this in several D11 projects for a while now. Marking as "Needs review" to get more eyes on this.

That said, testing this can be bit tricky, since we need the project's composer.json to support D11. You can use the issue fork repo and branch like so:

"repositories": [
    {
        "type": "vcs",
        "url": "https://git.drupalcode.org/issue/content_sync-3523475.git"
    },
[...]
"require-dev": {
    "drupal/content_sync": "dev-3523475-drupal-11-compatibility",
[...]

This is currently flagged as applying to 4.0.x-dev. However, we're currently at 4.0.0-RC2, which is the recommended release. Would we want to move to a 5.0.x branch for this?

I created a 5.0.x branch, and a 5.0.x-dev release, for easier testing and evaluation of the fixes in this issue.