Problem/Motivation

We want contrib modules to be able to define Normalizers, Encoders, Denormalizers and Decorders (NEDD) for different formats. Each of these specify the formats that it supports via a callable, and can also specify other things such as the objects it supports, etc.

The Serializer is instantiated with all of the NEDD that might be used. The Serializer figures out which one to use by iterating through all of the classes in the list and running the supports(N|E|D|D) function (e.g. supportsNormalization($object, $format)). Determining the correct NEDD to use should be left to the Serializer class. However, we need to compile the list of available NEDD to pass into the Serializer at instantiation.

We also need to provide a way to reorder this list. For example, the JsonldNormalizer would support normalization for entities. A contrib module might define a NodeJsonldNormalizer which only supports normalization for nodes. Since the Serializer returns the first matching class, we would want the Serializer to hit the NodeJsonldNormalizer before it hits the JsonldNormalizer.

Proposed resolution

This might be a use case for plugins or might not be. If it is, I believe we would need to use HookDiscovery. As described above, developers need to be able to reorder the list of NEDDs, so we will need an _alter. Unless the alter patten is supported by another plugin discovery method, we should go with HookDiscovery.

The hook implementation would probably look something like the following:

jsonld_serialization_info() {
  array(
    'jsonld_default_normalizer' => array(
      'type' => 'normalizer',
      'class' => 'Drupal\jsonld\Plugin\JsonldNormalizer',
    ),
  );
}

Followups

  1. #1831074: Send 415 Unsupported Media Type http header when serializer encounters an unsupported format
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

klausi’s picture

I think we should use plugins to register all available NEDD components. I still need to wrap my head around the plugin system overall, so I cannot tell if it supports the alter workflow we want. So just go with drupal_alter() for now.

Anonymous’s picture

This patch creates a Serialization module, which then uses the plugin system to find normalizers and encoders.

I'm not sure that this is the correct use of plugins, since we aren't doing any swapping between plugins. We are simply geting the list of Normalizers/Encoders and then passing that list to the Serializer.

The __for-review patch has the Serialization module. The __for-testing shows how the Jsonld module would define its plugins. The logic in JsonldController will eventually be in REST module.

Status: Needs review » Needs work

The last submitted patch, 1814864-02-serialization-module__for-testing.patch, failed testing.

klausi’s picture

I think jslond.module should be a submodule of serialization.module, i.e. serialization_jsonld.module.

Anonymous’s picture

After a discussion with neclimdul, EclipseGC, and Crell, I switched to using the DIC to register the normalizers and encoders. This patch depends on #1706064: Compile the Injection Container to disk and uses a compiler pass to add arrays of referenced normalizers and encoders as arguments for the serializer definition.

I like this approach a lot more because it is simpler on both ends. Serialization module doesn't need to set up a plugin manager and factory, and (more importantly) the API for modules that provide serializations is easier. Since we're not passing any configuration into the normalizers or encoders, I think this is a better solution.

The way serialization modules would register their normalizers and encoders is:

<?php
class JsonldBundle extends Bundle
{
  public function build(ContainerBuilder $container) {
    $container->register('serializer.normalizer.jsonld', 'Drupal\jsonld\JsonldNormalizer')->addTag('normalizer');
    $container->register('serializer.encoder.jsonld', 'Drupal\jsonld\JsonldEncoder')->addTag('encoder');
    $container->register('serializer.normalizer.drupal_jsonld', 'Drupal\jsonld\DrupalJsonldNormalizer')->addTag('normalizer');
    $container->register('serializer.encoder.drupal_jsonld', 'Drupal\jsonld\DrupalJsonldEncoder')->addTag('encoder');
  }
}
?>

This patch doesn't yet allow for the reordering of normalizers and encoders in the list. However, I think I should be able to do that using a 'priority' attribute as chained_matcher does in the compiled DIC patch.

fago’s picture

I'm wondering how the serialization API of symfony could play with the typed data API. I guess, we could use the typed data API as our normalized data representation. That way encoders would have all metadata available and could leverage them as necessary, e.g. to apply the RDF mapping. Makes sense?

Anonymous’s picture

@fago, you might want to take a look at #1811510: Enable JSON-LD entity serialization, which shows how Normalizers and Encoders work together.

I don't think that we can replace the Normalizers. Instead, the Normalizer should expect that typed data comes in, and should know how to handle that typed data. The encoder should be as dumb as possible.

Anonymous’s picture

I added priority handling and basic tests. The tests do not currently cover the new priority handling.

Anonymous’s picture

Forgot to commit my code style changes before diff-ing. Here's a patch with better code style.

Crell’s picture

The general structure in #9 looks good to me. I'm not sure if the intent is to commit it as is and then add stuff to it, or wait until we have more to add, but this looks conceptually sound.

Anonymous’s picture

TBH, I'm not sure whether we're even going to need to add any functionality to this, which does make it kind of strange as a stand alone module. I turned it in to a module based on feedback from the G+ hangout, but I would be just as happy to see it folded in elsewhere.

The only code that I could envision adding before commit is a test for the priority handling functionality.

effulgentsia’s picture

Status: Needs review » Needs work

Code looks good. We don't need this as a module though. We can add the classes to core/lib/Drupal/Core/Serialization and the 'serializer' service registration in CoreBundle.

I'd be fine with either this being committed on its own, or rolled into #1811510: Enable JSON-LD entity serialization.

Some nits:

+++ b/core/modules/serialization/lib/Drupal/serialization/RegisterSerializationClassesPass.php
@@ -0,0 +1,68 @@
+/**
+ * Adds servies tagged 'nested_matcher' services to the tagged_matcher service.
+ */

Copy/paste leftover.

+++ b/core/modules/serialization/tests/modules/serilization_test/lib/Drupal/serialization_test/SerializationTestEncoder.php
@@ -0,0 +1,43 @@
+class SerializationTestEncoder implements EncoderInterface {
+  /**

Add a linebreak between these lines.

+++ b/core/modules/serialization/tests/modules/serilization_test/lib/Drupal/serialization_test/SerializationTestNormalizer.php
@@ -0,0 +1,50 @@
+class SerializationTestNormalizer implements NormalizerInterface {
+  /**

Add a linebreak between these lines.

+++ b/core/modules/serialization/tests/modules/serilization_test/lib/Drupal/serialization_test/SerializationTestNormalizer.php
@@ -0,0 +1,50 @@
+    // If this is an Entity object and the request is for JSON-LD.
+    return static::$format === $format;

Copy/paste leftover of code comment.

klausi’s picture

Additionally:

+++ b/core/modules/serialization/lib/Drupal/serialization/RegisterSerializationClassesPass.php
@@ -0,0 +1,68 @@
+   * Adds services tagged with normalizer or encoder to the Serializer.

normalizer and encoder should have quotes around them?

+++ b/core/modules/serialization/lib/Drupal/serialization/RegisterSerializationClassesPass.php
@@ -0,0 +1,68 @@
+   * The highest priority number is the highest priority (reverse sorting).

Really? This is against the Drupal way of weight priorities where high numbers sink down. Is this necessary because the Symfony serialization stuff expects it this way?

+++ b/core/modules/serialization/lib/Drupal/serialization/RegisterSerializationClassesPass.php
@@ -0,0 +1,68 @@
+   * @param array $array
+   *   A nested array of Normalizers/Encoders.

$array is a really bad variable name. How about $components or similar?

+++ b/core/modules/serialization/lib/Drupal/serialization/RegisterSerializationClassesPass.php
@@ -0,0 +1,68 @@
+    foreach ($array as $a) {
+      $sorted = array_merge($sorted, $a);
+    }

why is that necessary? Why can't you just return $array after krsort()? Please add a comment.

+++ b/core/modules/serialization/lib/Drupal/serialization/SerializationBundle.php
@@ -0,0 +1,28 @@
+    $container->register('serializer', 'Symfony\Component\Serializer\Serializer')
+      ->addArgument(array())
+      ->addArgument(array());

why do you need the empty arrays? The default values of Serializer's constructor are arrays anyway?

+++ b/core/modules/serialization/lib/Drupal/serialization/Tests/SerializationTest.php
@@ -0,0 +1,46 @@
+  public function testSerilizerComponentRegistration() {

I think you should also test the encoder by actually invoking it and comparing the final serialized output.

+++ b/core/modules/serialization/tests/modules/serilization_test/lib/Drupal/serialization_test/SerializationTestEncoder.php
@@ -0,0 +1,43 @@
+  /**
+   * Implementation required. This function not used for testing.
+   *
+   * @param mixed $data
+   *   Data to encode
+   * @param string $format
+   *   Format name
+   */
+  public function encode($data, $format) {

As said, this function should be used for testing. And all parameter descriptions should en with proper punctuation.

Anonymous’s picture

Thanks for the reviews. @effulgentsia, I'll take care of those, and I will take care of the one's @klausi raised that I don't respond to below.

Really? This is against the Drupal way of weight priorities where high numbers sink down. Is this necessary because the Symfony serialization stuff expects it this way?

This is the way that the router handles priority. I would be fine with changing it, since ordering in first, second, and third priority (with first being highest) makes more sense to me. However, if we change it here we really need to change the router too.

why is that necessary? Why can't you just return $array after krsort()? Please add a comment.

Because the array needs to be flattened. Otherwise it would be a nested array, keyed by priority, which Serializer would not know how to handle. I will add a comment.

why do you need the empty arrays? The default values of Serializer's constructor are arrays anyway?

I'll test it again, but I believe that is necessary for the replaceArgument call in the compiler pass.

I think you should also test the encoder by actually invoking it and comparing the final serialized output.

This isn't meant to test the normalizer or the encoder, just that they are registered, which is why it tests with supportsEncoding() rathe than encode(). If others agree, though, I will change the test to use encode().

klausi’s picture

Thinking more about the term "priority": higher numbers mean higher priority, which is probably fine. So if we reverse that we should also rename "priority" to "weight". I think that is follow-up stuff, so just ignore that for now.

Ah, I see the replaceArgument() calls now, so better also add a comment to the empty addArgument() calls if they are required.

Regarding the test: right, so calling one method on the encoder suffices to make sure it is there. Never mind my comment about that then.

Anonymous’s picture

I've made the changes from the comments. I plan to change this to be a part of CoreBundle, so I'm going to leave this as needs work until I upload that patch.

effulgentsia’s picture

Regarding Symfony's convention of reverse sorting by priority vs. Drupal's convention of sorting by weight: #1825124: Provide an Drupal\EventSubscriber override that uses natural weights. Until that issue is resolved, I agree with doing what this patch does: for event listeners and things sorted in compiler passes, following Symfony's convention rather than Drupal's.

Anonymous’s picture

Status: Needs work » Needs review
FileSize
10.79 KB

Folded everything into CoreBundle and system.

effulgentsia’s picture

This isn't meant to test the normalizer or the encoder, just that they are registered, which is why it tests with supportsEncoding() rathe than encode(). If others agree, though, I will change the test to use encode().

I know there's some unit testing purists out there, but I think it's clearer to test normalize() and encode() rather than normalize() and supportsEncoding(), so I made that change here, along with some other minor stuff. If there's no objections to that, I think this can be RTBC.

effulgentsia’s picture

This one simplifies the test even further to just serialize(). That's the API that matters to us. Normalizers and encoders are implementation details that I don't think need to be tested individually, since the test can only pass if all is working properly, and if there's a failure, it's not that hard to step through a debugger to find out where it is. The interdiff is relative to #18.

Anonymous’s picture

Status: Needs review » Needs work
+++ b/core/modules/system/lib/Drupal/system/Tests/Serialization/SerializationTest.phpundefined
@@ -0,0 +1,61 @@
+    // Ensure the serialization fails for an unsupported format.
+    try {
+      $this->serializer->serialize($object, 'unsupported_format');
+      $this->fail('The serializer was expected to throw an exception for an unsupported format, but did not.');
+    }
+    catch (UnexpectedValueException $e) {
+    }

We should actually return 415 Unsupported Media Type in this case.

If we're OK with that happening in a follow up patch, then I'd say this is RTBC.

Anonymous’s picture

Status: Needs work » Reviewed & tested by the community

Actually, I believe the 415 will be handled by REST module, so this is fine.

webchick told me to RTBC if I was ok with Alex's changes, so here we go.

moshe weitzman’s picture

Code and approach look good to me too. I agree that 415 header can be a followup.I opened #1831074: Send 415 Unsupported Media Type http header when serializer encounters an unsupported format and added a link to it in the Issue Summary.

klausi’s picture

+1 from me as well. Code and comments look good.

webchick’s picture

Status: Reviewed & tested by the community » Fixed

We're slightly over thresholds atm, but this patch holds up other important stuff, and there were very smart people sprinting on it all throughout BADCamp, so...

Committed and pushed to 8.x. Thanks!

catch’s picture

Title: Provide way to register serialization classes » Change notice: Provide way to register serialization classes
Category: feature » task
Priority: Normal » Critical
Status: Fixed » Active

Just saw this when I git pulled:

b/core/modules/system/tests/modules/serilization_test/lib/Drupal/serialization_test/SerializationTestBundle.php

s/serilization/serialization/

could we get a quick patch to fix that?

Also doesn't this need a change notice?

Anonymous’s picture

Status: Active » Needs review
FileSize
11.44 KB

Just did a git mv to rename the folder.

Anonymous’s picture

Change record drafted at: http://drupal.org/node/1837872

chx’s picture

There is a really little problem about the change notice: I do not have the slightest clue what are you talking about if I am just reading that notice. It badly needs a sentence or two about it is for turning an entity into JSON-LD or whatnot -- I think that's what this is about, isnt it? "One Serializer is added as a service to the DIC. This is used by controllers to process entities or arrays to their serialized equivalent." -- and what a "serialized equivalent" would be? Why do I want one, where do I use one, etc. Not to mention that 'controller' is a word noone understands yet (link to the router change notice?). This whole change notice is an enigma.

Anonymous’s picture

Thanks for the feedback, I addressed it by adding a description of what benefit serializations provide in the first paragraph and adding examples of a controller that uses a serialization and what a serialized equivalent is. The function I cite as an example hasn't been added to core yet, but is in the patch in #1834288: RESTfully request an entity with JSON-LD serialized response.

chx’s picture

Title: Change notice: Provide way to register serialization classes » Provide way to register serialization classes
Category: task » feature
Priority: Critical » Normal
Status: Needs review » Fixed

Lot better, thanks.

Anonymous’s picture

Status: Fixed » Needs review

Great! I'm going to switch this back to needs review pending commit of the patch in #27.

effulgentsia’s picture

Status: Needs review » Reviewed & tested by the community
FileSize
2.21 KB

Same as #27, but with renames = copies.

webchick’s picture

Status: Reviewed & tested by the community » Fixed

Committed and pushed that follow-up. Thanks!

Automatically closed -- issue fixed for 2 weeks with no activity.

Anonymous’s picture

Issue summary: View changes

Updated issue summary.