Flatten Schema References During Schema Loading

Problem/Motivation

Currently, pattern schemas are loaded and undergo a "dereferencing" process to normalize all references to other schemas using namespace references and allow them to be loaded successfully from an API endpoint. The result of this converts "$ref" values in a schema from a namespaced value like @patternkit/atoms/example/src/example to a local link through the Patternkit API controller like /patternkit/api/patternkit/atoms/example/src/example instead. This allows schema processing to request the schema for the referenced pattern at runtime and populate it in place where the reference is. This is used both in the JSON Editor form as the form is being built, and in the server-side schema traversal validation and traversal logic where the schema has to be fully resolved.

When patterns become heavily interrelated and/or nested with lots of references to other patterns, you encounter issues with a great deal of processing or requesting required to fully resolve the schema. An example of this is described by #3268970: Fetching schemas takes very long time.

If, instead, schema references were resolved server-side when the schema is read in initially, the fully-resolved schema could be loaded in all uses and further requests and loading would not be required. This would also allow for caching of the full schema rather than just the top-level pattern without the contents of lower-level referenced schemas. As it exists now, patterns used in blocks have their schemas cached in Patternkit Pattern entities, but these cached schemas contain the normalized reference links instead of the resolved schema values, so changes to these referenced pattern schemas may result in breaking changes to the uses of this top-level pattern without manual refreshing of the cached data.

Proposed resolution

When a pattern is first parsed and the schema asset is loaded, the Swaggest library could be used to import the schema fully and then serialize the result for caching. This sidesteps the currently implemented complex traversal logic for replacing reference links, and incorporates the fully resolved schema, including sub-schemas, into all uses of the pattern.

This solution was initially implemented as part of the work in #3295824: Library Service Refactor and Cleanup, but as described in #3295824-8: Library Service Refactor and Cleanup patterns resolving with circular references lead to an infinite recursion and eventual failure. As a result, the changes for this feature were removed from that work, but the drafted change record still exists here.

The best place to resolve this may actually be in the underlying Swaggest library, so one of the early steps for this issue should be summing up the issue and reporting it there.

Remaining tasks

• Peer review
• Additional testing for updating on existing sites
• Add missing tests for new classes
• Document new classes and change records
• Browser tests covering updating existing pattern blocks

User interface changes

• All patterns containing references are having their loaded schemas updated to use bundled content. This results in existing block placements for them being flagged as needing schema updates when they are next edited.
• Patterns including a lot of references may currently take a long time to load the JSON Editor form as it requires all remote schemas to be loaded. Newly created blocks using these patterns or these blocks after updating the schemas should now load much more quickly.

API changes

See change record for more detail: https://www.drupal.org/node/3303797

New services

• patternkit.pattern.dependency_resolver: A new service to identify dependencies in patterns and load them in the most efficient order.
• patternkit.pattern.repository: A new service for loading pattern entities from their definitions and creating bundled schemas containing all referenced dependencies.
• patternkit.schema.context_builder: A new service split out from the recently introduced patternkit.schema.schema_factory service. This service is responsible for assembling and configuring Schema operation contexts consistently with necessary default options.
• patternkit.schema.schema_handler: A utility service for processing and manipulating references discovered within pattern schemas.
• patternkit.schema.data_preprocessor.factory: A factory service for instantiating schema DataPreProcessor utilities used in schema contexts for a variety of processing tasks.

New Way to Instantiate Patterns

Previously, the most direct path to loading a pattern and getting the schema was to use the \Drupal\patternkit\Entity\Pattern::create() method. The new patternkit.pattern.repository service provides a method to get the pattern entity directly and should be used for this instead:

$pattern = \Drupal::service('patternkit.pattern.repository')->getPattern('@patternkit/atoms/example/src/example');
$schema = $pattern->getSchema();

Loading a bundled schema

If a pattern's schema is needed with all dependencies included, the patternkit.pattern.repository service may also be used for this.

$schema = \Drupal::service('patternkit.pattern.repository')->getBundledPatternSchema('@patternkit/atoms/example/src/example');
$schema_json = json_encode($schema);

Listing all discovered patterns

If only a list of discovered patterns is needed, it is no longer necessary to load and iterate through all discovered pattern definitions. The patternkit.pattern.repository service provides a new method for retrieving a list of only the names of all known patterns:

$patternRepository = \Drupal::service('patternkit.pattern.repository');
foreach ($patternRepository->getAllPatternNames() as $pattern) {
  // Do something with each pattern.
}

Data model changes

New pattern entity dependencies field

An unlimited cardinality field on the patternkit_pattern entity populated during pattern discovery with references discovered immediately within a pattern's schema file.

This list of dependencies may be retrieved from an instantiated pattern entity with the new $pattern->getDependencies() method. Note this field is only populated with dependencies referenced directly within the pattern's schema. To identify any further dependencies referenced from those dependencies the new patternkit.pattern.dependency_resolver service may be used.

Testing Instructions

Patch-based workflow

Since this change introduces new Composer dependencies, testing it strictly in a patch-based workflow requires additional steps.

This work is also based off of the current work in the 9.1.x branch which includes work beyond the most recent release (beta4). To account for this and ensure the patch will apply, an existing installation of Patternkit will need to be updated to the latest dev release.

composer require drupal/patternkit:dev-9.1.x

You'll need to add the following to the patches section of your composer.json, but you will also need to manually require the dependencies listed below.

{
  ...
  "extra": {
    "patches": {
      "drupal/patternkit": {
        "#3297124: Flatten Schema References During Schema Loading": "https://git.drupalcode.org/project/patternkit/-/merge_requests/39/diffs.patch"
      }
    }
  }
}

You will also need to manually add these dependencies:

composer require marcj/topsort:\^2.0.0 symfony/polyfill-php80:\^1 symfony/polyfill-php81:\^1 webmozart/assert:\^1.0

Custom Git Repository

Alternatively, you may specify the fork repository directly to install from as shown below.

• The custom package definition must be added above the standard packages.drupal.org composer reference. This ensures the package reference for "drupal/patternkit" is loaded from the fork first.
• You'll need to change the required version of the patternkit module to dev-3297124-flatten-schema-references.
• After making these changes, you may install the fork with the command composer require drupal/patternkit:dev-3297124-flatten-schema-references.

{
    ...
    "repositories": [
        {
            "type": "package",
            "package": {
                "name": "drupal/patternkit",
                "type": "drupal-module",
                "version": "dev-3297124-flatten-schema-references",
                "source": {
                    "url": "https://git.drupalcode.org/issue/patternkit-3297124.git",
                    "type": "git",
                    "reference": "3297124-flatten-schema-references"
                },
                "require": {
                    "php": ">=7.4.0",
                    "ext-json": "*",
                    "marcj/topsort": "^2.0.0",
                    "swaggest/json-schema": "^0.12",
                    "symfony/finder": "^3.4 || ^4.0",
                    "symfony/polyfill-php80": "^1",
                    "symfony/polyfill-php81": "^1",
                    "symfony/yaml": "^3.4 || ^4.0",
                    "webmozart/assert": "^1.0",
                    "webmozart/path-util": "^2.1.0"
                }
            }
        },
        {
            "type": "composer",
            "url": "https://packages.drupal.org/8"
        },
        ...
    ],
    ...
    "require": {
        ...
        "drupal/patternkit": "dev-3297124-flatten-schema-references",
        ...
    },
    ...
}