This issue is about considering to use or not Doctrine ORM(Object-Relational Mappers) for entity mapping to database schema.

Problem/Motivation

We have built over the past home made Drupal ORM for Entity management. Right now, we are thinking about replacing hook_entity_info() arrays with annotations, requiring entities to be plugins. We should use the plugin system sparingly as it will set the tone for community.

Drupal as of version 6, implements an ORM. Hooks can add data to nodes from anywhere they want, whether it maps to a database table or not. However, the ORM is "naked": there is no abstraction on data access. Everything is loaded at once into a single big blob data that gets returned from node_load().

Drupal as of version 7 improved a lot this. The Entity API matches the entity structure to the relational database and adds an abstracted layer for CRUD operations, but, the loading of an entity is still not optimized.

Drupal in its 8-dev version changed the entity_info() arrays into annotations, so this is a good step and a good direction. Entities are currently re-factored against the plugin system, using a EntityManager to govern persistence. It is exactly how Doctrine ORM actually work.

Because an ORM is more robust, it can do a lot more optimization under the hood. Complex relationships can be modeled in a database-independent way, even tying to non-SQL systems like the file system, Services, etc.

  • The Entity system is really complex, you have to first extend a base class or implement the EntityInterface.
  • Entities themselves contains reference to controllers, via annotations, this should not appear in a data object.
  • Annotations are not used on a per-field basis, but as a meta-data stack which helps the entity manager to understand how to deal with the Entity.

Refactored solution

  • Does not need to extend a base class or to implement an interface.
  • Generates Physical Data Model from classes themselves.
  • No need for Schema API anymore in the long run.
  • Uses annotations the correct way.
  • Have a meta-data caching system included.

Entities are generated and looks like the following:

<?php

namespace Drupal\entity;

use Doctrine\ORM\Mapping as ORM;

/**
 * Node
 *
 * @ORM\Table(name="node")
 * @ORM\Entity
 */
class Node
{
  /**
   * @var integer
   *
   * @ORM\Column(name="nid", type="integer", nullable=false)
   * @ORM\Id
   * @ORM\GeneratedValue(strategy="IDENTITY")
   */
  protected $nid;

  /**
   * @var \Drupal\entity\User
   *
   * @ORM\ManyToOne(targetEntity="Drupal\entity\User")
   * @ORM\JoinColumns({
   *   @ORM\JoinColumn(name="uid", referencedColumnName="uid")
   * })
   */
  protected $user;
}
?>
<?php

namespace Drupal\entity;

use Doctrine\ORM\Mapping as ORM;

/**
 * User
 *
 * @ORM\Table(name="users")
 * @ORM\Entity
 */
class User
{
  /**
   * @var integer
   *
   * @ORM\Column(name="uid", type="integer", nullable=false)
   * @ORM\Id
   * @ORM\GeneratedValue(strategy="IDENTITY")
   */
  protected $uid;

  /**
   * @var string
   *
   * @ORM\Column(name="name", type="string", length=60, nullable=false)
   */
  protected $name;

  /**
   * @var string
   *
   * @ORM\Column(name="pass", type="string", length=128, nullable=false)
   */
  protected $pass;

}
?>

Remaining tasks

Decide if we want to go in this direction or not.

Resources

Comments

Sylvain Lecoy’s picture

Issue summary: View changes

Detailed 'arrays'.

pounard’s picture

+1

pounard’s picture

Issue summary: View changes

detailed 'do this'.

Crell’s picture

Status: Active » Postponed

I stumbled across this via Google in response to a twitter discussion. Bizarre... :-)

You forgot a link:

http://www.garfieldtech.com/blog/orm-vs-query-builders

This is also not even remotely close to possible for Drupal 8. You're talking about a wholesale replacement of the entity system. We're already most of the way through doing that in Drupal 8; doing it a second time with a month to feature freeze is not worth discussing.

There's no "9.x" version option, so marking postponed. Given how different Entity API and Doctrine are in their design (from what I understand of Doctrine, which is admittedly only cursory), it's probably a won't fix.

webchick’s picture

I've been doing this for D9 issues.

webchick’s picture

Issue summary: View changes

typo

tim.plunkett’s picture

Version: 8.x-dev » 9.x-dev
Sylvain Lecoy’s picture

Experimental work has started here: http://drupal.org/project/doctrine on the 7.x-1.x branch.

I implemented a Driver which from the Schema API + Entity API can translate to Doctrine meta data API. I am able to load a user entity from the doctrine entity manager created through a dependency injection container in Drupal 7.

You might check the code out as I am working on it. Now trying to complete the Schema + Entity API Driver for doctrine for further integration with ToMany relationships and 'foreign keys' column analysis.

geek-merlin’s picture

hot stuff. but surely the "orm as vietnam" paper is a must-read:
http://blogs.tedneward.com/2006/06/26/The+Vietnam+Of+Computer+Science.aspx

pounard’s picture

Very interresting article! Thanks

Sylvain Lecoy’s picture

Yes it is the problem of ORM, even if it was written back in 2006 this article is still valid.

But this issue is about creating and maintaining our own vietnam API versus using an existing vietnam API. :-)

Sylvain Lecoy’s picture

Some updates here: I was able to correctly translates the drupal_schema() to a conceptual data model understandable by Doctrine, thus the DatabaseDriver is now capable of reverse engineer the conceptual model to generates entities.

I am currently blocked by the way search module is conceptually defining relationships and getting this error:
Doctrine\ORM\Mapping\MappingException: It is not possible to map entity 'SearchDataset' with a composite primary key as part of the primary key of another entity 'SearchIndex#sid'.

It is because search_dataset table defines a composite primary key (sid, type), which is a subset of the primary key defined by search_index (word, sid, type). The foreign key defined by search_index (sid, type) is a part of the primary key, and it cause troubles as doctrine does not know how to map these entities together.

There is one solution which is to ignore this table, waiting for an upstream fix of the schema to fit doctrine's expectations, or to use a driver that I was writing - the Schema API Driver - which maps only entities tables and try to reverse engineer only tables from declared entities. This approach has the following drawback, entities must be declared through Entity API to be recognized by Doctrine.

The problem I encountered also is from a physical PoV (e.g. from a database perspective) Doctrine were unable to differentiate a OneToOne (bidirectional) relationship from a ManyToOne (unidirectional) relationship. They both are translated physically by the following:

User
id
address_id (FK)
Address
id

What is expressed is: "This entity has a property that is a reference to an instance of another entity". In other words, using a ManyToOne is the way to map OneToOne foreign key associations: the relation is treated like a toOne association without the Many part.

For the OneToOne (unidirectional) relationship its easier: the foreign key is constrained by a UNIQUE property.

Sylvain Lecoy’s picture

Long story short: Drupal Schema API cannot be translated as a CDM understandable for doctrine to generate entities.

What I propose here is to declare only the entities which will be managed through doctrine, that is for instance, entities declared through Entity API.

Sylvain Lecoy’s picture

Generated entities are now working:

You might check the result here: http://drupalcode.org/project/doctrine.git/tree/HEAD:/modules/entity/lib....

Here is a subset of the User object:

<?php

namespace Drupal\entity;

use Doctrine\ORM\Mapping as ORM;

/**
 * User
 *
 * @ORM\Table(name="users")
 * @ORM\Entity
 */
class User
{
  /**
   * @var integer
   *
   * @ORM\Column(name="uid", type="integer", nullable=false)
   * @ORM\Id
   * @ORM\GeneratedValue(strategy="IDENTITY")
   */
  protected $uid;

  /**
   * @var string
   *
   * @ORM\Column(name="name", type="string", length=60, nullable=false)
   */
  protected $name;

  /**
   * @var string
   *
   * @ORM\Column(name="pass", type="string", length=128, nullable=false)
   */
  protected $pass;
}
?>

The following code has been generated tahnks to the EntityAPIDriver, which for each entities defined through entity API, walk through the tables defined by the Schema API and generate proper fields and relationships reading the foreign keys meta-data.

Here is the interesting part in the Node entity: the relation to its author.

<?php

namespace Drupal\entity;

use Doctrine\ORM\Mapping as ORM;

/**
 * Node
 *
 * @ORM\Table(name="node")
 * @ORM\Entity
 */
class Node
{
  /**
   * @var integer
   *
   * @ORM\Column(name="nid", type="integer", nullable=false)
   * @ORM\Id
   * @ORM\GeneratedValue(strategy="IDENTITY")
   */
  protected $nid;

  /**
   * @var \Drupal\entity\User
   *
   * @ORM\ManyToOne(targetEntity="Drupal\entity\User")
   * @ORM\JoinColumns({
   *   @ORM\JoinColumn(name="uid", referencedColumnName="uid")
   * })
   */
  protected $user;

}
?>

The variable name 'user' is inferred from the Foreign Table Name, singularized if the association is a toOne relationship, and pluralized if the association is a toMany relationship.

Sylvain Lecoy’s picture

Issue summary: View changes

Updated issue summary.

Sylvain Lecoy’s picture

It is not feature complete by the way: unique constraint is missing, and I need to test what happen when you subclass Node with an Article class for instance.

Sylvain Lecoy’s picture

Issue summary: View changes

Added entity classes preview.

Sylvain Lecoy’s picture

Issue summary: View changes

Added further motivations.

Sylvain Lecoy’s picture

Issue summary: View changes

typo

Sylvain Lecoy’s picture

Status: Postponed » Active

Battle plan:

  • Polishing Drupal 7 integration and release something with tests and quality stuff.
  • Port it to Drupal 8 as a separate module.
  • Start investigating Drupal 9 possibilities.
Sylvain Lecoy’s picture

Issue summary: View changes

Added doctrine project.

geek-merlin’s picture

@sylvain: i'm really excited about your efforts!

Sylvain Lecoy’s picture

Created a roadmap for a 1.0 release here: #2243015: Roadmap 1.0.

I am actively looking for a co-maintainer to help implement the remaining issues.

Module usage is growing slowly but people are clearly using it (see page statistics).

Please contact me if you want to contribute with me, I already put guidelines in the issues, I will do intensive code review and help you if you have any questions.

geek-merlin’s picture

In above url i can only find the placeholder.
My workload is already way too high but i might help here or there.
Maybe i'll write a rules integration (like we have for EFQ)... ;-)

Sylvain Lecoy’s picture

Hello axel.rutz,

If you want to take one of these issues (which are actually child of the placeholder: I've updated the issue, thanks :))

Main challenges are revision and multi-lingual support, which can be approached by two different ways: bottom-up or top-down (see following issues).

For now the Entity API driver is 80% functional, and not yet automated tested. The Field API driver task can be a great way to enter in the Doctrine/Drupal field at the same time. General idea is to see fields as entities, with a special care to the cardinality in case its a 1-1 relationship.

Sylvain Lecoy’s picture

Issue summary: View changes
elvis2’s picture

I currently use symfony and really like doctrine. There are a few known problems though. Doctrine, with lots of schemas, is slow. When adding annotations on top, those annotations need to be read and cached.

I would be concerned about using doctrine on a large build. BUT, I could see a reverse engineering usage, to pull into Drupal to create schema files (yml now?).

Just a thought...

Sylvain Lecoy’s picture

Have a look at my work in Doctrine I wrote a driver for Drupal Schema API.

http://cgit.drupalcode.org/doctrine/tree/lib/Drupal/doctrine/Mapping/Ent...

and

http://cgit.drupalcode.org/doctrine/tree/lib/Drupal/doctrine/Schema/Sche...

The idea was to not use annotations (while entirely compatible) but schema API as meta-data. Of course both scenarios would need caching to be effective.

catch’s picture

Status: Active » Postponed (maintainer needs more info)
Issue tags: +Needs issue summary update

This needs an explanation of how things like:

field translation
field UI
views

and etc. would be handled. If they can't be then it's won't fix.

Sylvain Lecoy’s picture

Partially answered in:

Missing views, but after 2 years I have lost interest in this. Maybe someone will take the lead on this issue one day !

giorgio79’s picture

Version: 9.x-dev » 9.0.x-dev

The 9.0.x branch will open for development soon, and the placeholder 9.x branch should no longer be used. Only issues that require a new major version should be filed against 9.0.x (for example, removing deprecated code or updating dependency major versions). New developments and disruptive changes that are allowed in a minor version should be filed against 8.9.x, and significant new features will be moved to 9.1.x at committer discretion. For more information see the Allowed changes during the Drupal 8 and 9 release cycles and the Drupal 9.0.0 release plan.

xjm’s picture

Version: 9.0.x-dev » 9.1.x-dev

This is a minor-only addition. Since 8.9.x and 9.0.x are now in beta, I'm moving this to 9.1.x. Thanks!

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

catch’s picture

Status: Postponed (maintainer needs more info) » Closed (won't fix)

https://drupal.org/project/doctrine hasn't been updated for seven years.

In the meantime, we've had to fork Doctrine code into core, to support use-cases they no-longer wanted to #2631202: Doctrine no longer supports SimpleAnnotationReader, incorporate a solution into core.

Given this, the prospect of trying to replace the entity system with Doctrine, which we might then have to fork back into core, seems like a non-starter to me. Any work would need to start again with the 2023 version of the entity system compared to 2023 Doctrine too. Marking this 'won't fix'.