Create generalized relationship module

If you haven't already, you may also want to check what Jaza has proposed here: http://drupal.org/node/23730

He's writing a new categories module that combines taxonomy and book module functionality. It seems related to your proposed efforts.

Update: after speaking to Adrian about this, I rewrote clipper to use this idea. Code and files will b e released soon. In generla it will do most of what is described here.

Clipper module was just updated to use the new code.

Some todos still stand, but its well on the way to use this relation system.

Next step: get shazamgallery running with this relation stuff.

oh this might be of interest too:
/**
* About relations table:
* I use the terms parent-child and child parent for the realtionships.
* parent-child means that left_id is the parent of right_id.
* child-parent means that left-id is the child of right_id
*/

not sure if it was intended that way.

What would be the main differences between the proposed relationship module and node relativity module? Sorry if I am missing something obvious here.

Darius

Its not so obvious!

Clipper has been around since long, under different names, but stil. Then at one day someone contributed realtivity module.
It does its things in a very specific manner. You either like it, or you dont.

I aim my modules at APIness (as i call it). Let themes do the display, let drupal do the configuration and as much coding as possible (use existing APIs) and let the module do the left-over coding magic. Realitivity module does it all in iself. It renders numerous pages, interfaces etc. Just lok at the 300 lines clipper uses vs the near 1600 lines relativity uses. It has a huge lot of things you will probably never ever use or need.

But the main difference, is that clipper is the first module that uses a global relationship system. Any relation module should -in some way- use this in the near future, if it were up to me. Clipper is just one way of using and displaying these relations. More will follow, so I hope.

I hope that makes it a bit clearer.

I had a bit of an epiphany regarding the user interface for this, and how it could work / scale.

Essentially introduce a 'select box' , which is a list of nodes that are currently selected by you.

When you create a new node, you can automatically select it, and when you find a node (possibly with the 'can relate' permission) you can add it to your selection.

When you choose to create a certain relationship (possibly by selecting the add relationship tab local task), you have a filtered list of the selected nodes to go from. How i see the hooks working, is that each module defines a set of rules for what nodes it can relate to, based on node type, or whatever .. and there can be another module (possibly clipper?) that can be used to edit these possible relationships. What this means, is that you can set up a permission to only allow you to relate to your own nodes, or perhaps .. with some code .. only nodes created by a certain other type of user.

Example :
So say you create 10 song nodes, and selecting each of them as they happen .. Or you could search for the nodes, and select them manually. You then create an album node , and choose 'add songs'.. which then shows you a list of all the songs you have selected. You simply check which ones you want added, and wether or not you want to remove them from your selection, and there you go.

The other way around, you create an album node .. and you select it. And when you create the node there is a dropdown that says 'add to album'.
You might also want to set it so that the user can't remove this specific relationship .. so that they can always select from all the albums they have created.

Now, what makes this even more interesting .. is when we have profiles as nodes, as the selected_node is (you guessed it) another type of node relationship. It would essentially share 90% of the code that buddylist would use, infact .. we can automatically create blocks for each kind of relationship a user has. As an example : subscribing to a node, is just another relationship.

Furthermore, this could be AJAXed quite beautifully, by automatically replacing the select node, or add user to buddylist, or subscribe to this feed, with some javascript that updates the right block and have a realistic (and toggleable) ajax update for the blocks. Perhaps you can even create one callback that updates the user with all the applicable relationships.

Also, since there is a weight column, all these list can be draggable to sort. The subscriptions block for instance could automatically sort the list based on the latest activity (wouldn't that be neat *g*).

What this entire setup is missing though, is an über way of searching all nodes .. and filtering them, preferably in a table, preferably ajaxey. Which in turn could be used for the admin/content page.

Some great ideas in here. I'll be trying out the Clipper module soon :)

Talking about these relationships between nodes, does anyone else see value in having not just parent/child relationships but also allowing sibling or peer relationships to be defined as well?

Another thing I've noticed about Drupal hierachies is that they seem to focus around an item having just one parent associated with it. I suspect that is because designing a schema and an interface for that is pretty straightforward. One thing I'd like to see with these relation modules and future book modules is the ability for a node to appear in multiple heirarchies to allow content to be reused in different places. But I do see how that would complicate the interfaces somewhat.

These relations ships can be any relations. siblings and "cousins" are perfectly possible.You could even make a relation "married-to" if you want.

Adn there are as many relations per node possbile as you want. Not just one, as book module does.

Furthermore: clipper is just one module using this. It it definately not *the* relationship module. It will either be hijacked and called "relation.module" in near future, or b renamed to remix.module; when there is a more general relation module that uses this system.

The relationship type field is changeable. You can have parent-child, child-parent, sibling-sibling , or

in the case of buddylist, you have buddy-friend/buddy-colleague etc.
In the case of an audio album you have 'track' and 'album' relationships.
In the case of picture albums (maybe we should use the flickr'esque "set" name) you have 'picture' and 'set' relationships.

This is one of the best 'fundamental tweaks' to come out in months, IMO. I'm going to be refactoring two of my modules to take advantage of it.

Just a quick note.
we have left_id and rihgt_id and inbetween that the realtionship.
I found it good behaviour to make the 'relationship' explain the left-right relation:
12 parent-child 15 means that node 12 is parent of fifteen
45 child-parent 10 means that 10 is parent of 45
That was clearer to read then mymodule-parent-child or mymodule-hierarchy etc.

think think think.
While writing clipper and rewriting shazamgallery, i am doing quit some testing and implementing some more ideas. Conclusion: the current database scheme is too limiting and suboptimal.
Here is a new idea:

This reads as:
node 41 is parent of 41
user 870 has bookmarked (owns bookmark) node 50
user 870 added user 4 as his buddy

*_map is the table name, with the ID, where it maps to. *_relation is a splitted version of the 'relationship' column. This splitted version allows for easier and cleaner SQL queries. But it is clearer too.

ugh. I am SURE I set input format to full HTML. well. Ill retry in ascii then :(

left_id  left_map   left_relation   right_id  right_map   right_relation   weight
  40     node       parent          41        node        child             5
  50     node       bookmark        870       user        owner             0
  4      user       buddy           870       user        owner             0

I support this new table schema. it is an improvement.

Bèr & friends,

This generalized relationship module looks very interesting indeed. It would appear that this module-in-development has the potential of replacing a great deal of Drupal's functionality (as in books), as well as filling in some defecits (e.g. user groups & node-to-node relationships). Seems quite visionary.

I have almost finished a module which does basic node-to-node relationships using hook_nodeapi. I am realizing that my development efforts are redundant with this project. Not sure how I missed this project & Clipper until now- I did numerous searches & read every available module carefully & posted questions on this. Nonetheless, I will probably scrap this endeavor.

Can you point me toward the roadmap or vision for this module? To what extent is the planned functionality encompassed in the current Clipper module?

Thanks

dado:

we are busy developers, developers that have little time left to write roadmaps, extensive module descriptions etc. Sorry.

This thread is the best documentation you will get about the relationships-in-progress. Once it settles, we will -of course- spend time on documenting the code in doxygen and handbook pages.

Hey, and please do not throw away efforts. why not rewrite your module with this system? or merge some code into clipper? I am welcoming improvements, ideas and knowledge.

The ability to do user to node mapping is good for me, that's exactly what my module user_related_content.module is doing and I was about to write to Bér about the possibility of making the relationship table more generic to allow it.

This schema could possibly allow user_related_content to make use of this generic relationships module.

I do think it may make this more generic if we store information on the relationship type in a separate table, and providing a relationship type id allows other modules to join in extra information (such as the descriptions user_related_content stores). Currently in user_related_content I use a 'relationship_type' table to define different types of relationships and the relationships are stored in a 'relationships' table as (relationship_type_id, user_id, node_id).

Just adding some more ideas to the mix. If you look at the API and tables in user_related_content, apart from the specific definition of one field as uid, and the other as nid it supports just about everything we're talking about here.

Cheers,
Eric

Regarding a roadmap:

As best as I can understand, the extent of the system right now is a standardized relationships table, a working agreement on what type of content will be put into it, and a module -- the new version of clipper -- that uses it to do certain things.

Yes? A shared API for inserting and retrieving parents/children/siblings, as well as module-specific relationship types, is probably the next step. That sounds like a good candidate for a streamlined relationships_api.module, or something like that, neh?

I am quite amazed by all the people getting into this discussion in a positive way. It looks like we have quite some hands to do the hard work :)

So here is my small battleplan:
* Try out the Best Way To Write And Use APIs [tm] in several modules. in progress currently
* Once that is settled, move these APIs to a relation.inc.
* Then we should use that relation.inc in several contrib modules. Test it, improve it and finetune it. (think performance)
* Then try to get it in core ;)
* rewrite book.module, comment.module, taxonomy.module one by one to let them use the relation.inc

Just to clarify -- you mean that the relationships between taxonomy terms should be represented by this new relationships api, NOT that the relationships between nodes and taxonomies should, right?

The 'taxonomy' side is what scares me, personally. There's such a large base of stuff that works with the taxonomy system...

I don't want to add static, and my stuff is too in-progress to make lots of noise about, but I'm also working with a relationship/hierarchical data module. I started with clipper and I don't want to give up the ability to work with/using taxonomy terms. Then my concern was peoples' permissions to add child links. I managed to get around this with some crafty db_rewrite_sql action, but ran into enough trouble that I started fresh.

I wrote a very complex system based almost exclusively on this type of relationship system a few years back and I learned a lot! The important thing is not really how 2 nodes are related but how they *can be* related - and what happens to them when they are. The other important thing is the efficiency of any kind of tree traversal. There may be better ways to do it than recursive parent -> child links (which I used in that system and paid dearly for).

Real questions involve what happens to privileges from higher-up nodes -- do they cascade to child items, or are you stuck with (in)editable chunks of your tree? How should the children appear - as links or as embedded objects? If embedded, you can approximate some poor-man's cck functionality and accomplish tasks currently performed by node_image and friends. I'm attempting to build a system that defines the types of relationships that can exist and how they can link together.

The relationship table itself is quite small and optimized for hierarchical relationships

Again, I'm not trying to make noise or derail anybody, just letting y'all know I'm out there. I'll be able to explain myself better when I have something to show. I am open to ideas and discussion, but I'm on a critical path right now and may not be 100% receptive.

Thanks for all your hard work!

OK: there's a very, very, VERY basic implementation (read nonfunctional) available at:

http://cvs.drupal.org/viewcvs/drupal/contributions/sandbox/vauxia/hier/

I'm open to input, conceptual or otherwise!

I don't intend to muddy the waters w/ yet another approach, but I have finished the alpha version of my own module which permits nodes to be "tagged" to one another (Drupal 4.6). It meets my requirements: I am building a website for tracking community events. I wanted to tag events (flexinode + event.module) with places (flexinode + location.module). I wanted to be able to link from an event to its place node. I also wanted to similarly tag user reviews (another flexinode type) to events and places. So any node type can be designated to be taggable to any other node type. I do not need much explicit hierarchy.

I implemented a method similar to earlier version of clipper (as far as I can gather; I do not have a Drupal 4.5 environment to test). My module permits one to designate any taxonomy + any node type as defining a class of tagnodes as I call them. A class of tagnodes = a node type + a vocabulary. The module uses hook_nodeapi to ensure that each tagnode's node title is identical to the a corresponding term name. No new tables. You can then effectively "tag" tagnodes (like places) to other nodes (like events) by using Drupal's taxonomy system. When rendering nodes, the module displays links to the associated node(s).

I do not have CVS access yet (applied some days ago). If there is any interest in my module I would be happy to upload it or paste it in this project area or wherever you suggest. Thanks

I've long wanted something like this, and started writing it for my (now defunct) libdb module. I described what I particular wanted, for the most flexibility, on the Drupal dev list. Berkes has the same idea (as in his #16 and #17). But, after having extensive discussions with chx about it, I'd like to amend my previous thoughts with two new fields: uid and timestamp. Timestamp is self-explanatory, but the uid contains the uid (0 if none) of the user creating the relationship. This becomes vitally important with something like "(Morbus Iff) (likes) (Berkes)". Who made that relationship? Was it you? Me? Dries? Who makes a relationship is critical to how strong, relevant (occasionally "truthful") a particular relationship is (ownership of a fact is a big thing in the trust metric of an RDF "triples" world, which is what, ultimately, my desired table design is replicating).

To wrap up, this was the DB schema we arrived at: (id1, type1, id2, type2, module, delta, uid, timestamp).

In re-examining Berkes schema, I'm not entirely fond of the "left relation", "right relation" of parent/child, bookmark/owner. This means for every one type of link ("bookmark", "family"), two relationship types are required. I don't like that - it makes it far too easy to make the wrong sort of links ("parent/owner", for example, is particularly frightening). His layout tries to add meaning to the relationship *in the database itself* ("to be a child of something means that that something is a parent") which creates a rules based system - which'll mean extra validation and so on. I don't like that.

In RDF (and my schema) relationships are defined on an "is/has" basis. "Berkes is a parent of Morbus". "Morbus has a parent of Berkes". Thus, one entry in the database ("parent") constitutes two relationships (is/has). This then begins the process of inference. If "Berkes has sister of Dries" ("sister"), and "Berkes is a parent of Morbus" ("parent"), with just those two relationships (the "relationship type" in my schema), we can infer that "Morbus is a brother of Dries" and "Dries is a sister of Morbus" - two extra relationships that we didn't have before, based on inference. Yes, granted, we COULD have inference based on Berkes schema, but its not as clean - it's too much maintenance in the database itself, IMO.

it is become more and more complex. so KISS is a little gone. We should really introduce KISS in the APIS or else go for a much simpler database.
However, I really like the idea of a uid. I see that it is very much needed. It will save you going for two relations: a Foo to Bar relation and a user to relation relation. I can see the potential in date and uid for weighting stuff.
Mayor +1.

Who wants to make a new SQL file.... ?

@dado
I indeed used that node-taxo thing a lot in various module. But there aer two reasons why i abandoned it:
1) silly developers/drupaleers did not liek it. For no reason. Bu no-one 'liked the apprach'. Again: for no real reason. But people not liking it, is enough reason to move awaay from it, for no-one will contribute to a project he/she dislikes.
2) taxonomy is fairly limited. You can achieve the same wit a general relation API. I simply 'hijacked' taxonomy as relation API. But taxonomy wil never really become that.

CREATE TABLE relationships (
  rid               int(10) unsigned NOT NULL default '0',
  uid               int(10) unsigned NOT NULL default '0',
  created           int(11) NOT NULL default '0',
  referer_id_type   varchar(32) NOT NULL,
  referer_id        int(10) NOT NULL,
  module            varchar(128) NOT NULL,
  relationship      varchar(32) NOT NULL,
  referent_id_type  varchar(32) NOT NULL,
  referent_id       int(10) NOT NULL,
  INDEX (referer_id,referent_id)
) TYPE=MyISAM;

Here is a revised version of my original LibDB table, updated for all my recent thoughts. Column definitions are based on similar columns in other tables. Chx: note that I've swapped out "delta" (a number, like the block) for just a regular varchar (like the word "parent"). I think it's important for ease of exportation and readability (think node.types).

I really think you ought to at least consider defining relationship types. I have seen it mentioned a few times that this relationships module will be of big benefit to everything Drupal, but I haven't seen a whole lot of use cases to validate this model.

My biggest points are:

1) Per #30, you're looking at storing roughly 40 bytes per row. I know this sounds nitpicky, but if you intend to use the relationships table for lots and lots and lots of relationships, you want to minimize the data stored here.

2) I understand that any module could choose to start a relationship, but what are the access rules, and how would Drupal generically know to grab and present related items?

From a raw database standpoint, it's much more efficient for an album module to have a tracks table with an album_id, or a join table to relate albums to tracks. See, we're already able to pick the most appropriate relationship mapping based on business rules! Lots of modules do this in lots of different ways, and usually it's good to use the best tool for the job.

The justification for putting all these relationships into one table seems to be "because we can". Because lots of stuff relates to other stuff, there should be a single, best practice to manage these relationships. That's possibly true, but it requires a lot of forethought to replace lots of efficient tables with one big one.

One justification might be that Drupal can maintain a consistent user and security interface for selecting related items. Or that Drupal can have good default settings for displaying related items. If executed properly, these benefits justify the overhead cost of globbing all relationships together in one table. But Drupal needs some rules to validate and enforce relationship types, and those rules aren't present in your data model.

The only word I've heard on how user interfaces will manage selection and display is that these matters are up to the module. So we have a more expensive data model and similar work effort for module developers and no consistent UI. What do we gain from this?

I maintain that modules need to define relationship types to offload some of the data you're storing in the relationships table, and to provide Drupal with some clues for security, selection and presentation. Come at this from a UI perspective and think about how Drupal could generically provide an interface to select related and/or present related items.

FWIW, these concepts are already expressed in the hier code I posted and I did lots of elaborating in the readme. Feedback is welcome!

Allie, more later, but my original LibDB code DID have a specific relationships_types table, with rtids, etc., etc. I didn't address permissions in it, but it was originally intended to reduce the duplicity of module/relationship names in the relationships table itself. That schema, unmodified, was relatively unexciting (and naturally, the default relationships were related to FRBR, and not Drupal). It's attached.

Hi,

This has been a very interesting read. I'm interested in relations as the very foundation of drupal menu/navigation system as well as the enrichment of content with related content (It becomes one and the same after a moments thought). The discussion has "topic maps" and, alternatively, RDF (actually mentioned briefly) written all over it. I'm looking once again at drupal as a target app to add topic map and faceted classification functionality. I'm also exploring the current system to see how much I can achieve (equally effective functionality) with what drupal offers now.

In topic maps, relations are defined as: {relation-type, topicA, roleA, topicB, roleB}. For example: {actor-film, Gregory Peck, actor, To Kill A Mocking Bird, film}; a hierarchy: {superclass-subclass, Europe, superclass, Italy, sublcass}. This richly defines the relation adn is bidirectional. RDF is less rich but sufficient in many cases: {Gregory Peck, acted-in, TKM}. This is unidirectional, so a second db record is required: (TKM, has-actor, Gregory Peck}. In topic maps, content or content links ares associated to a topic, which means a system like drupal, heavily based on the concept of nodes, should lend itself comfortably to topic map integration (that's the theory).

Topic maps (and RDF) break away from the limiting and often misleading construct of hierarchy, something this thread seems to be trying to achieve. Does anyone here have an opinion on or interest in topic maps or faceted classification? Is it something of interest for drupal?

Pacoit,

What you propose does not seem to differ from what we want to achieve. What we want is what the title says: Create generalized relationship module.
If you think you have or know a database scheme that will suit your and our needs even better, thne please provide the scheme here.
I want to freeze the scheme very soon, so that we can focus on the APIs.
I don't care if its called unidirectional, RDF, topic map hierarchical or any other jargon-name. All we care about is a generalized relationship module that works, and serves as many cases an developers as possible.

This means two things:
It must be flexible enough to do complex relations
It must be simple enough to be used by any developer.

The first thing should be achieved in the DB scheme, the second in the APIs.

Bèr

Ber & friends,
Agreed that the data model comes first then API.

On the API, I have uploaded my relationships module called "tagnode" here
http://cvs.drupal.org/viewcvs/drupal/contributions/sandbox/dado/tagnode/

This module supports node-to-node relationships using Drupal's taxonomy system, and uses term name=node title to define a "tagnode" (no new tables). Each node type can be converted into a tagnode, so there can be multiple classes of tagnodes each taggable to any other node types. It is similar in concept to a recent version of Ber's clipper, and I revamped this module using some of Ber's methodology.

Boris Mann asked me to interface w/ Ber to see if we could merge efforts on this module.
Functionally the API needs to support what this module can do. Advantages of the module as is: it uses Drupal's (existing, solid) taxonomy system & code to handle all the UI elements of the tagging of 1 node to another.

Let me know. Thanks

Bèr and all,

FYI, I am not yet familiar with the drupal API, so it will be difficult for me to offer a concrete solution in a short period of time (I sense the need of an immediate solution).

Topic maps, along with ontology and classification theory, are an involved subject. I can offer links to topic map standards and info pages, as well as a datamodel that conforms to the whole standard. The first step, though, is to understand the concept behind topic maps, and understand how they provide a general solution for representing knowledge (relations among things), such as hierarchies, thesauri, faceted classification, arbitrary typed relations, and ontologies. Once we understand and are convinced of the value topic maps can provide to drupal, we'll be able to quite easily realize a well-chosen implementation.

The TM specification is very ambitious in scope; the Semantic Web is the goal. But don't be put off. With some background understanding of topic maps, it is easy to conceive a simplified, easy to understand, and easy to implement version, a conforming partial implementation of the standard that would provide 90% of the ambitious full standard and could be expanded on over time as desired. Meanwhile, we would already have a vastly more flexible system in place.

Personally, I see a relation module becoming the very core of the drupal CMS, taking over the menu system. The creation of richly defined relations among all content means that you get a context sensitive menu/navigation system for free. The concept of topic types allows one to render instances of each topic type according to the type's assigned template. Relation links within the topic page can also be templated according to their type, i.e.: narrower term links as left-hand menu; sibling links in right block, related links in related block, etc.. The beauty is that these link-enriched pages are created automatically; newly created relations automatically show up as links wherever they logically apply. The system implies that one could reconfigure the whole ebb and flow (organization of content) of a website by altering the relations. I hope this helps paint a picture. Drupal could become the first free TM driven CMS in the world.

Some links:
http://www.topicmaps.org/ (standard defined in XML format.)
http://www.topicmap.com/ (an info portal for TM)
http://www.ontopia.net/topicmaps/materials/tao.html (a popular intro to TM)
http://www.tmapi.org/ (common TM API project)
http://phptmapi.sourceforge.net/ (contact author for status)
http://xml.coverpages.org/topicMaps.html (many articles)

Article and sample TM application, plus datamodel:
http://www.geocities.com/xtopicmaps/
http://www.geocities.com/xtopicmaps/database_builder_for_mysql.sql.txt

~pacoit

Topicmaps seem to be very interesting.
But for now I rather focus on a real world solution. I think we might eventually aim for those topicmaps, but that we should now aim at something more practical. Something that has an immediate and clear result.

Maybe those interested in topicmaps should see this relations project as a first stepstone towards their general topicmaps and help with this first?

I followed Boris' advice and created a new project "tagnode".

I recognize that there might be significant overlap & redundancy between Tagnode & clipper + this project. And I dislike redundancy. So I'd like to help out if I can when the data model & other high level details are sorted out.

dado

Bèr, Boris,

I understand completely and I agree that you shouldn't loose focus on your current efforts.

Hi,

A little more in scope: 2 options; you might like the second one better.

-- a simple topicmap-ish relation table.
--
table tm_relations {
id       int auto_increment
type     varchar               -- TM instanceOf (eg, actor-film)
subject  varchar              -- TM topic (topic of current page/view)
subjRole varchar              -- TM roleSpec (eg, actor)
object   varchar              -- TM topic (object of association, 
                                    -- eg, "TO Kill a Mocking Bird")
objRole  varchar              -- TM roleSpec (eg, film)
}
--
-- roles are useful because topics may have multiple roles, eg, Gregory Peck
-- is/was(?) an actor, a father, a producer, a teacher, etc.
--
--
--
-- an RDF-like approach is very simply based on triples: Subject, predicate, 
-- object. it's amazing how powerful this simple construct is. The predicate
-- does all the work.
-- eg, {GP, is-actor-in, TKMB} for GP as subject; 
-- eg, {TKMB, has-actor, GP}   for TKMB as subject;
--
table rdf_relations {
id       int auto_increment
subj     int                  -- node_id
pred     varchar              -- or an id in predicate table (more normalization)
obj      int                  -- node_id
}
--
-- To get all relations for GP:
--       Select * from rdf_relations where subj='GP' order by pred;
--
-- You'll probably want to create a link for the object, so you'll want to
-- join with the node table on object to get node_id (or other needed link
-- creation information). Then, you output with 
-- (pseudo code): foreach $result, echo $pred [link:$obj].
--
-- Since the RDF concept is so simple and immediately intuitive, it may be a
-- good choice for an initial relation module. It can also be easily extended
-- later into richer TM while preserving the original fields and data.

I am sensing an urgency for arriving at an answer to a question that has not yet been asked.

As I asked in my previous comment, why are we introducing this feature? I can think of lots of reasons, but those reasons are not necessarily addressed by the models presented thus far. Instead, I see solutions that will introduce performance penalties without much justification in gains.

I have presented a data model and some discussion on the reasons and solutions for such an approach, as have others. There are similarities and differences among each of these models, just as there are similarities and differences in the problems they're each trying to solve.

A request to "create generalized ___ module" leads to a whole lot of questions, and it seems that the most proactive approach at this point is to clarify use cases so that we have a litmus test for comparing each proposed solution.

What problem or benefit does creating this one general relationship table address? In particular, without the rule-imposed relationship types and security rules (cascade?), how will this table benefit Drupal over having several such relationship tables which are defined by cooperating modules sharing an API and business rules?

Seems like it just buys one place to look at data for the cost of inefficient database performance and the continuing need to look at all the modules to find the relationship types and rules.

I don't think the idea of general meta-data about relationships is an easy problem, but I do think it has been well-studied by others. Maybe their work should be looked at?

Allie,
I see generalized relations from an information architecture point of view, a mechanism for rich linking and navigation among pages, which translates to the organization of content. In such a scheme, all drupal nodes are topics/subjects/resources/whatever; relations result in hyperlinks added to a page; the relation type indicates if link should be rendered as a hierarchical main nav, or a "See Also" block, or previous/next pager links, etc.. Menu is replaced by relation management (which offers more than hierarchy). So, the notion of a generalized relation feature suggests that it must be part of drupal core. All modules would then use the core functionality. "Relation" and "Node" become the drivers. In short:

1) Generalized relations is for website navigation (and content filtering);
2) therefore generalized relations only makes sense as part of core.

A non-core generalized relations helper module in the interim would be useful, elegant, and more efficient than the present state, but it's ability to unify data and display from different modules would likely remain limited.

Chris,
I'm not yet familiar with drupal authorization API; but if authorization is assigned to a resource, and the user accessing the resource link is not suitably authorized, then user would be asked to login/authenticate. Relation code needn't deal with permissions at all.

I invested some time in reading about Allie's hier concept in http://cvs.drupal.org/viewcvs/drupal/contributions/sandbox/vauxia/hier/R...

All I can say: yes. This is what we want.

Some ideas:

hier_item_types records all items we can manage in the system. Each item type (e.g. story, page, book, event, user, vocabulary, etc.) is listed here along with its driver (node, user, vocabulary). Story, page, book and friends all use the "node" driver, while user uses the "user" driver, and so on.

The hier system is not cognizant of nodes, users or taxonomies; only "items". Each item has a globally unique "hid", derived from the hier_map table. This mapping will be unnecessary for item types are are created to work with the hier system and use the hid sequence as unique identifiers.

hier_types table indicates item types can act as parents and children in a particular relationship type. What's the minimum and maximum count for this type of relationship? Apperance: an item may appear as part of its parent item (a picture or data field that appears on an item's display) or
linked (in "links" or some other reference). Privileges. These rules will be enforced by the hier system and the drivers for each item type.

Presentation will be handled by the drivers and the modules that use this system.

I tried to keep this post short so you can quickly see that she tought of really everything, general storage, API, UI, everything. I strongly advise to concentrate making this a reality. I know we can discuss anything 'till we are blue but what about writing some code instead :) ?

If that does not happen, I'll start my own thread and begin working with Allie towards an implementation.

Wow! I don't know where this is going, but i for sure will not let this effort die in some complex discussion about what sort of relationship designs there cna, could or should be.

People: let us focus on a real life example. I do not want to rewrite drupal to make it some relationship system. All we want is a method to say 'Foo is a buddy of Bar'. 'Bar is a remix of Foo' etc. Certainly not a method to replace menus, node etc in Drupal. Certainly not!

So, people, please focus on getting stuff done. Attach mysql dumps, point to existing modules that could use this relationship table/API. Rewrite your module to become a part of this relationship system etcetc. But please do not make this a place to discuss all your academical theories on what a relationship could be like.

The past followups are way over my head. And do not help us getting anywhere. Well, except for that hierarchy system. Though chx, ally, I fail to see if your proposal will influence the current relationship tables. Can you please give a concrete example how the tables should look to get your efficient hierarchy APIs top work with this relationship system?

REmember :here too: Code is gold, talk is silver.

Bèr,

Perhaps you missed my earlier reference to code, documentation and table schemas that illustrate my argument. I fully agree that we don't want to get locked with analysis paralysis, that's the reason I too the time to write the code and make it available.

I assume you missed the link, otherwise you would not have rejected my input citing your "talk is silver, code is gold" philosophy. So here it is again:

http://cvs.drupal.org/viewcvs/drupal/contributions/sandbox/vauxia/hier/

Whether or not you choose to look into this, I will ask two simple questions for the third time:

1) Introducing a shared relationships model is computationally expensive. What benefits are you introducing to justify the cost?

2) How does your model/code support these benefits?

If you can't answer these questions, no amount of coding will get this change committed. That's my favorite feature of Drupal.

If this was not a broad change then we wouldn't need much up-front analysis. But if you are trying to make an informed decision that benefits Drupal in the long term, "it's too hard" is not an excuse for failing to acknowledge input.

It's counterproductive to ignore solutions people have already written because you're too busy trying to get someone to write a new solution. At this point, it seems the biggest risk is not having the project die in complexity - it's having good people abandon it in frustration.

Allie, I did not post my previous comment with your APIs and ideas in mind. Sorry if you felt it that way.
I read trough your docs and liked what I see, but found it too far away (yet) to suit my current needs.

And to (finally) answer your two questions:
1) Introducing a shared relationships model is computationally expensive. What benefits are you introducing to justify the cost?
-- A set of program APIs: module developers can re-use this, instead of introducing all sorts of private relation stuff
-- These APIS can do some low level caching: I dont see simple contrib modules using cache a lot, currently.
-- Contrib modules now very often need .mysql files. Both clipper and shazamgallery now already use one table. You need to install only one. We did this for weblinks and already five modules use one shared table. I hope to convert more modules to use this relationship table soon. Users need to install only one database table now.
-- Because there are about 12 modules (that i see) that could use this table, we can focus on optimising code/queries it in a central place. As opposed to having 12 custom and often suboptimal systems. (re: cache et al)

2) How does your model/code support these benefits?
-- There are a lot of modules that will become very small after this central module gets out.
-- There will be a set of easy to use APIs to store, delete, load and update relations. that will help developers. But also a general available (cachable) array with relations can be made available. $node->relations can always be filled on node_load with a relations array for that node.
-- I fail to see why a single big database table is slower then a range of various tables that are all loaded on comment_load, userload, node_load etc. Centralising it will help a lot, IMO.

Also, note the questions to you, Allie, in my last comment: I cannot see yet if/how your hier code will effect my relationship code.

Most of all, though, my latest comment was to slow down all the ideas of 'converting taxonomy, menu, comments etc' to this relationship system. I would like to get this into contribs first. Then well see where to go.
Bèr

My basic rationale for introducing hier into a relationships discussion is this:

- A hierachy can represent relationships (after all, a relationship is just a short hierarchy)
- A relationship API cannot efficiently represent hierarchy (think recursion and adjacency models)

Hierarchy may or may not be the answer, but it bears investigation because it solves both problems.

I have tried to point out some problems you may have if you try to go forward with the relationships API as-is. Unless there are questions, I won't keep repeating myself. I'll just leave it at my opinion that it's irresponsible to get a dozen modules to use an API before it has been fully thought through.

Because of the required metadata, a shared relationships table is inherently more expensive even if only one module uses it. If lots of modules use it then it's even more expensive because of its ever-increasing size. Further, modules are not creating best-tool-for-the-job relationship tables based on their business rules. These are not just an IMO -- benchmarks would corroborate.

I fail to see how caching would do much to improve this problem. It's one thing to cache an entire tree of data, but how do you efficiently cache tens of thousands of links for thousands of nodes? Think back to the url_alias discussion (not THAT one, the other one)

Even if caching does help, this argument is similar to "we should buy this really expensive thing we don't need because we can get it at a discount". Until I see benchmarks or a data model that's a genuine performance improvement I won't be sold on this argument alone.

Making a module developer's job easier and eliminating the requirement for .mysql files is also a weak argument. A module is written once, installed once per site, and executed many thousands of times per day. Creating a performance penalty for the execution step is not justified by reducing workload on the other 2 activities.

All that said, I'm not opposed to a generalized relationships module, but I'm not in favor of this one because it introduces cost but no real benefit. If you take it a step further, you could really have something.

Maybe we all have slightly different ideas about what are the requirements for this generalized relationships module. Bèr's recent post describing the proposed module was enlightening & makes sense. I am unclear why Allie's solution would not work. I could well be missing something.

My only concern with Allie's high level approach is that it seems it is layering a hierarchical structure over the Drupal taxonomy ,which itself is hierarchical. I am sure there might be a sensible justification & path forward, but would we be adding redundant systems for categorizing things? Maybe not.

An advantage of Allie's system is that it could include users (& terms I guess) as the items that can be related. So the scope of this core relations module would support relating users to nodes. I believe this would potentially replace custom code of some existing user-centered modules.

Maybe the question comes down to this: should this generalized relations module support hierarchy or not? Here is an example where I might need hierarchy.

I have a community website. I have this structure of relationships among node types
EVENT node type [is_a_child_of] ANNUAL_EVENT node type, which [has_a] LOCATION node type

When rendering a given EVENT, I wish to show its LOCATION. This requires looking 2 levels up the tree of associations to find the location info. Ideally the relations module would support this. The 80-20 rule might argue not.

Sorry for finding this discussion so late, but many of these issues have been discussed before. There was discussion on on the devel mail list early this year about using a generic relationship construct across many different types (nodes, users, comments, whatever) and it seems that the biggest arguments against such a construct came down to performance. Though I didn't put up much of a fight at the time, I tend to agree with Bèr that a single table could suffice (if well indexed and kept fairly trim). This is where the Node Relativity module was going a few months ago (the last time I had available time).

I'm surprised to find that there are many people with different ideas about how relationships should be implemented. I know I've written two modules (simple, and complex) that do relationship management, myself. In some ways, I wonder if it even makes sense to have a generic approach, as each problem requires a different solution (hence the dozen or so relationship mgmt modules).

I do, however, like what was done around the node_access construct. A structure was built that allows multiple node_access modules to potentially coexist without breaking the system. The modules manage the data and drupal core queries it to determine access to a node for a given user. This lets modules be in control while centralizing a lot of overlapping code. I don't recall if multiple node_access-like modules existed prior to the development of this system, but several do now. This feature brought about some innovation that I doubt would have happened without the core code.

I agree that many of these relationship modules could share some code with each other, mine included. The list of use cases varies wildly (from simply attaching images to nodes to building giant tree-like graph structures). I know that much of the API will depend on the data structure, but I would hope that it remains small and simple. I would hate to burden users with excessive code / slow execution if they're not going to use the features.

To Allie's point, a relationship API could represent hierarchy if a hierarchy-oriented module was in control of the relationships that were created. In fact, using a simple relationship design, you could implement all kinds of graph/tree structures, enforcing whatever rules your structure requires or allows. I do agree that the relationship should be "qualified" in some way or another. I also agree that metadata should exist (my early thoughts) for relationships. What I question is whether the metadata should be maintained using the core API or not.

The level of interest in this topic shows just how great the potential of such a feature is. The fact that so many people have wandered off and made their own implementations gives a good indication of how complex the field of problems is. I think now is a good time to identify the common goals/logic/code/data and make something solid that can unify all of these efforts. Great discussion!

Didn't mean to change the version.

For us non coder people (useless as we are) use cases are the best way to understand a module. I've used node relativity, clipper in 4.5 and now tagnode. I'm also interested in the hier module. I figured I could throw out a real world problem and maybe the developers of these different modules could help explain how their module or system might solve it, or how some combination of what is in the works might.

I've been working on a site for awhile with a simple goal- to get more volunteers and dollars to relief organizations. The main idea is that people blog, posts news stories, join groups on shared interests, but somehow all that posting gets tied to real world action. The solution to tie all this together is creating a simple way to connect relief organizations and action items with content.

The idea is a user:

1. Reads a blog entry or news article about an urgent relief issue
2. Has a link to a related organization working on that issue
3. Finds links to specific action items they can do to help that organization fulfill their goal

From a Drupal perspective I needed a way to organize various organizations by categories, attach action items to those organizations, and allow the organization to blog and have their entries related back to their organizations overview (some organizations use this, most have own website so need to tie their rss feed in). I also needed a way for users to attach organizations and action items to a particular news story or blog entry, or if none existed that matched the need described in the article, create a new organization entry tied back to the story.

Most of what is needed can be done with node relativity, but it hasn't been updated. The advantage with node relativity was that I could define relationships in depth, and display, without coding new modules to interface with a system. I could tell node relativity that whenever someone views a news article, they should see all attached organizations, they should also be able to attach organizations themeselves or create an entry for a new organization. But there are still problems.

What if I want to create an organization entry for a group that does polio inoculations, but no "medical" category exists. The user has to create a new category for the organization before creating the organization. What they need is the ability to create that category (which is both a taxonomy term and node- ala clipper and tagnode) while they are writing their organization entry. Or what if I just read a heartbreaking story about polio in Africa and nobody has attached the polio organization to that post, if I click "attach organization" I see a list 5,000 titles that I can't possibly sort through to attach to that story.

Wow, I already wrote a bunch, but I hope you get a sense of what I'm working on enough to talk about how the various systems might work. I'd like to commit to something and use it and hope this sparks some fruitful discussion for you all.

I started collecting notes. If you have more pages bullets, references etc to add please tag them "drupal relations"

http://del.icio.us/tag/drupal+relations

After a long discussion with Vlado and Morbus aka relationMonster we concluded that a split in th tables might be the best one to go for.

alright, so there'd be two tables. one for the relationship types, that would contain "1|edit|has edited|was edited by", and then another, the actual relationships, that'd contain "user.uid|15|1|node.nid|20". With that one statement, and that relationship ID one, we can now infer two facts: node:20 "was edited by" user with uid 15, and user with uid 15 "has edited" node with nid 20.

This sounded like a great plan, the strings "has edited|was edited by" aer powerfull, flexible and even huamn readable, but will hit performance of the joins badly.

Thus we thow in a third table: the relationsships description mapper.
On a day-to-day operation, the module who wants to look for who edited what, will know the numbers for 'has edited' (eg 101) and 'is edited by' (e.g 102). That can be in a cached variable or a once time pulled in cached array.
Then the tables look a lot easier. only when we want to find human readable inference (http://en.wikipedia.org/wiki/Inference) will we need to hunt for the arrays.

Attached is an image that shows teh proposed layout. For the image i used only the tables node and users, but it can be any object, off course.

I was cratching my brains lately. I've put down descriptions of relationships [ http://dikini.net/node/55 ], and related indexing. It is related to Bers's diagramme above, although I'm not using the same names - they simply confuse me, but we are different. Nevermind that, we have different requirements when we use relations - one for navigation, menus, books, and similar - which can be sped up using specialised tree indexing, like nested sets. And the other major case is when we want to do distance or closeness measuring between things. So I tried to put down some of the cases in SQL, to see what is common and what is not.

I think we should stress that this relation system is NOT meant to replace menu's or even taxonomy. I do not (yet) care about books, nor about comment threads. If we want to cover all these, we are never goping to get anywhere, unless we have everyone behind this.

The first and for all aim of this project is to allow
* shazamgallery
* clipper
* buddylist
and such modules to benefit froma general pool and set of APIs.

Thus efficient nesting, optimised subqueries are all nice, but I think Dikini is very right when he states that we should recognise we are dealing with different cases.

I am, however, not saying that when people want to help to focus on optimised hierarchy building they are not welcome. In contrary. Just that I want to stress the main focus of this project. If we loose that focus, this project might end up going nowhere.

Bèr

There seem to be a lot of people interested in this, with slightly different ideas. I agree that a more inclusive rather than exclusive model is best.

Would it make sense to begin to define a group of developers who want to work on this, construct a basic design, assign some tasks, and get us developing? I'd be happy to help contact people who might be interested and get people on board.

An immediate challenge would be how do we reconcile existing code & other conflicts (Allie's vision vs. yours vs. javanaut's vs. dikini's vs. other). I suggest this needs to be defined prior to getting people to commit to working on this.

Dado

AFAIK Vlado (dikini) is working on this all inclusive thing, already.
Taking off from where i left this dangling, considering Allies great hierarchies, and a lot of maths, I think we might actually see some result next week!

I am letting this rest, untill Vlado is finshed with his mad scientists posts :) (http://dikini.net/node/64)

Yeah, I have been following Vlado's culinary ravings. You clearly don't want to interrupt (mad) genius while it's frothing like this. Will stay tuned.

Hi All. I posted a giant write-up on my progress in exactly this direction over in the forums.
3 big write-ups in fact, and then my own module.

Please have a look at my statement of the problem (with direct reference to a bucnh of real-world client requests) and follow up with my contribution to this discussion ... including another dozen user requirements this project should achieve.

:) ... I thought this thread was getting out of control - forums are more readable, IMO.

... and just think, if this facility was available, I'd be able to tag these two threads together!

.dan.

Sorry for cross posting, but I think people following this issue might want to know. I relesed some initial api code on my website. I intentionally haven't put it in cvs yet. I think it will be better to develop the initial stages outside, similar to the way the forms api was developed. I've posted a comment in the "rdf relations" thread with the announcement. As Dan ays, it is easier to discuss in the forums.

updated the title back. apologies

What is the status of the various projects on here? Time for a new thread?

Hi everyone,
I personally missed OSCMS/DrupalCon. I am wondering if any concensus was reached on this subject. So, Allie and Dan and some other folks have built or started general modules to deal with relationships, while others have made other modules that deal with more specific relationships. Then there is CCK, which allows nodes to reference nodes and other things. Overall, is a core relationship API a current goal, or a single contributed relationship API, or other?

cheers,
Ian

subscribing

Subscribing. What about a (entity)reference module that provides a entity reference field?

If anyone isn't aware of it, there's now a relation.module (not related to the one mentioned above). The relations defined by this module can relate any entities to any other entities (including self relations, directed relations, and n-ary "octopus" relations). Relations are entities, so they can relate relations to other entities, for example:
CompanyA->donation123->PartyB
donations123->transaction456->BankC
that is, "Company A made a donation to Political Pary B, via BankC".
Relations are NOT fields, however they are fieldable. This allows you to eg. give the donation a value, or a date. There is a plan to implement a dummy field that allows relations to be displayed on entities, as if they were fields, and possibly even to add relations while editing the entities that they relate.

Any comments on the issue queue are welcome.

I haven't yet read this issue, so I'm not sure if this suits all requirements. Will try to get back here soon.

The major points I get out of reading this thread that relation.module DOESN'T cover are:
- hierarchies, although it will do directed relations (half implemented). Also, see http://groups.drupal.org/node/134484
- efficiency: this is not one of my strong points, I don't know how efficient relation is at the moment, nor any decent metric by which to measure this.

The schema is pretty simple (but allows for high complexity relations):

• relation bundles define the possible relations that can be created. There are two tables:
• relation_type: much like node_type, contains predicate (bundle id), and a bunch of other settings (mostly limits on what can be related)
• relation_bundles: contains the entity bundles that are available to be related to by this relation type. m rows, keyed by predicate
• relations themselves are also made up or rows in two tables, in a similar way:
• relation: contains the core relation data {relation_id, predicate, arity} soon also author and creation date
• relation_data: contains n rows for an n-ary relation, each with {relation_id, entity_type, entity_id, r_index}

All of this is reasonably well documented in the relation.install file: http://drupalcode.org/project/relation.git/blob/refs/heads/reboot:/relat...

Theoretically, this could replace taxonomy and book, and even menu, when #916388: Convert menu links into entities gets in. Practically, I'm not sure if that's a good idea.

I found the current relation module to be quite unusable yet, UI-wise at least.

However, this new functionality with a decent UI can get Drupal into a new level of usability. But it's not going to be an easy task to work on UI for this. What could be accomplished would be a series of modules (for now) to deal with each UI task, like taxonomy, node reference, etc... But then there would be the Field UI, where there would be an overlapping, so perhaps it shouldn't rely on Field UI.

Marked #1270422: Meta: Core entities should all provide reference fields as a duplicate of this issue.

For reference: usage statistics of Entity reference and Relation.

Well, duh, Entity reference is used for Drupal Commerce. Also I am totally unconvinced that relation is core material. I am not sure whether ER is enough, you might need entity tree as well / instead / merged.

> Entity reference is used for Drupal Commerce

I don't think it is. At least, not according to its project page. Commerce uses a custom product reference field type.

Sorry then -- that does not change the meat of what I wanted to say: entity reference usage is immaterial. It might or might not be core ready, that's for the ER maintainers to decide. I am the relation maintainer and I have my doubts about putting that in core. It's a great module and by now quite well tested but I am rather unsure whether we want every relationship to be its own entity (although it'd certainly be handy...). On the other hand, hiearchy is not solved with ER (or relation) and entity tree is much more core needed than either ER or relation despite it's only PoC material at this point. However, if you put ET in core, do you need ER?

Here's what entity tree does.

I'm not entirely sure how ER and ET are related.

ET is about a hierarchy between like elements: any element can play any role in the hierarchy, parent or child.

Whereas ER is about pointing to from one element to another, and they can be different types on entities.

So while ET would give us an abstraction of taxonomy hierarchy, ER would give us an generalization of the taxonomy term reference field.

And while term ref fields may be the only reference field in core, the fact that they are different from ER means that any module in contrib that seeks to do interesting things with reference fields has to cater for two slightly different field types. So for me that's definitely a good reason for getting ER into core.

Look at a parent and a child in ET. Seems to me that relationship looks suspiciously identical to an ER.

Sure, from a graph theory perspective, an ER is just a two-element ET.

But I don't think it's practical to make the ER module a special case of an ET.

For hierarchies, Tree is what you are looking for. It's both a generic framework for storing trees and a specialized tree field built as an extension of Entity Reference.

> if you put ET in core, do you need ER?
>> I don't think it's practical to make the ER module a special case of an ET.
ET is only an optimization for tree data storage, which evidently can be done using the same field referencing API. Perhaps ET could be {past tense (-ed)} #1669052: Port to a backend storage mechanism for Entity Reference's Tree plugin.

I asked chx in irc if it would still make sense to have this type of optimization even for document-oriented databases, thinking that mongodb's bson engine would automatically do it and that this was primarily for relational databases. He said that this optimization is relevant not just for relational data, that it does make sense for document-oriented storage engines. I'm not convinced. Even if the document-storage engine didn't optimize the data representation automatically you could update indexes on CRUD ops, which pretty much automates it, but outside the engine. Perhaps this would be better to discuss in this in #1167144: Make cache backends responsible for their own storage and #504012: Index content when created. He also said that graph databases wouldn't need this optimization.

I think RDFx's use of ARC2 is the only Drupal implementation of a graph database, but there doesn't appear to be an effort to extend it into an entity storage backend, only fleshing out its RDF model and faux-fields implementation. It's not really the topic of this issue, but if we want graph database support, I think it would make sense to write an entity storage backend for ARC2 or another triple store library and/or perhaps a generic XML entity storage backend with RDF lifting and lowering, which is what I was thinking about for Views XML Backend. It wouldn't cover all of RDF's expressiveness, but this seems in sync with a Canonical Entity representation format.

I also want to mention a coinciding effort on keeping valid caches for relational data with a Materialization plugin for Views. Introducing an optimization method like this in core was already proposed by Damien and is further linked from #1261120-5: Deprecate Tracker module in D10 and move to contrib in D11.

A lot of that gets away from OP, so going back to taxonomy, here's a rabbit hole of issues related to this reference, orient, and optimize issue: #277312: Propose your change to taxonomy module in core and #1207326: Refactor taxonomy hierarchy API for performance, consistency, and convenience. I haven't tried Tree yet, but I'll check it out soon.

Here's chx's entity_tree presentation in pdf.

Edited. And, sorry about the tags; please re-add them in the next comment.

This issue here was initially filed back in 2005 and is quite lengthy too! Any reason to keep it open and not have an updated summary to go with it? As is one has to pointlessly go through the whole issue and read really outdated posts. Please either update the issue summary or close it in favor of a new issue. Thanx in advance.

Closing this a duplicate of current issues and trying to connect the ideas from 4.x to 8.x.

For relating entties, see:

• #1801304: Add Entity reference field
• #1803064: Horizontal extensibility of Fields: introduce the concept of behavior plugins
• #1805996: [META] Views in Drupal Core

For complex entity types (hierarchy / oop-style composition):

• #1797740: Introduce the EntityTypeAwareInterface interface for entity operations uniformisation
• #1782460: Allow to reference another entity type to define an entity type's bundles
• #100925: Add "OO" bundle inheritance features
• #1446600: EntityFieldQuery (pseudo-)join support

And for a D7, XML Entity / Field Storage Controller, see Salsa Entity and Salsa API.