Does anyone know of any modules that specifically wraps a custom node around an external object provided by a webservice? To be clear, this really doesnt have anything to do with the Services module; I'm not extending a service, but am on the receiving end of an existing one (SOAP interface to be exact).
We're trying to build a community site around a set of objects from an external source but of course have all the bells and whistles attached to these object representations locally, eg, with stuff like wishlists, favorites, and adding comments about the objects, all through what drupal offers up as an effect of having a custom node implementation.
Ive already run into a couple road blocks:
- the session favorites module that has the ability to email some one your favorites is, unfortunately just that, tied to a session only, not user based. The very similar flag module might work, but then doesn't support the emailing facility of session favorites.
- chicken and egg problem: i do a search which queries the external webservice that returns a result set that doesnt know anything about my nodes. When should I match up the resulting objects returned with my node wrapper? I dont know that creating the nodes during the query is very optimal performance wise. I considered handling automatic node creation on the fly when someone clicks a link to go to the (still non-existent) node's page. This also creates new problems because then if the node doesnt exist, theres no way to attach a "flag" from the flag module to nothing. Means i'd have to hack the flag module to accept other primary key ids than just node ids.
- There's more but thats just at the top of the list for the moment
Ugh. This is a real brain workout to architect. I'm wondering if anyone else has done this in the past and if they have any recommendations or architectural design suggestions to follow. Any help or suggestions very much appreciated!
Comments
Why not implement
Why not implement hook_cron(), make the request to remote server, then parse the response and run node_submit() + node_save()? Should work just fine...
---
Yuriy Babenko
www.yubastudios.com
My Drupal tutorials: http://yubastudios.com/blog/tag/tutorials
---
Yuriy Babenko | Technical Consultant & Senior Developer
http://yuriybabenko.com
Thats a good suggestion
Thats a good suggestion Yuriy. I do have a few questions about how I would implement that:
The number of objects in the external service is quite large, therefore the initial cron run(s) to bring the system up to sync would take forever and perhaps cripple the site during the update. How could I split up a transaction like this across cron runs so it doesn't time out, and doesnt lose track and repeat itself without having to query the database to "see" if a node already exists.
I suppose to even answer the second part there myself, I'd first need to figure out the repeatable set of webservice queries that I can effectively use to to build the nodes and keep the db in sync with the search mechanism. I'm not sure if there's one call to get all the current entries in the db and another call to get all the "updated" ones since a specified date.
Does the external service
Does the external service have some kind of primary key assigned to each object?
If so, you could make the remote requests just like any other pagination script by setting a limit in the query.
---
Yuriy Babenko
www.yubastudios.com
My Drupal tutorials: http://yubastudios.com/blog/tag/tutorials
---
Yuriy Babenko | Technical Consultant & Senior Developer
http://yuriybabenko.com
yes there are definitely
yes there are definitely primary keys. That's what I'm currently using to build the nodes on the fly:
it works like this:
1) execute the search, the result set gets returned with a limited set of information about each object, and its object_id.
2) I publish to the user the table of results adjoining a custom link to a url, eg, mysite.com/my_wrapper_node/1238472 (where 1238472 is the external object_id).
3) if someone clicks on the link i have a handler that checks to see if that id exists in my_wrapper_node table (basically at this point just a table that relates 1to1 nids and the external object_id (actually many-to-1 vids to object_id since ive incorporated support for versions into my node wrapper).
3a) if the node exists, then drupal_goto (node/$nid)
3b) if it doesnt exist, then create it that instant, and then go to the node/$nid
I can see this lazy loading isn't going to work.
One problem that shows up,
One problem that shows up, is what if someone performs a search, before the system has had time to create / update / syncronize itself, and returns a result set that doesn't in fact have a local node representation.
It looks like the cron idea is good, but I'm going to have to anyway, do a set of db calls to "see" if my local node exists and actually load each node in order to display the result set with my included local attributes (flag module links comes to mind).
The problem then arises if not just one node of the result set doesn't exist but say 10. Then the response is further delayed to the user in order to on-the-fly build the node wrapper in order to get a node id that the additional attributes can use.
what could be a way to reduce the perceived delay in this case?