Improving the OpenLayers 2.x module architecture

Hey David,

It's good to hear that another person's thinking about the module architecture on a complete level. I agree that there are some difficult choices here, and there's certainly a lot of work to be done before this module is generally consumable. I'll try to respond to this ticket - not that your observations are in any invalid, but a lot of the architecture choices have a bunch of thinking behind them which is non-obvious.

Let's start with the Views integration: Currently, you have to create Views with OpenLayers data displays. The user then creates a preset, and the OpenLayers module discovers and makes available all the possible data displays so that they can be added to the "preset" as layers. Then you create a view that uses the OpenLayers map display style, and select the proper preset in the display style's settings.

To my addled brain, this behavior and methodology is highly non-obvious.

Yes: this behavior is not obvious. It's the kind of thing that people do need to read the documentation for in order to understand. What we've made recently are baby steps - alpha7 gives a helpful note to people trying to create Openlayers Data layers via the OpenLayers interface, for example, but it's far from point and click.

First, calling such an entity (map settings, base layer(s), and Drupal-provided data layers) a preset is misleading. I expected a preset to correspond to a set of default settings on top of which I might wish to layer my own data. I suspect others may have a similar reaction when they see the word “preset”.

Yes - I would like to see the term 'preset' changed to 'map' in future versions. Since this isn't a bug, it's been prioritized behind bugs, but it needs to be done eventually. It was also reliant on a bit of the architecture that has been late in arrival - self-powered maps, without views.

Third, this behavior also seems very counter-intuitive and technically limiting from a Views standpoint. For example, given the current structure, how one possibly expose filters for the data layers, or sanely handle arguments?

Currently arguments are passed down from the OpenLayers Map view to the OpenLayers Data views - this can, and has, enabled users to do, say, a view in a block that shows the current node. There is a ticket for supporting exposed filters: #789668: Support exposed filters from OpenLayers Data displays, but the implementation of that feature is tricky so it hasn't been written yet.

With the current structure, the data layers and final display need not even live in the same View. While that may have some advantages, I don't think it should be the default behavior.

This has rather huge advantages, actually. Pushing data views into their own area is big, because with Views modes, one can present an OpenLayers Data view as a map, and then one of its attachments as a table - or allow KML or WFS download of the displayed map. It also has the benefit that OpenLayers Data Views can have different base tables - a map can display layers from nodes, taxonomy, data, and whatever else, without a hitch.

The alternative method would be that the user creates OpenLayers data layer attachments that can then be attached to an OpenLayers map display in the same view. The data layers could inherit the parent display's arguments (optionally, just like regular attachment), configure some of their own styles and behaviors, and provide exposed filters to the parent map. I asked a random sample of a couple of Drupal developers and asked how they might expect such a module to work, and everyone had some variation of this same idea.

I'm not sure how this differs from the current behavior: one can easily create an OpenLayers Map view with OpenLayers Data views as attachments. Is the difference that OpenLayers Views acquire their own styles and behaviors? It seems like there would also be very large blind spots to this system - in that a data layer can't be shared between maps, for instance.

While writing the module and wrestling with the OpenLayers module's functions, it became clear how much the module currently hinges on the idea of presets as the primary container of layer data. Why should the rendering function go through a build process that goes out and fetches the data? Shouldn't the rendering function just take data directly and render it instead of doing a stealth lookup?

Could you clarify this? The institution of layer types and the render() method is simple encapsulation, and personally it doesn't seem like a major stumbling block. Rendering functions shouldn't 'take data directly' when the taking of data directly can be cleanly encapsulated, it seems.

I know this module is already in alpha, but the current architecture choices seem likely to cause pain and trouble in the future – users are going to be frustrated, and the Views integration will have a hard time taking advantage of Views' featureset and API.

This module is in alpha because I, and a bunch of other module maintainers, feel like version numbers are a weak indication of status. Very mature modules like Feeds are in alpha - alpha15 to be specific. This isn't to say that OpenLayers 2.x is a very mature branch - it won't be that until many bugs are ironed out. But it's widely deployed - more deployed than the release candidate-quality 1.x version - and all compatibility-breaking changes are noted and avoided.

If the current functionality of “presets” is still desired, the module could introduce an additional concept of a “map” which could work very similarly to how the data layer selection process for presets works now. These “maps” themselves could be made exportable. The user interface would be far cleaner (select a preset, select a couple of layers provided by Views OpenLayers data displays or from non-Views sources), and such a system could potentially allow for maps that are
more sophisticated than those provided by Views.

You'll need to clarify how this could allow for more complex maps than are currently possible. I also think that one of the current faults of this module is that it provides two routes to map display - Views and CCK, of which the latter suffers because the developers (myself included) don't use or prefer CCK displays. Increasing, rather than decreasing, these routes, is not the direction of current development.

So, in conclusion:

I agree entirely that the current views-based setup is convoluted and nearly impossible to understand without reading the documentation: currently this module comes with the semi-requirement that users need to read documentation, which is very rare amongst Drupal modules (the rest just require you to write patches).

The solution to this problem which has been floating around for a while is to have self-sustaining maps. This means that presets are renamed maps, and they can present themselves on pages, blocks, etc., without requiring an extra view. This functionality will be made possible by CTools page management - which has recently become stable enough to be used.

It would also make a lot of sense to have a module that makes 'map making' easier by abstracting away the additional functionality to generate views based around common use cases.

However, the reasons behind the current complexity come into play often in real-world situations - there are use cases in which your module would make your life a lot easier, but there are others in which it would be impossible to do what this module does currently without a complete architectural rewrite. So, I'd rather the interface become simpler and the 'preset' dillema be clarified than to switch strategy at the level you're suggesting.

That said - I'm reading through this module and the workflow that it embraces is common. Currently a lot of the users who could be using this module are using Gmaps or something else instead because the learning curve is high. So I'm not trashing the idea of having an interface which supports this workflow better and doesn't require cracking open documentation to get started, but I'm suggesting that there are ways to support this workflow better and keep the flexibility of the current OpenLayers module that does actually come into play in real life. So, absolutely - I'll see if there are parts of this module which could be guides to the future UX of the OpenLayers module, and if you wish to write patches that clarify the muddy parts of the infrastructure, that that would be great - or even if you want to form a new module that hits another user segment and has some kind of compatibility with OpenLayers proper, then that would also be appreciated.

But, in this form, I think that it would help if we discussed the reasons behind the current architecture because the reasoning isn't immediately obvious but is incredibly needed in real situations.

@davideads, thanks for the thoughts. Like @tmcw says, outside opinions are always encouraged, as it can be difficult to remove oneself from their own project enough to have a objective perspective. Some initial comments:

* Presets should definitely be called Maps. Started new issue: #806318: Rename Presets to Maps

* The current workflow is very hard for most people to grasp, even after reading the documentation. Pretty much everyone I have tried to explain how it works to has been lost until I actually demo it for them. Even using attachments, a lot of people that are basic Drupal users and site builders will probably have a hard time. The Views interface and parts are complicated no matter what.

Having an attachment approach seems like a good idea. The ability to keep things in one interface is important. As @tmcw suggest, this could be possible right now. I have not looked at the attached code yet, but there could be some good UI directions there.

I personally think focusing on utilizing another display mechanism that can be managed in the map/preset interface is pretty crucial. Unfortunately it is not necessarily best to write our own, and nothing provides a generic way to handle this outside a separate complicated interface (ctools page manager is only for pages, I believe, and its interface is just as confusing as Views). I think considering writing our own is worth that conversation; remembering that 90% of cases just want to display maps in a block or on a page, and that really isn't all that hard to do.

* As far as the API, I think it could be cleaned up, but the goal as it sort of stand is that we "build" the map data first, then "render" it. "Building" is taking a basic data structure and filling it in with all its parts. The "rendering" is taking that data and creating output from it. I think there are definite points of cleaning up for this, like naming things differently accouting for the difference between a map (which is a preset) and the data array, as well as providing a function for processing a preset name, and spearating out the data array and preset_name parts more, etc.

I'm somewhat convinced by the concept of a 'automap' type functionality in which attached OpenLayers Data views are automatically added to a map - and I think that this could be somewhat accomplished by simply adding an option to the OpenLayers Map style plugin. But, of course, it becomes worse and worse when you think it through and realize that this conflicts with need for style and behavior configuration, and the duplication of interface required for that to work.

Anyway, I kind of feel like, despite that being a great idea and very implementable, the concern that's causing much of your dissatisfaction with the module is not strongly connected to the changes that you're suggesting on the UI level. It seems like there are two things - an architectural change, which I must say I don't support, and a UI change, which seems more reasonable.

This line of thinking prompted my initial post. It is why I was a little disappointed when I walked up to OL and was initially disoriented. I also must admit I didn't hit the docs until after I'd looked at the Views code for the map rendering ('return openlayers_render($preset->data);'), looked at the data member and didn't see any *data* as I expected it, and was all, like, "whoa, this isn't very transparent, what's going on?!"

Render is a poorly defined word. And the preset data doesn't mean the map content, much like how a View definition does not mean the View content. However, you really need to give another shot at understanding why data is not involved at this stage.

Layer types are encapsulation of data. Creating a new layer type allows you to have one map that pulls data from a Oracle database and another map that pulls data from who knows what, and to have the custom code for pulling the different kinds of data entirely encapsulated - so that one doesn't interfere with the other, and the map doesn't need to know the exact methods of pulling. Yes - it is more straightforward to simply allow people to hack around this limitation and shove data into the map array at any point in processing. And, don't tell anyone, but it's abundantly simple to hack it that way - it's simply a whole lot worse architecturally because layers are no longer objects that can be pulled into different maps, moved to different websites, exported in code, but now live as little code hacks in random places. Yes: there is boilerplate code to write in order to implement a layer type. Perhaps it needs better documentation but the difference in difficulty between pushing data into a map object and pulling that data using a layer type is not monumental and is overshadowed by the hackishness of the former and the sustainability of the latter.

Relative to how well the module handles your use case: I was involved in a recent project that involved a Views 3 query plugin with Sphinx backend. It was quite straightforward, and didn't require any patches to OpenLayers.

The way it currently works -- pushing map building into rendering, and then layer, behavior, and style processing into building, with data retrieval happening in the {FIND OUT} step -- just seems convoluted.

As far as layer, behavior, and style processing - in the current 2.x branch, all three have been boiled down to taking the stored data, calling the defined 'render()' method on layers, and pushing it to JSON. There is no smoke and mirrors here.

Data retrieval happens in the render() method of a layer type. It's the only place it happens in PHP, ever, and it's really quite obvious how it happens if you simply read the openlayers views layer type. Yes - you'll need to read code or documentation, but if you're going to write code, I think that's a reasonable assumption.

My main suggestion for rendering is to allow the map object passed to the render function to directly contain data, behaviors, styles, etc, not just pointers to stuff that will be filled in later. I understand this is a design goal, but if other 3rd party developers are like me and expect similar behavior out of their render functions, the current implementation is going to be frustrating and surprising.

This can be accomplished by using a 'generic vector layer' - see #734702: Make Vector layer generic for details. This may be exactly what you're looking for - a way to push arbitrary data into the map object from any source.

What I'm really suggesting is that by shifting the architecture a little, and decoupling a couple of the pieces, you will wind up supporting more real world use cases and significantly decrease the always alienating RTFM-factor.

It really seems that pulling data and pushing it into the map object directly is infinitely more coupled than to have the map be agnostic to data source particulars. That said, having a more obvious method of pushing data to the map would reduce the RTFM factor for moderate hacking.

It seems prudent to me to use both approaches, despite the risks of too much choice, and I think it would be beneficial to everybody -- developers and end users -- if the Views technique and the preset/map technique were not as tightly coupled, and the merits and drawbacks of each approach explained in the module help strings and documentation.

My main concerns with having both techniques in the module are that a great amount of code would be duplicated and that it would lead to an increase in code and a decrease in stability when the module is aimed, primarily, at at stability at this point in time.

This, if well supported, would effectively end my complaints and would allow me to write my own OpenLayers views integration that plays nice with the module's core API. I'd definitely be happy to contribute patches to see that one resolved.

That's great - because I feel like that ticket is very close to completion. What's already implemented in CCK for vector layers and the Javascript of OpenLayers Views is pretty much what that ticket aims for, just with a standardized features structure and minimal configuration.

I can't imagine that a "great amount" of code would actually wind up being duplicated, but there definitely would be an increased burden of maintenance and documentation for a system which allows both techniques.

I would just see the interface for assigning behaviors and styles to presets being duplicated, and there's already a tricky dance of being able to assign X to Y for behaviors, styles, and layers, which falters with CCK and would also be a little tricky with this scenario. Not impossible, but it'll be different code for all three cases.

I personally think it would make a great deal of sense to provide a wafer thin core API module for working with the OpenLayers library and displaying a map, and then layering other techniques (your map concept, my views-as-maps concept) on the same core.

The core map rendering process could be a bit split up - I mean, if only openlayers_build_map wasn't called by openlayers_render_map then you'd be able to sneak in at the last minute with comparable data.

Also, how would supporting both techniques compromise stability of the core? Nothing I'm suggesting changes the fundamental behavior of the preset concept in any way, nor does it entail extensive changes to the OpenLayers codebase. And, having developers building on top of the OpenLayers API seems like it should have the effect of make the core functionality more stable.

A solid core is a great thing, and this may uncover some more bugs in it. My fear is that the core functionality of this module is not about putting a map on a page, which is relatively simple, and more about managing map components and assembling them. So, the 'core' that we'd be building off when creating a new structure based on different principles would actually be a fairly trivial bit of code, rather than the meat of the module.

Okay - so here's the ticket for the generic vector layer: http://drupal.org/node/734702

Note that, although that's one way to do it, it's an inferior approach to writing a layer_type, which, for some reason, is cracked up to be difficult but truly is not.

My initial thought was to enable/disable behaviors on the main map display, and set styles in the data layer attachment display. Is that sensible? Are there gotchas?

There are a few gotchas - there are layer-specific behaviors and there are non-layer specific styles.

I understand that the core functionality of the module is not really about putting a map on a page and that the 'core' I'm speaking of is almost so lean as to wonder if there's any "there" there at all. But is that necessarily bad? My gut feeling is that even if the core is whisper thin, and only handles the very tip of the mapping iceberg (drawing a map and putting it in a page), creating a thin wrapper around the OpenLayers library in Drupal is good for the overall flexibility of the module, and allows people who know the underlying library to jump right in. This is very much a matter of personal philosophy, but I tend to expect a module which provides an integration with a 3rd party library to provide such a thin wrapper that exposes the basic functionality of the library, and then build off it to do fancier things.

Good points throughout. Yes - the core that puts a map on a page is whisper-thin, and I don't think that it should be any thicker. Duplicating the OpenLayers Javascript library - or adding a full 'translation layer' between it and PHP would be a crowd-pleaser but a thoroughly poor decision, because it would inextricably tie this module to the intricacies of the Javascript below it (and their variations and bugs), while also adding a larger Javascript translation layer - since, of course, this module will never try to write 'functional javascript' as a result of its labors.

Alas, in some places, this module basically follows what I can see somewhat as the Drupal philosophy - requests for 'maps from arrays' are kind of like requesting the Views module to work off of arbitrary arrays - it doesn't, and won't. Developers need to write query backends, display plugins, and style plugins to achieve their dreams, and the learning curve for doing so is balanced with the sustainability of the result.

You've seen that the generic layer type was committed: http://drupal.org/node/734702#comment-3030206 ? That hits the area of 'dynamically creating a layer', although really the ideal way to dynamically create a layer is to write either a layer type, a views backend, or both, so that it's not a one-off hack. Creating a layer type (as well a creating behaviors, etc.) really is not difficult.

And disappointment and frustration with the architecture is pretty commonplace. This is a module that got wonky in the 1.0 version, got a bit better in the 2.0 version, and is still attempting to solve problems that aren't solved elsewhere - complex problems in simple ways. If there are more specific concerns, post away.

Can you be more specific? As in, you have multiple layers on a map by clicking one? Or having layers from something? If you want layers to be provided dynamically, then implement hook_openlayers_layers like in openlayers_views.module. Actually providing layers is not in the domain of layer types.

If you're adding layers based on AJAX requests to a map, and you want to add them dynamically, then create KML or some other kind of remote point layers like GeoJSON, etc. to the map.

If the layers themselves, not just the data, need to be pulled from AJAX requests (for instance, if this traceroute all happens after a page is loaded, by some dynamic form submission), then use a behavior to create dynamic layers, since there's nowhere that the layer system would interact here.

Er, then that's not something that the module aims to address. Frustration with the OpenLayers Javascript library, perhaps? We've all got that (although, yes... it is the only thing that does (specifically) what it does).

Closing - open tickets for specific actions and patches.