Enable loading and rendering into multiple formats (html, xml, json, atom, etc.)

Interesting! :)

Rails 2 has AJAX and REST support natively built in. That is, a AJAX and REST API is built at the same time as the application is built. In future versions of Rails, Atom will also be natively built in, and other formats might be too. I think that is the direction we have to take. In future, things like an AJAX API, a REST API and/or an Atom API will be assumed and we currently don't have the system in place to make that happen.

This patch is a step in the right direction. I'd like to take this one step further -- but maybe in follow-up patches.

(I haven't looked at the actual implementation yet.)

Seems like a good idea. I read the code and it seems like it can be tightened up a little. i will look closer as time permits. thanks nedjo.

Very interesting patch. This will definitely open new use cases for Drupal.

Subscribing.

Well, in the first place I would love to see such a thing hit the trunk!

Question remains, what the best way is to test this patch.
I'm not a JS-guru, but as I really want to make use of this functionality, perhaps you explain to me what the best way would be to properly test this?

I'd be interested in seeing how this might connect to http://drupal.org/node/144608 -- perhaps allowing specific nodes to be rendered as JSON/ATOM/etc as needed. the 'styles' concept might work for that, as well as passing in $options = array('renderer' => 'json')...

@Eaton, yes. I included code to render nodes in JSON in my first draft of the patch but left it out as I didn't find an elegant approach. Part of the general issue is that the structured data we pass to theming generally includes a lot of data we don't want to expose (e.g., the full node object).

Absolutely. I think we might want to touch base and talk about this at some point -- between the 'node styles' patch, which would enable modules to explicitly request nodes in JSON format, etc... and the Node Rendering patch, which would allow them to be built without any unecessary chunks when that rendering mode is specified, I think we could do some really cool stuff.

This patch stands on its own without those pieces, but it's definitely an interesting nexus...

subscribe

I haven't looked at this patch much though I very much approve of the goal; my Magic Tabs module (see the New York Observer writeup comments) is a very poor approximation of where this patch could go.

Here's an offhand thought: why not have the request specify its desired output format as the first arg? So we register the paths json, xml, other_format, whatever as top-level paths. When they are called as (e.g.) json/rest/of/the/args it sets the global output format then re-invokes the menu handler for rest/of/the/args. If you do not happen to request a path registered as a format, you get the current default output format (XHTML).

It's a complex issue to grasp (which makes is also an interesting problem to solve). :)

I'd just try to be 'extreme' and set a 'simple' goal. For example: I want node, comments, users, profiles and taxonomies to be exportable using XML. In my view, the location/path of the callback doesn't really matter. It could be 'http://example/node/42/xml', it could be 'http://example/node/42.xml' or even 'http://example/xml/node/42'. What matters at this point, is that we have a system in place that allows objects to describe themselves in XML (Atom, JSON, etc).

The core of the problem is not how it integrates with the menu system, but how Drupal objects can be declarative. The answer to that might be in the schema API -- and in particular, an extension to that.

It's exactly why I committed the schema API patch -- I've never been a fan of abstracting the abstraction layer but I do believe in the things the schema API enables us to do with regard to a data API.

Maybe this question gets you started: take a schema defintion from system.scheme and ask yourself "how can we translate a record of that schema to a correct XML blurb?", or specifically, "how can use the node schema definition to turn node 42 into a valid Atom message"?

At what callback that message becomes available, doesn't really matter at this point.

Dries, I completely agree re: using the schema data structure to be able to load complete objects. I was working on this path in April until I was convinced to switch to getting the Schema API into core. (It turns out that none of the cool schema-driven features we want require the Schema API in core b/c the schema.module could just export the core tables as e.g. views does for node and other core tables. However, now that Schema API is in core it will get way more visibility, all other major modules will support it sooner, etc. etc.)

Look at http://drupal.org/node/136171#comment-229414 in the original schema issue. I talk about "loading nodes in a single query." That goal is basically isomorphic to "output any object in JSON/XML" and is a major step towards "incremental database migration." The code is still in schema.module in contrib. It's pretty broken but now that the dust has settled on getting Schema API into core I can revisit it.

This issue/patch, I thought, is primarily about the theme and menu issues. We already do have a somewhat data-driven node representation with $node->content and drupal_render. The ability to request *just* the rendered node contents (or rendered view contents, or whatever) instead of a fully rendered page is in itself quite valuable. Look at http://druplinars.com/seminars. The "All/Upcoming/Suggestions" tabs use a pretty hacky module custom called Magic Tabs that lets me set up static+dynamic tabs based on a defined set of menu paths. It would be great to make getting this kind of data and features out of Drupal easier without first having to tackle the harder problem a completely general schema structure query builder.

That said, I do agree with you, and I'm working on it. :-)

This needs a re-roll...

nedjo, I feel your pain re: the 'lots to do with less than a week until freeze'; we ran into the same problem with the node rendering patch. I'm beginning to wonder if we all need to spend some time to hammer out some sort of 'output type' or 'renderer' as a full-fledged Drupal core concept. Something that pages, nodes, comments, etc. could use for output. It seems like there is a lot of value to be found in making the concept expandable: pdf, email, rss, plain-text, html, xml, json... all those are potential output formats or rendering types.

Solving the problem of 'how a callback can say it wants content [foo] in format [bar] seems like something that needs to follow those other decisions, doesn't it..?

the foo.xml, foo.json, foo.rss, etc syntax seems like it makes sense for that...

Subscribing

Any chance this will make it into D6?

I have been doing some Mediawiki integration and they are pretty far along with their implementation of this. See http://www.mediawiki.org/wiki/API. Note the links on right hand side whic hdescribe further the features of the API.

If you download Mediawiki, api.php is in the root directory; thats the entrypoint.

Anyone gonna be speaking about this at DrupalCon? Such a juicy topic.

Hi moshe could you explain further what going on in the API

nedjo has been doing some sterling work on fleshing out the problems on the architecture and hopefully his work on javascript behaviors can help with this post

http://drupal.org/node/114774#javascript-behaviors

cant wait for drupal 6 to get out the door so we can see some more posts here
keep up the good work

> how can use the node schema definition to turn node 42 into a valid Atom message

Imagine Drupal can dispatch GET request of a node page to node_view_text_html = node_view, node_view_text_plain, node_view_application_json etc. according to url prefix/suffix or request mime type text/html, text/plain, application/json etc. (it is more RESTfull and do not violate opaque URI rule). Default implementation of this hooks in node module should then iterate over node model field definitions (node Schema fields and CCK fields) and concatenate markup. theme() will be called only in node_view_text_html. Other hooks should incapsulate other markup's rules and possibly add type info from model (node schema definition). Then all responce must be wrapped correspondent to markup type to provide valid message.

I found myself thinking about this topic recently and want to record my thoughts. I suspect I'm not the first to think them but I haven't seen them spelled out in exactly this way yet.

I started with: Why do menu callbacks return HTML? e.g.:

function myobj_page_view($obj_id) {
  return theme('myobj', myobj_load($obj_id)); // very simplified, of course
}

"Clearly," the menu callback should just return the data (myobj_load($obj_id)) or perhaps an array of theme-hook and data (array('myobj', myobj_load($obj_id)).

But returning the object itself isn't necessarily what we want. The menu callbacks are the business logic. They should turn data in the database into a nicely digested format for the renderer. This is what $node->content is: a nicely ordered list of data to be returned to the client, whether in HTML, JSON, XML, whatever.

So, perhaps menu callbacks should return a nested renderable content array along with its theme hook. Each element within the renderable content array would also contain its theme hook, a la $node->content. So:

function node_page_view($nid) {
  $node = node_view($nid);
  return array('#theme' => 'node', '#value' => $node->content);
}

The HTML version of #theme node (e.g. theme_html_node) would output the enclosing <div class="node ..."> and then call drupal_render(#value). #value ($node->content) of course could contain elements with plain markup, or #item_lists, or #tables, or whatever. The menu hook build up that data, tagging each #value with a theme hook name indicating its format, but not rendering any of it into any format.

This would seem to make it pretty easy to render into multiple formats. We select the format via some property of the request (path, headers, whatever). Say we choose JSON. Then, when rendering #theme node, it calls theme_json_node() (instead of theme_html_node), which happens to call drupal_render(), which calls theme_json_item_list(), or theme_json_table(), etc. There'd be a default theme_json that just outputs the #value as a string, or whatever the most useful default render is for that output format. And ditto for other output formats.

I used node as an example in my previous comment but, in case it wasn't clear, I was suggesting that every menu callback return data in that format, basically the format $node->content is already in. Each element in the returned array is tagged with its #theme hook, and each output format is responsible for implementing the core set of standard theme hooks. Modules would still provide base HTML functions, and hopefully for most tagged-data formats (json, xml), a standard per-output format theme function would suffice for most cases, but if not the module would also have to provide a custom theme function per output format, not just HTML; ultimately, someone has to write the code to render the data into an output format. And, of course, a theme could override not just the HTML theme function but any output format theme function for a given hook.

I actually doubt I'm making any sense. Good night.

yes, we discussed the idea of the whole page being an array that gets passed to drupal_render(). eaton and dries and others have nodded and agreed that it is a good idea. eaton has worked on node content and was trying to expand that to include the whole node (node links ...) but we ran out of time. we might want a parallel effort to do the page which shouldn't be too hard if you just define the page as a the set of regions (and lets make the center content a region already).

your notion of #theme is a little different from what we have today but it sounds like a good proposal to prepend an output format when looking for the proper theme function.

Eaton's node_view refactor patch is at http://drupal.org/node/134478

Subscribing

Subscribing

Subscribing

This sounds like it could help produce facebook canvas pages. For those who don't know, they can be written in FBML, which is yet another XML document type.

In my work with facebook so far, I've made a theme to produce FBML, and this does a pretty good job. In fact, as I read in this thread about renderers, its not clear to me how they are different from themes. Is there some shortcoming of themes that makes a renderer necessary and if so, why not improve themes to solve the problem? How do I know, in the case of FBML, whether its best to make an FBML theme or an FBML renderer (or both)?

In producing FBML for facebook canvas pages, I've run into a problem I haven't seen mentioned here, so I will try to explain it. Let's say my node body contains:

link to <a href="internal:node/123">another node</a>

where the "internal:node/123" noramlly expands to something like /drupal/node/123. Now, when producing FBML, I need that link href to expand to something like "http://apps.facebook.com/myapp/node/123". This is tricky with Drupal, because the results of node filters are cached. I end up with two filter cache tables; one is used for normal pages and the other for facebook canvas pages. I mention that filter issue because with the content described in this thread it is likely to be an issue, too. For example, if I'm rendering a page as atom, do I want that link to be "/drupal/node/123/atom"? This is something to think about, perhaps the renderer needs to be known early in the request, and affect the way content is filtered.

Note also that with FBML, I would not want to have to append "/fbml.xml" to every local URL (I can tell by the incoming request that FBML is expected).

I'm just catching up with this, so please edify me if I'm missing any of the big picture.

Subscribing

@Nedjo: What does this need to get into the spotlight again? I would love to see this into core (D7 this time), but after reading the whole treath I'm unsure how I could help you...

I bet this issue could also be extended to cover what I am trying to do over at #218830: Popups in Drupal 7: Plugable renderers for generating content

There might also be synergy with chx's: #218770: Drupal Pipes

Subscribe.

Subscribing.

This might overlap with the Services stuff in Drupal 7. JSON Server is a good example of an implementation of a web server implemented using the Services API.

track

There should be two parts of this Services API:

Servers

The servers provide means of communication between the local server, and the external source requesting the data. They are the medium in which the data is communicated. This would include a XML-RPC server, a JSON server, and a RSS server, like nedjo mentioned. Contributed modules, as well as potential future core services, could provide a REST server, a SOAP server, an RDF server, and many others.

Services

The services would describe what data could be transported. The Data API will help us here, as it will (probably) provide all CRUD methods for any data we want to manipulate (users, taxonomy, nodes, blocks, etc). The drupal_load(), drupal_save() and drupal_delete() functions will help us load/manipulate information from any data source within the Drupal database, and pass it along a service through to the various servers available (XML-RPC, JSON, RSS, etc). The service I described here would end up being called the Data Service.

All of this should adhere to the user permissions system. This means that in order for services to provide administrative rights (to do things like save new nodes), they should authenticate with a session, and then hold onto their own session ID.

suscribe

I somewhat agree with Rob Loach's comment about the Services API having two distinct parts. It would be great to have an incarnation of services.module in core. I'll be keeping a close eye on how this issue unfolds.

I haven't really looked at the services module code in detail. I agree that it provides a useful functionality template but I am not sure it is necessarily the model we would want to use for some of the web services we might provide in Drupal core.

The reason for this is that after reading the O'Reilly REST book, it is clear that 'resource oriented' web services (primarily RESTful, be it XML, XHTML, JSON etc) have a fundamentally different model to 'remote procedure' based services (XML-RPC, SOAP etc).

I think the 'resource oriented' service types are best organized around our page creation and serving mechanism (with existing URLs as the 'base'). It seems that this can best be done through refactoring the rendering engine in a way that will allow us to (as much as possible) transparently provide multiple delivery formats (XML, XHTML, JSON etc) with basically zero additional code in each content module. In other words, we would ensure that there would be a 1-1 mapping between the data format returned by the module to Drupal core, and the output format (I think I need to put together and example to demonstrate what I mean here). REST libraries in other languages normally work by automatically translating the data object/array into the desired format - similar to how language native serialization works, but using open formats instead. I propose that we do just this, but enhance it by allowing per-format override callbacks to allow a module (or theme) to manually create this format's textual output. This could (I think) completely replace our standard theme functions for HTML output, and could also be used when more control is needed over the exact output structure for other formats (e.g. when you need to adhere to a standard XML format).

For the 'remote procedure' based services (which I would add, can also be RESTful, but are not resource oriented) there is a need to map the exposed remote API call to an internal function call. This obviously needs some coding for each remote call to be implemented correctly (and securely) and so I think needs a fundamentally different architecture to the resource oriented services. We already have something available for XML-RPC in Drupal core, although I am sure that there are techniques from services module that could be used to make this easier and more flexible.

I was going over this whole thread, and was basically thinking exactly what you just wrote. We're talking about 2 different needs here. Personally, I prefer RESTful resource oriented services over RPC, but there still will be a need for both in Drupal. Ruby on Rails does it so nice and simple, it'd be great to have the same thing. For CRUD, it doesn't get much better ... especially if it were to support client libraries like Rails ActiveResource. But, for things that aren't crud based like sending an email, or hooking into various contrib module functionality, or even just being able to provide alternate web service protocols, we need something else. This is where Services can come in and replace XML-RPC. Services is essentially the same concept as existing XML-RPC, it just expands upon it, splitting the serving mechanism and service callbacks into separate swappable components, which allows us to do SOAP, AMFPHP, etc.

Really happy to see all this moving forward.

A minor code-related point.

+        // This construct ensures that we can keep a reference through
+        // call_user_func_array.
+        $args = array(&$variables, $hook);
+        foreach ($info['preprocess functions'] as $preprocess_function) {
+          if (function_exists($preprocess_function)) {
+            call_user_func_array($preprocess_function, $args);
+          }
+        }

I am not convinced that call_user_func_array is required here. The following should accomplish the same thing more elegantly, unless I'm missing something here:

foreach ($info['preprocess functions'] as $preprocess_function) {
  if (function_exists($preprocess_function)) {
    $preprocess_function(&$variables, $hook);
  }
}

Subscribing

Wim Leers was kind enough to point out this thread in the comments to Centralized module to control feeds and feed types.

It seems that in some ways many people share an understanding of what this will accomplish. It would be great if someone can break down how this will make generating Atom and RSS feeds easier. Also, will this make my patch for a Syndicate module in core not necessary?

I am working on this at the code sprint...anyone else wanna work on this let's get together. I am in a black SoC t-shirt ;)

A number of us at Drupalcon put together a quick design sketch of how the proposed API would look. There must, of course, be better ways of doing it. Any input would be greatly appreciated.

I unfortunately missed Boris' session regarding RDF, but any input from that crowd would be good. Seeing how the RDF module already implements its rdf_service method, it seems like both would seem fit for collaboration.

Several of us discussed this, and various use cases that may want to use it at the sprint today - here are my notes.

Here is a small random sample of the use cases:
- Pulling structured content into another site - e.g. XML (possibly with XHTML rendered fields), but just the content - not the whole page
- Getting a representation of resource that can be PUT (or POSTed) back to edit the resource
- Jumping in on a form submit and offering a user a FAPI form in a modal dialog (e.g. confirmation of deletion)
- Updating the page following a FAPI submit or other operation
- Submitting a random form from javascript

There are several aspects that need to be considered for this to handle the use cases we discussed.
1. Data format (XHTML, XML, JSON, RSS, RDF...)
2. Part of page (Full page, a part of a page, data associated with a form)
3. Load vs. render (If you want to submit to forms you want 'load', if you want to show content to users you mostly want prerendered stuff)

Most of the use cases would probably break down something like this.
Page
- xhtml

Part page
- xml
- json
- xhtml

Form Data (input and output)
- xml
- json

We noted that the hook registry is really critical to implementing a lot of the more lightweight calls rapidly. We also discussed a lot of the workflow for AJAX/AHAH stuff, but this is out of scope for this issue.

Regarding URLs and format selection the preferred approach for most people is to integrate with our current schema, and build off that. The desired format could ideally be selected either through the HTTP Accept header, or a extension suffix. A lot of the details still need to be discussed and defined, but here are some examples:
- node/123 gives the page in the default format (normally XHTML). This is the canonical URL for the 'user-friendly' representation of this resource.
- node/123.xml gives the node in XML format with parts rendered where appropriate. The format definition would control if the full page or just the resource itself is included in the output (normally only XHTML would need the full page with blocks etc).
- node/123/edit is the canonical URL for the data representation of this resource (normally a HTML form by default).
- node/123/edit.xml is the XML representation of this resources data, that can be PUT (or POSTed, if we can't get PUT to work right!) in the same XML format to submit edits to this resource.
- node/123.dc.xml is a variant XML representation of this resource, normally to conform with an alternative representation (e.g. Dublin Core), or perhaps to request a subset of the entire resource.
- search is the canonical URL for a form. Each form that we want to access using APIs needs a URL - I think pretty much all core forms have this already.
- search.xml is an XML representation of this form (perhaps using WRDL), which could be submitted to and results would be returned in this format.

We could also provide alternative, more function oriented URLs, such as form/formid - perhaps as an intermediate step.

Implementation notes still to come....

Subscribing.

subscribing

subscribing

subscribing

Here is a reroll of the latest patch. I am still working out the smaller details though and will post a more detailed writeup soon.

Wow, there is a lot of overlap between this approach and Quicksketch's proposed reworking of theme() over at: http://drupal.org/node/218830#comment-791590
(Which I used in my latest popup patch over at: http://drupal.org/node/193311#comment-835854)

The cleanest way to tell the system what representation do you want is setting the Accept http header. This is what other systems use and is a recommendation for RESTful services.

Also, this is what your browser already does and is completely inline with other HTTP headers used for other purposes, e.g. Accept-Language for l12n and i18n and Accept-Encoding for output compression.

Comparing requests headers with response ones we have something like this:

Accept (e.g. text/xml,application/xml,...) + Accept-Charset (e.g. ISO-8859-1,utf-8...) -> Content-Type (e.g. text/html; charset=utf-8).

Accept-Encoding (e.g. gzip,deflate) -> Content-Encoding (e.g. gzip).

To make every jQuery ajax call request a JSON representation, we can use something like this:

jQuery.ajaxSetup({
  'beforeSend': function(xhr) {
    xhr.setRequestHeader('Accept', 'text/javascript');
  }
});

But to tell the system if this is an ajax call the history is a bit different, so maybe we can rely on X-Requested-With header or explore other possibilities.

Subscribe

After much thinking on it, here is a simplified version:

- it eliminates the special handling of page data, which was somewhat copied from theme preprocessing, and instead truly rely on the proven theme preprocessor
- keeps printing the page output, as by that time, we need to generate the output... it would make much more sense to keep structures before, but by this time, we generate the output and send it out, so no need to work with those arrays
- adds a new "success" parameter/variable to the page theme, so we don't need to try and make that up ourselves... those generating the page should know whether it was a success or not... also, the resulting HTTP response includes the error code (404, 403, etc), so no need to encode that ourselves in a special format

Still works fine for me. Please empty your cache before you test (after you apply the patch).

And the actual patch.

Subscribing. Need tie to test. *sigh*

Added XML rendering support with the XML-RPC formatting we built into Drupal already. This is a "generic" but well defined XML serialization format. It could be simpler, but this was already there and the XML generated should be easy to handle. There is no renderer trigger for that, I am looking for ideas on what should trigger that. (Accept: text/xml is something browsers send along, so it should not trigger the XML generation itself).

subscribing.

I am not sure we'd like to use the exact XML-RPC serialization, eg. Java does simpler formats, but that does not include type information, even not as much as this one. Although there you can rely on the data being loaded into an object, so it is properly casted from/to the define data types there. We don't have that thing to do here (and we are only managing output).

I was also wondering on maintenance_page, but I was not sure whether we intend to have that in XML or JSON. Although we will probably should, thinking of external applications, right.

Also agreed that services module integration is a separate issue, this one is "just data export", while services would be configurable, named endpoints, with possibly customized protocols.

I also like using a file extension as trigger. So node/12.xml returns an xml doc.

I actually favor the HTTP Header mechanism. HTTP headers exist for this sort of thing, let's use them. Pseudo-file-extensions are a hack.

There's also the envelope question, though. We need a way to differentiate between "HTML rendered version of a node as an HTML fragment" and "HTML rendered version of a node within an HTML page". For that, I see no good alternative to a custom HTTP header.

I guess this isn't an either or question. We could honor an http header is sent, and if not fall back to file extension. The utility of file extension is that you don't need code to properly make a request. You can't easily alter http headers in a browser. I know some extensions/tools exist for this, but they aren't nearly as ubiquitous as an address bar. So, lets support both.

@Gàbor: "... I am looking for ideas on what should trigger that. (Accept: text/xml is something browsers send along, so it should not trigger the XML generation itself)."

Yes, but it is up to the application to weight what format it prefer to render, e.g. if a browser send 'Accept: text/xml, text/html, */*' the application can choose to serve 'text/html'. But if an client application requests 'Accept: text/xml' the system should serve a xml representation or return a 406 (Not Acceptable) error if a representation of this type is not available.

Some browsers Accept headers:

Internet Explorer 7: */*

Firefox 2.0.0.14 / Safari 3.1: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5

Firefox 3.0b5: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

More here: http://johnex.se/header_accept.php?cmd=list

As you can see, Firefox 3 moved to preferably accept text/html.

JQuery is well alware of Accept headers, e.g. if you call $.ajax({dataType: 'xml'...}) it will request with 'Accept: application/xml, text/xml, */*', and 'Accept: application/json, text/javascript, */*' for dataType = json.

@Gàbor: "I was also wondering on maintenance_page, but I was not sure whether we intend to have that in XML or JSON. Although we will probably should, thinking of external applications, right."

IMO, in this case we should issue an 503 (Service Unavailable) error along with a representation of the maintenance page on the requested format.

@Crell: "There's also the envelope question, though. We need a way to differentiate between "HTML rendered version of a node as an HTML fragment" and "HTML rendered version of a node within an HTML page". For that, I see no good alternative to a custom HTTP header."

I'm too think this should be a header, like 'X-Requested-With' that many Ajax libraries sets to 'XMLHttpRequest'. Maybe we should check for other values or abstract that too?

@moshe: "I guess this isn't an either or question. We could honor an http header is sent, and if not fall back to file extension."

I think that should be the reverse, if a file extension or query param is set, use that. If not, fall back to header. If no Accept header is found, then fall back to the default renderer (html).

As a person who routinely has to debug stuff like this, I'm not crazy about using http-headers to select content. It's very convenient to be able to go through a broad range of functionality using URLs and manipulating headers isn't as convenenient when things go wrong - which they do. Extensions can get confusing, as can anything tacked onto the end; because if you think of a URL like 'command +varargs' things always get added onto the right to provide more granular results, and in the most degradative case these are human phrases.

Which brings me back to the left. xml/node/21, atom/node/21, rss/node/21, json/node/21, mobi/node/21, print/node/21, lisp/node/21 - you get the idea; with the default representation being xhtml. This is fairly trivial to parse at a high level for a finite set of representations, and the URL accurately conveys what is being represented. If this parsing takes place before processing, it can be stripped out of the query string (and a global variable set) so that from a module perspective it's all node/21. They don't care until it's time to theme the data.

Atom publishing protocol might force how some of the underlying things are done, so it would be a good idea to consult that documentation as this progresses - mostly to avoid painting oneself into a corner. It's kind of an application wrapper itself around the site data and fully supporting it could have a bit of impact how the data and XML architecture should be best organized so it's tightly integrated rather than an awkward bolt-on package as it is with WordPress (for instance). Many other representations are purely structural renderings and don't have as much impact on how things are done.

I agree with macgirvin that using Atom as a use case makes a ton of sense.

I also think we should support both headers as url-based format requests.

Attached is a draft implementation of drupal_requested_renderer() function that returns a renderer based on a. and 'format' query argument or b. the Accept HTTP headers.

Two more thoughts:

- Can we use mod_rewrite to change from a extension to a query argument. E.g. from http://somesite.com/node/1.xml to http://somesite.com/?q=node/1&format=xml.

- Can we pass the 'format' argument as the first (optional) argument for a menu_callback? This would allow menu callbacks to respond to different requested formats.

regarding #72, there are two modules I know of which rely on values prepended to the path. They are i18n and Drupal for Facebook. Both use custom_url_rewrite in accomplishing this.

Custom_url_rewrite is handy to have, but its implementation does not support multiple modules. (It's not a hook, its a single function you can define in settings.php). So it would be a pain to combine two (or more) modules which rely on custom_url_rewrite. We have an opportunity here to improve this situation, perhaps completely replacing custom_url_rewrite with something more flexible. As we consider it, let's bear in mind...

• Multiple prepended values need to work together, i.e. either atom/en/node/21 or en/atom/node/21.
• If mobi/en/node/21 links to "internal:node/42" (using pathfilter style syntax), the link that's generated might need to be mobi/en/node/42. While rss/en/node/21 might link to xhtml/en/node/42. That is, some links need to be of the same type, while other do not.
• The previous point applies to links that come from node bodies, and also blocks, breadcrumbs, etc. Currently, drupal caches the results of most input filters, making this sort of treatment impossible.
• Whatever technique is used to introduce these prepended paths, third-party modules should be able to define their own additional ones.

I've given this some thought lately, because Drupal for Facebook has a complicated url rewrite scheme. Because a request for a facebook canvas page can come from facebook directly, or from a user's browser, AND because both fbml and iframe type pages need to be support, AND because a single drupal instance can support multiple facebook apps, I end up with paths like "fb_cb/1/fb_cb_type/iframe/node/21". What I mean is, there's more use cases for this than initially meets the eye.

@Dave Cohen - thanks for the info, though it may pretty much invalidates my reasoning. (Which was basically that putting a representation on the left of the URL would be less messy and easier to implement than putting it on the right - so we wouldn't have to tag it.)

If we can't rely on having the pole position in the URL I agree that a 'format=xxx' tagged option somewhere in the path unfortunately makes the most sense. I would still prefer this as a pseudo path element rather than an option string so that it doesn't negate clean URLs completely... something like is done with the page=2 option.

Is there an issue for making custom_url_rewrite a proper hook? Though it isn't needed here, it's going to keep coming up. It was intended as a kind of administrator's pathauto swiss army knife IIRC, but now that we've got multiple modules using it they're all going to keep colliding until it's fixed.

Is there an issue for making custom_url_rewrite a proper hook?

Not that I know of. I think in some cases custom_url_rewrite needs to do its thing before modules are loaded. Perhaps a new method should be introduced in addition to custom_url_rewrite than than trying to replace it.

Most of the issues I mentioned are relevant whether we use the pole position or not. For example, links from ?format=mobi pages would still want ?format=mobi appended to their URLs and I think there will be some challenges to doing that.

Clean URLs only matter to the "HTML / page" use case, I think. There's no SEO advantage to an AHAH request or an XML-RPC request having a non-clean URL. So that point is moot.

Using the HTTP Accept header with an override from a GET parameter makes sense to me. That gives us both a "proper standard" mechanism and a "probably actually works in the real world" mechanism, and we can use whichever makes the most sense. Trying to cram it into the path on either end is going to be too messy.

Regarding passing the request type into the page handler, I disagree. We should allow a given path to have multiple request handlers depending on the request type. See: http://drupal.org/node/218830#comment-792841 (Early version, it's evolved a bit in my head since then but I haven't written it down.) That way node/$nid always returns a node; it could be HTML/page, HTML/fragment, HTML/json, json/json, SOAP, AMF, or or whatever else, all at the same path, depending on the headers and GET parameters. We also get envelope handling that way.

subscribing

I also want to put in another vote for settings headers as one of the defaults mechanism to trigger a different render mode. This'll make Tao's work on popups possible (#193311: Ajax Popups in Drupal 7: Adding Modal Dialogs to Help, Confirmations and Filter tips (Unified)). The approach taken here is nearly identical to what I came up with in #218830: Popups in Drupal 7: Plugable renderers for generating content (I wish I would've found this issue first!) I'll reroll the popups patch to use this patch instead of mine in #218830.

subscribing

@quicksketch: I am super excited to see that patch. I should have a window of time opening soon to try pushing on 193311 again.

I would also prefer to see the render mode indicated as a query string (as a backup to the header info). e.g. /node/1?render=xml NOT /xml/node/1

I think this will be simpler to code and maintain, and we won't have to deal with conflicts with i18n where language codes are appended already to the left of the path.

I have rerolled Gabor's patch and integrated much of the feedback. This new patch accepts the query string field "render", which can currently be either 'xml' or 'json'.

The header field is honored if it is provided (thus selection of json if the request is made from an XmlHttpRequest via jquery), but this can be overridden by the query string field.

Glad to see this moving along.

function drupal_render_json() {
  $args = func_get_args();
  drupal_json();
  print drupal_to_js(array_pop($args));
}

is that not the same as

function drupal_render_json() {
  $args = func_get_args();
  drupal_json(array_pop($args));
}

?>

In order to get a feel for how this renderer system works, I'd love for us to change one or two pages to use it. Are there any suitable pages in core? Can we change the node/feed link so that it is http://example.com?render=rss. We can keep the node/feed path as an alias for existing sites.

I need to run this through a debugger to see when/if preprocess functions are called for non HTML functions.

You are absolutely right about the drupal_json function taking the arguments. I have made the modification and rerolled the patch.

@moshe - yes I think the latter form is a more compact version of the same thing.

I'm not clear why we are using array_pop() - drupal_json() and xmlrpc_value_get_xml only take a single argument, so why not just declare the two arguments and pass the second? i.e.:

function drupal_render_json($template, $value) {
  drupal_json($value);
}

I think this can be changed due to registry magic - maybe use module_invoke?

include_once 'includes/xmlrpc.inc';

This looks funny - is this to try to put system module first?

array_reverse(module_implements('renderer'))

Some other code here could be simplified if we can fix module_invoke.

also do we want to use echo() in preference to print()?

I think a killer user story for this is to output a node as Atom when render=xml. http://tools.ietf.org/html/rfc4287#section-1.1 has some examples of single entry Atom feeds. This could make it a lot easier to import/export a Drupal site as well as to syndicate content.

To export a Drupal site, you'd do:

  foreach ($all_nodes as $node) {
    grab_the_atom_version_of_the_node();
  }

How cool would that be?

http://drupal.org/project/atom might have code to lift to generate Atom.

apparently this xmlrpc render is expecting a full node object:

notice: Trying to get property of non-object in /Users/Shared/www/drupal-7/includes/xmlrpc.inc on line 91.

re-rolled the patch per my suggestions above.

hmm, still not quite right, but at least this gives some output for xml.

I think this approach is really missing something - we want to be able to render each node/user/etc into XML or JSON - when all we have now is the already-rendered content, I think it's too late. We should postpone this until we make progress on: http://drupal.org/node/134478

#230710: Centralized module to control feeds and feed types is on hold until this issue is resolved. Does postponed mean later or does it mean post Drupal7?

postponed means I think we need to deal more fundamentally with issues of rendering, so parts of this patch may get rolled into http://drupal.org/node/134478. Hopefully, all within the D7 scope.

subscribe

#193311 is waiting on this patch as well. Just to add to the pressure :)

@starbow - I'm planning to do this work in the other issue.

Currently, the testing framework uses HTTP User Agent, combined with the approach described here the system could render metadata to be consumed by tests, see: #266220: Exposing metadata to tests

Does this really have to wait for #134478: Refactor page (and node) rendering?

Yes, because we need structured data coming out of modules (not html blobs) so we can redirect theming to an alternate format at a global level. Otherwise we are left with a pre-theming and per-module "switch and die" approach (as we do now for json etc), which tends to duplicate code and is plain ugly.

#99, NO this doesn't need to wait. Look, one thing is rendering a page in multiple formats, useful for e.g. enabling Drupal to work without full page reloads (this issue). Another is to be able to give developers (and themers) a way to selective render pieces of node content. IMO non rendered data is just useless for AJAX, i.e. it would need a theme implemented in both server side (PHP) and client side (Javascript). In the other hand, non rendered data is useful for external data APIs.

Subscribing.

Could some one explain with the hook_page_alter commit to core yesterday how does this effect JSON XML loading of content
http://drupal.org/node/351235
I looked over the patch and it breezed over me ....

Is there any link on people trying to implement this functionality

later
M

I think the best chance of moving this functionality forward is going on over at:
#355236: Refactor drupal_render theming - docs

Subscribing. Wanting to get my head around what this would mean for presenting a simple javascript popup to elicit user confirmation before changing something. e.g. http://drupal.org/node/107038#comment-1228507

#355236: Refactor drupal_render theming - docs and #351235: hook_page_alter() have both been committed. This should make some things here quite a bit simpler.

Subscribing. Any breakthrough on the user access questions?

I know I could really make use of a way to integrate an AMF server using the new Zend_AMF PHP library. It's an Adobe sponsored successor to AMFPHP, written by the author of AMFPHP Wade Arnold. As a Flash developer, I get very excited about service based architecture and want to help make sure that I can bolt on an AMF server to Drupal 7.

Any renewed interest here?

The Services module is, without a doubt, core-worthy. It's well written and highly adopted.

spell check

Anyone still working on this? (Also, I added the old title back in a subtitle, because I think it's a little clearer, and clarified that this issue was about REST services more than XML-RPC a la Services module.)

Yes, I am and will submit a patch in a couple days. Assigning the issue to myself.

Thanks for changing the title, but the patch I'm working on will only do the rendering, not a whole REST API with PUT AND POSTS. So I'm changing the title to the original.

Will update here soon.

That's great to hear!

And thanks for changing the title again - I would've just out-and-out reverted the title, but I didn't want to limit the scope of the issue any further than necessary. I just wanted to get it back on track, since I didn't think the original issue was about Services API in core at all.

Any update on this?

Hi. I just found this issue. I like some of the thinking that's been done here, but it doesn't look like anything is close enough to converging in time for the D7 API freeze on 10/15. However, I submitted a different, but related issue: #599804: Unify page, AJAX 'path', and AJAX 'callback' callbacks. The purpose of that issue is to bring consistency to page callbacks, but the solution I'm proposing I believe lends itself to some cool stuff along the lines of what's being discussed here being done in a contrib module. Please check it out and comment on whether a small change to that patch would make the work of this issue easier.

Agreed that this is D8 material at this stage.

Ok, unfortunately I had no time to work on that in this development cycle.

Cross-linking and trying to promote http://groups.drupal.org/node/93644

Subscribing

Subscribe

Subscribing

To move this forward: What do you say about trying to incorporate the Services module into Drupal 8 core? I think that would be a good starting point for this.

@voxpelli: the starting point for this will very likely be http://groups.drupal.org/butler - which then can serve as a base architecture for native web service interfaces in D8.

Butler and Services are quite a good fit for one another and work has already been made to make Services fit well into a Butler world :)

subscribe, Ive been working services lately and would LOVE to get some of this into core. Ill write some stuff up as far as battle plans are concerned

Subscribe.

Upping to major, because Dries identified this as one of the major priorities for Drupal 8 (see http://chicago2011.drupal.org/keynote-dries-buytaert, 43 minutes into the video).

Setting to "active", because the last patch is from several years ago, so time to rethink our approach in light of work already committed to D7 and goals for D8.

I can think of several things we'll be wrestling with here:

• Detecting which format is wanted: probably part of the scope of the Butler initiative, and there's existing content negotiation web standards we can base this on.
• Replacing the 'delivery callback' API with something much more elegant and robust: also part of the Butler initiative.
• We'll probably identify some improvements needed to drupal_render() and render arrays in general to make certain things easier. For example, if we want drupal_render() (or a replacement function) to be able to output a JSON string from a render array, how can the render array identify which of its properties (i.e., keys that start with "#") should be included or omitted? We have #access to control at the element level, but nothing to control at the property level. http://drupal.org/project/services and http://drupal.org/project/restws are already currently dealing with similar questions, just not necessarily at the render array level.

In addition to the above, the part I've been thinking about lately (based on some of the conversation earlier in this thread) is what is needed at the theme layer? For example, do we need to expand template naming to be per format, so that we have stuff like node-html5.tpl.php, node-html4.tpl.php, and node-atom.tpl.php. I sure hope not. I hope that the following two assumptions are true:

• That a given theme is responsible for just 1 html format. This is yet to be decided, but most likely, for D8, html5 will become the default. But, for example, if a site requires html4 to be sent to IE8 so that it can look good even for visitors who disable JavaScript, then that should be handled with an html4 based theme. Similarly, if lighter-weight markup is wanted for bandwidth constrained devices, then it should be done with an alternate theme. In other words, we should provide a mechanism by which the page renderer can select the appropriate theme to use, but we do not need to provide a mechanism by which a single theme can output different formats via differently named functions/templates.
• That no format other than html requires theming. For example, JSON, XML, ATOM, etc. can be output via standard serialization functions (with sufficient opportunities for module hooks to customize), rather than something that needs tweaking at the theme level. In other words, no need for a node-atom.tpl.php file.

What do you all think? Are the above assumptions valid? Or will adding complexity to the theme layer be necessary to properly support real-world use-cases? If you have experience building a real-world multi-format website, your input would be very helpful.

There is really no such thing as having an "HTML5 version" and "HTML 4 version" of a site. Dries misspoke during his keynote. Let us not confuse the issue by even discussing such a concept.

More to the point, the theme layer as we know it today simply becomes one output plugin among many. That it's the most configurable is, in a sense, coincidental. :-)

There is really no such thing as having an "HTML5 version" and "HTML 4 version" of a site.

We'll see how html5 in D8 is implemented, but as I understand it, it may be the case that some html5 tags that are desired by a site will be problematic when rendered on IE<9 without JS. Many people might choose to not care about this, but some sites will care, and will want a mechanism by which to not output any tags not defined by HTML 4. If we don't want to call this an html4 theme or an html4 version of a site, what terminology do you suggest to capture this concept?

More to the point, the theme layer as we know it today simply becomes one output plugin among many.

That's awesome. Really, really awesome. But core will continue to ship with a specific theme layer implementation, and we'll continue to have the concept of theme hooks, and therefore, expected naming of theme functions and templates. So I think the questions in #130 are still relevant. I guess knowing that it will be pluggable means we don't have to solve every possible use-case. We just need to make smart default decisions that cover the majority of use-cases.

@Crell - effulgentsia is asking an additional question. He is asking if alternate output formats like JSON need to be themeable (it is a given that they are hookable). HTML output format certainly does. I agree with @effulgentsia that I don't see a need for any other formats to have themeing.

And yeah, we need different name besides output format. Thats taken. Maybe 'response format'.

@Crell - effulgentsia is asking an additional question. He is asking if alternate output formats like JSON need to be themeable (it is a given that they are hookable). HTML output format certainly does. I agree with @effulgentsia that I don't see a need for any other formats to have themeing.

And yeah, we need different name besides output format. Thats taken. Maybe 'response format'.

"Response Formats" is what Flickr uses, so that might be a good naming convention to go on. Note that XML-RPC, REST and SOAP use special "Request Formats" other then just GET/POST, so we'll need to accomodate for that in here too.... Services module uses "Servers", but Response and Request Formats does seem appropriate.

+1 for Response and Request Formats.

About theming, could someone bring an use case where we will need to theme for something besides HTMLx?

Let's not confuse this issue with html5 support. I think the consensus is pretty clear that we will not support switching HTML versions based on user-agents - but either way it's a topic better suited for http://groups.drupal.org/html5, #963832: Decide how to use HTML5 in Drupal 8, #1077356: HTML5 in core battle plan

Services module uses "Servers"

Correction, Services module 2.x uses Servers, Services module 3.x uses REsponse and Request formats by looking at Accept and Content-Type headers, as well as the path, ie node/1.json or node/2.xml, etc and all of this is extensible so you can write your own response format.

I don't think we should have any theming of JSON etc., it adds a tonne of complexity and there's no use case. If someone can come up with a use case, then we should look at whether that can be met without a templating system (let alone the actual theme layer) first.

We are going to need to cleanup a lot of places to do this properly - like drupal_exit() drupal_page_footer(), hook_init() - some of these have stuff in that's specific to rendering HTML - this would can be done parallel to the overally implementation here, for example #1029460: Find a better hook than hook_init() for drupal_add_js() and drupal_add_css() is already a valid issue in Drupal 7 and 8 even without response formats.

Two more things, which might be off-topic but fwiw, we currently render images via the page system, then call exit(), the same has to happen for RSS feeds (which I'd hope would be covered by this anyway), and css/js aggregates (if I get my way in other issues), and private files. It's possible we might want to use stream wrappers or similar to do the file paths (maybe), but we should try to figure out whether we want to include binary file requests here at some level.

Subscribe.

…it may be the case that some html5 tags that are desired by a site will be problematic when rendered on IE<9 without javascript (js). Many people might choose to not care about this, but some sites will care, and will want a mechanism by which to not output any tags not defined by HTML 4.

IE doesn't have any problems with HTML5 elements. IE 5/6/7/8 just ignores them completely (on a site that does not use the js shiv), while still seeing the things between the elements. For example, <figcaption>This is the caption.</figcaption> is seen by IE8 as This is the caption.

The problem with not using the js shiv (or having js turned off) comes into play when you target that element with CSS. If you put this in your stylesheet figcaption { color:blue; }, no-js IE8 would not apply it to the <figcaption>, and your caption would not be blue.

A solution to this is to just not target any CSS to new HTML5-only elements. Write your code like this: <div class="figure-caption"><figcaption>This is the caption.</figcaption></div> and .figure-caption { color:blue; }. Then you support styling content in IE 6/7/8 with js turned off.

I do not believe Drupal needs a way to revert to HTML4-only elements, replacing new elements with old ones. Besides, by the time Drupal 8 comes out, there will be tons of contrib modules that rely HTML5 elements. Anyone who uses any of those modules will end up using new elements. A revert-to-HTML4-option would have to do something with all of those as well. It'd be a mess.

Developers who use Drupal 8 will have a choice to either not worry about the edgecase of IE 6/7/8 without js needing styling, or support non-js IE 6/7/8 styling by targeting their CSS to classes, not elements. In both cases, the site is not broken. It doesn't explode. It doesn't crash the browser. No ponies die. The content is still fully accessible. It works functionally just fine, just without all the visual styling.

I feel dirty subscribe....

So a few thoughts on the last few comments, in no particular order:

• While this issue is about loading and rendering (and not C-UD) I think it is worth noting that we not only need to be able to get the render array as a data response to be used for AJAX-style page refreshing and mangling, we also need to be able to get the underlying data for the underlying entity (if any). The underlying data can allow for RESTful C-UD operations (another issue), which the render array cannot. Among other things, this implies that render format plugins may eventually also need a corresponding response format plugin (for handling POST/PUT).
• I am not sure how we would distinguish render/page-oriented requests from entity oriented requests, but from what I have read of the Butler plans it seems like this should be fairly easy. Perhaps the convention could be something like /page/path.json gives the render array data, plus a list of entity data URLs that occur on that page (which could have URLs like /node/123.json or perhaps something slightly distinct so as to not conflict with render/page-oriented requests).
• While I don't think we need a theme function for json, I think we still do need a fair bit of flexibility - for example even though you may want the render array, you may want specific branches (or all branches after the Nth level) to be rendered down to HTML (wrapped in json) rather than having every separate #value as raw json. This can probably be done fairly easily using our alter hooks and/or callbacks (could override the default output generators). With this we need to consider API stability, so perhaps there is a case a layer of "data structure" abstraction in addition to the "response format".
• Other examples to consider are ATOM, which is a semantic format - which will need some context around what counts as an "item", what is it's "title" and other metadata.
• One way to handle this flexibility would be for response formats to define a set of "hints" (although hopefully core could figure out a set of useful general ones and defaults) that affect how the default response generators structure their response, in the form of additional render API properties. These would affect what is included/excluded in the output, ATOM-style semantics, XML property vs. element etc.
• There are some touchpoints with the deployment/config-management work - for example it would be great if URLs and IDs (at least when outputting entities) used DRUIDs consistently, rather than auto-increment IDs. We would need the ID mapping layer for this anyway for entities (or persistable thingies), and it would mean that the web API is less tied to the site internals, potentially allowing some crazy/cool stuff (e.g. allowing external deployment systems that simply GET from one site and PUT on another, or having "distributed nodes" that sites cross-aggregate node updates around git style). General thoughts on this at http://groups.drupal.org/node/134569#comment-454584

Of course, starting simple is good - I think a good first goal for this specific issue could be to get basic content-negotiation (with a .ext option) for render arrays into an initial json response format plugin. Of course we should keep our eye on the prize of other render format plugins, as well as eventual handling of entity data and CRUD.

Correction of #139:

Services module 2.x uses Servers, Services module 3.x uses REsponse and Request formats

Services 3.x does also uses Servers to distinguish between different request types - like REST, XML-RPC, SOAP, AMF - and they are pluggable so that you can have different implementations of the same request types. There eg. exists multiple REST-like modules, multiple RPC-style modules and multiple AMF-modules.

Services response formats isn't simply encoding data into a data format - they are also paired with data models which transforms the data into what is needed to build eg a RSS-response. It also allows these data models to alter the request arguments, to make sure that eg. all fields needed for a node is actually fetches if there's a possibility of defining which fields to fetch. The data models can also take arguments of their own to eg. define if a RSS-feed should include full text or an excerpt and these can optionally be exposed as arguments in the request as well.

I think drupal core should support REST requests in a similar way to the REST server - but I also think drupal core should ensure that those who need other request types like RPC, SOAP or AMF can reuse as much as possible of the system that's in place for the REST server. Use the same resource callbacks etc.

@Owen Barton: Take a look at how the REST server in Services 3.x solves it - it supports complex semantic response formats like rss and ical and also supports crud on resources.

I would actually argue that render arrays are unique to the HTML output format. They are too rigid to use effectively for anything else, and the meta data we would need to pass around, as Owen notes, would get ugly very fast.

My initial sense is that JSON and other such formats don't need theming at all. return json_encode(node_load($nid)); is all we need. Ibid for other data-centric response formats.

That said, Owen makes a good point that you may want JSON delivery of "partially HTMLified" data. I'm not sure at the moment how we would handle that. One option I suppose is to simply allow people to swap in their own JSON responder that does exactly what they want, but I don't know if "subclass and write your own code" is an adequate solution. Maybe it's a first-cut solution at least?

We probably want to consider defining Drupal-standard ways of representing Drupal data in some standard formats, such as JSON, Atom, etc. Entities are moving more and more toward having a standard data structure and API, with Fields and, potentially, a standardized system for properties. If we can agree on a standard way to represent that structure in other formats (JSON, Atom, custom XML, RSS, etc.), then we can automagically have new Fields supported in all output formats, which would be really really nice.

It would also make it possible for 3rd party systems to support communicating with "Drupal content" rather than just "the particular Drupal content on this one site I built custom support for."

RSS at least needs something more than rss_something(node_load($nid)); since we have configurability for how much of, and what, gets output in the RSS feed (and then there's things like enclosures etc.). For this we definitely want something like renderable arrays (including caching of the rendered/formatted output - I was able to save around 50% of time rendering time for RSS feeds in D7 by applying render caching to the feed items, need to write up that bit elsewhere though).

Also it's not just formatting one entity, it formatting a list of entities, with a wrapper around that list (which will include channel information etc.) - this lends itself very well to an array which contains a list of entity IDs and instructions for rendering plus some other bits and pieces.

Also even in core, what you get from node_load() is not necessarily what you want to output in json - do we want to know whether the current user as voted on a poll when requesting JSON? This needs cleanup, documentation and possibly some enforcement so we don't waste both cycles and bandwidth sending cruft to requesters.

I would actually argue that render arrays are unique to the HTML output format.

My initial sense is that JSON and other such formats don't need theming at all.

I agree with the second sentence, but not the first. Theming does seem like an HTML-oriented process. But rendering does not. You can certainly do a lot just with serialization of entities. But that alone isn't enough. Consider, for example, a use-case of wanting to build a native iPhone app interface to Drupal administration. How do you get back some kind of serialized representation of the admin/modules page or any other administration page?

Owen makes a good point that you may want JSON delivery of "partially HTMLified" data.

This isn't hypothetical. This is what the D7 AJAX system already does. By default, it currently makes the split at the page callback level. In other words, what the page callback returns gets rendered as HTML, but then that gets enveloped into JSON in a way that's usable by ajax.js. There's a couple ways we can carry this forward to D8. One is to split at the block level. Since Butler will basically refactor all of the flow that happens between index.php and block display, it can decide on a block by block basis what format to render each block in, and then call the corresponding display/render plugin. This alone will give us what we already have and more. If we want to go deeper though, another approach is to support something like a #format on render arrays, and have drupal_render() itself capable of either calling #theme/#theme_wrappers as it does now, or else some other function appropriate for the #format of the element. Child elements can inherit their parent's #format by default. If we think we might want to go down this path though, it would be good to develop some use-cases, so we don't build too much abstraction without clear goals in mind.

I would actually argue that render arrays are unique to the HTML output format.

If render arrays make sense for HTML, I would argue they make sense for any XML format, and any structured markup. What would be nice is if when building the render array the code knows in advance what output format is expected. A smart node render hook might build the render array differently depending on the output format.

I'm not sure we can speak simply of HTML as if it were just one kind of output. Owen brought up the possibility of "partially HTMLified". I suggest there are multiple "flavors" of HTML. While a simple HTML link might be

<a href="node/42">view the node</a>

If that HTML were appearing in an iframe, we might want this "flavor":

<a href="http://example.com/node/42" target="_top">view the node</a>

And if that iframe were a facebook application, we might want this link:

<a href="http://apps.facebook.com/example/node/42" target="_top">view the node</a>

Right now, modules/fb jumps through hoops to make links like that. I'd love to see Drupal build that more efficiently in the future.

I just marked #1133984: Formatting fields values not only for html output. as a duplicate of this issue, but there is some good explanation over there so I do recommend reading it over.

A couple remarks coming from the issue Crell just mentioned :

- for file or image fields; drupal_json(node_load($nid)) would only give you something like :
"filed_file : {fid : 21, path: 'public://my_image.jpg'}"
--> +1 on the need for "partially massaged data". node_load() is a tool for internal PHP code, we cannot expect it out of the box to return data ready for the outside world.

- how could you specify you need a json with the url for field_image *in a given image style* ?
Right now, the only concepts we have for 'configurable output' in this area are formatters / formatter settings / view modes. All of which are very much HTML-oriented right now.
Field formatters are a "massaging tool" towards HTML. It so happens that the basic ones are suitable for JSON/XML/... (plain text, number, file URL...), but many others very much not (an actual image with <img>, a carousel, some script to show a location on a Gmap...).

I closed the issue and marked it as duplicate since.

I am reading the 151 posts before contributing to this thread, just to say that my idea to separate domain logic and presentation layer, with swappable presentation layers still stands.

• AMF: with strongly styped object (Flash/Flex)
• JSON: with raw data (Javascript/PHP/...)
• XML (for XML/RPC which should die and be replaced by SOAP)
• XSD (for SOAP where we need strongly typed object as well)
• Atom
• HTML4
• HTML5
• HTML Fragments
• etc.

Before being a Drupal developer I came from a Java background, then Flash and this is both 2 potential client for a Drupal backend system. Those languages are strongly typed and in any serious WebService application, we'll need some notion of Class Mapping. In this issue I introduce a system to annotate classes before sending it to a flash client: #1048838: Class Mapping from Drupal Entities to Flash Object.

Attached three screenshots to demonstrate the needing of typed object through flash debugger:

1. Generic Object: http://drupal.org/files/issues/Generic.jpg
2. Semi-Typed Object: http://drupal.org/files/issues/Semi%20typed%20Object.jpg (here the node content type is declared as teo.vo.Gallery, which is a mapping from $node->type (gallery) within Drupal).
3. Fully-Typed Object: http://drupal.org/files/issues/Fully%20typed%20Object.jpg (I mapped d6 cck fields to Flash class).

Also, we should speak to the guys from WSDL group: http://groups.drupal.org/taxonomy/term/11176. Because they will have probably some good ideas about services description language.

I am very interested in how a Drupal web service might respond as an example with a list of items in JSON format as opposed to the current HTML output of a full page. Hence, I am subscribing to this thread to hopefully help move this initiative forward.

subscribe

I would actually argue that render arrays are unique to the HTML output format. They are too rigid to use effectively for anything else, and the meta data we would need to pass around, as Owen notes, would get ugly very fast.

Agreed.

My initial sense is that JSON and other such formats don't need theming at all. return json_encode(node_load($nid)); is all we need. Ibid for other data-centric response formats.

I don't think it's sufficient, as usually more access checks are involved and some additional data-massaging is needed.
Regarding the access checks, consider field access or just private properties some users are not allow to see. Similarly, we need full CRUD access on data property level as you don't want a regular user to be suddenly be able to write to $node->status.
Then, the additional data massaging I think is in particular necessary for properly outputting resource references (-> URIs), which you want to look slightly different for different formats in practice. If you look at the references in the example outputs from http://wolfgangziegler.net/restful-web-services-for-Drupal you'll note, that the PHP array equivalent of the JSON and XML output have to be different. Not mentioning that other output formats, like RDF or some XML output compliant to a given scheme, require customized formatters and/or more knowledge about the data which is actually output (data type, RDF mapping, ..).

We probably want to consider defining Drupal-standard ways of representing Drupal data in some standard formats, such as JSON, Atom, etc. Entities are moving more and more toward having a standard data structure and API, with Fields and, potentially, a standardized system for properties. If we can agree on a standard way to represent that structure in other formats (JSON, Atom, custom XML, RSS, etc.), then we can automagically have new Fields supported in all output formats, which would be really really nice.

The exactly the idea behind the entity property metadata system of the entity API. We need to have a standardized representation of any data property based on its data type, such that formatters as well as other APIs can rely on that. That way, we'd finally have knowledge of our data model and could start building things on top of it.

That said, I really think we should put something like the entity metadata system in d8 - maybe more deeply integrated with the field API. So we know whether a data property may be accessed or updated, whether it should be output as a reference to something else.

Also note, that the entity property metadata system of the entity API is not restricted to entities at all. E.g. Rules as well as the WSClient and RestWS modules use it to describe and work with any data. Thus, it really establishes a way to describe any data in Drupal (and to work with it based on that information).

@restws:
Perhaps I shortly describe how it works in restws here.
There, for a request the "resource", the "format" and the operation (one of CRUD) is determined. Then the request is totally handled by the format controller, while it makes use of the resource's controller (which abstracts CRUD) and the resource's metadata (specified using the entity API-way).
For core I think it should probably work similar, but for our general-purpose request router we don't want to be limited to certain operations. What's left then, I think is basically that what is already proposed for butler: Identify the resource in question, use that plus the other HTTP request information to "set up" the context and use it to determine the right response(format) callback.

My initial sense is that JSON and other such formats don't need theming at all. return json_encode(node_load($nid)); is all we need. Ibid for other data-centric response formats.

Certainly not, services works like that and its not the way to go. Image is always a good example, but it applies to lot of fields. If you got a node with a field_image taken from your camera, and want to display it as a thumbnail in your AJAX application, passing the result of node_load would send the 4MB raw file (or URL). Unless you hard coded the styled URL of the image to actually look, your application will takes ages to load. Moreover, if you change your style machine name, your application will not display anymore images; Unless, again, that you do the update in your AJAX client as well, which is absolutely not maintainable.

Subscribe.

This sound like the need of having a real fully featured MVC implementation in Drupal. Provided patches here are just patches, not a real refactor. I did myself some work about this topic here.

As I understand, some of this is already fixed in D7.
Should we not start a new issue with an up-to-date summary?
I'm totally lost here.

No its not, as far I know. But it would be good to summarize in an up-to-date topic.

sub..

sub..

We haven't had a workable patch in over 2 years and I'm not seeing any real promising approaches in this issue. I'm on the side that renderables got us far along this path, at least as much as I've ever experienced needing it. I think this is pretty much a moot point now, I'd suggest opening a new issue with a dedicated purpose to any suggested approaches. This issue is spinning wheels with a bunch of theoretical use-cases with no actual reason why we need further refactoring.

As effulgensia said in #148:

If we think we might want to go down this path though, it would be good to develop some use-cases, so we don't build too much abstraction without clear goals in mind.

Downgrading to "normal" since we don't actually have anything to execute on in this issue. Per http://drupal.org/node/1201874, major tasks are now delaying all development of new features and this task with no direction to move forward doesn't make sense to postpone the rest of core's development.

Invalid JSON: Click registering not implemented. i downloaded the version 8 and i still get this. it is driving me crazy for it pop ups all the time. i do not know how to repair it for i dont know that much about how to do that. please help.

@jacqulinej please do not highjack threads with something totally unrelated to the issue.

I'm going to discuss some use-cases that would suggest we need to re-write/ditch/whatever Render API during my core conversation next week: http://london2011.drupal.org/coreconversation/1300-1330-responsive-conte...

Regarding theming of non (X)HTML content, one case I think of is a number of other XML output formats. For instance, someone may want to "theme" a SVG or raw XML output by linking it to a stylesheet after the fact. Case in point: XML sitemaps. These are currently designed to be consumed by search engine crawlers, which could not care less for styling, but are also usable - to some extent - in a browser thanks to the (currently bundled) raw XML styling, but this was not always the case. Or one might want to alter a SVG rendering by redefining strokes attributes and whatnot in an attached CSS file.

Does this just fall under "altering" or is it "theming".

I would argue that non-HTML formats do not count as "theming". The are rarely, if ever, fluid or flexible enough that you'd want to customize per-site your output the way you do with HTML.