I wonder why you chose not to use Services for this? Is it not possible to expose entities through Services resources the way you like? If not - why so?

I think it would be great if a project like this was built as an extension of Services. If it proved successful then it could be included in upcoming major versions of the Services module. As it looks now no such thing will be possible and also no modules built to support Services will be usable with this module.

The Services module need more contributors - there's already more to be done there than the current contributors has time to do. Why not instead of spending time on duplicating Services help the Services guys out in getting that module as good as possible?

Comments

kylebrowning’s picture

+1 we could really get this into the already existing services module, no need to fork.

This is easily handled by creating your own Resources and just as an FYI those could easily be brought into core services

klausi’s picture

Version: » 7.x-1.x-dev

This is definitively not a fork of services as we go after fundamentally different design goals. Instead of describing each entity type as resource, we want to leverage the full potential of the Entity API that already contains property information, CRUD functions etc. This generic approach is capable of dealing with entity types that do not even exist yet or that come with any third party module. Furthermore we do not believe in the concept of endpoints - resources should always be available on the same URL path pattern. /node/1.json gives me the JSON representation of a node, /user/1.xml gives me the XML representation of a user, /profile/1.pdf gives me the PDF representation of a user profile. This is simple, robust and eases client development.

Maybe this module can be an inspiration for services 4.x, but I don't see that it fits to the design of services 3.x right now. I don't want to piss you brave services developers off, but rather suggest a radically different approach how RESTFul web services can be exposed in Drupal. However, I'm open to discussion and look forward to talk to you guys at the Drupal Developer Days Brussels next week.

voxpelli’s picture

Services is resource agnostic - it provides a few built in resources just like Views provides a few built in Views definitions but the core functionality of Services is not its resources - it's the framework for exposing resources through API:s. Resources can be wrappers for entities or wrappers for something entirely different. You can eg. create resources that replicates the Twitter API if you'd like - or you can create resources that exposes Drupal entities in a different way.

Endpoints are one of the main new features in Services 3.x and was born out of the need to have different authentication mechanisms and permissions for different API:s. The feature originates from real world client work made by us at Good Old so there's real use cases for it. If you eg. want to expose an AJAX API with Session authentication but at the same time want a REST API that requires OAuth authentication then you need to have one endpoint for each so they can have different authentication mechanisms.

Endpoints are exportable which makes it a tool for increasing interoperability between Drupal sites. A module like this one could contain an exported standard endpoint for its resources that would be shared by all sites using the module. We used this in http://drupal.org/project/oauthloginprovider to make sure that all sites using that module would get the log in resource exposed at the same path so that clients consuming the resource would know where to look. If anyone for any reason wouldn't like our endpoint though they would be free to create their own - which I can't see as something bad?

I can still so no reason why this module couldn't be created as an extension to Services. I see nothing new added by it except in how it exposes entities which it could just as well do through Services recourses?

I unfortunately won't be at the Drupal Developer Days as I will be in Berlin for the Transmediale conference.

klausi’s picture

I think authentication should be ignored completely in a core REST service module and should be handled by separate modules. HTTP is a stateless protocol, so an authentication module must always read the credentials somehow from the HTTP request and perform a user login. I see currently 3 common ways for REST services:

1) Session cookie: Drupal handles that for us and does the user login. Nothing to do here, yay!
2) HTTP Authorization header: see restws_basic_auth.module (a submodule of this project), 8 lines of code to login the user.
3) OAuth: A module could read the access_token from the URL, map it to a user account and do the login.

The authentication must be done at the beginning of page request, then the RESTful web services module is able to work with users and the permission system. I see no problem even when all 3 authentication methods are active, a resource representation (/node/123.json) can live at the same place in any case. Am I missing something?

voxpelli’s picture

I think what you're saying makes sense - and I think that is what we're doing with Services and when we're not I think that's because we've discovered that we needed to do solve it differently. If you can prove us wrong and show that we can simplify Services - then great! But if not - then don't assume that we did what we did just because we wanted to make things more complex - we almost certainly did it out of necessity.

One of the things you seem to assume to be simpler than it is is OAuth. You don't log people in with OAuth - if you do then you're using it wrong. You give clients access tokens containing a subset of the issuing user's permissions. That way the user knows that clients can't change its password or see its email address - which is very important. OAuth also makes it possible for the user himself to choose which level of privacy he wants against a client.

klausi’s picture

I'm pretty sure you had your reasons for developing services as it now is, no offense.

I think it is a good idea to handle authorization completely with the user/permission system. For an OAuth module this could mean to create a new (sub)user that has restricted permissions and that is used to log into when an access_token is passed. That would protect the user's privacy. I can see that this could lead to problems with the "own" permissions (e.g. "edit own content") ... *sigh* yeah, OAuth is not trivial.

voxpelli’s picture

So you're more or less justifying duplication of the Services module by it being "a good idea to handle authorization completely with the user/permission system" or am I missing something?

klausi’s picture

No:
1) Neither is the code nor the intended functionality a duplicate of the services module.
2) Athorization via the user/permission system is just one aspect among the differences between restws and services. Using entity metadata from entity.module instead of explicit resource information for entities is another. Convention over configuration (i.e. no explicit endpoints) also. We know that in Drupal the HTML representation of a node lives at /node/123, so we also want to have the JSON representation there (either by sending "Accept: application/json" in the HTTP header or by appending ".json" to the URL). As I said, we have some fundamentally different design decisions here.

klausi’s picture

Coincidentially Jeff Eaton just made a blog post about Drupal 8 and missing APIs:

The second missing API is a built-in foundation for web services. Drupal's early automatic support for standards like RSS and BlogAPI helped push it ahead of competition that required grafted-on solutions. As next generation standards for data exchange emerge, though, we risk falling behind. In Drupal 8, we can build on the foundation of the Entity API: automatically exposing all Drupal entities in a RESTful fashion would dramatically simplify the work of building desktop and mobile apps, complex cross-site publishing tools, and more.

That is exactly what we are trying here with restws.

kylebrowning’s picture

Actually klausi, you can use ACCEPT headers for application/xml or application/yaml or json in services 3.x and you can even do things like node/1.php for serialized options.
I dont see how it is fundamentally different.

You can already do everything you have described in services, and we have an extensible system for adding any type of Response format or auth system as well as the ability to write your own resources.

DId you even look at services 3.x?

voxpelli’s picture

@klausi: That's exactly what Services tries to do as well - but Services doesn't limit itself to the Entity API but also tries to be a framework for providing standard API:s like the Blog API:s.

Google and others won't change their API:s to match any Drupal provided API so we need a framework to be able to expose API:s like the ones Google, Twitter, Flickr and the others provide - only then can we achieve what eaton describes. Services can do that and hopefully any core provided API framework can as well.

Exposing all entities through Services is a pretty obvious future direction and as you know there's an issue for that: #745046: make use of entity metadata

voxpelli’s picture

klausi’s picture

@kylebrowning: I was not able to create an endpoint in services that serves resources at "/" or "<front>", can you explain how I can do it?

@voxpelli: What do you mean by Google-like APIs? Of course "artificial" resources should be possible at any path, but well known Drupal entities should live at their default path.

fago’s picture

The Services module need more contributors - there's already more to be done there than the current contributors has time to do. Why not instead of spending time on duplicating Services help the Services guys out in getting that module as good as possible?

As you know I've started contributing to Services 7.x-3.x a year ago. I tried to move Services in that direction I think it should go, but without any results (see #745046: make use of entity metadata).

ad #10:
>DId you even look at services 3.x?
See above.

See also #745046-12: make use of entity metadata for why we are not building upon services.

heyrocker’s picture

As voxpelli said, Services is for more than just entities (although it doesn't disallow them at all) and thus has to support a much wider range of use cases. Multiple endpoints was a widely requested feature for people implementing their own product-specific APIs. This allows them to expose versioned APIs, as well as having some features be authenticated and some not. Also note that by default Services uses Drupal authentications invisibly just like Restws, additional authentication methods are optional. While Services 3 does optionally support alternate server implementations, we are squarely focused on REST for the most part.

Ultimately I don't really care who does what or whatever. We have got an amazing release with 3.x and people will use whatever modules they want and that is as it should be. We will fully support Entities in the next dot-release or two most likely, and what would have been nice is if this effort could have been focused on making that happen, but no big. Entities are obviously a big part of Drupal 7 but I don't believe for a second that we are moving to a world where everything in Drupal core and contrib for all use cases ever is entitized, and if you want to support USERS rather than THEORIES then acting as such is incredibly short sighted. That's just my opinion tho, and if there's anything there is plenty of room for in Drupal it is opinions. Best of luck on the project.

fago’s picture

Entities are obviously a big part of Drupal 7 but I don't believe for a second that we are moving to a world where everything in Drupal core and contrib for all use cases ever is entitized, and if you want to support USERS rather than THEORIES then acting as such is incredibly short sighted.

While restws strongly builds upon entities and the APIs provided by the entity API, it doesn't force resources to be entities either.

As voxpelli said, Services is for more than just entities (although it doesn't disallow them at all) and thus has to support a much wider range of use cases. Multiple endpoints was a widely requested feature for people implementing their own product-specific APIs. This allows them to expose versioned APIs, as well as having some features be authenticated and some not.

I'm sure there are situations you need that, however we haven't seen the need for it in our use-cases, so we prefer to keep things as simple as possible. Also having some resources authenticated and others not works fine with a single "endpoint".

That's just my opinion tho, and if there's anything there is plenty of room for in Drupal it is opinions. Best of luck on the project.

Agreed! Wish you (and the services team) all the best too! I'm sure both projects can learn and benefit from each other, so in the end Drupal will win.

kylebrowning’s picture

"Agreed! Wish you (and the services team) all the best too! I'm sure both projects can learn and benefit from each other, so in the end Drupal will win."
or be fragmented...

eaton’s picture

Fago and I had a chat yesterday in IRC about the endpoint URL issue; although I can't speak for him, I think we nailed down a useful idea regarding the URLs. Versioning via different endpoint URLs is pretty much standard at this point, although it's also standard to have /api as the url of the current api version. Using native entity URLs by default makes moving TO a versioned API very very difficult, but it would be possible to supply optional menu callbacks on the entities themselves, aliasing those to the normal API callback URLs.

For example, node/%node/json could simply be an optional pointer to api/node/%node/json.

voxpelli’s picture

Regarding the entire endpoint discussion - can be interesting to read the original posts that introduced them into Services: http://groups.drupal.org/node/45636 and http://goodold.se/blog/tech/future-services-3x

No one criticized them then - the first criticism comes now - a year later.

@eaton: You still wouldn't be able to support the old version, would you? With Services you can have multiple versions of an API deployed at the same time - that solved our versioning problem completely.

Regarding having an unversioned API endpoint as standard - I don't think most API:s do that? Eg. the Twitter API is versioned by default. The entire point of versioning an API is to not break clients when deploying a new API and always pointing api/ at the latest API would do that for everyone using that endpoint.

eaton’s picture

@eaton: You still wouldn't be able to support the old version, would you? With Services you can have multiple versions of an API deployed at the same time - that solved our versioning problem completely.

Not with the code that's present in this module. It was more a discussion about the philosophy of having explicit endpoints versus node/%node/$format style endpoints. I'm only just sinking my teeth into the complexities of the various web services issues ;-)

voxpelli’s picture

Regarding the authentication system - Services uses drupals access system for everything. The only thing an authentication module like Services OAuth does is to define what should be required by the client - if a resource method should require consumer key, signing with consumer secret and perhaps require a token of a certain kind.

There are other examples of authentication modules for Services that adds extra authentication for clients - my colleague Simon is about to release one on Drupal.org, Services Accept Origin: http://drupal.org/node/976620

So it seems like the main difference between RestWS and Services when it comes to authentication is that RestWS ignores client authentication - right?

jherencia’s picture

Subscribe.

fago’s picture

@eaton:
I've opened #1046470: Prepare for versioned services for that, so let's discuss that there. I'm still unsure about what's best way to proceed though.

@voxpelli: This is not about criticism services' approach. With multi-server support I agree with you, you'll need multiple endpoints. But this is about finding the best way to support versioned APIs in RestWS.

kylebrowning’s picture

#23, Again it just seems like you are recreating the wheel since services already does this.

joachim’s picture

> Entities are obviously a big part of Drupal 7 but I don't believe for a second that we are moving to a world where everything in Drupal core and contrib for all use cases ever is entitized

My feeling on this is that on the one hand, having a standardized system of services for entities is great. Yay standardization and consistency.

But having *just* services for entities can't be the whole picture. Consider: when was the last time you built a Drupal site where every public-facing URL was a node, user, or taxonomy term, and showing nothing more than that entity?

We're always building sites that show multiple things at once, with complex relationships. In the same way, we're always building services that pull out some bizarre combination of data from the site, depending on use cases often unique to the site in question.

So a service system that has only CRUD operations on entities seems to be to be a very nice demonstration, but not that useful practically.

klausi’s picture

Maybe we haven't been clear on this, bur restws will of course support also "artificial" resources that developers create on demand. I'm currently working on such a resource and will publish an example how it can be done with restws (basically you just have to implement your custom resource controller class and register it with the restws_resource_info hook).

So don't worry ;-)

P.S.: Jep, that would be something for restws.api.php which does not exist yet.

kylebrowning’s picture

"bur restws will of course support also "artificial" resources that developers create on demand"

Which makes this a COMPLETE fork of the existing services solution.

fago-- and klausi--

Dave Reid’s picture

Yeah this has gone from very interesting alternate idea to essentially a fork with the added feature set. It's quite obvious that the maintainers just want to do their own thing for the sake of doing their own thing and there is nothing we can do besides continue the broader community support behind the Services effort. Good luck with the project.

fago’s picture

Status: Active » Closed (works as designed)

>Yeah this has gone from very interesting alternate idea to essentially a fork with the added feature set.

a) The feature set has not been changed at all since we've published the project.
b) We've already explained the different design decisions as well as the difference of the functionality of the modules, all of that is still valid.
c) As there is no common source of the code base, there is no fork.
d) Sad, that the discussion has moved to "that level" - I see no purpose to continue with that.

kylebrowning’s picture

Status: Closed (works as designed) » Active

A. What does that have to do with ANYTHING?
B. Not once have you explained any design decisions that differ from what Services can already achieve. They are NOT valid in the sense that you continue to ignore you can do this with Services and we would love your help to do this in 4.x #1044162: Move to Services, initial port and #745046: make use of entity metadata
C. Its not a fork in the sense you are using the same code, its a fork in the sense that you are attempting to provide everything services already provides. You chose to NOT combine efforts just because we said this feature cannot be in 3.x. Im sorry you got butt hurt about that but it can be in 4.x. We proved you can do entities with Services here #1044162: Move to Services, initial port.
D. Uhmm ok.

Whatever Its cool, ill just make sure to create a project and import Voxpellis code. That will probably get included into core Services in 4.x

Have fun reinventing the wheel.

kylebrowning’s picture

Status: Active » Closed (works as designed)

Sorry, didnt mean to switch this.

klausi’s picture

Status: Closed (works as designed) » Active

For the record, I'll list some differences to services (again). Maybe we can use this for a documentation page later.

* You cannot expose the node resource at /node/1.json with Services (at least I could not do it and nobody showed otherwise).
* Resources are not automatically exposed in Services (you have to create endpoints). In RestWS you just enable the module and setup user permissions.
* RestWS is authentication agnostic while Services provides authentication integration (read: if you need OAuth go to Services, at least now)
* Services is resource API version agnostic (for a new API version a new endpoint is created), while RestWS plans to implement versioning differently (see #1046470: Prepare for versioned services)
* RestWS is strictly REST principles only and does not contain any code for message-oriented service types (read: if you need SOAP, XML-RPC etc. go to Services). This makes the code base smaller and easier to maintain.
* Any custom entity is automatically available via a uniform URL interface (/entity_type/entity_id) in RestWS, no configuration needed. Incoming data is handled by the Entity API wrappers. Insert all the Entity API advantages here.

I can see that you are interested in integrating the Entity API for Services 4.x and we will of course help. However, I don't think that even this step will render RestWS useless nor obsolete.

fago’s picture

* RestWS has data property information baked into the API, such that formatters can/have to make use of it. This allows formatters for e.g. to render resource references differently, as required for the format. Also formatters can format a resource by making use of any additional information available about a resource, as the RDFa example shows by applying the RDF mapping the core RDF module provides for entities.

voxpelli’s picture

@klausi:

1. As shown in #1044162: Move to Services, initial port this is indeed possible with Services
2. As shown in #1044162: Move to Services, initial port the RestWS module could include a Services endpoint activated by default
3. RestWS and Services uses the same authentication mechanism for users - but Services also allows authentication of clients - something that RestWS doesn't.
4. Services has support for multiple endpoints which suits versioning very well
5. Services 3.x is completely REST-oriented but contains some minor code that translates resources into RPC-style code. I see no drawback in that.
6. As shown in #1044162: Move to Services, initial port this is indeed possible with Services as well

kylebrowning’s picture

Fago, #33 is achieved by changing your Content-Type header to application/json or application/yaml or xml. Alternatively you can make your own formatters, alter existing formatters and extend others. It even has support for calling an endpoint as node/1.json or node/1.xml if you dont want to send headers.

How is #33 or for that matter, anything else listed different from Services?

fago’s picture

>How is #33 or for that matter, anything else listed different from Services?

The architecture is different than in services. Services has resources, models and formatters - but restws just has resources and formatters, whereas the formatter has full control / knowledge of the context. That enables the formatter to lookup the RDF mapping for the resource in question and use it to generate RDF that takes the mapping into account.
Then as said multiple times, the availability of property information to formatters enables them to take the info into account, e.g. for formatting resource references as ideal for a certain format.

ad #34:
Well, as your patch has started to show we could handle the requests on our own and then pass the request on to services, where then our resource implementations are invoked. But that way Services isn't a great help any more for what we are doing, as we'd have to do most parts ourselves anyway. Next, being bound to the services API would form an obstacle for us when formatting the resource in the way we want it (see above). So, as Services doesn't help us, we don't use it (for now).

entrigan’s picture

"Why not using Services?" Clearly the central theme in the responses is that using services would add extra complexity. This is not to say that services is overly complex, but that it is possible to build a simpler module by accepting certain assumptions about how the service will work. This is not illogical, hardly a duplication of work, and has precedent in the community (see emf vs. campaignmonitor for one of several examples).

The only concern I see is a fragmentation of the user base and support modules (see the variety of video related modules in drupal 6). Hopefully this will not be an issue!

Thanks for the great work Klausi and Fago, and also to everyone who has worked on services.

entrigan’s picture

"Why not using Services?" Clearly the central theme in the responses is that using services would add extra complexity. This is not to say that services is overly complex, but that it is possible to build a simpler module by accepting certain assumptions about how the service will work. This is not illogical, hardly a duplication of work, and has precedent in the community (see emf vs. campaignmonitor for one of several examples).

The only concern I see is a fragmentation of the user base and support modules (see the variety of video related modules in drupal 6). Hopefully this will not be an issue!

Thanks for the great work Klausi and Fago, and also to everyone who has worked on services.

skyredwang’s picture

So, this project sounds like Views_datasource

klausi’s picture

Yes and no. It is similar to views_datasource as it also exposes content in various formats (as far as I can see). However, RESTWS goes beyond that: you can also create, update and delete content via an interface. And it is not only limited to content (nodes), as all kind of entities known to the system can be exposed. Also artificial resources (as shown in restws.api.php) can be CRUD enabled.

dgtlmoon’s picture

The comment from Jeff Eaton scares me..

dgtlmoon’s picture

@klausi nice simple solution for REST-only, keep up the good work!

dgtlmoon’s picture

Additionally Services module uses form submission functions for inserting new data, makes it impossible for images/files etc

kylebrowning’s picture

Its not impossible at all go look at the .test files. Uploading files to a node is easy.

dgtlmoon’s picture

I think you're wrong kyle, services node_resources only let you add a file and a node as seperate things, you still need to connect them somehow right? How do i submit the file/image for a content type node along with all the other fields in the same submission?

dholzgang’s picture

Please don't let the negative feedback from the Services developers stop this useful project. I spent the better part of a day trying to get Services to run. It was large, unreliable, complex and has virtually no useful (that is, not out of date) documentation. I eventually got it running for a sample "endpoint" using the supplied specialty "noteresource" module outlined in http://drupal.org/node/1246470 with the corrections noted in the Comments and some other fixes. But I realized that to make it work for my project, I would have to write a bunch of custom code. And all I want it to do is expose some node data as a REST service.

I used this module and Entity and had it up and running in 15 minutes. Simple, reliable (so far anyway ;-) and completely understandable. I've already done the demo version for my boss to review. So thanks for this and for all your work.

MichaelCole’s picture

Thanks for the hard work on this great alternative!

RobLoach’s picture

I'm all for working on alternative solutions! You should update the project page though, because Services module in the 3.x branch does do Restful. Full CRUD credentials are covered.

amitaibu’s picture

> I'm all for working on alternative solutions!

I fully agree :), especially as I think there is one big difference in terms of entity create/ update -- Entity metadata Vs form API

From my own experience, I prefer restws for entity CRUD as it passes my data to entity_metadata_wrapper() and not to a form_submit() e.g. populating a date-field using wrapper is much easier, and IMO more correct via wrapper then FAPI.

bendiy’s picture

Title: Why not using Services? » Why not use Services Module?

I've been playing with the services module for a few days to see if I can use it for a project. I ran into an issue. The Services module does not return custom fields when you request a resource. It might work for nodes, but support for comments is not there and that's what I need. When requesting a comment resource, I want the full resource including any custom fields attached and the data they contain.

So I discovered the restws module today. It works as expected right out of the box.

Thanks!

dasjo’s picture

people interested in this discussing might want to join forces on creating a common basis for services in drupal 8.

see the following posts by klausi on g.d.o: