Port Data Module to Drupal 7

I'll have to take a close look at the field API to decide what's going to make sense in Data in Drupal 7.

Data is a very weird animal with a colorful bag of functionalities that don't necessarily need to live on within Data for Drupal 7:

1. A table editor (hum, I'm still not sure whether I should have ever built this :)
2. A CRUD API for manipulating tables.
3. A schema manager with the ability to harmonize Drupal schema info with the actual table schema by either adjusting the Drupal schema or the table (much of the functionality is from Schema module, only the UI lives in Data).
4. A CRUD API for manipulating data in tables (competes with nodeapi; this is the piece that very likely may be able to move to Field API).
5. Automatic Views integration for 'Data managed' tables (this is sth that other modules like Table Wizard do, too. Not too much of rocket science, though).
6. Integration modules with Drupal API's like node's or taxonomy's.

Some of the advantages of Data:

- It's fast on run time.
- You can build storage for random, flat data very fast (this is where building a CCK content type becomes arduous).

Some of the disadvantages of Data:

- it sucks for certain types of multi-dimensional data (related data is usually fine, but suckage starts when there is a notion of 'records', that are a logical unit, spreading across tables).
- there is no CRUD *UI* for data in tables.
- its CRUD API competes with nodeapi, the difference being speed.
- the integration with Feeds is a little odd as Feeds allocates data tables on the fly. This is awesome if you understand Data, but confusing if you don't.

My next actions are to evaluate Field API in Drupal 7 and see to what extent we can transfer Data functionality (most importantly (4)) to a FieldAPI based approach.

jmiccolis pointed out offline that entity API is a related candidate to evaluate on the path to Drupal 7: yes, it is.

Doesn't data seem more like entities than fields? Is the idea to attach the stuff made accessible by the data module to nodes, users, and other entities? Or, to be exposed as top level objects?

If you go the entity route you may want to look at the commerce product module for an example of dealing with CRUD. Specifically see the controller include and crud functions. See http://github.com/rszrama/drupalcommerce/blob/master/modules/product/com... and http://github.com/rszrama/drupalcommerce/blob/master/modules/product/inc...

I would imagine that each data table & record will be a 'fieldable' entity. Each column for each table would be a 'field' that makes use of the field API.

subscribe

I would really like to see the Data module ported to Drupal 7. Here are some reasons why:

1) Simply showing raw table data in Drupal 7 is hard without the Data module. Views integration is important.
2) Third party software doesn't integrate well with Drupal 7's field concept. Eg. using Navicat to access data.
3) Speed. The field system is flexible but not fast.
4) The possiblty to show denormalized data. Not everything fits well into contenttypes.

+1 for D7 and data module.

subscr

Suscribe

+1 for D7 version of drupal

subscribe

+1 for D7 version of drupal

+1 for D7 version.

i would recommend review if entities with entity api suffice your needs before asking for a data d7 version
http://drupal.org/project/entity

+1 for D7

Subscribe.

I'm currently using Table Wizard to expose external DB fields to Views. The tw module is going to be deprecated and they suggested to use the Data module instead.
I tried out the Data module but I don't see a way of exposing the external DB fields.
The Data module maintainers are not confirming if there will be a D7 version.
Does anyone knows how to achieve this functionality and/or this will be a new feature for Data if they decide to create a D7 version?

Changed titel for better recognition in dashboard.

@luthien: Data creates Views integration by "adopting" tables that you create manually in the Drupal database (or which have been created by other contrib modules)

Once you have created your table, go to admin/build/data/overview and click on the "adopt tables" link.

It will take you to a page where you can choose what table you want to adopt.

This will autogenerate Views integration code for your table. If you create a new View, you should see an option for creating a "View type" based on your tables data.

thanks for the info about the Data module. Basically, Data module is not a replacement for the tw module, which allows us to expose external DB fields to Views. There are several posts that mentions to use Data module once tw is deprecated but it looks like it does not works for this particular case scenario.

+1 for a port!

+1 for port to DX7

+1 for a D7 port. Thanks!

sub

With the change in ownership of this module could we get an update from the new maintainers as to what the intentions are for D7?

+1 for D7

subscribe

subscribe

Can someone give a timeschedule for a D7 release?

subscribe

+5 for D7 port (I'm from Chicago - we vote early & often...)

+1 - would be very keen to see some feedback from the new maintainers on their plans here.

+1 for D7 version

+1 for D7 version

+1 for D7 version

subscribe

sub

+1 for D7 version

+1

how bout a migrate from 6? I think this would be a great module.

+1 sub

Has anyone tried to run this through Coder?

subscribing and setting proper title for better tracking.

Subscribing

I need the "adopt" data module feature in Drupal 7. I'm using the webform mysql view module and need a way of notifying drupal of the view that is created. Is there an equivalent way of doing so? Is entity the solution and if so what it the process

THanks

+1

Drupal is a great content management system, if by content we mean material whose structure can be appreciated by an intelligent agent (natural language text, images, etc.). But the world - especially of business - is full of so-called "structured" data. Very often, we need to combine a structured approach: where we can ask precise questions or queries and get a precise answer, because the semantics of the data are explicit as tables and their attributes / columns. Such database querying is at the heart of many business processes. Those same processes can of course already benefit from Drupal exploitation for their less-structured content.

Get the structured data and querying aspect right, and Drupal will have another competitive advantage - for use cases like learning management (teaching, learning and assessment) and multimedia applications.

How has it been possible for a new version of Drupal, D7, to be produced without the architects seeing the essential need for Data or Table View functionality? These would ideally be combined, so that Data can integrate tables managed by other applications rather as Table View does. Is such a scenario being considered for Drupal 8 core?

I'm not a competent PHP coder - sorry! - but will happily contribute on requirements specification, documentation, etc.

@MarkRennes

I don't want to fault the programmers, I think those of us involved in this field (beyond just simple make it look pretty design) are transitioning into a new realm. Well, maybe not "new" for those that have been grappling with this idea for the past 20+ years in academia and data industry. My point being, I think web developers in general are facing the need to deal with mounds of structured data. The work I do, at least, requires data mashups. I'm a one man shop, so definitely having tools like Data and Feeds is very helpful for me. Like, I know what I need and want, but I don't necessarily know how to program it from the ground up as a properly trained and experienced database admin might.

So I whole heartily agree with you - I would love to see more data functionality built into the core of Drupal.

tracking

Thank you @morningtime.

Using entities, I can build - painfully - the equivalent of one table. But what if I need the equivalent of two (or more) tables which are related on a primary-foreign key? So far as I can see, I cannot (using an entities-based approach) meet one of Codd's original two integrity constraints, which is referential integrity: which I take to be fundamental to the proper structuring and exploitation of (structured) data.

Am I wrong?

Subscribing

I think it's a higher priority than "minor" the ease of manipulation, feeds integration and views integration. It's much better and easier than Entity API.

This is a highly recommended port to D7.

sub

@morningtime:

No doubt you CAN enforce foreign keys. That's still infinitely harder than the end-user ease-of-use that I for one - spoilt no doubt as I am by years of good experiences with Microsoft Access in a desktop environment - would like to see. I'm with Mohammed J. Razem et al: why not a straight port of the data module? Anybody?

subscribe

sub

LOL, Todd you just made my day with the vote early and often comment - I'm also from Chicago.

I also need to figure out how to replicate this functionality in D7. I'm trying to convert some perl scripts loading flatfiles into Drupal modules using database tables. This is a sample of the data:

60000,002891,7,40.69,2.3,1.72,3.437,3.375,0.125,8,C,na,na,na,0,,,,,,
60035,002900,6,32.08,1.4,2.18,1.239,4.65,0.125,8,C,na,na,na,0,,,,,,
60050,002907,6,30.6,1.6,2.28,1.25,4.75,0.19,8,B,na,na,na,32,93.75,23.44,13.66,8,32,56
60055,002912,6,48.15,2.2,1.45,2.625,4.75,0.19,8,A,na,na,na,132,93.75,23.44,22.89,8,32,56
60060,002919,8,71.03,4,0.98,4.5,4.75,0.312,8,C,na,na,na,16,,,,,,

Doesn't make any sense to me to use nodes for that - too much overhead for such simple data.

subscribing

I agree with MarkRennes and aceinthehole that structured data is an important function needed by Drupal.

As with seanr, I want to display data in database tables. I could do it with nodes and Views, but it would require hundreds of nodes for simple data.

No need to subscribe. Click the green follow button at the top of the page instead.

finally ... unsubscribe :)

subscribe

> I would imagine that each data table & record will be a 'fieldable' entity. Each column for each table would be a 'field' that makes use of the field API.

That would sort of be doable... but surely you'd need to create a different field storage backend module that puts the field data into the DB table in question rather than make a new table per field.

I've made some progress on this.

I've started with the Coder Upgrade conversions, and then worked through the core data module and data_ui. I've not touched data_node & co.

Here's a sequence of patches from my local git branch for the D7 port, and the summary of my commits in the patch:

cc87239 Applied Coder upgrade coding standards conversion.
fb5547d Applied Coder upgrade Core API conversion.
b75046f Upgraded for changes to CTools.
34ca3f9 Fixed incorrect menu path.
899e019 Replaced obsolete schema_invoke() for upgraded Schema module.
af776f8 Added file list to info file; removed obsolete data_include().
5d9f63d Upgraded for FormAPI #markup elements.
2debfc4 Commented out hook_schema_alter() for the time being.
a3d2e7b Fixed adding a field to a table.
96aa7e7 Fixed warning on empty field labels.
06c5481 Fixed for changes to Database API functions.

What works so far:
- adopting a table
- viewing its schema
- adding a field
- changing primary keys & field labels
- Views fields and filters

Newer version of the above patch with a small UI fix and tentatively uncommenting hook_schema_alter() which now seems to work without blowing up!

> I would imagine that each data table & record will be a 'fieldable' entity. Each column for each table would be a 'field' that makes use of the field API.

I've made some progress towards that here: #1342294: add a module to create entity types and entities from tables
(UPDATE: these are now fieldable!)

Further fixes.

Turns out my tentative uncommenting of hook_schema_alter() was rather premature.
I've taken a new approach which works, though I recommend enabling CTools first, then Data, rather than together (or try it on a test site and see if you can make it work! :)

I've contacted the maintainer of this module about co-maintaining, so I can get this ongoing work into git and available as a dev release.

+1 on joachim getting co-maintainership. I think this is a big important suite that needs some love.

Agreed - @joachim if you're willing and open an issue requesting it we'll all RTBC it :-)

Done. #1348546: Data: request for co-maintainership.

Hi

First : big thanks to you to port data module on D7.

I tryed to patch but it doesn't work. I'm not familiar with Terminal... sorry.
Could we have a version. tar or. Zip patched module?

Thanks

@MatNk run these commands.
cd sites/all/modules && git clone --branch 6.x-1.x http://git.drupal.org/project/data.git && cd data && wget http://drupal.org/files/827000-3.data_.port-d7.patch && patch -p1 <827000-3.data_.port-d7.patch && rm -iv 827000-3.data_.port-d7.patch

@joachim gj man!

@joachim I have made you a maintainer as per #1348546: Data: request for co-maintainership thanks and feel free to get this going....

Ok I've made a 7.x-1.x branch and merged my local branch into it, so the changes from my patch above are visible as a sequence of commits -- which should help anyone looking at the changes I've made.

I've made a dev snapshot release which will be available as soon as the d.org bot makes it :)

I'll apply the fix for #1056470: Data contains multiple SQL injection and XSS vulnerabilities and also #1342294: add a module to create entity types and entities from tables and make an alpha release.

Closing this, as I reckon further porting work is best handled in separate issues.

Please bear in mind that help is still very much needed with the porting of this module, in particular CTools support and the submodules for nodes, search, and taxonomy.

Follow-on issue: #1363554: deprecate data_node, data_taxonomy on D7, upgrade path?.