Problem/Motivation
Data stored in tablefield can be an efficient means of storing viewing and editing tabular data to enable complex structures on drupal entities. However, accessing the data in SQL can be difficult because it is stored in PHP serialized format. Recent advances in PostgreSQL and MySQL have made enabled tabular data storage in JSON that is queryable. I propose to:
- Enable a backend data storage options of JSON
- Provide functionality to let views use JSON data extracts (i.e., show a specific row/column in Views) for more rapid rendering and analysis in Views
Proposed resolution
Allow users to specify the backend storage type, serialized or json on a per field basis.
Figure 1: Example table values (used in queries in Figure 2).
Figure 2: SQL queries and results using JSON against table in Figure 1.
-- Get the data in the 2nd row (first data row)
select (field_test_value::json->'tabledata')::json->'row_1' from field_data_field_test;
?column?
----------------------------------------------------------------------------------
{"col_0":"for","col_1":"20","col_2":"19","col_3":"18","col_4":"17","weight":"2"}
(1 row)
-- Get the header in the 3rd column
drupal.alpha=# select ((field_test_value::json->'tabledata')::json->'row_0')::json->'col_2' from field_data_field_test;
?column?
----------
"1985"
(1 row)
-- Get the value in the 3rd coluimn of 2nd row
drupal.alpha=# select ((field_test_value::json->'tabledata')::json->'row_1')::json->'col_2' from field_data_field_test;
?column?
----------
"19"
(1 row)
Figure 3: Setting page.
Remaining tasks
- (done) Add storage format option to field configuration screens
- Replace calls to "serialize" and "unserialize" with "tablefield_serialize" and "tablefield_unserialize", and then handle choice of serialization format in helper functions "tablefield_serialize()" and "tablefield_unserialize()".
- Develop code to transform data from serialized to json and back if administrator decides to switch backend data formats for a field.
- Add Views code to allow extraction of JSON data at the SQL level
User interface changes
n/a
API changes
n/a
Data model changes
Fields may have data stored as JSON if site builder desires it, but default behavior users serialize() so no automatic changes to data storage are required.
Comment | File | Size | Author |
---|---|---|---|
#17 | interdiff-2883127-12-17.txt | 1.92 KB | lolandese |
#17 | optional_json_storage-2883127-17.patch | 9.25 KB | lolandese |
#12 | 2883127-optional-json-storage_12.patch | 9.48 KB | robertwb |
#11 | 2883127-optional-json-storage_11.patch | 9.42 KB | robertwb |
#10 | 2883127-optional-json-storage_10.patch | 9.2 KB | robertwb |
Comments
Comment #2
robertwb CreditAttribution: robertwb commentedComment #3
lolandese CreditAttribution: lolandese at HCL Technologies Limited commentedNice.
I guess that at some point a conditional is needed to detect if the used database type and version supports the JSON Data Type.
Comment #4
robertwb CreditAttribution: robertwb commentedYeah - most definitely. Attached is a first draft - tested a good bit on the latest release, and tested only briefly against the latest dev. It does not include your db capability check, but I will get that in next week.
Comment #5
lolandese CreditAttribution: lolandese at HCL Technologies Limited commentedPHP Fatal error: Maximum function nesting level of '256' reached, aborting! in /sites/all/modules/contrib/tablefield/tablefield.module on line 2439
This seems to happen when using xdebug because of the recursive use of the function tablefield_unserialize().
Furthermore it gives an undefined index error because the field setting initially does not exist. This can probably be avoided in the same function with:
Also on https://simplytest.me/project/tablefield/7.x-3.x?patch[]=https://www.dru... we get a 500 Internal Server Error. Unsure if this has the same cause.
Comment #6
robertwb CreditAttribution: robertwb commentedThanks -- will take a look at this asap.
Comment #7
robertwb CreditAttribution: robertwb commentedPicking through the nesting error, I am a little stumped (but will dig deeper). For what it's worth, the functions tablefield_serialize() and tablefield_unserialize() do not call themselves recursively, so there may indeed be recursion happening, but not directly from those methods. I DO seem to be encountering that error when attempting to over-ride the tablefield contents in a custom form plugin (somewhat unrelated project) -- continuing to dig.
Comment #8
robertwb CreditAttribution: robertwb commentedUpdated patch due to changes in code base in latest dev version. Do we have test bot coverage on this? Any idea HOW we might do that, or rather, is there a thought on best ways to enable this and test cases we might consider? If there is not yet, I will add a test coverage issue and link back here.
Comment #9
robertwb CreditAttribution: robertwb commentedComment #10
robertwb CreditAttribution: robertwb commentedRemoved a dpm() call from patch #08 -- this may be why there was an error in the 03 version? Testing on simplytestme and will report back.
Comment #11
robertwb CreditAttribution: robertwb commentedFixed missing $field info in calls to tablefield_unserialize() (discovered in testing).
Comment #12
robertwb CreditAttribution: robertwb commentedAdapted settings detection from the code that @lolandese provided in #3:
Comment #13
lolandese CreditAttribution: lolandese at HCL Technologies Limited commentedCool. I had to change the conditional to:
because the function is dropped in PHP 7 (see http://php.net/manual/en/function.mysql-get-client-info.php). In the case the function does not exists we can be pretty sure we're dealing with PHP 7.
EDIT:
We should find an alternative for mysql_get_client_info in PHP 7 to determine the MySQL version.
Comment #14
lolandese CreditAttribution: lolandese at HCL Technologies Limited commentedUff!! Looks like it can be as simple as:
That should work for both PHP 5 and 7.
Comment #15
lolandese CreditAttribution: lolandese at HCL Technologies Limited commentedFurthermore:
Warning: Missing argument 2 for watchdog(), called in /home/martin/www/yellow/sites/all/modules/contrib/tablefield/tablefield.module on line 2512.
See https://api.drupal.org/api/drupal/includes%21bootstrap.inc/function/watc....
Comment #16
lolandese CreditAttribution: lolandese at HCL Technologies Limited commentedAlso:
Notice: Undefined index: backend_storage_format in tablefield_unserialize() (line 2489 of /home/martin/www/yellow/sites/all/modules/contrib/tablefield/tablefield.module).
This could be solved by using an update hook in the .install file giving a field settings value to all existing table fields. You can use the existing update hooks as examples.
Alternatively use the code below that assigns a value if no field settings have been set.
Comment #17
lolandese CreditAttribution: lolandese at HCL Technologies Limited commentedRolled suggestions of #14 and #16 into a new patch.
Comment #18
robertwb CreditAttribution: robertwb commentedAwesome @lolandese! I will get this tested on my alpha system asap.
Comment #19
lollypopgr CreditAttribution: lollypopgr commentedWhat is the status with this?
Comment #20
lolandese CreditAttribution: lolandese at HCL Technologies Limited commented@lollypopgr:
Nice to see interest in this awesome new feature built by @robertwb.
Feel free to test. Keep in mind your MySQL version should be 5.7+ or PostgreSQL 9.2+.
If this goes in it will likely be a new 7.x-4.x branch if it would have big implications for an upgrade on existing sites. To analyse that as well.
Comment #21
lolandese CreditAttribution: lolandese at HCL Technologies Limited commented