Experimental project
This is a sandbox project, which contains experimental code for developer use only.
Introduction
This Drush extension is meant for individual field data manipulation from commandline. It uses Drush commandline interface. Aim is to be able to perform all CRUD operations and more on all data in all core field types and contributed field types. Some fields have field-specific support other fields have general support.
New functionality is added by feature requests.
This project is currently located at:
- For a full description of the Drush extension, visit the project page:
https://drupal.org/sandbox/ragnarkurm/2274189 - To submit bug reports and feature suggestions, or to track changes:
https://drupal.org/project/issues/2274189
Status of the project
This project has been decided to be duplicate of Drush Entity, but until the merge actually happens download this Drush extension.
Meanwhile new features are added here.
This was project application.
Here is merge proposal issue.
Code review by Clemens and further thoughts on integration.
Currently working on merge.
Audience
This Drush extension is mostly for developers, admins, but not limited to.
- Admins
- Quick data inspection
- Quick data fixes
- Helpful in data analyses
- Helpful for finishing up migration
- Developers
- Integration with other systems
- External data processing
- Asynchronous data manipulation
- Testing for module development
- Regular user
- Importing directoryful of files or images
Supported field types and operations
Update = Add, Set, Import.
Field | Read | Update | Find | Replace |
---|---|---|---|---|
Date | Yes | Yes | ||
Datestamp | Yes | Yes | ||
Datetime | Yes | Yes | ||
Yes | Yes | |||
Entity Reference | Yes | Yes | ||
File | Yes | Yes | ||
Image | Yes | Yes | ||
Link | Yes | Yes | ||
List: Boolean | Yes | Yes | ||
List: Float | Yes | Yes | ||
List: Integer | Yes | Yes | ||
List: Text | Yes | Yes | ||
Number: Decimal | Yes | Yes | ||
Number: Float | Yes | Yes | ||
Number: Integer | Yes | Yes | ||
Taxonomy Term Reference | Yes | Yes | ||
Text | Yes | Yes | ||
Text Long | Yes | Yes | ||
Text with Summary | Yes | Yes | ||
General | Yes | Import | Yes | |
Node title | Yes | Yes | Yes | Yes |
All field types:
- Count, Delete, Read, Find
Requirements
- Drush is installed
- Field module (in core) is enabled
- Entity module in installed and enabled
- Relevant field module to be enabled. (Example: If you want to add/del/read a file field, then you need File module enabled.)
- This Drush extension assumes there are some fields with that type to work with.
- You have commandline (SSH) access to your Drupal installation.
- Understanding about following concepts:
- Entity type (Node, User, Taxonomy Term, Comment, Profile2, ...)
- Bundle (Article, Basic Page, ...)
- Entity ID (1, 100, 1234, ...)
- Field name (field_pdf, field_thumb, ...)
- Machine name (only_lowercase_letters_and_underscores)
- Field type (File, Image, Email, Text, Integer, ...)
- Drush
- Unix shell
An Introduction to Entities: https://drupal.org/node/1261744
About Drush: http://www.drush.org/about
GNU Bash: http://www.gnu.org/software/bash/
Install
- Download the Drush extension
cd ~/.drush git clone --branch 7.x-1.x http://git.drupal.org/sandbox/ragnarkurm/2274189.git drush_fields
- Refresh Drush commands cache
drush cc drush
Configuration
This Drush extension has nothing to configure.
Usage
General
Generally command line consists of 5 parts:
- drush
- command - For example: fields-update, fields-node-title-read, etc
- target arguments - This specific to each command and it specifies which data to access. Please check
drush command --help
. For example: node 1234 field_myfile 6 - required options - For example: --value=/tmp/a.txt
- other optional - For example: --mime=text/plain
Detailed help:
drush help
drush fields-count --help
drush fields-del --help
drush fields-read --help
drush fields-info --help
drush fields-update --help
drush fields-node-title-find --help
drush fields-node-title-read --help
drush fields-node-title-replace --help
drush fields-node-title-set --help
fields-find
To search field values there is a command for that fields-find
. It searches within one field instance, but it can be across many entity types.
There are two sets of values available for filter and for output.
General
- bundle
- delta
- entity_id
- entity_type
- language
- revision_id
Field-specific
It varies from field to field what kind of values are available. For example, for date(iso), there are additional field-specific values: value, value2, timezone, offset, offse2.
Output format
It is almost identical what fields-read has. It is based on template and --template and --header options work similarly. Differences are: here are no structural output like JSON or XML etc and secondly no ~key|key|key data structure traversal.
Conditions
But how does actually find works. Maybe more accurate is to say filtering. It works by adding command-line arguments, each being a condition. Each condition consists of 3 parts: value, operator and operator argument. There are 3 value types supported: string, integer, float. Each type has separate set of operators available:
- String: =, <>, like, not-like, ~, !~
- Integer: =, <, >, <>, <=, >=, in, not-in, between, not-between
- Float: <, >, <=, >=, between, not-between
There are 2 categories of operators: database level and php level. Database-level filtering is fast, PHP-level filtering requires fetching all data and then perform matching and this is slow. Most of operators work on database level and ~ and !~ work on PHP level.
Tilde ~ performs regular expression match and !~ performs negative regular expression match.
Examples
Suppose you want to find common misspelling 'wich' to replace it later manually:
$ drush fields-find field_text 'value ~ /wich/'
Type Bundle ID Delta Value Format
node page 1 4 Test wich (NULL)
Lets make life easier:
drush fields-find field_text 'entity_type = node' 'value ~ /wich/' --template='http://example.com/node/{%d:id}/edit' --header=n
http://example.com/node/1/edit
fields-update
import
It is possible to import all field data (not individual elements in multi-value field) at once using following formats:
- serial
- json
- xml
It is possible to read input from file (absolute path) or from stdin.
In order to perform import, use --op=input
. Use --value
to supply input source and format. Examples: --value=/tmp/location.serial
or --value=-.json -y
. The input data type is determined by extension, including stdin (-.json
). For stdin must supply -y
.
Note that during importing only rudimentary data checks are done. If the data has wrong structure for the field (for example export date field and import as file field) then results are unspecified.
Examples:
$ drush fields-update node 1 field_geo --op=import --value=/tmp/a.json
$ drush fields-update node 1 field_geo --op=import --value=/tmp/a.serial
$ cat /tmp/a.xml | drush fields-update node 1 field_geo --op=import --value=-.xml -y
Advanced example - clone/copy field from one entity to other and change case on the fly:
drush fields-read node 1 field_text --template=json \
| tr a-z A-Z \
| drush fields-update node 2 field_text --op=import --value=-.json -y
fields-info
While working with fields, coding PHP, there is often need to inspect field itself (not it's data). Most frequent inspections are:
- field general info - field_info_field()
- field instance info - field_info_instance()
General info example:
$ drush fields-info field_text
Array
(
[translatable] => 0
[entity_types] => Array
(
)
[settings] => Array
(
[max_length] => 20
...
Instance info example:
$ drush fields-info field_text node page
Array
(
[label] => text
[widget] => Array
(
[weight] => -3
[type] => text_textfield
[module] => text
[active] => 1
[settings] => Array
...
fields-read
Command 'fields-read' lists data in one field in one entity. Data is printed out in tabular form. Each field type has its own default template how to print data. User can specify own format (good for automation). template consists of text and data-placeholders which will be filled with data. Placeholders consist of two parts. First printf()-style formatter (like %s or %-10d) and key. Key may be preset keyword which is constant over Drupal versions or data structure specific keys which lead to data. If user-specified template is empty, then all possible keys are listed specific to selected field. Header is attempted to print aligned to columns, but does not always work properly with custom template.
drush fields-read node 1 field_image
Delta FID UID Width Height Created Size Filename
0 65 0 400 225 2014-Jul-04 00:03 23543 images-background.jpg
1 74 1 (NULL) (NULL) 2014-Jul-11 00:33 379 aaa.txt
2 75 1 1920 1200 2014-Jul-11 00:33 380478 img.jpg
This is default output.
fields-read node 1 field_image --template='{%5d:delta} {%05d:width} {%-8.8s:filename} {%s:~metadata|height}'
Delta Width Filename Value
0 00400 images-b 225
1 00000 aaa.txt (NULL)
2 01920 img.jpg 1200
Here you can see custom template with custom formatting in tabular form. Not vary practical, but serves purpose of demonstrating power of formatting options. First, delta is 5 digits wide integer, secondly width is zero-padded 5 digits wide integer, thirdly filename is 8 characters wide and truncated at 8 chars aligned left, finally string is fetched from value data structure $value[delta][metadata][height]
.
drush fields-read node 1 field_image --template='{%d:width}:{%d:height}:{%s:filename}' --header=n
400:225:images-background.jpg
0:0:aaa.txt
1920:1200:img.jpg
This is more practical example. Data is listed csv or delimited format which is excellent for shell scripting and works well with cut, grep, sort, uniq and other standard text processing tools.
$ drush fields-read node 1 field_geo --template=print_r
Array
(
[0] => Array
(
[geom] => POINT (22.33 11.22)
[geo_type] => point
[lat] => 11.220000000000
[lon] => 22.330000000000
[left] => 22.330000000000
[top] => 11.220000000000
[right] => 22.330000000000
[bottom] => 11.220000000000
[geohash] => s3zzu4
)
)
It means, it is possible to use following template placeholders:
- {%s:~geom}
- {%s:~geo_type}
- {%f:~lat}
- {%f:~lon}
- etc.
For example --template='{%f:~lat}:{%f:~lon}'
gives following result:
Value:Value
11.220000:22.330000
12.120000:23.230000
16.160000:24.240000
Generic data handling
There exists few generic ways to export data from fields. Use --template=format
to get desired format. Hopefully format IDs are self-explanatory. Available formats are:
- print_r
- serialize
- var_export
- var_dump
- json
- xml
Examples:
$ drush fields-read node 1 field_date --template=print_r
Array
(
[0] => Array
(
[value] => 1999-12-31 22:00:00
[value2] => 1999-12-31 22:00:00
[timezone] => Europe/Tallinn
[offset] => 7200
[offset2] => 7200
[timezone_db] => UTC
[date_type] => datetime
)
)
$ drush fields-read node 1 field_text --template=serialize
a:3:{i:0;s:10:"/tmp/a.php";i:1;s:8:"asdfasdf";i:2;s:4:"test";}
$ drush fields-read node 1 field_text --template=var_export
array (
0 => '/tmp/a.php',
1 => 'asdfasdf',
2 => 'test',
)
$ drush fields-read node 1 field_text --template=var_dump
array(3) {
[0]=>
string(10) "/tmp/a.php"
[1]=>
string(8) "asdfasdf"
[2]=>
string(4) "test"
}
$ drush fields-read node 1 field_text --template=json
[
"\/tmp\/a.php",
"asdfasdf",
"test"
]
$ drush fields-read node 1 field_text --template=xml
<?xml version="1.0"?>
<field_text>
<value_0>/tmp/a.php</value_0>
<value_1>asdfasdf</value_1>
<value_2>test</value_2>
</field_text>
Note that the values are always printed as arrays, even if min (required) = 1 and max (cardinality, number of values) = 1.
Note that if field is empty no data structure is returned, but indicative text is issued.
Note that the formatted data is based on $wrapper->field->value()
.
Versions, milestones, todo
Current code is 7.x only.
Following features are planned:
- Find operation implementation.
- Replace operation implementation.
- Read, Update, Find, Replace are supported for common entity type's (node, user, comment, taxonomy term, profile2) pseudo fields (title, published, username, email, etc).
- Server-side counterpart (a module) for assisting file handling (file permissions).
Please submit your feature requests!
Misc, todo
- Merge into Drush Entity
- Tests
- Resolve class autoloading with drush.
- Drush extension updating
- Change File handling, since files are entities and file fields are references to entities
Security and data access
Drush default Anonymous user and user 1 can do all operations for practical reasons.
All other users (-u
or --user
) are checked if they can access particular data.
This data access approach may change according to feedback.
Troubleshooting
- This Drush extension contains a lot of checks and verbose error messages which should be helpful for solving problems.
- Check:
drush command --help
- Check watchdog log
- Search issues
- Post an issue:
https://drupal.org/project/issues/2274189
Related projects
Use cases
If you develop a system base on this Drush extension or find some other valueable use, it would be nice to share a short description of it. Also get a link to your site.
Simple cases
- Creating invoice .pdf in shell and add to Drupal Commerce, Order entity.
- Adding (importing) collection of files or images to a field.
Case: Export images or files from node
I had situation where one of my sites accommodated many galleries and it grabbed too much disk space (drush backups were too big). So I decided to move gallery images to Flickr. So how to export images from node at once? The problem is that I have all images mixed in the directory - how to separate them easily?
$ cd /www/myprojectname/html/sites/example.com/files/images
$ drush fields-read node 15 field_images --template='{%s:filename}' --header=n | tar -cf /tmp/15.tar -T -
I have configured field_images to save images into files/images directory. With help on Drush Fields I extracted all image file names and fed them into tar for archiving, -T -
means reading filenames from stdin. Now the archive file can be downloaded and then uploaded to Flickr.
Case: Linux distribution ISO compilation system
A system has been developed where user enters parameters (Language, region, packages, etc) and after a while will be able to download customised Linux distribution ISO. Techinically user posts parameters to node edit form. Cron runs regularly, exports the data (Views Data Export) and compiles ISO images (ca 700mb). These images are transferred back to system by Drush Fields. Users will be notified when the task is done and distribuition is ready for download. It is very easy and handy to perform the compilation and insertion to Drupal by shell script.
Maintainers
Current maintainers/developers:
- Ragnar Kurm (ragnarkurm) - https://drupal.org/u/ragnarkurm
Project information
- Module categories: Administration Tools, Content Editing Experience, Site Structure
- Created by ragnarkurm on , updated