A big goal for Drupal 7 is to enable text formats (formerly known as "input formats") on as many textareas as possible, both in Drupal core and contrib. This gives fine-grained control over allowed HTML tags and opens up new possibilities for different ways to input the text, including much better integration with rich text editors (see http://drupal.org/project/wysiwyg).

A list of places in core that don't use text formats but should is at http://groups.drupal.org/node/9072. What's stopping them? It's probably just the fact that dealing with text formats is kind of a pain - you have to figure out how to store the data correctly, display it correctly, etc.

This patch takes the first steps towards making it easy. Since the new Field API in core provides native support for text format handling within text.module, and since it also supports attaching fields to any kind of "object", all we have to do is treat some of the random textareas in Drupal -- that are currently stored in the variable table -- as field-enabled objects, and we get everything else for free! (We also get the benefit of not filling up the variable table with gigantic paragraphs of text that get loaded on every page view, even though they are rarely needed...)

The attached patch is basically working code, although it's still rough around the edges and the exact implementation strategy is certainly open to debate. However, it currently does the following:

  • Adds an API for storing text-format-enabled data that works very similarly to variable_get/set. If you want to store/display any random piece of text with an associated text format, you can simply call functions like drupal_save_text('variable_name', $text, $format), drupal_get_text('variable_name') and drupal_get_raw_text('variable_name'), and Drupal's field system takes care of everything else for you.
  • If you're using system_settings_form(), it gets even easier. If you want to convert a textarea on one of these forms to be text-format-enabled, all you have to do is add this single line of code to the form element:

    #text_format => TRUE,

    Then, when displaying the text, find places in your code that used to be along the lines of this:

    filter_xss(variable_get('variable_name'))

    and replace them with this:

    drupal_get_text('variable_name')

    That's it, no more work required. (Note that this is also a big security improvement, since using this system means you no longer have to remember to sanitize the text yourself every time you display it.)

  • As proof of concept, the attached patch converts a single textarea in Drupal (the site mission statement) to use the new system.

Comments?

CommentFileSizeAuthor
text_formats_everywhere.patch13.42 KBDavid_Rothstein
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

yched’s picture

Interesting idea. A few initial remarks :

- Overall model : You create one fieldable entity type per 'variable', with one single object in each type ($object->id = 1). Makes the number of fieldable entities explode, where we currently have only a few 'big' stuff like 'node', 'user' (to be continued...). I think we're best keeping that list short and uncluttered, with only well identified entities.
It seems one single fieldable type ('variable'), each variable being it's own bundle would fit more nicely to the usual mind model. Of course, the difficult part is then to generate ids, since all variables then share the same id space. If and when #368082: Non-numeric object ids and revision ids gets done, this becomes a non-issue.

- drupal_get_text() : I'm not sure we need to run through field_attach_view(), which goes through a drupal_render, the theme layer and the default formatter. A straight check_plain [edit : check_markup, of course] should be enough IMO - we don't want to allow different formatters, do we ? I mean, this bypasses widgets, it could as well bypass formatters and use fields as a storage only mechanism.

- storing this as fields will give localisable variables for free if / when #367595: Translatable fields lands. In such a case, the interest is not only limited to formatted text, but to all text fields.

- next step / feature creep part : non-text variables ? could be interesting if we get a core file field, date field...

kika’s picture

bjaspan’s picture

This is not really a Field API issue, it is just something that is using the Field API. I'm not sure where it belongs, just not here. :-) I did cleverly manage to subscribe myself while saying so, of course...

catch’s picture

Very interesting. I agree with yched about the fieldable 'object' - best to have one if possible. Subscribing.

David_Rothstein’s picture

Thanks for the comments. Here are some responses:

1. Agreed that it makes more sense for each variable to be a bundle. I'm guessing it's not even that hard to convert the patch to do this now, regardless of whether #368082: Non-numeric object ids and revision ids gets in. Because we already has to store an array of the variable names, so storing an integer with those too shouldn't really be any harder.

2. Regarding check_markup() vs field_attach_view(), my general goal here was to use as much of the Field API as possible, since that makes things consistent. For example, field_attach_view() calls a hook which lets other modules modify the field after it's rendered... do we want to prevent them from doing that here?

3. It might indeed make sense to expand this to all text fields, not just format-enabled ones. The only complication I can think of there is what to do about system_settings_form(). For now, I have it pretty simple -- module developers who use system_settings_form() don't get a choice about how their data is stored. (Text-format enabled data gets stored and accessed using the Field API, and everything else is done with the variable_get/set API.) If we expanded this to include everything, it seems like we might have to/want to give them more of a choice?

4. Also agreed that allowing this for non-text variables would be good too, but yes, let's save that for a followup patch :)

5. Thanks for the pointer to #365934: Handle big texts / module text files / readable and localizable / get rid of _user_get_text() -- good to know about that. I think there's a tiny bit of overlap, but mostly it seems very complementary -- since that focuses on how the default values of variables are stored, whereas this focuses on how the user-entered values are stored. I'll add a comment to the other issue, though.

6. Agreed this doesn't really belong as a Field API issue, but I don't know where else it belongs either ("base system" ... "other"? ... ugh). My main goal in filing it under the field system was to get some of the "Fields people" to look at it, which I seem to have accomplished :)

David_Rothstein’s picture

Also, Gabor pointed out that the site mission statement might not really be a good candidate to convert to this system, since if we convert it to anything we could convert it to a block instead, and that way you get the layout/visibility features along with the text format.

However, I just did it as an example, so no worries for now :) There should definitely be plenty of other text variables throughout core and contrib that could benefit from this.

David_Rothstein’s picture

Issue tags: +wysiwyg

Adding tag.

yched’s picture

Thinking about this now : this approach make those ex-variables unreachable by settings.php $conf override.

David_Rothstein’s picture

Title: Use Fields to store variables that require text format support » Use Fields to store text variables, especially those that require text format support
Component: field system » base system

I'm thinking of postponing work on this for at least a little while (ha, as if it hasn't already been postponed for the last few weeks!!) to see what happens with #414424: Introduce Form API #type 'text_format'. A lot of the ugly hacks that this patch requires for system_settings_form() would no longer be necessary if that goes in.

Regarding @yched's concern with settings.php overrides, I'm not sure if it's that big of a deal, since this will usually be used for long strings of text that people are probably not going to want to override in settings.php too often anyway? However, if it is a concern, I think it's probably easy to deal with, as long as we enforce the restriction that both types of "variables" live in the same namespace. Then, I think functions like drupal_get_text() could just call variable_get() at the beginning, and if they find anything there, return that? I think that would do it.

What I'm more worried about in general with this patch is the case of upgrades from Drupal 6. Some of these variables were previously using things like filter_xss() or filter_xss_admin() to output their text. If we're converting them over to use text formats, what do we do about that? Do we have to add a new text format to Drupal during the upgrade, that gives the same output as filter_xss_admin()? Not sure if there's a simpler way to handle this...

moshe weitzman’s picture

Very clever. I'd like to see this in core.

sun’s picture

Version: 7.x-dev » 8.x-dev

Before proceeding here, let's revisit the list of variables that actually still use text formats. >90% of that can and should be eliminated through other means.

We even considered to remove {cache_filter} for D7 already, just were not sure if it's really not used anymore.

Leaving us with <10% of variables that may store text with a format. With #type 'text_format', the only remaining question is why the variable can't simply be an array instead of text. Though that leaves us with maintenance problems, in case text formats need to change over time.

David_Rothstein’s picture

Yeah, back while I was working on this, right at that same time people were already starting to convert a lot of those one-off variables (mission statement and the like) to other things, like blocks, and more of that will be possible as time goes on.

I still think this would be useful, though. There are a lot of places - especially in contrib - that have e.g. a single admin screen with a one-off textarea that might make sense to have text format support.

This patch stalled for D7 because it really needed the #type 'text_format' stuff in order to reasonably proceed, and that didn't happen until too late in the cycle. I'm definitely interested in trying again for D8, though. There's a fair amount of work left to do, but I think it could still be very interesting.

Gábor Hojtsy’s picture

For translatability, it would make most sense if **ALL settings on earth** (think which contact forms you have, your block titles and bodies, your views empty text, etc) would be fields. This is not really just "text" fields. At least if date format possibilities are fields, we can cleanly make them translatable with translatable fields. In fact I have a whole post that illustrates that most of i18n module's heavy lifting evolves around defining "field"-like behavior for these settings elements: http://groups.drupal.org/node/152929 and best for this use case would be if all those would be fields.

sun’s picture

@Gábor Hojtsy: I'm on board with that direction.

Roughly speaking, it sounds like one way would be to abstract Field API more from Field Attach API. Whereas Field Attach API would use fields to attach them to entities. Non-entity use-cases would also use Field API to attach fields to non-entity stuff, possibly configuration.

Another way would be to simply turn everything into entities and just simply use fields. Requires no further abstraction and all tools are in place already.

Gábor Hojtsy’s picture

@sun: great, I posted a discussion piece about this within the drupal initiatives groups: http://groups.drupal.org/node/154434

fago’s picture

>Non-entity use-cases would also use Field API to attach fields to non-entity stuff, possibly configuration.

I really see no point in doing that as it would just complicate things. Then, if configuration would even have fields, for what's sake shouldn't it be an entity too?
Instead I'd like to go the other way and integrate the field API and the entity API much more - see http://groups.drupal.org/node/154434#comment-515409.

jhedstrom’s picture

Version: 8.0.x-dev » 8.1.x-dev
Status: Needs work » Postponed (maintainer needs more info)
Issue tags: +Needs issue summary update

Much has changed since this was last updated.

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.0-beta1 was released on March 2, 2016, which means new developments and disruptive changes should now be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.0-beta1 was released on August 3, 2016, which means new developments and disruptive changes should now be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.0-alpha1 will be released the week of January 30, 2017, which means new developments and disruptive changes should now be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.0-alpha1 will be released the week of July 31, 2017, which means new developments and disruptive changes should now be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.0-alpha1 will be released the week of January 17, 2018, which means new developments and disruptive changes should now be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.6.x-dev » 8.7.x-dev

Drupal 8.6.0-alpha1 will be released the week of July 16, 2018, which means new developments and disruptive changes should now be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.7.x-dev » 8.8.x-dev

Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

smustgrave’s picture

Status: Postponed (maintainer needs more info) » Closed (outdated)

If still a valid task please reopen addressing issue summary update from #17