Background

Now that we have a new release cycle, we have the possibility of new features in minor releases, i.e. although we are in RC1 for 8.0, that doesn't mean we can't add new features until 9.0. Provided they are backwards-compatible, we can add new features in 8.1 and 8.2.
After recently taking over maintainer-ship of the core contact module @andypost and I, in consultation with@berdir have formulated a draft roadmap for the features we'd like to see in contact module in the future.
Why not make this the guinea pig for how these initiatives and features might work in future versions of D8.

High-level goal

To provide the 80% use-case of webform. i.e. allowing creation and submission of feedback forms from site-users; and providing editing, listing and administration of submitted form values.
Webform contains lots of features, we're only after expanding contact module slightly to add storage and administration and in the process meet the basic use-case of webform in core.
Note that some of these items have been developed in contrib as the contact storage module. They can continue to mature there during 8.0 with the view to include in point releases eg 8.1, 8.2.

  1. The proposal: phase 1

    1. #1856562: Convert "Subject" and "Message" into Message base fields
    2. #2289063: Change contact message entity to behave more like a normal entity
    3. #2285083: Rename contact category to contact form
  2. The proposal: phase 2

    Bring in features from contact_storage and finalise those missed in D8.0.

    1. #306662: Add redirect option to site-wide contact forms
    2. #2750613: Add option to store messages from contact forms
    3. #2750617: Add views integration and listing page for contact module Message entity.
    4. #2750621: Add bulk delete action plugin for contact module Message entity
    5. #2750629: Add view builder for contact module's Message entity
    6. #2750627: Add Edit and Delete forms for contact module Message entity
    7. #2750633: Add view builder for contact module's ContactForm entity
    8. #2689265: Allow changing the 'Send message' contact form string and optionally hiding the preview button
    9. #2750649: Add option to disable form submissions for contact forms
  3. The proposal: phase 3

    Future features

    1. Support for file-fields attached to emails - requires formatter for file-field.
    2. Ability to edit format of messages bodies including tokens
    3. #2325801: Abstract contact module mail delivery out of form into a service
    4. #2325805: Make email sending optional on a per-contact form basis
    5. Path integration to allow simple alias management of contact categories
    6. #2325799: Add per contact-form permissions for contact module.
    7. Provide a menu-link per category in a custom menu - auto builds menu of contact category links leveraging the menu link API to solve the category selector regression.
    8. Ability to set submission limits
    9. Per form access controls
    10. Ability to configure mail recipients based on form values
    11. Change default value of 'send me a copy' field.
  4. The team

    @larowlan
    @andypost
    @jibran
    @naveenvalecha
    ... more to come

    The process

    The plan is to get this issue to RTBC to get product-owner/manager approval of the plan. Then to start work.

    It is still early days for 8.0, there are still some major bugs open in contact. Tending to those is obviously the priority. But getting sign-off/buy-in on the future features and building a cross-functional team in preparation can all happen in parallel to fixing bugs so that when feature development recommences, we're ready to go.

    If you're interested in getting involved, please let me know. We're looking to build a multi-disciplinary team as follows:

  • Backend developers
  • Frontend developers
  • UX/design
  • Reviewers
  • Project management
  • Manual testers

The other 20%

So if we're aiming for the 80% use-case of webform, what's the other 20%. Webform does these exceptionally well:

  • Lots of discrete webforms (100,000s in case of webform.com)
  • Lots of fields on a form (100+)
  • Multi-page forms (although fieldgroup might enable this for Field API and therefore contact messages.
  • Faster loading (although a custom storage handler might enable this for contact messages too).

Cheers

Files: 

Comments

larowlan created an issue. See original summary.

larowlan’s picture

Issue summary: View changes
jibran’s picture

Issue summary: View changes

This is so exciting +1 to the idea.

andypost’s picture

The related blocks us on proper handling of submissions

larowlan’s picture

Issue summary: View changes

Elaborated on what was the 20% not covered

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.0-beta1 was released on March 2, 2016, which means new developments and disruptive changes should now be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

larowlan’s picture

Title: Contact module 8.1 and beyond roadmap » Contact module 8.2 and beyond roadmap
Assigned: Unassigned » webchick
Issue tags: +Needs product manager review

We missed 8.1 here

jibran’s picture

Title: Contact module 8.2 and beyond roadmap » Contact module 8.2.x and beyond roadmap

I think contact storage is up to date with most all of changes in core and creating a patch will not be an issue.

andypost’s picture

Issue summary: View changes
naveenvalecha’s picture

Issue summary: View changes

I would love to help into this to push this functionality into core.

xjm’s picture

Title: Contact module 8.2.x and beyond roadmap » Contact module roadmap: 80% usecase of webforms in core

Retitling to make the "why" of this plan obvious. :)

dawehner’s picture

@larowlan and myself talked about it one day and we realized that at some point "renaming" the contact module to something like form would be a big help for users to realize how flexible the contact module actually is in Drupal 8. Maybe we could add it to the agenda.

yoroy’s picture

Issue tags: +Usability, +DevDaysMilan
larowlan’s picture

larowlan’s picture

Issue summary: View changes

Starting to add the new phase 2 issues.

larowlan’s picture

Issue summary: View changes
larowlan’s picture

Issue summary: View changes
larowlan’s picture

Issue summary: View changes
larowlan’s picture

Issue summary: View changes
larowlan’s picture

Issue summary: View changes
larowlan’s picture

Issue summary: View changes
jibran’s picture

Assigned: webchick » Unassigned
joachim’s picture

catch’s picture

How much of phase 2 could be done in an experimental module (i.e. contact_storage for example)?

Also would it be worth making that new experimental module called 'forms'? We could then move some things out of contact module, or even deprecate it.

jibran’s picture

As per our(@larowlan, @berdir, @pameeela and me) discussion with @webchick and @xjm, all these things will only go into feature branches and in contact module. We removed all those issues/features from the roadmap which can't be included to the contact module i.e. HTML mails, per contact form permissions and e.t.c.

The idea is to keep maturing and adding those features in contrib i.e. contact_storage and if we think after user testings that we need some of these features in core then we'll consider adding a new experimental module to core.

Berdir’s picture

IMHO, an experimental new module doesn't make too much sense. Most of those features aren't really experimental, they're already being tested in contact_storage and they're AFAIK all opt-in.

Actually moving any relevant code like the entities and all code around them would lead to API changes or duplication of those entities. One thing we could consider is renaming contact module to forms module just in the UI, like we did with Simpletest, although that has downsides too.

My suggestion would be to add more features to contact.module now and then just rename it in 9.x to forms.

Forgot the most important part: An experimental module gives warnings on the status page and we essentially tell non-experienced users to not to use it. That's IMHO not the message we want to send.

larowlan’s picture

Assigned: Unassigned » webchick

Ok - so is that consensus on the approach for phase 2?

I like the idea of a rename to forms, but I don't think we can do it until Drupal 9 either.

@xjm / @catch / @webchick?

dawehner’s picture

I like the idea of a rename to forms, but I don't think we can do it until Drupal 9 either.

Well, if we are super radical we could copy contact module to forum module and call contact module as deprecated as of now, and just work on the new forms module.
Not sure this is a good idea though, because, at least at that point in time, it would require new patches for all existing issues.

catch’s picture

Yeah I was thinking of something like @dawehner's #29. On top of that we could probably have contact module depend on forms module and subclass bits to minimize duplication.

ndobromirov’s picture

+1 on #29 and #30

larowlan’s picture

I like this idea - but we need sign-off from product managers - @xjm/@webchick - thoughts?

andypost’s picture

Issue tags: +Needs design

I like idea of Forms but once we fork experimental module better to reaseach ymlform module ideas as well, there's ui builder working

I think forms approach needs design to decide what is 80% of use cases

Contact storage needs UI to construct, list and expose
The main issue in this are suck for years #1997692: Regression: category selector missing on site wide contact form

pameeela’s picture

Another +1 to #29 and #30

Looking at this as an end-user, even just changing to Forms from Contact form would be a major improvement. It just feels wrong to "Add contact form" when you're not working on a contact form.

The UI is contact-specific in a few ways, including the "Selected" column, which is confusing. I think it is a blocker to adoption of contact as webform's successor, not just because of the name/UI, but things like every form has a subject and message field that you have to hide on display. It feels like a workaround rather than a solution.

We discussed in the call that we would spin up a demo for some client-types to test but tbh I don't think it's ready for that currently.

Berdir’s picture

Yeah, renaming in the UI probably makes sense.

But I still don't fully get how we'd benefit from duplicating the contact.module into a new form.module?

We'd have to do it as the very first thing, as a result, we'd have two identical modules, one deprecated and one experimental (?). Only then we could start adding new features one by one. The alternative would be to develop the forms module as a contrib module first, possibly based on a merge of contact + contact storage and only get it into core when we consider it complete (enough). I suppose it *would* be a good match for the development model proposed by Dries in Amsterdam but so far, we haven't done that.

Also, switching from contact to forms module for existing sites sounds pretty painful. They'd have to re-create or migrate all configured forms and data if they want to keep that. And existing contrib modules like contact_storage would no longer work, we do not plan to move every feature it has into core, e.g. HTML mails/swiftmailer integration. Keeping contact_storage working against a 8.x-2.x moving target would of course also be a challenge. We also have a ton of custom code integrating with contact forms, I suppose other sites as well that would have to be rewritten if they want to use the new module.

As I wrote above, none of the phase 2 features nor most of the phase3 things seem to be an issue for BC and adding them in smaller steps would result in immediate improvements for existing and new sites. Some challenges would exist either way (configurable storage, I suppose that simply means always create storage, and then save or not. otherwise views integration and so on would be really fun). One that possibly only exists with a non-experimental module is to figure out how to keep views configurations up to date without breaking customizations. But that's a problem that contrib already has to fight with and having better support for that in core would be a win for everyone (#2684081: Modules need a way to keep their configuration up to date).

A bit of feedback on the specific steps in phase 3, 2 seems fine:
7: Still not a fan of the menu suggestion for the category regression/use case. I think the options email field type that we have in contact_storage now is a much better replacement for that. As mentioned before, once we have that in core, I'd even go as far as suggesting to migrate 6.x/7.x contact forms into a single contact form with a category field.
9 seems to be a duplicate of 6
10/2 are again related and being worked on in #2734289: Configurable mail subject with token support (Actually, plenty of the phase 3 features have issues/patches in contact storage already)

catch’s picture

I don't think we have to do that rename now, but it's something that doesn't have to wait until 9.x either - we could do it towards the end of the 8.x cycle for example.

pameeela’s picture

Issue summary: View changes
pameeela’s picture

Issue summary: View changes
pameeela’s picture

Issue summary: View changes

Before assessing the technical challenges, let alone starting to write patches, I think we need to decide whether this is actually supported as a core initiative.

My 2c is that we need to completely remove all of the contact assumptions, and on some levels that might be easier starting from scratch that retrofitting what we have. Otherwise, it's going to be a half-baked solution. So maybe this belongs in contrib (or at least a feature branch) for the time being?

I created a few more issues to this end and discussed with @larowlan but the piecemeal approach seems daunting the longer I spend looking at what needs to be done. It will be hard to get the UX right by undoing one thing at a time.

Bojhan’s picture

I would suggest to schedule a call/discussion with the product maintainers and potentially release managers. We need to have a clear understanding of what we want to achieve, the exact "what we are gonna name it" is it "experimental or not" should be secondary to that discussion.

Patrick_McMahon’s picture

Make a Forms Module, moving all the base code from Contacts module into Forms Module. Have Contacts Module utilize Forms modules APIs to continue to work without duplicating code. We then leave Contact Module the way it is as a layer on top of Forms Module. At this point we can put all our focus on Forms Module and built it as a true WebForms replacement.

Should at any time the API of Forms change in a way that breaks Contacts Module we then make a patch to Contacts Module to fix the problem. Contacts Module will no longer have any real underlining APIs of its own and will be 100% reliant on Forms Module. At some point we can start to phase out contact Module as Forms module takes shape.

Forms Module should strive to meet 100% of what WebForms can do with the additions of all popular add-on modules for WebForms. This will give a powerful Forms API that other modules can work off and build on.

Forms Module should have a simple API for building forms but also have a drag-n-drop form builder UI. Forms should be easy to assign to menus, blocks, users and roles. There should be full integration with the Rules Module for complex form logic.

Storage, redirect, and external mail services should be built into Forms Module.

andypost’s picture

Good points in #41, yamlform already has nice form builder, I'd like to get comments from maintainer here

jrockowitz’s picture

Hi,

I am the maintainer of the YAML form module, which has gradually turned into a somewhat D8 port of Webforms. I am 100% on board with the goal that the Contact Storage module (aka an Entity/Field based form builder) can address 80% of the form building use cases for D8/D9. My module is only trying to address the other 20%.

As I finish up my module, I would like to start helping out with this initiative. For now, I want to share my experience researching and building a form builder for Drupal 8.

The most successful and popular form builders (Wufoo, Google Forms, etc...) are generally very simple to use. There use case is basically, build a form, collect results, and download them. The general challenge with using Drupal's Entity/Field system as a form builder is that it is too robust (and complex). I feel for a form builder in core to be successful it needs to feel simple and hide Drupal's robust Field API.

For example, Drupal's Field API requires multiple steps to create a form input (aka a field). Ideally this should be streamlined into a single form with reasonable default values. The default form display, submission display, and email display could be part of an input's main single form. Ideally, there should also be one single UI for organizing the form and submission display.

Also, my form builder started out by sending plain text email and then I added support for HTML emails. I realized that this was the wrong approach, because almost everyone can view HTML email and most people prefer HTML emails. Drupal's form builder should focus on building the HTML email first and then just downgrade the HTML to plain text. BTW, Drupal core really should support sending HTML email out of the box. One challenge is that the Field API templates that ship with core are not exactly HTML email friendly.

I feel like I am just touching the tip of the iceberg when it comes to the challenges of implementing a form builder for core. I have to agree with @Patrick_McMahon the best approach is probably to start with new module that focuses on the form build UI first, submission handling and viewing second, and finally sending emails. At the same time, I think Contact Storage module should move ahead with an MVP (Minimum Viable Product) so that the community has something to use, which then can easily be migrated to the new form builder in core.

Finally, if I got a little more help and general code review with my YAML form module, it could also help calm the grumblings about Webforms not being available for D8.

Hope this helps. ~jake

pameeela’s picture

So I actually think that the name bike shedding was quite useful in shaping my thinking on this. It spawned a lot of conversation about forms in general and how they can be improved in Drupal core across the board. These ideas are great and I hope that we can progress toward a more intuitive UI for all forms.

But for me it highlighted the disconnect between what Field UI does and what the average editor wants/needs. The main distinction is, with contact forms, the submission is the entity. With Webform, the form is the entity. I believe that we need a "form is the entity" solution, not necessarily in core, but somewhere in the ecosystem.

Until now that has been Webform, and Webform is awesome. I can understand the many reasons that it hasn't been ported yet, and the many reasons to try to use Field UI as the solution rather than building a separate one. But it is very telling that there is already a Webform-style module in D8 in YAML form, which is a different approach to the same problem. YAML form is also awesome and I'm really impressed with how advanced the functionality is already. The fact that this module exists and is stable and actively supported is evidence that we need this functionality.

There is already some great work on Field UI with Dream Fields, and to make Contact more generic. But I can't really see a point where it supersedes the Webform/YAML form functionality. Most people who want to make a form just want to make a form, and creating a whole entity type for that is overkill.

Of course I could be wrong about that. Even so the point where the Field UI form stuff can replace the simple(ish) form builder is probably still pretty far off and in the meantime we need a YAML form to fill the gap. So thank you @jrockowitz for providing it!

Sam152’s picture

Without trying to downplay yamlform (which is frankly a phenomenal effort and wonderful solution to this problem), the developer in me that sees the huge overlapping functional and technical challenges of a full-featured form builder/submission manager and entity/field UI can't really reconcile that these are two different problem spaces. If the amount of effort that went into solving problems twice (the entity way and the webform way) went into improving making entity API less daunting, then all of Drupal would benefit, not just the form-builder use case.

Yamlform is awesome, but it doesn't help shop owners add attributes to their commerce products. The conditional functionality awesome, but it doesn't help clean up /node/add pages for complex content types.

I realise it's asking a lot, but I feel this initiative is the "right" way , forward thinking and the D8 way, free from special snowflake pockets of code in walled garden contrib modules.

jrockowitz’s picture

@Sam152 My goal with the YAML form module is to provide the Drupal community with an Open Source customizable equivalent to a product like SurveyMonkey and Wufoo. Webform and YAML form are contrib modules that are products. Drupal core is a framework that helps build products. Drupal core needs to improve its form building framework and hopefully this will eliminate the need for a form builder in contrib. At the same time, I think that Drupal's adoption has benefited from products like webforms and also distributions.

You are right the YAML form module "doesn't help clean up /node/add pages for complex content types". I feel things like this simple task and important feature needs to be addressed in Drupal core. Maybe there needs to be an 'Entity customization' module that allows any entity form's layout, submit buttons, confirmation action, etc… to be altered using a UI.

A better example, is entities should have a global reusable way to trigger notifications (via the Rules module) that can be sent via email (and even SMS). Right now, both the Contact module and the User module are sending custom email notifications when new entities are created and updated. If a unified and pluggable mechanism to send notifications was added to core it would provide the confirmation functionality required by a form/survey builder.

The exporting of submission data could be addressed by a generic entity export module, that could just live in contrib, which would also be extremely useful for managing other entities like user, comments, and redirects.

A super powerful concept would be the automatic generation of the initial manage submissions view for new forms (aka bundles). My first d8 migration actually generated admin and teaser views/displays for all 30+ content types on the website.

It is also very likely that the handling of form submission drafts and previews could be addressed by the workflow initiative. Also the workflow initiative would benefit from a global notification/messaging API

Everyone is agreeing that there needs to be an initiative to improve Drupal's form and survey building capabilities. We now need to figure out the best approach.

I am suggesting smaller sub initiatives that address discreet pieces of reusable form builder related functionality. Ideally the Forms module could just contains a form config entity and a submission content entity. that leverages cores APIs to create a form/survey building UX

So far the possible new functionality/modules/initiatives I have potentially outlined are …

Entity customization: Allows site builders to customize an entity's form and confirmation handling.

Entity notification: Allows site builders to add notifications to an entity's CRUD operations.

Entity export: A simple UI that allows any entity and fields to be exported (as a CSV).

Entity Views: Automatically generates admin views for new content entity bundles.

...and Rules probably needs to be in core
…and core should support sending HTML email.

Above are just my initial thoughts. I also think Ted Bowman's Entity form/Eform module is another good starting point because he has had to address most of these requirements.

pameeela’s picture

@Sam152 Let's say you're right that this is the "right" way long term, but if we need a form builder now, contact does not suffice. So what do you propose in the meantime, if not YAML form or something similar?

kclarkson’s picture

Very interesting points on this thread.

One of the things that has always saddened me with Drupal community is how often there are duplication of efforts that can divide our collective contributions. (Of course I get that can also inspire innovation too).

At this point my vote would be to start with YAMLForms and grow from there. As of right now YAMLForms is the most comprehensive and closest thing to webform out there. With D8 adoption rate increasing, more and more site builders are going to reach for the webform replacement and stumble upon YAMLForms. If you look as the YAMLForms downloads statistics you can see the spike in just the past few months.

The more the YAMLForms maintainer can get help from the community the better this solution can be and maybe even inspire the creation of a better API like mentioned earlier.

I heard keynote from Josh Koenig from Pantheon talk about how WordPress works a bit differently where a featured doesn't' get added unless it helps the blogger. I think this is an interesting lesson that the Drupal Community could learn from. How can developers help the site builders build sites?

andypost’s picture

I totally agree that we need to define actionable items and reorder priorities on current state of contact module issues

As of now I though plan as
- storage (any form submission needs storage with current core's /dev/mail)
- html mails (suppose improvements in decoupling mail, book module already have render of print version) #2223967: Do not encode a contact message twice
new topics
- form builder
-- that could cooperate with layout plugin issue #2296423: Implement layout plugin type in core and with #1787942: Allow assigning layouts to pages to make contact pages reusable for simple contact landing pages
-- could collaborate with https://www.drupal.org/project/dream_fields to improve field UI
- submission logic customizations

So I'm focused on

8.2.0 - decide on mail issues and extensions like #2689265: Allow changing the 'Send message' contact form string and optionally hiding the preview button also I'd say we can move storage from tests to module
8.3.0 - builder/layout - experimental module
... tbd

Finally we can polish "Message" entity to incorporate sort of https://www.drupal.org/project/collect

I am suggesting smaller sub initiatives that address discreet pieces of reusable form builder related functionality. Ideally the Forms module could just contains a form config entity and a submission content entity. that leverages cores APIs to create a form/survey building UX

Can you please elaborate your's opinion on order and steps for core to try achieve in minor versions of core?

andypost’s picture

Technically speaking I'd simple implemented following

- polished storage plugability
- decoupled submission
-- introduced submission event (mail subscriber is core default implementation)
-- Introduced Submission API
- finished api of form building
- forms api somehow... builder should create some entity that implements FormSubmissionInterface

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.0-beta1 was released on August 3, 2016, which means new developments and disruptive changes should now be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

webchick’s picture

Issue tags: +Drupal core ideas

Hi there! This is an issue that we'd ideally like to move to the new "Drupal core ideas" queue when/if #2785735: [policy, no patch] Agile process for bigger changes in core (Part A: Ideation) goes through (hoping for the next week or two). If you could read that issue and provide your thoughts on that, that would be great!

webchick’s picture

Project: Drupal core » Drupal core ideas
Version: 8.3.x-dev »
Component: contact.module » Plan

Doing that. :)

MartinMa’s picture

@phase 3 issue 5. Path integration to allow simple alias management of contact categories

Why not integrate contact form into pathauto so that i can generate custom paths automatically?

dawehner’s picture

@MartinMa
Well, pathauto allows you to generate path aliases automatically. This particular issue is about being able to set a custom path, which is the opposite of automatic :)

Berdir’s picture

Pathauto does only support content entities at the moment and changing that will not be trivial.

Contact storage adds a alias field to contact forms now, and it would be pretty easy to move it to contact.module in core. Just like many of the other features it has.

jibran’s picture

mikejw’s picture

I have just been talking with the maintainer of eForm which it looks like this is going to replace.

@andypost the eForm module pretty much does what you mention. It has a config entity and a content entity and then some other "formy" bits.

I have had to create quite a few forms for my employer and as such I have created a bunch of modules (in sandbox and only local at present - haven't written any tests yet) to make the form building process a lot nicer. Mainly make it quite easy to create forms from the manage form display (mfd) page including adding fields (using presets to make it a one page affair), edit/delete from the (mfd), controlled fields (basic conditional fields for radios/checkboxes), draft support even with required fields etc.

screenshot

I really really like the idea of renaming this to something like form module and then contact could just be an addon/sub module that leverages this. Just point me at the form (groan) to sign up at. Happy to help where ever needed.